Molecular structure recognition system, control method thereof, and method of training molecular structure recognition system

Abstract

The present invention relates to a molecular structure recognition system, a control method thereof, and a method of training the molecular structure recognition system. More specifically, the present invention relates to a method of training a molecular structure recognition model of a molecular structure recognition system.

Description

Molecular structure recognition system and control method thereof, and learning method of the molecular structure recognition system

[0001] The present invention relates to a molecular structure recognition system, a control method thereof, and a learning method for the molecular structure recognition system. More specifically, the present invention relates to a learning method for a molecular structure recognition model of the molecular structure recognition system.

[0002] Effectively retrieving and utilizing molecular structural formulas is a critical task in the fields of chemistry and pharmaceutical research. These molecular structural formulas are stored in image form in various research documents, patents, and academic papers.

[0003] While these molecular structures in image form are intuitive for humans to understand, they present significant limitations for automated search and analysis by machines. In particular, researchers developing new materials need to consult existing research or explore existing molecular structures to avoid infringing on existing patents. However, when molecular structures are stored in image form, retrieving (or browsing) them is impossible, significantly hindering the research and analysis process.

[0004] To address this, AI technologies (e.g., AI models or algorithms based on optical character recognition (OCR)) have recently been developed to convert molecular structures in image form into searchable string formats based on sequence representations. However, some molecular structures are extremely complex to convert into conventional string representations, limiting the ability of these technologies to convert complex molecular structures.

[0005] As an example, the Markush structure is a way to represent molecules that have the same basic skeleton but can be transformed into various forms, which is difficult to fully capture using conventional linear string representations.

[0006] As another example, since polymers contain repeating substructures, converting them to a simple SMILES format increases their complexity and may result in data loss.

[0007] Therefore, there is still a need for a method that enables AI models to directly understand molecular structures based on images, just as humans understand molecular structures through images.

[0008] The present invention provides a molecular structure recognition system and a control method thereof, which can be usefully utilized in the fields of chemical and pharmaceutical research, and a learning method of the molecular structure recognition system.

[0009] More specifically, the present invention is intended to provide a molecular structure recognition model that can improve the efficiency of molecular structure search and analysis in the fields of chemical and pharmaceutical research.

[0010] In addition, the present invention is to provide a molecular structure recognition model capable of directly understanding and interpreting molecular structure images.

[0011] Furthermore, the present invention provides a learning method of a molecular structure recognition model capable of learning and recognizing a molecular structure from a molecular structure image itself without a process of converting the molecular structure image into a string.

[0012] In order to solve the problem described above, a learning method of a molecular structure recognition system in which a memory and at least one processor are cooperatively performed according to the present invention comprises the steps of: inputting a molecular structure image of each of a plurality of molecular structures into an image encoder; inputting a string according to a sequence representation method of each of the plurality of molecular structures into a molecule encoder; obtaining an embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures from the image encoder; obtaining an embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures from the molecule encoder; storing the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures in the memory; and performing contrastive learning on a molecular structure recognition model of the molecular structure recognition system using the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures stored in the memory and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures. Learning methods for recognition systems.

[0013] In an embodiment, the method further includes a step of constructing a learning data set including a molecular structure image of each of the plurality of molecular structures and a string according to a sequence representation method of each of the plurality of molecular structures, wherein the learning data set can be constructed such that the molecular structure image of each of the plurality of molecular structures and the string according to the sequence representation method of each of the plurality of molecular structures form a pair.

[0014] In an embodiment, at least one molecular structure image having a different visual appearance for each string according to the sequence expression method of each of the plurality of molecular structures may form a pair.

[0015] In an embodiment, the molecular structure image of each of the plurality of molecular structures includes molecular structure images having different visual appearances generated using a preset molecular structure image generation method, and in the step of constructing a learning data set, molecular structure images having different visual appearances can be generated while maintaining the characteristics of each of the plurality of molecular structures.

[0016] In an embodiment, the method further includes a step of specifying, from the learning data set, a molecular structure image of each of the plurality of molecular structures forming a pair and a string according to a sequence expression method of each of the plurality of molecular structures, wherein the plurality of molecular structures may include at least one of a first molecular structure and a second molecular structure.

[0017] In an embodiment, the step of specifying may include a step of specifying a molecular structure image of the first molecular structure forming a pair and a string according to a sequence expression method of the first molecular structure, and a step of specifying a molecular structure image of the second molecular structure forming a pair and a string according to a sequence expression method of the second molecular structure.

[0018] In an embodiment, the step of inputting to the molecular encoder may include a step of obtaining a molecular graph for each of the plurality of molecular structures by converting atoms into nodes and bonds between atoms into edges based on a string according to a sequence representation method of each of the plurality of molecular structures, and a step of inputting the molecular graph for each of the plurality of molecular structures into the molecular encoder.

[0019] In an embodiment, the step of obtaining the molecular graph includes a step of converting atoms constituting the first molecular structure into nodes and a bonding relationship between atoms constituting the first molecular structure into edges using a preset tool, thereby obtaining the first molecular graph including nodes and edges corresponding to the first molecular structure, and a step of converting atoms constituting the second molecular structure into nodes and a bonding relationship between atoms constituting the second molecular structure into edges using the tool, thereby obtaining the second molecular graph including nodes and edges corresponding to the second molecular structure, and in the step of inputting to the molecular encoder, the obtained first molecular graph and the second molecular graph can be input to the molecular encoder.

[0020] In an embodiment, in the molecular encoder, an embedding vector corresponding to the first molecular graph is generated using nodes and edges corresponding to the first molecular graph, and in the molecular encoder, an embedding vector corresponding to the second molecular graph is generated using nodes and edges corresponding to the second molecular graph, and an embedding vector corresponding to the first molecular graph generated from the molecular encoder and an embedding vector corresponding to the second molecular graph can be obtained.

[0021] In an embodiment, the step of inputting to the image encoder may include inputting a molecular structure image of the first molecular structure and an image of the second molecular structure to the image encoder, and obtaining an embedding vector corresponding to the molecular structure image of the first molecular structure and an embedding vector corresponding to the molecular structure image of the second molecular structure from the image encoder.

[0022] In the embodiment, in the step of performing the contrastive learning, learning for the molecular structure recognition model may be performed using a preset loss function so that the similarity between the molecular structure image of the first molecular structure forming the pair and the string according to the sequence representation method of the first molecular structure is maximized, and learning for the molecular structure recognition model may be performed using the loss function so that the similarity between the molecular structure image of the second molecular structure forming the pair and the string according to the sequence representation method of the second molecular structure is maximized.

[0023] In an embodiment, the similarity between the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the first molecular graph may be maximized so that the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the first molecular graph are included in the first embedding space, and the similarity between the embedding vector corresponding to the molecular structure image of the second molecular structure and the embedding vector corresponding to the second molecular graph may be maximized so that the embedding vector corresponding to the molecular structure image of the second molecular structure and the embedding vector corresponding to the second molecular graph are included in the second embedding space.

[0024] In the embodiment, in the step of performing the contrastive learning, learning for the molecular structure recognition model can be performed using the loss function so that the similarity between the molecular structure image of the first molecular structure forming a pair and the string according to the sequence representation method of the first molecular structure and the molecular structure image of the second molecular structure forming a pair and the string according to the sequence representation method of the second molecular structure is minimized.

[0025] In an embodiment, the similarity between the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the molecular structure image of the second molecular structure, which is different from the embedding vector corresponding to the first molecular graph, and the embedding vector corresponding to the second molecular graph can be minimized.

[0026] In an embodiment, the molecular structure recognition model may be trained to recognize a molecular structure corresponding to a molecular structure image of each of the plurality of molecular structures, to recognize a molecular structure corresponding to a string according to a sequence expression method of each of the plurality of molecular structures, or to recognize a molecular structure corresponding to a molecular graph for each of the plurality of molecular structures.

[0027] The present invention relates to a molecular structure recognition system, comprising a memory and at least one processor, wherein the memory and the processor cooperate to input a molecular structure image of each of a plurality of molecular structures into an image encoder, input a string according to a sequence representation method of each of the plurality of molecular structures into a molecule encoder, obtain an embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures from the image encoder, obtain an embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures from the molecule encoder, and perform contrastive learning on a molecular structure recognition model of the molecular structure recognition system using the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures.

[0028] The program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium, the program comprising the steps of: inputting a molecular structure image of each of a plurality of molecular structures into an image encoder; inputting a string according to a sequence representation method of each of the plurality of molecular structures into a molecule encoder; obtaining an embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures from the image encoder; obtaining an embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures from the molecule encoder; storing the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures in the memory; and performing contrastive learning on a molecular structure recognition model of the molecular structure recognition system using the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures stored in the memory. May contain commands.

[0029] A control method of a molecular structure recognition system according to the present invention may include the steps of: receiving a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal; processing the user query as an input of a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures; extracting information on the specific molecular structure corresponding to the user query from the molecular structure recognition model; processing the extracted information on the specific molecular structure as an input of a large-scale language model (LLM); obtaining a response to the user query from the large-scale language model; and providing the response to the user query to the user terminal.

[0030] A molecular structure recognition system according to the present invention includes a control unit that receives a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal, and processes the user query as an input of a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures, wherein the control unit extracts information on the specific molecular structure corresponding to the user query using the molecular structure recognition model, processes the extracted information on the specific molecular structure as an input of a large-scale language model (LLM), obtains a response to the user query from the large-scale language model, and provides the response to the user query to the user terminal.

[0031] The program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium, and may include instructions for performing the steps of: receiving a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal; processing the user query as an input of a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures; extracting information on the specific molecular structure corresponding to the user query from the molecular structure recognition model; processing the extracted information on the specific molecular structure as an input of a large-scale language model (LLM); obtaining a response to the user query from the large-scale language model; and providing the response to the user query to the user terminal.

[0032] As described above, according to the molecular structure recognition system and the control method thereof, and the learning method of the molecular structure recognition system according to the present invention, contrastive learning can be performed on a molecular structure recognition model using an embedding vector corresponding to a molecular structure image of each of a plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures. The molecular structure recognition model learned through this can accurately recognize molecules even in molecular structure images of various styles and orientation changes, and can accurately distinguish subtle molecular structure differences even in molecular structure images that contain small differences such as atomic changes, bond changes, and chirality changes. In addition, the molecular structure recognition model can clearly identify gradual differences (or degrees of deformation) from small structural changes to large changes in molecules, and can precisely analyze similarities and differences between molecules.

[0033] In addition, according to the molecular structure recognition system and control method thereof, and the learning method of the molecular structure recognition system according to the present invention, learning can be performed so as to maximize the similarity for pairs of molecular structure images and strings of the same molecular structure, and to minimize the similarity for pairs of molecular structure images and strings of different molecular structures. Through this, the encoder of the molecular structure recognition model learned can extract meaningful features containing a deep understanding of the molecular structure from the molecular structure formula image without going through a direct image-to-molecule conversion process. In addition, the encoder of the molecular structure recognition model can more accurately grasp the details of complex molecules and minimize information loss during the translation process. In other words, the present invention can greatly expand the applicability in the fields of chemistry and pharmaceutical research by enabling the direct understanding and interpretation of molecular images.

[0034] Furthermore, according to the molecular structure recognition system and control method thereof, and the learning method of the molecular structure recognition system according to the present invention, it is possible to construct a learning data set by generating molecular structure images having different visual appearances while maintaining the characteristics of each of a plurality of molecular structures. In other words, the present invention can provide an environment in which a model can perform learning on molecular structure images of various styles while maintaining the structural consistency of molecules. Through this, the molecular structure recognition model can utilize molecular structure images of various styles as learning data, and can maintain consistent molecular recognition performance on molecular structure images of various styles without being biased toward molecular structure images of a specific style. In other words, the molecular structure recognition model can maintain robust molecular recognition capabilities through learning on molecular structure images of various styles, and thus can be widely utilized in patent searches, new drug development, chemical research, etc.

[0035] As such, the molecular structure recognition model according to the present invention can more clearly maintain structural similarities between molecules even in images of various styles. In particular, the encoder of the molecular structure recognition model is robust to style changes and can effectively distinguish molecular structures. Furthermore, it exhibits strong generalization performance even in new, untrained molecular structures, and can effectively recognize and search molecules through image embedding. In other words, the molecular structure recognition model according to the present invention can easily understand the structural features of molecules from molecular structure images themselves, just as humans can easily understand molecular structure images, thereby effectively extracting molecular-related features even for complex molecular structures.

[0036] Figure 1 is a conceptual diagram for explaining a molecular structure recognition system according to the present invention.

[0037] Figures 2a and 2b are conceptual diagrams for explaining a learning data set according to the present invention.

[0038] FIG. 3a and FIG. 3b are conceptual diagrams for explaining a molecular structure recognition model of a molecular structure recognition system according to the present invention.

[0039] Figure 4 is a flowchart for explaining a learning method of a molecular structure recognition system according to the present invention.

[0040] FIG. 5a, FIG. 5b, FIG. 6a, FIG. 6b and FIG. 7 are conceptual diagrams for explaining a learning method of a molecular structure recognition system according to the present invention.

[0041] Figure 8 is a flowchart for explaining a control method of a molecular structure recognition system according to the present invention.

[0042] FIGS. 9 and 10 are conceptual diagrams for explaining a control method of a molecular structure recognition system according to the present invention.

[0043] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0044] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0045] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0046] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0047] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0048] The present invention relates to a molecular structure recognition system, a control method thereof, and a learning method of the molecular structure recognition system. The molecular structure recognition system, the control method thereof, and the learning method of the molecular structure recognition system according to the present invention can be usefully utilized in various situations. For example, the present invention can be usefully utilized in the field of chemical research for designing new materials or developing new drugs. The present invention provides a system, a control method, and a learning method for recognizing a molecular structure, which recognizes a molecular structure corresponding to a molecular structure image of a molecular structure, recognizes a molecular structure corresponding to a string according to a sequence representation method of a molecular structure, or recognizes a molecular structure corresponding to a molecular graph of a molecular structure. The molecular structure recognition system, the control method thereof, and the learning method of the molecular structure recognition system according to the present invention are implemented based on a “molecular structure recognition model.” For convenience of explanation, the system, the method, and the learning method will not be separately named, but will be uniformly referred to as a “molecular structure recognition model.”

[0049] Meanwhile, the molecular structure recognition model according to the present invention can be applied to various industries and services, and for example, it can be usefully utilized by being applied to a system based on at least one of a language model (LM) and a large language model (LLM). Recently, along with the development of deep learning technology, generative artificial intelligence (Generative AI) technology has been attracting attention. More specifically, a generative AI model can generate new data in various forms such as text, images, and voice from given data, and this provides a different level of application potential than simply classifying or predicting existing data. The molecular structure recognition model according to the present invention can also be applied to a system implemented based on such a generative AI model, and can be usefully utilized in various fields such as designing new materials, developing new drugs, and requiring molecular structure search and analysis.

[0050] The molecular structure recognition system according to the present invention includes a molecular structure recognition model capable of directly understanding and interpreting molecular structure images. The present invention aims to provide a molecular structure recognition model that can enhance the efficiency of molecular structure search and analysis in the fields of chemistry and pharmaceutical research. In particular, the molecular structure recognition model according to the present invention can learn and recognize molecular structures from molecular structure images themselves, without converting the molecular structure images into strings.

[0051] Hereinafter, the present invention will be described in more detail with reference to the attached drawings. Fig. 1 is a conceptual diagram for explaining a molecular structure recognition system according to the present invention. Figs. 2a and 2b are conceptual diagrams for explaining a learning data set according to the present invention, and Figs. 3a and 3b are conceptual diagrams for explaining a molecular structure recognition model of the molecular structure recognition system according to the present invention. Fig. 4 is a flowchart for explaining a learning method of the molecular structure recognition system according to the present invention, and Figs. 5a, 5b, 6a, 6b, and 7 are conceptual diagrams for explaining a learning method of the molecular structure recognition system according to the present invention. Furthermore, Fig. 8 is a flowchart for explaining a control method of the molecular structure recognition system according to the present invention, and Figs. 9 and 10 are conceptual diagrams for explaining a control method of the molecular structure recognition system according to the present invention.

[0052] Meanwhile, as discussed above, if the molecular structure recognition system (100) according to the present invention is a system that recognizes a molecular structure corresponding to a molecular structure image of a molecular structure, recognizes a molecular structure corresponding to a string according to a sequence expression method of a molecular structure, or recognizes a molecular structure corresponding to a molecular graph for a molecular structure, it is possible for some of the components described in FIG. 1 to be excluded.

[0053] First, as illustrated in FIG. 1, the molecular structure recognition system (100) according to the present invention may include at least one of an input unit (110), a storage unit (120), a graph transformation unit (130), a control unit (140), and a molecular structure recognition model (200).

[0054] The molecular structure recognition system (100) according to the present invention may include at least one processor and at least one memory containing computer program code. In this case, the memory may function as the storage unit (120) described above. In the present invention, the memory and the program code may cooperate with the processor to perform a series of processes described below.

[0055] Although not shown, the molecular structure recognition system (100) according to the present invention may include one or more processors, which may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., a digital signal processor, a tensor processing unit (TPU), a graphics processing unit (GPU), a neural network processing unit (NPU), an application-specific integrated circuit, an application-specific integrated circuit (ASIC), etc.). The one or more processors may be configured to execute instructions stored (or included) in the storage unit (120), computer-readable instructions, and / or other instructions described herein. The molecular structure recognition method and system according to the present invention may allow the memory and at least one processor to cooperate to perform data processing as described below. The processor may perform a series of operations and data processing using data and information stored in the memory. At this time, the memory may be a component of the storage unit (120).

[0056] Meanwhile, the input unit (110) may be configured as a means for data input and may be configured in various types. For example, the input unit (110) may be configured to receive user input. The input unit (110) may be configured to receive user input (or user query) from the user terminal (10). Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on the input made by the user through the input unit configuration provided in the user terminal (10).

[0057] In addition, in the present invention, the input unit (110) does not necessarily mean a hardware means, and can be understood as a passage for receiving input from a user.

[0058] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, computer mouse, keyboard, keypad, touch pad, trackball, joystick, voice recognition module, or other similar device. However, the present invention does not limit the type of input unit (110).

[0059] Here, the user input may include documents, text, images (or videos), voice, etc. In this case, the molecular structure recognition system (100) may further include a module that converts voice into text.

[0060] Next, the storage unit (120, or memory) serves to store various data related to the present invention, and may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors.

[0061] The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage (120) may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage device), while in other examples, the storage (120) may be implemented using two or more physical devices.

[0062] The storage unit (120) may include computer-readable instructions and additional data. The storage unit (120) may include storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the devices and networks.

[0063] Furthermore, at least a portion of the storage unit (120) may be a cloud storage or a cloud server. At least a portion of data corresponding to user input received from the input unit (110) and learning data may be stored in the storage unit (120).

[0064] That is, it can be understood that the storage unit (120) is sufficient as a space in which information necessary for the operation of the molecular structure recognition system (100) according to the present invention is stored, and there are no restrictions on the physical space.

[0065] Furthermore, the storage unit (120) may store a computer program including computer program instructions. Furthermore, the storage unit (120) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (140) when loaded onto the processor of the system (100).

[0066] Next, the graph conversion unit (130) may be configured to generate a molecular graph for each of a plurality of molecular structures by converting atoms into nodes and bonds between atoms into edges based on strings (e.g., SMILES, InChI, etc.) according to the sequence representation method of each molecular structure.

[0067] More specifically, the graph transformation unit (130) can create a molecular graph including nodes and edges corresponding to the molecular structure by transforming atoms constituting the molecular structure into nodes and transforming bonding relationships between atoms constituting the molecular structure into edges.

[0068] This graph conversion unit (130) may include at least one tool used to handle and analyze molecular structures in cheminformatics, for example, the graph conversion unit (130) may include at least one of RDKit and Indigo used for analysis of molecular structures (or chemical structures), molecular structure representation, molecular structure visualization, sub-structure search, molecular fingerprint generation, physicochemical property prediction, chemical reaction processing, etc. In addition, the graph conversion unit (130) in the present invention may also be named as “pre-set tool” or “RDKit” or “Indigo”.

[0069] Next, the control unit (140) may perform a role of controlling the overall operation of the molecular structure recognition system (100) related to the present invention. The control unit (140) may process signals, data, information, etc. input or output through the components of the molecular structure recognition system (100) discussed above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (140) may be physically implemented by the processor discussed above.

[0070] Meanwhile, the control unit (140) can build a learning data set to be used in the learning process of the molecular structure recognition model (200).

[0071] The present invention can increase the diversity of learning data by generating molecular structure images of various styles for the same molecule. That is, in order for the molecular structure recognition model (200) to learn the essential structure of a molecule without relying on a specific style or visual representation, the present invention generates molecular structure images of various styles for the same molecule, thereby enabling the molecular structure recognition model (200) to learn consistent representations for molecular structure images of various styles.

[0072] In the present invention, this method (or technique, method, etc.) may also be referred to as “DPV (Diverse Positive Molecular Image Views).” DPV can be understood as a method that generates images of various styles for the same molecule, thereby providing various positive samples for the same molecular structure when learning to match molecular structure images and molecular graphs (or strings of molecular structures). For convenience of explanation, DPV is referred to herein as a “pre-configured molecular structure image generation method.”

[0073] The established molecular structure image generation method (or technique, method, etc.) may include at least one of various rendering techniques, tools, image generation algorithms, image transformation or augmentation techniques to secure various visual appearances (or various styles) of the molecular structure image.

[0074] In this regard, molecules (or molecular structures) may have unique characteristics, such as bonding patterns, atomic arrangements, physical and chemical properties, or molecular structures. Furthermore, even identical molecules may be rendered differently (e.g., atomic number display, line thickness, color, etc.). In this case, cropping, rotating, or flipping a molecular structure image, or other modifications, may misrepresent the molecule or alter its structural characteristics (or chemical and physical properties, or unique characteristics).

[0075] Accordingly, the molecular structure image generation method set in the present invention may include techniques such as color change (or modification), line thickness and size adjustment, contrast adjustment (or brightness adjustment), Gaussian blur, noise addition, etc., or various chemical structure rendering tools (or software) that enable generation of molecular structure images having different visual appearances while maintaining the characteristics of the molecular structure.

[0076] Such a preset molecular structure image generation method may include, for example, at least one of preset tools (e.g., “RDKit: a tool that expresses atoms and bonds as lines and applies black and white or simple colors,” “Indigo: a tool that applies line thickness, color, and style different from RDKit”) and preset augmentation techniques (e.g., SoftAug, HardAug, etc.).

[0077] The control unit (140) can build a learning data set including a molecular structure image of each of a plurality of molecular structures and a string according to a sequence expression method of each of the plurality of molecular structures.

[0078] In this case, the control unit (140) may use (or utilize, use, etc.) at least one of a string according to the sequence representation of the molecular structure and / or a molecular structure image of the molecular structure as data for generating a molecular structure image having different visual appearances. At this time, the molecular structure image used in the process of generating the molecular structure image having different visual appearances may be an image collected from various sources (e.g., a database, web crawling, API, a server linked to the molecular structure recognition system (100), an external server, etc.) or at least one of images generated using a preset tool.

[0079] Specifically, the control unit (140) can generate at least one molecular structure image having a different visual appearance for each of a plurality of molecular structures by using a preset molecular structure image generation method.

[0080] As discussed above, the control unit (140) can generate a molecular structure image having a different visual appearance for each of the plurality of molecular structures while maintaining the characteristics (e.g., structural characteristics or chemical characteristics, etc.) of each of the plurality of molecular structures.

[0081] In one embodiment, as illustrated in FIG. 2A, the control unit (140) can generate a plurality of molecular structure images (M1-2, M1-3, M1-4, M1-5) having different visual appearances for a specific molecular structure (M1) while maintaining the characteristics of a specific molecular structure (or the first molecular structure (M1)) among a plurality of molecular structures by using a preset molecular structure image generation method.

[0082] In this case, among the plurality of molecular structure images (M1-2, M1-3, M1-4, M1-5) having different visual appearances generated for the first molecular structure (M1), at least some of the images (M1-2, M1-5) may be images generated by inputting a string (T1) according to a sequence expression method of the first molecular structure (M1) into a preset tool (e.g., RDKit, Indigo, etc.). That is, the some of the images (M1-2, M1-5) may be images generated using a preset tool for the string (T1) of the first molecular structure (M1).

[0083] In addition, among the plurality of molecular structure images (M1-2, M1-3, M1-4, M1-5) having different visual appearances generated for the first molecular structure (M1), at least some of the images (M1-2, M1-5) may be images generated by applying a preset augmentation technique (e.g., soft augmentation (SoftAug), hard augmentation (HardAug), etc.) to the molecular structure image (M1-1) of the first molecular structure. In this case, the molecular structure image (M1-1) of the first molecular structure to which the augmentation technique is applied may be an image collected from various sources or an image generated using a preset tool.

[0084] In another embodiment, the control unit (140) can generate a plurality of molecular structure images having different visual appearances for the second molecular structure (M2) while maintaining the characteristics of the second molecular structure (M2) among the plurality of molecular structures by using a preset molecular structure image generation method.

[0085] In this case, among the plurality of molecular structure images having different visual appearances generated for the second molecular structure (M2), at least some of the images may be images generated by inputting a string (T2) according to a sequence expression method of the second molecular structure into a preset tool. That is, the some of the images may be images generated using a preset tool for the string (T2) of the second molecular structure.

[0086] Additionally, among the plurality of molecular structure images having different visual appearances generated for the second molecular structure (M2), at least some images may be images generated by applying a preset enhancement technique to the molecular structure image of the second molecular structure. In this case, the molecular structure images of the second molecular structure to which the enhancement technique is applied may be images collected from various sources or images generated using a preset tool.

[0087] Furthermore, according to the embodiment discussed above, a learning data set including a molecular structure image of each of a plurality of molecular structures and a string according to a sequence representation method of each of the plurality of molecular structures can be constructed.

[0088] The control unit (140) can construct a learning data set by pairing molecular structure images of each of a plurality of molecular structures and strings according to the sequence representation method of each of the plurality of molecular structures. For example, as illustrated in FIG. 2b, the learning data set (300) can be constructed by pairing molecular structure images of each of a plurality of molecular structures (M1, M2, M3, M4) and strings (T1, T2, T3, T4) according to the sequence representation method of each of the plurality of molecular structures.

[0089] In this case, at least one molecular structure image having a different visual appearance may be paired for each string (T1, T2, T3, T4) according to the sequence expression method of each of the plurality of molecular structures included in the learning data set (300).

[0090] The molecular structure images having different visual appearances may include at least one of a molecular structure image corresponding to data used to generate the molecular structure images having different visual appearances and a molecular structure image generated using a preset molecular structure image generation method.

[0091] For example, a plurality of images (M1-1, M1-2, M1-3, M1-4, M1-5) having different visual appearances may form a pair in a string (T1) according to a sequence representation method of a first molecular structure, and a plurality of images (M2-1, M2-2, M2-3, M2-4, M2-5) having different visual appearances may form a pair in a string (T2) according to a sequence representation method of a second molecular structure.

[0092] For another example, a plurality of images (M3-1, M3-2, M3-3, M3-4, M3-5) having different visual appearances may be paired with a string (T3) according to a sequence representation method of a third molecular structure, and a plurality of images (M4-1, M4-2, M4-3, M4-4, M4-5) having different visual appearances may be paired with a string (T4) according to a sequence representation method of a fourth molecular structure.

[0093] Accordingly, the molecular structure images of each of the plurality of molecular structures paired with each string (T1, T2, T3, T4) of each of the plurality of molecular structures may include molecular structure images having different visual appearances generated using a preset molecular structure image generation method. These molecular structure images having different visual appearances may include various images (synthetic images) generated while maintaining the characteristics of each molecular structure.

[0094] Meanwhile, in the present invention, the molecular structure recognition model (200) can be trained to recognize a molecular structure corresponding to a molecular structure image of a molecular structure, recognize a molecular structure corresponding to a string according to a sequence expression method of the molecular structure, or recognize a molecular structure corresponding to a molecular graph for the molecular structure.

[0095] The molecular structure recognition model (200) may include an image encoder (210) and a molecular encoder (or graph encoder, 220). In the present invention, the molecular structure recognition model (200) may also be referred to as a “molecular understanding model” or “molecular structure understanding model” or “molecular recognition model” or “molecular analysis model” or “molecular structure analysis model”.

[0096] As illustrated in FIGS. 3A and 3B, when a molecular structure image (30) of a molecular structure is input, the image encoder (210) can be configured to generate (or output) an embedding vector (or feature, 30a) corresponding to the molecular structure image (30). For example, the image encoder (210) can convert (or divide) the molecular structure image (30) into patch units, and then process (or embed) each patch to extract features (i.e., embedding vectors) of the molecular structure.

[0097] The embedding vector (30a) corresponding to the molecular structure image (30) includes vector values ​​that contain important features (or characteristics) of the molecular structure image, and can be represented as a vector of 512 dimensions or 1024 dimensions, etc. For example, as illustrated in FIG. 3b, in the embedding vector (30a) corresponding to the molecular structure image (30), “I1, I2, I3”, etc., are vectors for images of each molecule, and may be vector expressions extracted from the molecular structure image. Here, I1 may mean the image vector of the first molecule, and I2 may mean the image vector of the second molecule.

[0098] This image encoder (210) may be based on various artificial intelligence models (or networks), and for example, the image encoder (210) may be an encoder based on ViT (Vision Transformer).

[0099] When a molecular graph (ex: “Molecular Graph”, 31a) for a molecular structure is input, the molecular encoder (220) may be configured to generate an embedding vector (or feature, 31b) corresponding to the molecular structure graph (31a). In this case, the graph conversion unit (130) may convert a string (ex: “C1CCCCC1C2CCCCC2”, 31) according to a sequence expression method of the molecular structure into a molecular graph (31a) expressed as a node (atom) edge (bond), and input the converted molecular graph (31a) to the molecular encoder (220). For example, a molecular encoder (220) can learn structural information (e.g., information on nodes and edges) of a molecular structure included in a molecular graph (31a) using a message passing technique, and generate an embedding vector (31b) that takes into account the surrounding environment (connectivity) of each atom that constitutes the molecular structure.

[0100] The embedding vector (31b) corresponding to the molecular graph (31a) may include vector values ​​that contain important features of the molecular graph. For example, as illustrated in FIG. 3b, in the embedding vector (31b) corresponding to the molecular graph (31a), “T1, T2, TN,” etc., may be unique vectors reflecting the features of each molecule and may be vector representations extracted from the molecular graph. Here, T1 may refer to the vector representation of the first molecule, and T2 may refer to the vector representation of the second molecule.

[0101] Such a molecular encoder (220) may be based on various artificial intelligence models (or networks), and for example, the molecular encoder (220) may be an encoder based on a GIN (Graph Isomorphism Network).

[0102] Furthermore, in the present invention, contrastive learning can be performed on the molecular structure recognition model (200) so that if the graph representation (T) of a molecule and the molecular structure image representation (I) are the same molecule, they are located in a close vector space, and if they are different molecules, they are located at a far distance. For example, the control unit (140) can generate a similarity matrix (32) by calculating the inner product of each image vector (I_i) and each molecular graph vector (T_j). Each element (e.g., I_n and T_j) of the similarity matrix (32) can represent the similarity between the ith image and the jth molecule. In this case, if the molecular structure image and molecular graph are of the same molecule, the values ​​of each element (I_i and T_j) can be extracted to be large, and if the molecular structure image and molecular graph are of different molecular structures, the values ​​of each element can be learned to be small. That is, the molecular structure recognition model (200) can be trained through contrastive learning to be close to a positive pair corresponding to a molecular structure image and a molecular graph of a specific molecular structure that is a learning target, and far from a negative pair corresponding to a molecular structure image and a molecular graph of a molecular structure different from the specific molecular structure.

[0103] Meanwhile, the present invention provides a molecular structure recognition system, a control method thereof, and a learning method for the molecular structure recognition system. More specifically, the present invention provides a molecular structure recognition model that can enhance the efficiency of molecular structure search and analysis in the fields of chemistry and pharmaceutical research. Below, the learning method for the molecular structure recognition model (200) will be examined in more detail.

[0104] As discussed above, the learning data set (300) according to the present invention can be constructed so that molecular structure images of each of the plurality of molecular structures and strings according to the sequence representation method of each of the plurality of molecular structures form pairs. In this case, the molecular structure images of each of the plurality of molecular structures included in the learning data set (300) can be understood as including at least one molecular structure image having a different visual appearance.

[0105] The control unit (140) can specify, from the learning data set (300), a molecular structure image of the first molecular structure forming a pair and a string according to the sequence expression method of the first molecular structure, and can specify a molecular structure image of the second molecular structure forming a pair and a string according to the sequence expression method of the second molecular structure.

[0106] In this case, each of the molecular structure image of the first molecular structure and the molecular structure image of the second molecular structure may include a plurality of molecular structure images having different visual appearances.

[0107] Accordingly, in the present invention, it can be understood that the molecular structure image input to the molecular structure recognition model (200) is at least one of molecular structure images having different visual appearances. That is, so that the molecular structure recognition model (200) can perform consistent learning on molecular structure images having different visual appearances generated for the same molecular structure, any one of the molecular structure images having different visual appearances and a string can be input to the molecular structure recognition model (200).

[0108] First, the control unit (140) can input the molecular structure image of each of the plurality of molecular structures forming a pair and the string of each of the plurality of molecular structures into the molecular structure recognition model (200). More specifically, the control unit (140) can input the molecular structure image of each of the plurality of molecular structures into the image encoder (210).

[0109] As illustrated in FIGS. 5a and 5b, the control unit (140) can input a molecular structure image (M1-1) of a specified first molecular structure to the image encoder (210). For example, the control unit (140) can input at least one image (M1-1) among a plurality of molecular structure images having different visual appearances of the first molecular structure to the image encoder (210).

[0110] And, as illustrated in FIGS. 6A and 6B, the control unit (140) can input a molecular structure image (M2-1) of a specified second molecular structure to the image encoder (210). For example, the control unit (140) can input at least one image (M2-1) among a plurality of molecular structure images having different visual appearances of the second molecular structure to the image encoder (210).

[0111] Next, the control unit (140) can input a string according to the sequence expression method of each of the plurality of molecular structures into the molecule encoder (220).

[0112] In this case, each string of multiple molecular structures can be converted into a molecular graph and input to the molecular encoder (220). For example, the string (T1) of the first molecular structure can be converted into the first molecular graph, and the string (T2) of the second molecular structure can be converted into the second molecular graph and input to the molecular encoder (220).

[0113] The control unit (140) can obtain a molecular graph for each of the plurality of molecular structures by converting atoms into nodes and bonds between atoms into edges based on a string according to a sequence expression method of each of the plurality of molecular structures, and input the molecular graph for each of the plurality of molecular structures into the molecular encoder (220).

[0114] Specifically, as illustrated in FIGS. 5A and 5B, the control unit (140) can use a preset tool (or graph conversion unit (130)) to convert atoms constituting a first molecular structure (M1) among a plurality of molecular structures into nodes and convert bonding relationships between atoms constituting the first molecular structure into edges, thereby obtaining a first molecular graph (M1-G) including nodes and edges corresponding to the first molecular structure (M1).

[0115] In addition, as illustrated in FIGS. 6A and 6B, the control unit (140) can use a preset tool to convert atoms constituting the second molecular structure (M2) into nodes and convert bonding relationships between atoms constituting the second molecular structure (M2) into edges, thereby obtaining a second molecular graph (M2-G) including nodes and edges corresponding to the second molecular structure (M2).

[0116] And, the control unit (140) can input the acquired first molecular graph (M1-G) and second molecular graph (M2-G) into the molecular encoder (220), respectively.

[0117] Furthermore, the control unit (140) can obtain an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures from the image encoder (210), and can obtain an embedding vector corresponding to a string according to a sequence expression method of each of the plurality of molecular structures from the molecular encoder (220).

[0118] The control unit (140) can obtain an embedding vector (or a first embedding vector, an image embedding vector, etc., 510) corresponding to a molecular structure image (M1-1) of a first molecular structure from an image encoder (210), and can obtain an embedding vector (or a first embedding vector, an image embedding vector, etc., 610) corresponding to a molecular structure image (M2-1) of a second molecular structure from an image encoder (210).

[0119] The process of obtaining an embedding vector corresponding to each molecular structure image of the plurality of molecular structures discussed above may include a process of obtaining, from the image encoder (210), an embedding vector corresponding to each of the plurality of molecular structure images having different visual appearances of the first molecular structure and an embedding vector corresponding to each of the plurality of molecular structure images having different visual appearances of the second molecular structure.

[0120] And, the control unit (140) can obtain an embedding vector corresponding to each of the first molecular graph (M1-G) and the second molecular graph (M2-G) from the molecular encoder (220).

[0121] The molecular encoder (220) can generate an embedding vector (or a second embedding vector, a graph embedding vector, a molecular embedding vector, etc., 520) corresponding to the first molecular graph (M1-G) using the nodes and edges corresponding to the first molecular graph (M1-G). In addition, the molecular encoder (220) can generate an embedding vector (or a second embedding vector, a graph embedding vector, a molecular embedding vector, etc., 620) corresponding to the second molecular graph (M2-G) using the nodes and edges corresponding to the second molecular graph (M2-G).

[0122] Through this, the control unit (140) can obtain an embedding vector (520) corresponding to the first molecular graph (M1-G) generated from the molecular encoder (220) and an embedding vector (620) corresponding to the second molecular graph (M2-G), respectively.

[0123] Furthermore, the control unit (140) can store, in the storage unit (or memory, 140), an embedding vector (510, 610) corresponding to a molecular structure image (M1-1, M1-2) of each of the plurality of molecular structures and an embedding vector (520, 620) corresponding to a string (T1, T2) according to a sequence expression method of each of the plurality of molecular structures.

[0124] Meanwhile, the control unit (140) can perform contrastive learning on the molecular structure recognition model (200) of the molecular structure recognition system (100) by using the embedding vectors (510, 610) corresponding to the molecular structure images (M1-1, M1-2) of each of the plurality of molecular structures stored in the memory and the embedding vectors (520, 620) corresponding to the strings (T1, T2) according to the sequence expression method of each of the plurality of molecular structures.

[0125] The control unit (140) can define a similarity matrix for each of the molecular structure images of the plurality of molecular structure images forming a pair input to the molecular structure recognition model (200) and for each of the string pairs of the plurality of molecular structures. For example, the control unit (140) can define a similarity matrix for the molecular structure image (M1-1) of the first molecular structure forming a pair input to the molecular structure recognition model (200) and the string of the first molecular structure (T1, or a pair of the molecular structure image and string of the first molecular structure, a molecular structure image-string pair of the first molecular structure, etc.) and the molecular structure image (M2-1) of the second molecular structure forming a pair and the string of the first molecular structure (T2, or a pair of the molecular structure image and string of the second molecular structure, a molecular structure image-string pair of the second molecular structure, etc.).

[0126] In the present invention, a pair of a molecular structure image and a string of each of a plurality of molecular structures input to the molecular structure recognition model (200) can be represented as in (a) of FIG. 7, and a similarity matrix can be represented as in (b) of FIG. 7. At this time, the element of the ith row and the jth column in the similarity matrix can be represented as in (c) of FIG. 7. This represents the similarity (e.g., cosine similarity) between the ith text (or string) embedding and the jth image embedding, and the cosine similarity can be represented as in (d) of FIG.

[0127] The embedding (I) corresponding to the molecular structure image and the embedding corresponding to the string are projected into the same space (i.e., embedding space, vector space, etc.). In this case, the molecular structure recognition model (200) can be trained so that positive pairs become closer to each other and negative pairs become farther away from each other.

[0128] Here, a positive pair may include a pair of molecular structure images and strings of the same molecular structure, and a negative pair may include a pair of molecular structure images and strings of different molecular structures.

[0129] More specifically, a positive pair may include an image representing the same molecular structure and a graphical representation of the molecule (molecular graph). That is, different representations of the same molecule may exist, and the molecular structure recognition model (200) is trained to recognize them as the same.

[0130] For example, molecular structure images generated with different visual appearances for the same molecule discussed above can all be regarded (or defined) as positive pairs (or positive samples) corresponding to the same molecule. The molecular structure recognition model (200) learns to position the representations of these samples close together, thereby understanding that various representations of the same molecule all have the same meaning. The goal of this process is to minimize the embedding distance between molecular structure images representing the same molecule and the molecular graph by maximizing (or increasing) the similarity.

[0131] In contrast, a negative pair may refer to a molecular structure image and molecular graph containing different molecular structures. That is, since different molecular structures have different meanings or characteristics, the molecular structure recognition model (200) can be trained to make the embeddings between different molecular structures far apart.

[0132] For example, when a pair of a molecular structure image and a string of a first molecular structure is input into a molecular structure recognition model (200), and the molecular structure recognition model (200) learns about the pair of the molecular structure image and the string of the first molecular structure, the pair of the molecular structure image and the string of the first molecular structure can be defined (or regarded) as a positive pair. On the other hand, the pair of the molecular structure image and the string of the second molecular structure can be defined as a negative pair. The molecular structure recognition model (200) learns to position the representations of these samples far apart, so as to clearly distinguish the differences between molecular structures. In this process, the goal is to maximize (i.e., make far) the embedding distance between molecular structure images representing different molecules and molecular graphs by minimizing (or lowering) the similarity between the pair of the molecular structure image and the string of the first molecular structure and the pair of the molecular structure image and the string of the second molecular structure.

[0133] That is, it can be learned so that the similarity is maximized for pairs of molecular structure images and strings of the same molecular structure, and it can be learned so that the similarity is minimized for pairs of molecular structure images and strings of different molecular structures.

[0134] Meanwhile, when a similarity matrix (S) is given, a preset loss function for learning (i.e., single encoder learning) of either the image encoder (210) or the molecular encoder (220) included in the molecular structure recognition model (200) of the present invention can be represented as in (e) of FIG. 7. This loss function of (e) of FIG. 7 may also be referred to as a “preset first loss function.” Hereinafter, the description will be made on the assumption that the image encoder (210) is trained using the first loss function. However, the encoder trained using the first loss function is not necessarily limited to the image encoder (210). The molecular encoder (220) may also be trained using the first loss function.

[0135] The control unit (140) can perform contrastive learning for at least one of the image encoder (210) and the molecular encoder (220) using a preset loss function.

[0136] Specifically, as illustrated in FIGS. 6A and 6B, the control unit (140) may perform learning on at least one of the image encoder (210) and the molecular encoder (220) included in the molecular structure recognition model (200) using a preset loss function so as to maximize the similarity between the molecular structure image (M1-1) of the first molecular structure forming the pair and the string (T1) according to the sequence expression method of the first molecular structure.

[0137] Here, maximizing the similarity between the molecular structure image of the first molecular structure forming the pair and the string according to the sequence expression method of the first molecular structure may mean maximizing the similarity between the embedding vector (510) corresponding to the molecular structure image (M1-1) of the first molecular structure and the embedding vector (520) corresponding to the first molecular graph (M1-G) so that the embedding vector (510) corresponding to the molecular structure image (M1-1) of the first molecular structure and the embedding vector (520) corresponding to the first molecular graph (M1-G) are included in the first embedding space.

[0138] The control unit (140) can train the image encoder (210) in a direction that maximizes the similarity between the embedding vector (510) corresponding to the molecular structure image (M1-1) of the first molecular structure and the embedding vector (520) corresponding to the first molecular graph (M1-G) so that the embedding vector (510) corresponding to the molecular structure image (M1-1) of the first molecular structure and the embedding vector (520) corresponding to the first molecular graph (M1-G) are included in the first embedding space.

[0139] That is, the control unit (140) can train the image encoder (210) in a direction that maximizes the similarity between the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the first molecular structure and the embedding vectors corresponding to the first molecular graph (M1-G) for the first molecular structure, so that the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the first molecular structure and the embedding vectors corresponding to the first molecular graph (M1-G) for the first molecular structure are located (or included) in the first embedding space.

[0140] Accordingly, the molecular structure image of the first molecular structure and the molecular graph for the first molecular structure are mapped to vector representations having the same meaning, and the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the first molecular structure (M1) and the embedding vectors corresponding to the first molecular graph (or the string of the first molecular structure) can be clustered based on structural similarity in the first embedding space corresponding to the first molecular structure (M1). Alternatively, the molecular structure image of the first molecular structure (or the molecular structure images having different visual appearances) and the first molecular graph learned through the molecular structure recognition model (200) can be grouped (or sorted) based on similarity in the first embedding space.

[0141] In addition, as illustrated in FIGS. 6A and 6B, the control unit (140) may perform learning on at least one of the image encoder (210) and the molecular encoder (220) included in the molecular structure recognition model (200) using a loss function so as to maximize the similarity between the molecular structure image (M2-1) of the second molecular structure forming the pair and the string (T2) according to the sequence expression method of the second molecular structure.

[0142] Here, maximizing the similarity between the molecular structure image (M2-1) of the second molecular structure forming the pair and the string (T2) according to the sequence expression method of the second molecular structure may mean maximizing the similarity between the embedding vector (610) corresponding to the molecular structure image (M2-1) of the second molecular structure and the embedding vector (620) corresponding to the second molecular graph (M2-G) so that the embedding vector (610) corresponding to the molecular structure image (M2-1) of the second molecular structure and the embedding vector (620) corresponding to the second molecular graph (M2-G) are included in the second embedding space.

[0143] The control unit (140) can train the image encoder (210) in a direction that maximizes the similarity between the embedding vector (610) corresponding to the molecular structure image (M2-1) of the second molecular structure and the embedding vector (620) corresponding to the second molecular graph (M2-G) so that the embedding vector (610) corresponding to the molecular structure image (M2-1) of the second molecular structure and the embedding vector (620) corresponding to the second molecular graph (M2-G) are included in the second embedding space.

[0144] That is, the control unit (140) can train the image encoder (210) in a direction that maximizes the similarity between the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the second molecular structure and the embedding vectors corresponding to the second molecular graph (M2-G) for the second molecular structure, so that the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the second molecular structure and the embedding vectors corresponding to the second molecular graph (M2-G) for the second molecular structure are located (or included) in the second embedding space.

[0145] Accordingly, the molecular structure image of the second molecular structure and the molecular graph for the second molecular structure are mapped to vector representations having the same meaning, and the embedding vectors corresponding to each of the plurality of molecular structure images having different visual appearances of the second molecular structure (M2) and the embedding vectors corresponding to the second molecular graph (or the string of the second molecular structure) can be clustered based on structural similarity in the second embedding space corresponding to the second molecular structure (M2). Alternatively, the molecular structure image of the second molecular structure (or the molecular structure images having different visual appearances) and the second molecular graph learned through the molecular structure recognition model (200) can be grouped (or sorted) based on similarity in the second embedding space.

[0146] The similarity maximization process discussed above can be understood as training the molecular structure recognition model (200) in a direction that increases the similarity between the embedding vector corresponding to the molecular structure image of the positive pair and the embedding vector corresponding to the molecular graph.

[0147] Meanwhile, as illustrated in FIGS. 6A and 6B, the control unit (140) can perform learning on the image encoder (210) included in the molecular structure recognition model (200) using a loss function so as to minimize the similarity between the molecular structure image (M1-1) of the first molecular structure forming a pair and the string (T1) according to the sequence representation method of the first molecular structure and the molecular structure image (M2-1) of the second molecular structure forming a pair and the string (T2) according to the sequence representation method of the second molecular structure.

[0148] Here, minimizing the similarity between the molecular structure image (M1-1) of the first molecular structure forming the pair and the string (T1) according to the sequence representation method of the first molecular structure and the molecular structure image (M2-1) of the second molecular structure forming the pair and the string (T2) according to the sequence representation method of the second molecular structure may mean minimizing the similarity between the embedding vector (510) corresponding to the molecular structure image (M1-1) of the first molecular structure and the embedding vector (520) corresponding to the first molecular graph (M1-G) and the embedding vector (610) corresponding to the molecular structure image (M2-1) of the second molecular structure and the embedding vector (620) corresponding to the second molecular graph (M1-G) that are different from each other.

[0149] Accordingly, the molecular structure image-string pair of the first molecular structure and the molecular structure image-string pair of the second molecular structure are distinguished (or distinguished, classified, separated, etc.) as vector representations having different meanings, and the molecular structure recognition model (or image encoder, 200) can perform learning on this.

[0150] The similarity minimization process discussed above can be understood as training the molecular structure recognition model (200) in the direction of reducing the similarity between the embedding vector corresponding to the molecular structure image of the negative pair and the embedding vector corresponding to the molecular graph.

[0151] Meanwhile, a preset loss function for bidirectional learning of the image encoder (210) and the molecular encoder (220) included in the molecular structure recognition model of the present invention can be represented as shown in (f) of Fig. 7. This loss function of (f) of Fig. 7 can also be referred to as a “preset second loss function.”

[0152] Here, bidirectional learning can be understood as performing symmetrical learning from a molecular structure image to a molecular graph, or from a molecular graph to a molecular structure image. If the loss function of Fig. 7(e) discussed above is a loss function for learning a single encoder, the loss function of Fig. 7(f) can be a loss function that considers bidirectional learning of the encoder.

[0153] That is, the control unit (140) can train the image encoder (210) and the molecular encoder (220) included in the molecular structure recognition model (200) using the loss function of FIG. 7 (f) so as to learn which molecular graph the input molecular structure image is connected to (the direction of the molecular structure in the molecular structure image) or to learn which molecular structure image the input molecular structure is connected to (the direction of the molecular structure image in the molecular structure).

[0154] The molecular structure recognition model (200) can be trained to recognize a molecular structure corresponding to a molecular structure image of each of a plurality of molecular structures, to recognize a molecular structure corresponding to a string according to a sequence expression method of each of a plurality of molecular structures, or to recognize a molecular structure corresponding to a molecular graph for each of a plurality of molecular structures, using the loss function of FIG. 7 (f).

[0155] The molecular structure recognition model (or image encoder, 200) trained through the learning process described above can extract meaningful image features (or embedding vectors) from molecular structure images. In other words, features related to molecular structure images (or molecular structural formulas, molecular structures) can be extracted.

[0156] In this way, the learning process of the molecular structure recognition model according to the present invention may include, as illustrated in FIG. 4, a process of inputting a molecular structure image of each of a plurality of molecular structures into an image encoder (S410), a process of inputting a string according to a sequence representation method of each of the plurality of molecular structures into a molecule encoder (S420), a process of obtaining an embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures from the image encoder (S430), a process of obtaining an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures from the molecule encoder (S440), and a process of storing the embedding vector corresponding to the molecular structure image of each of the plurality of molecular structures and the embedding vector corresponding to the string according to the sequence representation method of each of the plurality of molecular structures in a memory (S450). Furthermore, the above learning process may include a process (S460) of performing contrastive learning on a molecular structure recognition model (200) of a molecular structure recognition system (100) by using an embedding vector corresponding to a molecular structure image of each of a plurality of molecular structures stored in memory and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures.

[0157] Meanwhile, the molecular structure recognition model (200) learned according to the learning method described above can be configured in the inference stage, excluding the molecular encoder (220), as shown in FIG. 10. In the inference section, the control method of the molecular structure recognition system according to the present invention may include, as illustrated in FIG. 8, a process of receiving a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal (S810), a process of processing the user query as an input of a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures (S820), a process of extracting information on a specific molecular structure corresponding to the user query from the molecular structure recognition model (S830), a process of processing the extracted information on a specific molecular structure as an input of a large-scale language model (LLM) (S840), a process of obtaining a response to the user query from the large-scale language model (S850), and a process of providing a response to the user query to the user terminal (S860).

[0158] Here, the user query may include at least one of a molecular structure image of the molecular structure, a string according to a sequence representation of the molecular structure, a graph of the molecular structure, a molecular structure formula, and a description related to the molecular structure. For example, as illustrated in FIG. 9, the control unit (140) may receive a user query (e.g., “Please explain the molecular structure”, 901) including a molecular structure image (901a) through a service page (or user terminal (10)).

[0159] In one embodiment, as illustrated in FIG. 10, the control unit (140) may input a molecular structure image (901a) included in a user query (901) into an image encoder (210) of a molecular structure recognition model (200) trained based on contrastive learning between an embedding vector corresponding to each molecular structure image of a plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures. When a specific molecular structure image (901a) is input, the image encoder (210) may extract information on a specific molecular structure corresponding to the molecular structure image (901a) from the storage unit (120) or DB, and process the extracted information on the specific molecular structure as an input to a large-scale language model (1000). Furthermore, the control unit (140) can obtain a response to a user query (901) (ex: “This molecular structure chemically appears to be one of the derivatives of indole-3-carboxylic acid. The core structure has an indole ring structure, and is a form in which a benzene ring and a pyrrole ring are combined...”, 910) from the large-scale language model (1000), and provide a response to a user query (902) generated from the large-scale language model (1000) to the user terminal (10).

[0160] In another embodiment, as illustrated in FIG. 1, the molecular structure recognition model (200) can receive a molecular structure image (1) of a molecular structure as input as a user query, and recognize a molecular structure corresponding to the molecular structure image (1). Then, the control unit (140) can search for a document (e.g., a patent document, a paper, etc., D) including a molecular structure (2) corresponding to the molecular structure image (1) based on the recognized result, and provide the searched document (D) to the user terminal (10).

[0161] In another embodiment, the molecular structure recognition model (200) can receive a string according to a sequence representation of a molecular structure as an input for a user query, and recognize a molecular structure corresponding to the string of the molecular structure. Then, the control unit (140) can search for a document containing a molecular structure corresponding to the string of the molecular structure based on the recognized result, and provide the searched document to the user terminal (10).

[0162] As described above, according to the molecular structure recognition system and the control method thereof, and the learning method of the molecular structure recognition system according to the present invention, contrastive learning can be performed on a molecular structure recognition model using an embedding vector corresponding to a molecular structure image of each of a plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures. The molecular structure recognition model learned through this can accurately recognize molecules even in molecular structure images of various styles and orientation changes, and can accurately distinguish subtle molecular structure differences even in molecular structure images that contain small differences such as atomic changes, bond changes, and chirality changes. In addition, the molecular structure recognition model can clearly identify gradual differences (or degrees of deformation) from small structural changes to large changes in molecules, and can precisely analyze similarities and differences between molecules.

[0163] In addition, according to the molecular structure recognition system and control method thereof according to the present invention, and the learning method of the molecular structure recognition system, learning can be performed so as to maximize the similarity for pairs of molecular structure images and strings of the same molecular structure, and minimize the similarity for pairs of molecular structure images and strings of different molecular structures. Through this, the encoder of the molecular structure recognition model learned can extract meaningful features containing a deep understanding of the molecular structure from the molecular structure formula image without going through a direct image-to-molecule conversion process. In addition, the encoder of the molecular structure recognition model can more accurately grasp the details of complex molecules and minimize information loss during the translation process. In other words, the present invention can greatly expand the applicability in the field of chemical research by enabling the direct understanding and interpretation of molecular images.

[0164] Furthermore, according to the molecular structure recognition system and control method thereof, and the learning method of the molecular structure recognition system according to the present invention, it is possible to construct a learning data set by generating molecular structure images having different visual appearances while maintaining the characteristics of each of a plurality of molecular structures. In other words, the present invention can provide an environment in which a model can perform learning on molecular structure images of various styles while maintaining the structural consistency of molecules. Through this, the molecular structure recognition model can utilize molecular structure images of various styles as learning data, and can maintain consistent molecular recognition performance on molecular structure images of various styles without being biased toward molecular structure images of a specific style. In other words, the molecular structure recognition model can maintain robust molecular recognition capabilities through learning on molecular structure images of various styles, and thus can be widely utilized in patent searches, new drug development, chemical research, etc.

[0165] As such, the molecular structure recognition model according to the present invention can more clearly maintain structural similarities between molecules even in images of various styles. In particular, the encoder of the molecular structure recognition model is robust to style changes and can effectively distinguish molecular structures. Furthermore, it exhibits strong generalization performance even in new, untrained molecular structures, and can effectively recognize and search molecules through image embedding. In other words, the molecular structure recognition model according to the present invention can easily understand the structural features of molecules from molecular structure images themselves, just as humans can easily understand molecular structure images, thereby effectively extracting molecular-related features even for complex molecular structures.

[0166] Meanwhile, the present invention discussed above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a medium (or recording medium) that can be read by the computer.

[0167] Furthermore, the present invention discussed above can be implemented as computer-readable code or instructions on a program-recorded medium. In other words, the present invention can be provided in the form of a program.

[0168] Meanwhile, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0169] Furthermore, the computer-readable medium may include a storage device and may be a server or cloud storage device accessible via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage device via wired or wireless communication.

[0170] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and there is no particular limitation on its type.

[0171] Meanwhile, the above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A learning method of a molecular structure recognition system in which a memory and at least one processor are cooperatively performed, A step of inputting a molecular structure image of each of a plurality of molecular structures into an image encoder; A step of inputting a string according to the sequence expression method of each of the above multiple molecular structures into a molecule encoder; A step of obtaining an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures from the image encoder; A step of obtaining an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures from the molecular encoder; A step of storing, in the memory, an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence expression method of each of the plurality of molecular structures; and A learning method of a molecular structure recognition system, characterized in that contrastive learning is performed on a molecular structure recognition model of the molecular structure recognition system by using an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures stored in the memory and an embedding vector corresponding to a string according to a sequence expression method of each of the plurality of molecular structures.

2. In paragraph 1, Further comprising a step of constructing a learning data set including a molecular structure image of each of the plurality of molecular structures and a string according to a sequence expression method of each of the plurality of molecular structures, The above learning data set is, A learning method of a molecular structure recognition system, characterized in that a molecular structure image of each of the plurality of molecular structures and a string according to a sequence expression method of each of the plurality of molecular structures are constructed to form a pair.

3. In paragraph 2, A learning method of a molecular structure recognition system, characterized in that at least one molecular structure image having a different visual appearance is paired for each string according to the sequence expression method of each of the plurality of molecular structures.

4. In paragraph 3, The molecular structure image of each of the above plurality of molecular structures is, Including molecular structure images having different visual appearances generated using a preset molecular structure image generation method, In the step of building the learning data set, A learning method of a molecular structure recognition system characterized in that it generates molecular structure images having different visual appearances while maintaining the characteristics of each of the plurality of molecular structures.

5. In paragraph 2, Further comprising a step of specifying a molecular structure image of each of the plurality of molecular structures forming a pair and a string according to a sequence expression method of each of the plurality of molecular structures from the above learning data set, The above plural molecular structures are, A learning method of a molecular structure recognition system, characterized in that it comprises at least one of a first molecular structure and a second molecular structure.

6. In paragraph 5, The above specific steps are: A step of specifying a molecular structure image of the first molecular structure forming a pair and a string according to a sequence expression method of the first molecular structure; and A learning method of a molecular structure recognition system, characterized by including a step of specifying a molecular structure image of the second molecular structure forming a pair and a string according to a sequence expression method of the second molecular structure.

7. In paragraph 1, The step of inputting to the above molecular encoder is: A step of obtaining a molecular graph for each of the plurality of molecular structures by converting atoms into nodes and bonds between atoms into edges based on a string according to a sequence representation method of each of the plurality of molecular structures; and A learning method of a molecular structure recognition system, characterized in that it comprises a step of inputting a molecular graph for each of the plurality of molecular structures into the molecular encoder.

8. In paragraph 7, The step of obtaining the above molecular graph is: A step of obtaining the first molecular graph including nodes and edges corresponding to the first molecular structure by converting atoms constituting the first molecular structure into nodes and bonding relationships between atoms constituting the first molecular structure into edges using a preset tool; and Using the above tool, a step of converting atoms constituting the second molecular structure into nodes and converting bonding relationships between atoms constituting the second molecular structure into edges is included, thereby obtaining the second molecular graph including nodes and edges corresponding to the second molecular structure. In the step of inputting to the above molecular encoder, A learning method of a molecular structure recognition system characterized by inputting the first molecular graph and the second molecular graph obtained above into the molecular encoder.

9. In paragraph 8, In the above molecular encoder, an embedding vector corresponding to the first molecular graph is generated using nodes and edges corresponding to the first molecular graph, In the above molecular encoder, an embedding vector corresponding to the second molecular graph is generated using nodes and edges corresponding to the second molecular graph, A learning method of a molecular structure recognition system, characterized in that it obtains an embedding vector corresponding to the first molecular graph generated from the molecular encoder and an embedding vector corresponding to the second molecular graph.

10. In paragraph 1, The steps to input to the above image encoder are: Inputting the molecular structure image of the first molecular structure and the image of the second molecular structure into the image encoder, A learning method of a molecular structure recognition system, characterized in that an embedding vector corresponding to a molecular structure image of the first molecular structure and an embedding vector corresponding to a molecular structure image of the second molecular structure are obtained from the image encoder.

11. In paragraph 1, In the step of performing the above contrastive learning, In order to maximize the similarity between the molecular structure image of the first molecular structure forming the pair and the string according to the sequence expression method of the first molecular structure, learning is performed on the molecular structure recognition model using a preset loss function, A learning method of a molecular structure recognition system, characterized in that learning is performed on the molecular structure recognition model using the loss function so that the similarity between the molecular structure image of the second molecular structure forming the pair and the string according to the sequence expression method of the second molecular structure is maximized.

12. In paragraph 11, Maximizing the similarity between the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the first molecular graph so that the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the first molecular graph are included in the first embedding space, A learning method for a molecular structure recognition system, characterized in that the similarity between the embedding vector corresponding to the molecular structure image of the second molecular structure and the embedding vector corresponding to the second molecular graph is maximized so that the embedding vector corresponding to the molecular structure image of the second molecular structure and the embedding vector corresponding to the second molecular graph are included in the second embedding space.

13. In paragraph 1, In the step of performing the above contrastive learning, A learning method of a molecular structure recognition system, characterized in that learning is performed on the molecular structure recognition model using the loss function so that the similarity between the molecular structure image of the first molecular structure forming a pair and the string according to the sequence representation method of the first molecular structure and the molecular structure image of the second molecular structure forming a pair and the string according to the sequence representation method of the second molecular structure is minimized.

14. In paragraph 13, A learning method for a molecular structure recognition system, characterized in that the similarity between the embedding vector corresponding to the molecular structure image of the first molecular structure and the embedding vector corresponding to the molecular structure image of the second molecular structure, which is different from the embedding vector corresponding to the first molecular graph, and the embedding vector corresponding to the second molecular graph is minimized.

15. In paragraph 14, The above molecular structure recognition model is, A learning method of a molecular structure recognition system characterized in that it learns to recognize a molecular structure corresponding to a molecular structure image of each of the plurality of molecular structures, to recognize a molecular structure corresponding to a string according to a sequence expression method of each of the plurality of molecular structures, or to recognize a molecular structure corresponding to a molecular graph for each of the plurality of molecular structures.

16. In the molecular structure recognition system, The system comprises a memory and at least one processor, The above memory and the above processor cooperate to: Input the molecular structure image of each of the multiple molecular structures into the image encoder, A string according to the sequence expression method of each of the above multiple molecular structures is input into the molecule encoder, From the image encoder, an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures is obtained, From the above molecular encoder, an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures is obtained, A molecular structure recognition system characterized in that contrastive learning is performed on a molecular structure recognition model of the molecular structure recognition system by using an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence expression method of each of the plurality of molecular structures.

17. A program that is executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of inputting a molecular structure image of each of a plurality of molecular structures into an image encoder; A step of inputting a string according to the sequence expression method of each of the above multiple molecular structures into a molecule encoder; A step of obtaining an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures from the image encoder; A step of obtaining an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures from the molecular encoder; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing contrastive learning on a molecular structure recognition model of a molecular structure recognition system by using an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence expression method of each of the plurality of molecular structures.

18. A step of receiving a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal; A step of processing the user query as input to a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures; In the above molecular structure recognition model, a step of extracting information about the specific molecular structure corresponding to the user query; A step of processing information on the above-mentioned extracted specific molecular structure as input to a large-scale language model (LLM); A step of obtaining a response to the user query from the large-scale language model; and A control method for a molecular structure recognition system, characterized in that it comprises a step of providing a response to the user query to the user terminal.

19. Receive a user query related to a specific molecular structure among multiple molecular structures from a user terminal, A control unit that processes the user query as an input of a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures, The above control unit, Using the above molecular structure recognition model, information about the specific molecular structure corresponding to the user query is extracted, The information about the specific molecular structure extracted above is processed as input to a large-scale language model (LLM), From the above large-scale language model, obtain a response to the user query, A molecular structure recognition system characterized in that it provides a response to the user query to the user terminal.

20. A program executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of receiving a user query related to a specific molecular structure among a plurality of molecular structures from a user terminal; A step of processing the user query as input to a molecular structure recognition model learned based on contrastive learning between an embedding vector corresponding to a molecular structure image of each of the plurality of molecular structures and an embedding vector corresponding to a string according to a sequence representation method of each of the plurality of molecular structures; In the above molecular structure recognition model, a step of extracting information about the specific molecular structure corresponding to the user query; A step of processing information on the above-mentioned extracted specific molecular structure as input to a large-scale language model (LLM); A step of obtaining a response to the user query from the large-scale language model; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing a step of providing a response to the user query to the user terminal.

Images