Molecular structure recognition method and apparatus, electronic device and storage medium

Abstract

The present disclosure provides a molecular structure recognition method and apparatus, an electronic device and a storage medium. The method comprises: acquiring a molecule image; initializing an empty angle set and, when a molecular structure is decoded on the basis of image features of the molecule image and a branch angle is obtained for the first time, storing the branch angle into the angle set; extracting one branch angle from the angle set, using the branch angle as a guide to decode the molecular structure at the branch angle on the basis of the image features of the molecular image, and updating the angle set on the basis of a new decoded branch angle for decoding the molecular structure at the next branch angle until the angle set is empty; and, on the basis of decoding results at each branch angle, determining the molecular structure corresponding to the molecule image.

Description

Molecular structure recognition method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2023111369517, filed on September 4, 2023, entitled “Molecular structure recognition method, device, electronic device and storage medium,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the field of image recognition technology, and in particular to a molecular structure recognition method, device, electronic device, and storage medium. Background Art

[0004] Chemical molecular structure recognition can be widely used in fields including pharmaceutical research and development, human-computer interaction, biochemistry, education, organic synthesis, etc.

[0005] Currently, research on handwritten chemical molecular structure recognition relies too much on rule-based post-processing methods, which simplifies the complexity of molecular structure recognition itself. As a result, complex handwritten molecular layouts are difficult to parse and recognize under existing schemes.

[0006] Summary of the Invention

[0007] The present disclosure provides a molecular structure recognition method, device, electronic device and storage medium to solve the defect of the existing technology in facing the difficulty of complex handwritten molecular structure recognition.

[0008] The present disclosure provides a molecular structure recognition method, comprising:

[0009] Acquire molecular images;

[0010] Initializing an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, storing the branch angle in the angle set; extracting a branch angle from the angle set, using the branch angle as a guide, decoding the molecular structure at the branch angle based on the image features of the molecular image, and updating the angle set based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0011] Based on the decoding results at each branch angle, the molecular structure corresponding to the molecular image is determined.

[0012] According to a molecular structure recognition method provided by the present disclosure, the updating of the angle set based on the new branch angle obtained by decoding includes:

[0013] Perform chemical bond detection on the new branch angle and each existing branch angle in the angle set;

[0014] The angle set is updated based on the detection result of the chemical bond detection.

[0015] According to a molecular structure recognition method provided by the present disclosure, updating the angle set based on the detection result of the chemical bond detection includes:

[0016] If the detection result indicates that no chemical bond exists, storing the new branch angle in the angle set;

[0017] In the case where the detection result indicates the presence of a chemical bond, the branch angle that forms a chemical bond with the new branch angle is deleted from the angle set.

[0018] According to a molecular structure recognition method provided by the present disclosure, the molecular structure at the branching angle is decoded based on the image features of the molecular image using the branching angle as a guide, including:

[0019] Determining visual context features at a current decoding moment at the branch angle based on the decoding features of the branch angle and a decoding state at a previous decoding moment at the branch angle;

[0020] Based on the visual context features, the image features are decoded for molecular structure at the current decoding moment to obtain a decoding state at the current decoding moment, and the current decoding moment is returned as the previous decoding moment for decoding until the decoding at the branch angle is completed.

[0021] According to a molecular structure recognition method provided by the present disclosure, a branch angle is taken from an angle set, the molecular structure at the branch angle is decoded based on the image features of the molecular image using the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty, including:

[0022] Based on the recognition model, a branch angle is taken from the angle set, and the molecular structure at the branch angle is decoded based on the image features of the molecular image using the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0023] The recognition model is trained based on the sample image and the molecular structure label corresponding to the sample image;

[0024] The molecular structure label is obtained by connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image to form a molecular structure graph and then performing graph traversal.

[0025] According to a molecular structure recognition method provided by the present disclosure, the step of determining the molecular structure tag includes:

[0026] connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image into a molecular structure diagram;

[0027] The molecular structure graph is traversed, and the molecular structure label is generated based on the atomic groups, chemical bonds, angles, nesting symbols and reconnection marks obtained through the traversal.

[0028] According to a molecular structure recognition method provided by the present disclosure, the training step of the recognition model includes:

[0029] Based on the initial model, molecular structure recognition is performed on the sample image to obtain a chemical bond detection result between the sample branch angle decoded during the molecular structure recognition process and the existing branch angles in the sample angle set, as well as a structural recognition result of the sample image;

[0030] Based on the structure recognition result and the molecular structure label, as well as the chemical bond detection and the reconnection mark in the molecular structure label, the initial model is iterated to obtain the recognition model.

[0031] The present disclosure also provides a molecular structure recognition device, comprising:

[0032] an acquisition unit, for acquiring molecular images;

[0033] an identification unit, configured to initialize an empty angle set, and when decoding a molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, store the branch angle in the angle set; extract a branch angle from the angle set, use the branch angle as a guide, decode the molecular structure at the branch angle based on the image features of the molecular image, and update the angle set based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0034] The output unit is used to determine the molecular structure corresponding to the molecular image based on the decoding results at each branch angle.

[0035] The present disclosure also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the molecular structure recognition methods described above is implemented.

[0036] The present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the molecular structure recognition methods described above.

[0037] The present disclosure also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned molecular structure recognition methods.

[0038] The molecular structure recognition method, device, electronic device and storage medium provided by the present disclosure add a decoding mechanism for each branch angle in the chemical molecular structure and a maintenance mechanism for the branch angle to be explored, and use the branch angle in the chemical molecular structure as a guiding condition during decoding to enrich the information of molecular structure decoding, improve the reliability of molecular structure decoding, and improve the decoding accuracy for complex chemical molecular structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] FIG1 is a schematic diagram of a molecular structure recognition method according to the present invention;

[0041] FIG2 is a schematic diagram of molecular structure decoding provided by the present disclosure;

[0042] FIG3 is a handwritten molecular image provided by the present disclosure;

[0043] FIG4 is a molecular structure diagram provided by the present disclosure;

[0044] FIG5 is a molecular structure tag provided by the present disclosure;

[0045] FIG6 is a schematic structural diagram of a molecular structure recognition device provided by the present disclosure;

[0046] FIG7 is a schematic structural diagram of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of this disclosure more clear, the technical solutions of this disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of this disclosure without creative effort shall fall within the scope of protection of this disclosure.

[0048] In recent years, advances in computer vision technology in object detection and semantic segmentation have led to satisfactory recognition performance for simple and controllable printed molecular images. Simultaneously, significant progress has been made in the recognition of handwritten chemical molecular structures. However, these approaches may overlook the importance of handwritten molecular image recognition in human-computer interaction, oversimplify the complexity of molecular structure recognition, and rely too heavily on rule-based post-processing methods, resulting in the recognition of large and complex molecular structures remaining a major challenge.

[0049] FIG1 is a schematic diagram of a molecular structure recognition method provided by the present disclosure. As shown in FIG1 , the method includes:

[0050] Step 110: Acquire a molecular image.

[0051] The molecular image herein refers to an image with a chemical molecular structure drawn in. Furthermore, the chemical molecular structure in the molecular image may be printed or handwritten, which is not specifically limited in the present embodiment.

[0052] Step 120, initialize an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding to a branch angle for the first time, store the branch angle in the angle set; take out a branch angle from the angle set, and use the branch angle as a guide to decode the molecular structure under the branch angle based on the image features of the molecular image, and update the angle set based on the new branch angle obtained by decoding for decoding the molecular structure under the next branch angle, until the angle set is empty.

[0053] Taking into account the complexity of the chemical molecular structure itself, when performing molecular structure recognition on molecular images, a decoding and maintenance mechanism for the angles of each branch in the chemical molecular structure can be added to the conventional encoder + decoder architecture, and the branch angles in the chemical molecular structure can be used as guiding conditions during decoding to improve the decoding reliability and accuracy for complex chemical molecular structures.

[0054] In a specific implementation, branch-by-branch decoding can be performed based on the image features of the molecular image. It can be understood that the branch-by-branch decoding here is a graph traversal of the molecular image.

[0055] Figure 2 is a schematic diagram of molecular structure decoding provided by the present invention. As shown in Figure 2, in (a) of Figure 2, at the beginning of the graph traversal, an empty angle set Memory can be initialized. During the traversal process, a branch point is encountered. At this branch point, two branch angles can be decoded, namely branch angle ① and branch angle ②, where branch angle ① points upward and branch angle ② points to the lower right. Branch angle ① and branch angle ② are the branch angles decoded for the first time and are both stored in the angle set Memory.

[0056] Subsequently, in Figure 2(b), branch angle ① is retrieved from the angle set Memory, and the graph is traversed upward along branch angle ① until a new branch point is encountered. In Figure 2(c), two branch angles are decoded, namely branch angle ③ and branch angle ④, where branch angle ③ points to the right and branch angle ④ points downward. Both branch angles ③ and ④ are stored in the angle set Memory. Next, in Figure 2(d), branch angles ①, ③, ⑤, and ⑦ are retrieved from the angle set Memory in sequence to traverse the graph until a branch traversal is completed.

[0057] In Figure 2 (e), after completing the traversal of a branch, the branch angle ⑥ is taken out from the angle set Memory, and the traversal continues along the branch angle ⑥. At this time, it is found that the traversed atom is connected to the previously visited atom, that is, the atoms marked as two solid circles in Figure 2 (e) are connected. For this situation, in Figure 2 (f), it can be predicted whether the branch angle ⑧ stored in the angle set Memory is a loop key. If so, it can be considered that the branch angle ⑧ has completed the traversal and is taken out and deleted from the angle set Memory. The loop key this time is understood as whether the two branch angles form a reconnection relationship (reconnection), that is, a loop of branch angles is formed.

[0058] Based on the above traversal method, after completing the branch decoding traversal under all branch angles in the angle set Memory, it can be considered that the graph traversal for the molecular image is completed.

[0059] It can be understood that the angle set is a set for storing branch angles to be traversed. At the beginning of the graph traversal, an angle set can be initialized first. At this time, the angle set is empty, that is, at the beginning of the graph traversal, there are no branch angles to be traversed in the angle set. In the process of graph traversal, when the branch point is traversed for the first time, that is, when the branch angle is decoded for the first time, the branch angle decoded for the first time can be stored in the angle set. At this time, the angle set is no longer empty, and the branch angle to be traversed is stored in the angle set. In subsequent traversals, the branch angle can be taken out from the angle set for exploration. Thus, the angle set is used to store and maintain the branch angles to be traversed, and the angle set can be updated and adjusted accordingly based on the new decoded branch angle during the decoding process of the branch angle to facilitate the decoding of the next branch angle.

[0060] Furthermore, in the process of decoding any branch angle, it is necessary to first take out the branch angle from the angle set, that is, confirm that the branch angle is the branch angle being explored or has been explored, so that the angle set can always maintain the branch angle that has not been explored.

[0061] After taking out the branch angle, the molecular structure under the branch angle can be decoded. Here, when decoding, the molecular structure can be decoded along the branch angle, that is, the branch angle is used as conditional information to guide the decoding of the molecular structure, thereby enriching the information of the molecular structure decoding and improving the reliability of the molecular structure decoding.

[0062] In the process of decoding the molecular structure under the branch angle, if a new branch angle is decoded, the angle set can be updated based on the new branch angle. Specifically, the new branch angle can be added to the angle set as a branch angle that has not been explored for subsequent decoding. In addition, before adding the new branch angle as a branch angle that has not been explored to the angle set, the relationship between the branch angle and the existing branch angles in the angle set can be judged, that is, whether the new branch angle and the existing branch angles constitute a reconnection relationship in the molecular structure. If a reconnection relationship is formed, it can be considered that the new branch angle and the existing branch angles that form a reconnection relationship with it have been explored. The new branch angle does not need to be stored in the angle set, and the existing branch angles that form a reconnection relationship with it can also be deleted from the angle set, thereby enabling the angle set to always maintain the branch angles that have not been explored.

[0063] Step 130: Determine the molecular structure corresponding to the molecular image based on the decoding results at each branch angle.

[0064] Specifically, after decoding the molecular structure at each branch angle in the molecular image, the decoding results at each branch angle can be obtained. The decoding results here may include chemical bonds, angles, atomic groups, etc. at the branch angle. On this basis, combined with the decoding results at each branch angle, the molecular structure corresponding to the molecular image can be obtained. The molecular structure here can be a string recorded in a markup language based on an organic chemical structural formula, or a representation diagram of the molecular structure, or a vector representation of the molecular structure under a specific coding language. The embodiments of the present disclosure do not specifically limit this.

[0065] The method provided by the embodiments of the present disclosure adds a decoding mechanism for each branch angle in the chemical molecular structure and a maintenance mechanism for the branch angle to be explored, and uses the branch angle in the chemical molecular structure as a guiding condition during decoding to enrich the information of molecular structure decoding, improve the reliability of molecular structure decoding, and improve the decoding accuracy for complex chemical molecular structures.

[0066] Based on the above embodiment, in step 120, updating the angle set based on the new branch angle obtained by decoding includes:

[0067] Perform chemical bond detection on the new branch angle and each existing branch angle in the angle set;

[0068] The angle set is updated based on the detection result of the chemical bond detection.

[0069] Specifically, considering that there may be reconnection relationships in the molecular structure, for example, the single bonds of 6 in the benzene ring are connected end to end, if we do not consider whether the new branch angle obtained by decoding constitutes a chemical bond with the branch angle that has not been explored in the angle set, it is very likely to lead to repeated traversal, that is, decoding the molecular structures under the two branch angles separately, so that the molecular structure that should have been under one branch is incorrectly decoded into two molecular structures, affecting the accuracy of molecular structure recognition.

[0070] Therefore, after decoding and obtaining a new branch angle, chemical bond detection can be performed on the new branch angle and each existing branch angle in the angle set. Here, for the new branch angle and any branch angle in the angle set, a test can be performed to determine whether a chemical bond is formed between the two branch angles. The test result obtained can include whether a chemical bond exists and, if a chemical bond is determined to exist, further include the type of chemical bond, such as a single bond, double bond, etc.

[0071] After obtaining the chemical bond detection results between the new branch angle and each existing branch angle in the angle set, the angle set can be updated based on this. It is understandable that if there is no chemical bond between the new branch angle and each existing branch angle, it means that a reconnection relationship has not yet been decoded, and the new branch angle can also be placed in the angle set as an unexplored branch angle; if there is a chemical bond between the new branch angle and one of the existing branch angles, it means that a reconnection relationship has been decoded, and the existing branch angle that has a chemical bond with the new branch angle will be deleted from the angle set, and the new branch angle will not be stored in the angle set.

[0072] Based on any of the above embodiments, in step 120, updating the angle set based on the detection result of the chemical bond detection includes:

[0073] If the detection result indicates that no chemical bond exists, storing the new branch angle in the angle set;

[0074] In the case where the detection result indicates the presence of a chemical bond, the branch angle that forms a chemical bond with the new branch angle is deleted from the angle set.

[0075] Specifically, for a new branch angle, if all the detection results of the chemical bond detection between the branch angle and each existing branch angle indicate that there is no chemical bond, that is, there is no chemical bond between the new branch angle and each existing branch angle, it means that the reconnection relationship has not been decoded yet, and the new branch angle also needs to be explored. At this time, the new branch angle can be stored in the angle set, so that the angle set can continue to maintain all the branch angles that have not been explored;

[0076] If there is a detection result indicating the existence of a chemical bond, that is, there is a chemical bond between the new branch angle and an existing branch angle in the angle set, then there is a reconnection relationship between the new branch angle and the existing branch angle. It can be considered that the new branch angle and the existing branch angle that has a reconnection relationship with it have been explored. The new branch angle does not need to be stored in the angle set, and the existing branch angle that has a reconnection relationship with it can also be deleted from the angle set, so that the angle set can always maintain the branch angles that have not been explored.

[0077] Based on any of the above embodiments, in step 120, the decoding of the molecular structure at the branching angle based on the image features of the molecular image using the branching angle as a guide includes:

[0078] Determining visual context features at a current decoding moment at the branch angle based on the decoding features of the branch angle and a decoding state at a previous decoding moment at the branch angle;

[0079] Based on the visual context features, the image features are decoded for molecular structure at the current decoding moment to obtain a decoding state at the current decoding moment, and the current decoding moment is returned as the previous decoding moment for decoding until the decoding at the branch angle is completed.

[0080] Specifically, when decoding a molecular structure at a branch angle, the branch angle can be used as a guiding condition for molecular structure decoding. When a branch is encountered, decoding can continue along the direction of the branch angle extracted from the angle set, and the branch angle applied at this time can guide the generation of visual context features when decoding the molecular structure at that branch angle.

[0081] That is, when generating visual context features at a decoding moment, compared to the prior art method of only using the decoding state at the previous decoding moment to determine the visual context features at the current decoding moment, the decoding features of the branch angle are also considered. It is understood that the decoding features of the branch angle here are stored when the branch angle is decoded during the decoding process of the molecular structure. The decoding features of the branch angle can specifically include the attention weight information and state features of the branch angle.

[0082] For example, the calculation of the visual context features at the current decoding moment can be expressed as the following formula:

[0083] Where, e t,i is the energy of the i-th position in the feature map at the t-th decoding moment, w, W x 、W y 、W S 、W a 、W sp 、W ap are the projection parameters of the attention module. i is the image feature of the molecular image at the i-th position, is the decoding result at the t-1th decoding time, s t-1 is the decoding state at the t-1th decoding moment, a t-1 、a t are the attention weights at the t-1th decoding moment and the tth decoding moment, respectively, a b 、s b are the attention weight information and state features when decoding the branch angle. It should be noted that when there is no branch angle for decoding, a b and s b are all zero vectors, then e t,i The calculation formula is the same as the conventional string decoder visual context feature acquisition method et,i The calculation formula is consistent.

[0084] a t,i is the attention weight at the ith position at the tth decoding moment, h and w are the height and width of the molecular image respectively; c t is the visual context feature at the t-th decoding moment.

[0085] After obtaining the visual context features at the current decoding moment, the molecular structure decoding at the current decoding moment can be performed based on this, thereby obtaining the decoding state at the current decoding moment, which can be specifically expressed as the following formula: p t =softmax(W c s t );

[0086] Where s t That is, the decoding state at the current decoding moment, and GRU represents the computational unit of the recurrent neural network. t Represents the probability distribution of the decoding result output at the current decoding moment, W c are the parameters of the classification layer.

[0087] It is understood that after completing molecular structure decoding at the current decoding moment, the decoding state at the current decoding moment can be used as the decoding state at the previous decoding moment, and the molecular structure decoding at the next decoding moment can be executed. Moreover, in this process, the decoding features of the same branch angle are applied at different decoding moments of the molecular structure at the same branch angle.

[0088] In the disclosed embodiments, the neural network architecture used to decode molecular structures is referred to as a stochastic conditional guidance decoder. This decoder significantly outperforms traditional string decoders in generalizing to complex structures without adding significant additional parameters or computational burden. The use of conditional guidance mechanisms and path selection effectively improves the accuracy of multi-path decoding.

[0089] Based on any of the above embodiments, step 120 includes:

[0090] Based on the recognition model, a branch angle is taken from the angle set, and the molecular structure at the branch angle is decoded based on the image features of the molecular image using the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0091] The recognition model is trained based on the sample image and the molecular structure label corresponding to the sample image;

[0092] The molecular structure label is obtained by connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image to form a molecular structure graph and then performing graph traversal.

[0093] Specifically, step 120 can be implemented using a pre-trained recognition model. Here, the recognition model can be in the form of an encoder-decoder. Training of the recognition model can be implemented through supervised learning. It is understood that the sample for supervised learning is a sample image depicting a chemical molecular structure, and the label for supervised learning is the molecular structure label corresponding to the sample image. Here, the molecular structure label reflects the molecular structure in the form of language.

[0094] Conventional markup languages ​​for organic chemical structures can be Chemfig, SMILES, etc. Compared with the most commonly used structural markup language SMILES, Chemfig requires almost no abstract chemical knowledge. Its design purpose is to use the chemical structure graphics themselves to improve relevance and accuracy. Chemfig's syntax also provides a description of "angles", which can be used to specify the direction of bonds in molecules. In the handwritten molecular image shown in Figure 3, the atom "N" is connected to two double bonds, one pointing to the upper right and the other pointing to the lower right. The angle codes in the Chemfig syntax, such as "[1]" and "[-1]", can be used to indicate the approximate orientation of the two double bonds, thereby making the visual presentation of the molecule more accurate and comprehensive.

[0095] However, if the original Chemfig markup language is directly used as the label for recognition model training, the following problems still exist:

[0096] First, Chemfig suffers from grammatical ambiguity: due to different starting points and traversal orders, there are multiple correct Chemfig sequences that can represent the same molecular image. This ambiguity is unavoidable and increases the difficulty of model learning.

[0097] Secondly, Chemfig syntax is complex: a priori rules and domain knowledge need to be incorporated into the syntax to keep it simple. For example, rules such as using brackets to group atoms and removing redundant chemical symbols and valence dashes need to reduce grammatical complexity. In addition, domain knowledge such as IUPAC (International Union of Pure and Applied Chemistry) naming rules is necessary to ensure that compound structures are correctly labeled. The use of these rules and domain knowledge increases the complexity of Chemfig syntax. For example, a benzene ring is simply represented as "*6(––)", and the directions of the six single bonds need to be automatically inferred based on the rules for the benzene ring. These a priori rules are complex and increase the difficulty of learning the model.

[0098] To overcome the above problems, the disclosed embodiment proposes parsing the atomic groups and chemical bonds in the molecular formula corresponding to the sample image and connecting them into a molecular structure graph, and then performing graph traversal to obtain a molecular structure label.

[0099] Here, considering that the molecular formula corresponding to the sample image can be represented by different character strings, in order to resolve this ambiguity at the grammatical level, in the embodiment of the present disclosure, the atomic groups and chemical bonds are manually parsed and restored from the molecular formula character string, and then all the obtained atomic groups and chemical bonds are connected to generate a graph, which is recorded here as a molecular structure diagram.

[0100] For example, "HO-**6(---(-COOH)---)", "COOH-[4]**6(---(-OH)---)", and "**6([:30]-(-OH)---(-COOH)--)" are different string representations of the same molecular formula. By parsing and restoring the atomic groups and chemical bonds, a molecular structure diagram as shown in Figure 4 can be constructed, thereby achieving unified representation in the form of a structure diagram.

[0101] In Figure 4, character strings are represented as atom groups, such as "HO" and "COOH." The ring-shaped benzene molecule is considered a special atom group. The lines between atom groups represent chemical bonds, which can be single bonds "-," double bonds "=", or triple bonds. An atom group can be connected to another atom group through a single or multiple chemical bonds. By using atom groups as vertices and chemical bonds as edges, a molecular structure diagram can be generated. This allows the graphical representation of the same molecule to be identical even when using different Chemfig label strings, eliminating potential ambiguity in labeling.

[0102] The resulting molecular structure graph is a complex data structure that requires graph traversal to convert it into a label format suitable for model training. Specifically, the graph traversal is used to convert the molecular structure graph into an equivalent one-dimensional text representation. The conversion rule can be a fixed rule, specifically starting from a specific point in the molecular structure graph and appending the traversed nodes (atomic groups) and edges (chemical bonds) to the end of the string, thereby obtaining a concise and grammatically ambiguous molecular structure label.

[0103] The method provided in the embodiment of the present disclosure connects the atomic groups and chemical bonds in the molecular formula into a molecular structure graph and then performs a graph traversal to obtain a concise and grammatically ambiguous molecular structure label, which can reduce the difficulty of supervised learning of the recognition model and improve the reliability of molecular structure recognition based on the recognition model.

[0104] Based on any of the above embodiments, the step of determining the molecular structure tag includes:

[0105] connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image into a molecular structure diagram;

[0106] The molecular structure graph is traversed, and the molecular structure label is generated based on the atomic groups, chemical bonds, angles, nesting symbols and reconnection marks obtained through the traversal.

[0107] Specifically, the molecular structure graph obtained by structural analysis and concatenation of molecular formulas is a complex data structure that must be converted into a form suitable for model training. For encoder-decoder recognition models, suitable training formats are typically one-dimensional strings or standardized labels. Standardized labels involve converting the molecular structure graph into an equivalent one-dimensional text representation.

[0108] In order to ensure that the rules for converting molecular structure labels are consistent and unambiguous, the specific rules can be pre-set fixed rules. For the traversal of the molecular structure graph, the molecular structure graph can be traversed from a specific point, and the nodes (atomic groups) and edges (chemical bonds) are appended to the end of the string. After the traversal is completed, a standardized label can be obtained. For example, by traversing the molecular structure graph shown in Figure 4, the following standardized labels can be obtained:

[0109] H O-[:0]? [a](-[:60]-[:0]-[:300](-[:0]COOH)-[:240]-[:180]?[a,{-}](--[:60]\circle))

[0110] Each unit is separated by a space. Graph traversal can use Depth-First Search (DFS). The above standardized labels can contain four types of elements, namely "atom group", "chemical bond", "angle", "nested symbol" and "reconnection mark". Specifically, in the above standardized labels, "HO" represents the atom group, "?[a]", "?[a,{-}]" are reconnection marks, "-" represents a chemical bond, "[:0]" represents an angle, and "(" represents a nested symbol.

[0111] Based on this, standardized labels can be integrated to create structure-specific molecular structure labels. Atom groups can be omitted if they do not contain any characters. Entity characters can be represented using LaTeX notation. Because "angle" and "chemical bond" are closely related in visual information, the standardized labels can be appended with the corresponding angle as "[:]" to represent the "bond angle" modeling unit. The angle value can be calculated based on the format during graphic construction and output. Furthermore, the nesting symbol "()" can be used to represent nested branches. Furthermore, the original Chemfig syntax can be retained for reconnection notation, represented by "?[a]", which represents a unit surrounded by a cyclic unit represented by "?[a,-]". When reconnection occurs, the reconstructed angle value is not displayed. Finally, "○" can be used to represent specific atoms in benzene, and a virtual bond angle unit "–[:]" can be constructed to represent the connection between "○" and any atom on the ring.

[0112] For example, the standardized labels shown above can be integrated to form the molecular structure label shown in Figure 5. The molecular structure label shown in Figure 5 marks the nesting relationship and reconnection relationship at different angles. Among them, "\eob" indicates that the decoding is completed at a branch angle, the arrow line between "\angle[:60]" and "-[:60]" indicates the nesting relationship, and the double-arrow line between "\angle[:300]" and "\angle[:120]" and "reconnection" indicate the reconnection mark.

[0113] The method provided by the embodiment of the present disclosure connects the atomic groups and chemical bonds in the molecular formula into a molecular structure diagram, and forms a molecular structure label by traversing the molecular structure diagram. The molecular structure label thus formed has the advantage of being more consistent with the image than the mainstream language SMILES, and is more conducive to model learning. In addition, the syntax of the molecular structure label is not restricted by chemical knowledge, which enables it to represent incorrect or non-existent molecular structures and makes it more versatile. The above method provides an innovative method for the characterization of molecular structures, and has potential application prospects in various fields of chemical structure recognition and analysis.

[0114] Based on any of the above embodiments, the steps of training the recognition model include:

[0115] Based on the initial model, molecular structure recognition is performed on the sample image to obtain a chemical bond detection result between the sample branch angle decoded during the molecular structure recognition process and the existing branch angles in the sample angle set, as well as a structural recognition result of the sample image;

[0116] Based on the structure recognition result and the molecular structure label, as well as the chemical bond detection and the reconnection mark in the molecular structure label, the initial model is iterated to obtain the recognition model.

[0117] Specifically, for the recognition model, an initial model of an encoder + decoder structure can be obtained, where the model parameters of the initial model can be initialized. During the training process, molecular structure recognition can be performed on the sample image based on the initial model. Specifically, image features can be extracted from the sample image based on the initial model, and then a sample branch angle is taken from the sample angle set. Guided by the sample branch angle, the molecular structure under the sample branch angle is decoded based on the image features of the sample image, and the sample angle set is updated based on the new sample branch angle obtained by decoding for the molecular structure decoding under the next sample branch angle, until the sample angle set is empty.

[0118] It can be understood that when the initial model is decoded to obtain a new sample branch angle, the sample angle set needs to be updated based on the new sample branch angle obtained by decoding. When updating the sample angle set, the new sample branch angle needs to be chemically bonded with each existing branch angle in the sample angle set respectively, thereby obtaining the chemical bond detection result between the sample branch angle and the existing branch angles in the sample angle set.

[0119] Here, the presence or absence of chemical bonds in the chemical bond detection results is used to reflect whether a reconnection relationship exists between the branch angles of the two samples. The chemical bond detection results can be compared with the reconnection mark in the molecular structure label to measure the training loss of the initial model from the perspective of reconnection relationship detection.

[0120] In addition, the structure recognition results and molecular structure labels can be compared to measure the training loss of the initial model from the perspective of molecular structure recognition.

[0121] Furthermore, the loss L of molecular structure recognition can be expressed based on the following formula: ce :

[0122] Where, L ce is the cross entropy loss, V represents the number of model unit characters, y ti Is the molecular structure label the i-th character at the t-th decoding moment, p ti It represents the probability that the structure recognition result at the tth decoding moment is the i-th character.

[0123] The detection loss L of the reconnection relationship can be expressed based on the following formula bc , that is, the loss caused by the cycle: q tb=softmax(W m s b +W o s t );

[0124] Where s t is the decoding state of the branch angle obtained by decoding at the current decoding time t. m and W o All of them are parameters used for chemical bond detection. The state characteristics s of the existing branch angles b stored in the angle set b The probability distribution of chemical bond detection between t and the branch angle t is expressed as Where N is the total number of chemical bond types, and N+1 includes N chemical bonds and the case where there is no chemical bond.

[0125] B is the total number of branch angles in the angle set, z tbi Is the chemical bond type between the molecular structure label and the branch angle b detected at the tth decoding moment i, q tbi It represents the probability that the chemical bond type i between the branch angle b is detected at t decoding moments in the chemical bond detection results.

[0126] Combining the above two losses, we can get the overall loss for recognition model training. The overall loss L here can be expressed as L ce +L bc , which can also be expressed as L ce and L bc The result of the weighted summation is not specifically limited in the embodiment of the present disclosure. After obtaining the overall loss, the parameters of the initial model can be iterated based on the overall loss to obtain a fully trained recognition model.

[0127] The method provided in the embodiments of the present disclosure realizes molecular structure recognition for molecular images through end-to-end modeling and training.

[0128] Based on any of the above embodiments, the initial model is an encoder + decoder structure. The encoder is a DenseNet, which includes three dense blocks for converting the input RGB three-channel image into high-dimensional features. The growth rate and depth of each dense block are set to 24 and 32, respectively. In addition, both the encoder and decoder use a GRU with a hidden state dimension of 256 as the recurrent unit of the RNN, and the dimension of the attention projection parameter is set to 128. In addition, the dimension of the embedding parameter is set to 256, and a drop-out rate of 15% is applied. For the decoder, the projection parameter dimension of the chemical bond detection classification is set to 256.

[0129] The optimizer applied for parameter iteration of the initial model can be Adam, the initial learning rate is 2e-4, the learning rate decay strategy is multi-step decay, Pytorch's MultiStepLR is used to adjust the learning rate, and the decay factor gamma is set to 0.5.

[0130] Based on any of the above embodiments, the present disclosure provides a molecular structure recognition method implemented based on a recognition model. The recognition model includes a random conditional decoder, distinct from a string decoder. The random conditional decoder incorporates three mechanisms to address existing issues: conditional attention, looping, and path selection.

[0131] Among them, the conditionally guided attention mechanism is able to take advantage of the natural graph structure of molecular structures and regard its recognition process as a graph traversal problem. When the recognition model traverses the graph, it encounters multiple branch angles, and the order of these angles in the proposed modeling unit follows a fixed counterclockwise direction. However, if decoding is performed only according to a fixed angle order, the model may forget which angle units have not yet been decoded due to the extension of the later decoding time. To solve this problem, under the conditionally guided attention mechanism, branch angles can be used as conditional information to guide the decoding process. When the recognition model encounters a branch, it will continue decoding along the specified branch angle direction. The updated decoder no longer uses "(" and ")" to indicate the start and end of the branch. Instead, the branch angle of each branch is first predicted, that is, "\angle[:<angle value> ]", the decoded features of each branch angle obtained by decoding are stored separately in the angle set Memory, that is, the context and attention weight information of each branch angle.

[0132] Subsequently, in the decoding process of the molecular structure at a branch angle, when “\eob” is decoded, it indicates that there are no additional branch angles to be predicted, and a decoding feature of a branch angle can be selected from the angle set Memory as a condition for continuing the decoding process.

[0133] In addition, the main difference between the graph structure and the tree structure is that the graph structure has a cyclic characteristic. In the molecular structure recognition method provided by the embodiment of the present disclosure, the branch angle corresponding to the loop has been stored in the angle set Memory and has not been decoded. Therefore, a simple multi-label classification module can be constructed for chemical bond detection to determine the direction corresponding to the loop angle and classify the type of loop bond (such as single bond, double bond, etc.). The result obtained by chemical bond detection may be empty or the type of bond, and empty means that there is no loop with a corresponding branch angle that has not yet been decoded. If there is a cyclic chemical bond in the test result, the corresponding branch angle is deleted from the angle set Memory, and the branch angle obtained by decoding at the current moment is not stored. On the contrary, if there is no loop, that is, the test result is empty, the branch angle at the current moment is saved in the angle set Memory for future decoding selection.

[0134] Finally, the recognition process of the random conditional decoder can be viewed as a graph traversal problem, where different traversals of the graph can produce multiple target sequences. Therefore, a single graph can have multiple training labels. Although the conditionally guided attention mechanism still decodes according to a fixed counterclockwise order, it may cause overfitting and recognition errors in complex or uncommon structures. Therefore, the disclosed embodiment can be combined with a path selection mechanism for decoding, which randomly selects different paths during training to improve the alignment between visual information and decoded characters using a conditionally guided attention mechanism. During the inference process, the recognition model can be made to attempt to decode all branch angles stored in the angle set and participate in the calculation of the beam search path score PK, thereby selecting the path with the highest score to continue decoding.

[0135] Through the above method, the recognition model can have better versatility and recognition efficiency.

[0136] Furthermore, for training the recognition model, the disclosed embodiments use the publicly accessible CASIA-CSDB dataset as a benchmark to establish a handwritten dataset for sample images. Specifically, real-world handwritten molecular structures can be obtained from educational environments. This dataset includes many examples with writing errors and non-existent structural data.

[0137] Furthermore, the handwritten dataset consists of 52,987 images of handwritten molecular structures collected in educational settings. These images were acquired using various devices, including cameras, scanners, and screens, and labeled with native Chemfig strings. Furthermore, the dataset contains a mixture of molecular structures and regular molecular formulas. In addition to isolated molecular structures, many instances in the dataset contain combinations of formulas and molecular structures. Furthermore, the dataset incorporates different structural writing styles, such as the use or absence of abbreviations, types of Keckler ring representations for benzene, and the inclusion or exclusion of hydrogen atoms. The dataset also contains numerous examples of human-written errors that violate chemical principles, or even non-existent structures. The model trained on this data can potentially be used to identify such structures and correct handwritten answers. Furthermore, to verify the generalization performance of the model, the molecular structures in the dataset need to maintain a certain level of complexity. Specifically, the total number of atoms and bonds can be used to assess the complexity level of the molecular structure. Approximately 10% of the dataset has a complexity level exceeding that of the most complex examples in the conventional training set.

[0138] Based on any of the above embodiments, after the recognition model training is completed, an experimental test can be performed on the recognition model.

[0139] Specifically, considering that the molecular structure label itself is concise enough, we can directly compare the predicted recognition result string with the label string to see if they are the same. We can convert both the label string and the recognition result string into molecular structure diagrams, and then compare the two molecular structure diagrams to see if they match. Assuming T is the number of samples and R is the number of recognition results that match the label, the Exact Match (EM) score can be calculated using the following formula:

[0140] In addition, considering that there is a mixture of formulas and molecular structures in handwritten datasets, a single image may contain multiple molecular structures. Therefore, the embodiment of the present disclosure defines two auxiliary indicators as follows:

[0141] Structure EM: For samples containing mixed molecular structures and regular formulas, when all molecular structure recognition results match the labeled graphics, the structure in the sample is determined to be correctly recognized. Let T be the number of samples, R struct Represents the number of correctly identified structure samples, and the structure EM score is obtained as follows:

[0142] Structural EM Fractional EM struct And a single EM score EM can measure the recognition performance of the recognition model for molecular structures in handwritten datasets under mixed modes.

[0143] From the experimental results, the molecular structure recognition method provided by the disclosed embodiment significantly improves recognition performance compared to the SMILES-based recognition method. This is mainly due to the reduced ambiguity of the molecular structure label and the stronger consistency between the image and the label. In addition, the random conditional decoder proposed in the disclosed embodiment has significant advantages over the string decoder. It should be emphasized that the computational and parameter costs of the random conditional decoder are almost the same as those of the string decoder.

[0144] Based on any of the above embodiments, FIG6 is a schematic structural diagram of a molecular structure recognition device provided by the present disclosure. As shown in FIG6 , the device includes:

[0145] An acquisition unit 610 is configured to acquire a molecular image;

[0146] The identification unit 620 is configured to initialize an empty angle set, and when decoding a molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, store the branch angle in the angle set; extract a branch angle from the angle set, use the branch angle as a guide, decode the molecular structure at the branch angle based on the image features of the molecular image, and update the angle set based on the decoded new branch angle for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0147] The output unit 630 is configured to determine the molecular structure corresponding to the molecular image based on the decoding results at each branch angle.

[0148] The device provided by the embodiment of the present disclosure adds a decoding mechanism for each branch angle in the chemical molecular structure and a maintenance mechanism for the branch angle to be explored, and uses the branch angle in the chemical molecular structure as a guiding condition during decoding to enrich the information of molecular structure decoding, improve the reliability of molecular structure decoding, and improve the decoding accuracy for complex chemical molecular structures.

[0149] Based on any of the above embodiments, the identification unit 620 includes a set updating unit configured to:

[0150] Perform chemical bond detection on the new branch angle and each existing branch angle in the angle set;

[0151] The angle set is updated based on the detection result of the chemical bond detection.

[0152] Based on any of the above embodiments, the set updating unit is specifically configured to:

[0153] If the detection result indicates that no chemical bond exists, storing the new branch angle in the angle set;

[0154] In the case where the detection result indicates the presence of a chemical bond, the branch angle that forms a chemical bond with the new branch angle is deleted from the angle set.

[0155] Based on any of the above embodiments, the identification unit 620 includes a branch decoding unit configured to:

[0156] Determining visual context features at a current decoding moment at the branch angle based on the decoding features of the branch angle and a decoding state at a previous decoding moment at the branch angle;

[0157] Based on the visual context features, the image features are decoded for molecular structure at the current decoding moment to obtain a decoding state at the current decoding moment, and the current decoding moment is returned as the previous decoding moment for decoding until the decoding at the branch angle is completed.

[0158] Based on any of the above embodiments, the identification unit 620 is specifically configured to:

[0159] Based on the recognition model, a branch angle is taken from the angle set, and the molecular structure at the branch angle is decoded based on the image features of the molecular image using the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0160] The recognition model is trained based on the sample image and the molecular structure label corresponding to the sample image;

[0161] The molecular structure label is obtained by connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image to form a molecular structure graph and then performing graph traversal.

[0162] Based on any of the above embodiments, the device further includes a label conversion unit, configured to:

[0163] connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image into a molecular structure diagram;

[0164] The molecular structure graph is traversed, and the molecular structure label is generated based on the atomic groups, chemical bonds, angles, nesting symbols and reconnection marks obtained through the traversal.

[0165] Based on any of the above embodiments, the device further includes a model training unit, configured to:

[0166] Based on the initial model, molecular structure recognition is performed on the sample image to obtain a chemical bond detection result between the sample branch angle decoded during the molecular structure recognition process and the existing branch angles in the sample angle set, as well as a structural recognition result of the sample image;

[0167] Based on the structure recognition result and the molecular structure label, as well as the chemical bond detection and the reconnection mark in the molecular structure label, the initial model is iterated to obtain the recognition model.

[0168] FIG7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG7 , the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute a molecular structure recognition method, which includes:

[0169] Acquire molecular images;

[0170] Initializing an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, storing the branch angle in the angle set; extracting a branch angle from the angle set, using the branch angle as a guide, decoding the molecular structure at the branch angle based on the image features of the molecular image, and updating the angle set based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0171] Based on the decoding results at each branch angle, the molecular structure corresponding to the molecular image is determined.

[0172] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0173] On the other hand, the present disclosure further provides a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the molecular structure recognition method provided by the above methods, which includes:

[0174] Acquire molecular images;

[0175] Initializing an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, storing the branch angle in the angle set; extracting a branch angle from the angle set, using the branch angle as a guide, decoding the molecular structure at the branch angle based on the image features of the molecular image, and updating the angle set based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0176] Based on the decoding results at each branch angle, the molecular structure corresponding to the molecular image is determined.

[0177] In another aspect, the present disclosure further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the molecular structure recognition method provided by the above methods, the method comprising:

[0178] Acquire molecular images;

[0179] Initializing an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding a branch angle for the first time, storing the branch angle in the angle set; extracting a branch angle from the angle set, using the branch angle as a guide, decoding the molecular structure at the branch angle based on the image features of the molecular image, and updating the angle set based on the new branch angle obtained by decoding for decoding the molecular structure at the next branch angle, until the angle set is empty;

[0180] Based on the decoding results at each branch angle, the molecular structure corresponding to the molecular image is determined.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0182] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A molecular structure recognition method comprising: Acquire molecular images; Initialize an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding to a branch angle for the first time, store the branch angle in the angle set; take a branch angle from the angle set, and decode the molecular structure under the branch angle based on the image features of the molecular image with the branch angle as a guide, and update the angle set based on the new branch angle obtained by decoding for decoding the molecular structure under the next branch angle, until the angle set is empty; Based on the decoding results at each branch angle, the molecular structure corresponding to the molecular image is determined.

2. The molecular structure recognition method according to claim 1, wherein: The updating of the angle set based on the new branch angle obtained by decoding includes: Perform chemical bond detection on the new branch angle and each existing branch angle in the angle set; Based on the detection result of the chemical bond detection, the angle set is updated.

3. The molecular structure recognition method according to claim 2, wherein: The updating of the angle set based on the detection result of the chemical bond detection includes: When the detection result indicates that there is no chemical bond, storing the new branch angle into the angle set; When the detection result indicates the presence of a chemical bond, the branch angle that forms a chemical bond with the new branch angle is deleted from the angle set.

4. The molecular structure recognition method according to claim 1, wherein: The decoding of the molecular structure at the branching angle based on the image features of the molecular image and guided by the branching angle includes: Based on the decoding features of the branch angle and the previous solution under the branch angle The decoding state at the decoding moment determines the visual context features at the current decoding moment under the branch angle; Based on the visual context feature, the molecular structure of the image feature at the current decoding moment is decoded to obtain the decoding state at the current decoding moment, and the current decoding moment is returned as the previous decoding moment for decoding until the decoding at the branch angle is completed.

5. The molecular structure recognition method according to any one of claims 1 to 4, wherein: The step of taking a branch angle from the angle set, decoding the molecular structure at the branch angle based on the image features of the molecular image with the branch angle as a guide, and updating the angle set based on a new branch angle obtained by decoding for decoding the molecular structure at the next branch angle until the angle set is empty comprises: Based on the recognition model, a branch angle is taken out from the angle set, and the molecular structure under the branch angle is decoded based on the image features of the molecular image with the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding of the molecular structure under the next branch angle, until the angle set is empty; The recognition model is trained based on the sample image and the molecular structure label corresponding to the sample image; The molecular structure label is obtained by connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image to form a molecular structure graph and then performing graph traversal.

6. The molecular structure recognition method according to claim 5, wherein: The step of determining the molecular structure tag comprises: Connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image into a molecular structure diagram; The molecular structure graph is traversed, and the molecular structure label is generated based on the atomic groups, chemical bonds, angles, nesting symbols and reconnection marks obtained through the traversal.

7. The molecular structure recognition method according to claim 5, wherein: The training steps of the recognition model include: Based on the initial model, molecular structure recognition is performed on the sample image to obtain a chemical bond detection result between the sample branch angle decoded during the molecular structure recognition and the existing branch angles in the sample angle set, as well as a structural recognition result of the sample image; Based on the structure recognition result and the molecular structure label, as well as the chemical bond detection and the reconnection mark in the molecular structure label, the initial model is iterated with parameters to obtain the recognition model.

8. A molecular structure recognition device, comprising: An acquisition unit, used for acquiring a molecular image; The identification unit is used to initialize an empty angle set, and when decoding the molecular structure based on the image features of the molecular image and decoding to a branch angle for the first time, store the branch angle in the angle set; take out a branch angle from the angle set, and decode the molecular structure under the branch angle based on the image features of the molecular image with the branch angle as a guide, and update the angle set based on the new branch angle obtained by decoding for decoding the molecular structure under the next branch angle, until the angle set is empty; The output unit is used to determine the molecular structure corresponding to the molecular image based on the decoding results at each branch angle.

9. The molecular structure recognition device according to claim 8, wherein: The updating of the angle set based on the new branch angle obtained by decoding includes: Perform chemical bond detection on the new branch angle and each existing branch angle in the angle set; Based on the detection result of the chemical bond detection, the angle set is updated.

10. The molecular structure recognition device according to claim 9, wherein: The updating of the angle set based on the detection result of the chemical bond detection includes: When the detection result indicates that there is no chemical bond, storing the new branch angle into the angle set; In the case where the detection result indicates the presence of a chemical bond, the new branch angle Branch angles that form chemical bonds are deleted from the angle set.

11. The molecular structure recognition device according to claim 8, wherein: The decoding of the molecular structure at the branching angle based on the image features of the molecular image and guided by the branching angle includes: Determining visual context features at a current decoding moment at the branch angle based on the decoding features of the branch angle and a decoding state at a previous decoding moment at the branch angle; Based on the visual context feature, the molecular structure of the image feature at the current decoding moment is decoded to obtain the decoding state at the current decoding moment, and the current decoding moment is returned as the previous decoding moment for decoding until the decoding at the branch angle is completed.

12. The molecular structure recognition device according to any one of claims 8 to 11, wherein: The step of taking a branch angle from the angle set, decoding the molecular structure at the branch angle based on the image features of the molecular image with the branch angle as a guide, and updating the angle set based on a new branch angle obtained by decoding for decoding the molecular structure at the next branch angle until the angle set is empty comprises: Based on the recognition model, a branch angle is taken out from the angle set, and the molecular structure under the branch angle is decoded based on the image features of the molecular image with the branch angle as a guide, and the angle set is updated based on the new branch angle obtained by decoding for decoding of the molecular structure under the next branch angle, until the angle set is empty; The recognition model is trained based on the sample image and the molecular structure label corresponding to the sample image; The molecular structure label is obtained by connecting the atomic groups and chemical bonds in the molecular formula corresponding to the sample image to form a molecular structure graph and then performing graph traversal.

13. The molecular structure recognition device according to claim 12, wherein: The step of determining the molecular structure tag comprises: Connect the atomic groups and chemical bonds in the molecular formula corresponding to the sample image into molecules Structural diagram; The molecular structure graph is traversed, and the molecular structure label is generated based on the atomic groups, chemical bonds, angles, nesting symbols and reconnection marks obtained through the traversal.

14. The molecular structure recognition device according to claim 12, wherein: The training steps of the recognition model include: Based on the initial model, molecular structure recognition is performed on the sample image to obtain a chemical bond detection result between the sample branch angle decoded during the molecular structure recognition and the existing branch angles in the sample angle set, as well as a structural recognition result of the sample image; Based on the structure recognition result and the molecular structure label, as well as the chemical bond detection and the reconnection mark in the molecular structure label, the initial model is iterated with parameters to obtain the recognition model.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the molecular structure recognition method according to any one of claims 1 to 7 when executing the program. 16 . A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the molecular structure recognition method according to any one of claims 1 to 7.

Images