Molecular formula recognition method and related device, equipment and storage medium
Through the molecular formula recognition method, the molecular formula recognition model is used to identify organic chemical image data, which solves the problem of insufficient recognition ability in the prior art, and achieves the accurate identification and generalization ability of organic chemical data.
Patent Information
- Application Number
- CN202111630143.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-28
AI Technical Summary
Existing graphic recognition technologies are difficult to accurately identify image data in organic chemistry disciplines, especially due to their significant differences from Chinese and English data and mathematical formulas, resulting in insufficient recognition capabilities.
The molecular formula recognition method is adopted to identify images to be recognized through the molecular formula recognition model, and the sample symbol sequence is constructed using the graphic visual morphology of the sample molecular formula, thereby restoring the target molecular formula to improve the accuracy and generalization ability of recognition.
Accurate recognition of organic chemical data is achieved, the original information of the molecular formula in the image is fully retained, the accuracy and generalization ability of recognition are improved, and suitable for complex chemical image data.
Smart Images

Figure CN114359086B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a molecular formula recognition method and related devices, equipment and storage media. Background Art
[0002] With the development of deep learning, image and text recognition technology has become more mature and has begun to empower more and more industries. Especially in the education industry, the ability to recognize images of homework, answers, and test papers has become a very important part. On the one hand, it can be used for efficient electronicization of teaching knowledge, and on the other hand, it can serve automatic correction, realize learning situation analysis, and teach students in accordance with their aptitude. It has undoubtedly become an absolute necessity.
[0003] At present, the image and text recognition technology has basically matured in the recognition of Chinese and English data, and the recognition ability of structured data such as formulas has basically reached a usable state, which can meet most scenarios in Chinese and English mathematics, physics and chemistry. However, for some special scenarios, the image and text recognition technology has not yet reached a usable state, and organic chemistry is one of them. Because the appearance of organic chemistry questions and answer image data is very different from Chinese and English data and mathematical formulas, it does not yet have the ability to recognize organic chemistry data. Summary of the invention
[0004] The main technical problem solved by the present application is to provide a molecular formula recognition method and related devices, equipment and storage media, which can improve the accuracy of molecular formula recognition and the generalization ability of molecular formula recognition.
[0005] In order to solve the above technical problems, the first aspect of the present application provides a molecular formula recognition method, including: using a molecular formula recognition model to recognize an image to be recognized to obtain a symbol sequence; based on the symbol sequence, recovering the target molecular formula in the image to be recognized; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.
[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a molecular formula recognition device, including: a sequence recognition module and a formula recovery module; the sequence recognition module is used to use a molecular formula recognition model to recognize an image to be recognized and obtain a symbol sequence; the formula recovery module is used to recover the target molecular formula in the image to be recognized based on the symbol sequence; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.
[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the molecular formula recognition method in the above first aspect.
[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the molecular formula recognition method in the above first aspect.
[0009] The above scheme uses a molecular formula recognition model to recognize the image to be recognized, obtains a symbol sequence, and based on the symbol sequence, recovers the target molecular formula in the image to be recognized; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. Therefore, the symbol sequence obtained by the formula recognition model can accurately reflect the graphic visual state of the target formula, that is, it can truthfully reflect the content of the molecular formula in the image to be recognized, and fully retain the original information of the molecular formula in the image to be recognized, thereby improving the accuracy and generalization ability of recognizing the target molecular formula. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 It is a schematic diagram of a flow chart of an embodiment of a molecular formula recognition method provided by the present application;
[0011] Figure 2 is a schematic diagram of an embodiment of a SMILES recognition result provided by the present application;
[0012] Figure 3 It is a schematic diagram of an embodiment of the Chemfig annotation result provided by the present application;
[0013] Figure 4 It is a structural schematic diagram of an embodiment of a molecular formula recognition model provided by the present application;
[0014] Figure 5 It is a schematic diagram of an embodiment of sample data provided by this application;
[0015] Figure 6 yes Figure 1 The flowchart of step S11 is shown as an embodiment;
[0016] Figure 7 This is a flow chart of an embodiment of processing an original label sequence to obtain a sample symbol sequence provided by the present application;
[0017] Figure 8 This is a flow chart of an embodiment of determining whether an original tag sequence is correct provided by the present application;
[0018] Fig. 9 is a schematic diagram of an embodiment of a sample symbol sequence provided by the present application;
[0019] Fig.10 yes Figure 6 The flowchart of step S112 of an embodiment is shown;
[0020] Fig.11 is a schematic diagram of another embodiment of a sample symbol sequence provided by the present application;
[0021] Fig.12 It is a flow chart of an embodiment of a training molecular formula recognition model provided by the present application;
[0022] Fig.13 It is a schematic diagram of an embodiment of the molecular formula recognition model decoding process provided by the present application;
[0023] Fig.14 This is a flowchart of an embodiment of determining the end of decoding provided by the present application.
[0024] Fig.15 It is a schematic diagram of the framework of an embodiment of a molecular formula recognition device provided by the present application;
[0025] Fig.16 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;
[0026] Fig.17 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0027] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0028] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0029] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0030] See also Figure 1 , Figure 1is a flow chart of an embodiment of the molecular formula recognition method provided in this application. It should be noted that if there are substantially the same results, this embodiment is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:
[0031] Step S11: using the molecular formula recognition model to recognize the image to be recognized, and obtaining a symbol sequence.
[0032] The method of this embodiment is used to identify the image to be identified so as to recover the target molecular formula in the image to be identified. The image to be identified described herein may be an image including any molecular formula, which may be obtained from local storage or cloud storage. It is understandable that in other embodiments, the image to be identified may also be obtained by real-time acquisition by an image acquisition device, which is not specifically limited here.
[0033] In this embodiment, the molecular formula recognition model is used to recognize the image to be recognized, and a symbol sequence can be obtained. The molecular formula recognition model is trained using a sample image containing a sample molecular formula, and the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. In other words, the molecular formula recognition model is trained based on a sample image annotated with a sample symbol sequence of the sample molecular formula constructed from the graphic visual form of the sample molecular formula; in addition, the molecular formula recognition model completed by training can process the image to be recognized to obtain the target molecular formula.
[0034] For example, if Figure 2 As shown, Figure 2 is a schematic diagram of an embodiment of the SMILES recognition result provided by the present application. The deep learning molecular formula recognition method generally directly applies the existing Encoder-Decoder model for modeling. The model can directly learn the molecular formula image to the recognition result. The recognition result is generally stored in the SMILES string format; however, Figure 2 As shown in the figure, the input images on the left and right are the handwritten molecular formulas of two students. The molecular formula in the left input image is written correctly, while the molecular formula in the right input image is written incorrectly (the dashed box in the right input image can only be 3, otherwise it violates the valence rule). At this time, using the SMILES specification, only the molecular formula on the left can be correctly expressed, but not the molecular formula on the right. That is, using SMILES cannot truthfully express some handwritten answers that do not conform to chemical rules. In addition, Figure 2In the input image on the left, the atomic group "H3C" is written, but in the SMILES recognition result, "H3C" is not reflected, but is simply omitted according to the rule of "carbon and hydrogen are omitted"; and the "N" atom is adjusted to "N+", and the "O" atom is adjusted to "O-", all of which are also based on chemical principles. Therefore, the SMILES recognition results are standardized, but they are all based on a lot of professional chemical knowledge, without considering the form of writing, and cannot truly reflect the content in the image, so they are not based on the visual form of the graphic, that is, the picture and text are not related.
[0035] And as Figure 3 As shown, Figure 3 is a schematic diagram of an embodiment of the Chemfig annotation result provided by the present application. Figure 3 In the left input image and the right input image, the "H3C" atom group symbol in the left input image complies with the chemical valence rule, and the "H4C" atom group in the right input image does not comply with the chemical valence rule, which is an incorrect writing. The writing error can be reflected in the Chemfig annotation result, that is, the Chemfig annotation result corresponding to the right input image contains the words "H_{4}C". Since the Chemfig syntax relies less on abstract chemical knowledge, but is completely customized based on the molecular formula image itself, the sample symbol sequence obtained using the Chemfig syntax is constructed from the graphic visual form of the sample molecular formula, that is, the Chemfig syntax is a "what you see is what you get" molecular formula image annotation method, which can fully retain the original information of the sample molecular formula, so that the trained molecular formula recognition model has the possibility of being corrected. It can be understood that in other embodiments, other rule grammars can also be used to construct a sample symbol sequence that can reflect the graphic visual form of the sample molecular formula, which is not specifically limited here. It should be noted that, for ease of description, the following will take the example of constructing a sample symbol sequence using Chemfig syntax as an example, but it can be understood that this does not limit the syntax rules for constructing the sample symbol sequence.
[0036] The network architecture of the molecular formula recognition model is not limited and can be specifically configured according to actual use needs. Figure 4 As shown, Figure 4 : is a schematic diagram of the structure of an embodiment of the molecular formula recognition model provided by the present application. The molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model. The image-text encoding model is trained using a sample image containing at least one of a sample text and a sample formula. For example, Figure 5 As shown, Figure 5This is a schematic diagram of an embodiment of the sample data provided by this application. The sample text is the Chinese and English text "and passes through point A", and the sample formula is a mathematical formula. Therefore, the image-text coding model is trained using sample images containing at least one of Chinese and English texts and mathematical formula images and texts. In other words, some parameters in the image-text coding model can directly reuse the trained parameters of at least one of the Chinese and English text recognition model and the mathematical formula image-text recognition model, thereby accelerating the convergence speed of the molecular recognition model and, to a certain extent, compensating for the problem of insufficient molecular formula training data caused by the high difficulty of labeling, thereby improving the recognition accuracy of the molecular formula recognition model.
[0037] In one embodiment, the sample symbol sequence includes a string representing an atomic group in the sample molecular formula and a string representing a chemical bond in the sample molecular formula, and the string representing the chemical bond at least includes the angle of the chemical bond. Figure 3 As shown in the figure, the sample image input on the left is annotated using Chemfig syntax to obtain a sample symbol sequence of "H-{3}CN(=[1]O)=[-1]O", which includes the string "H-{3}C" representing the atomic group in the sample molecular formula, and the strings "-" and "=" representing the chemical bonds in the sample molecular formula; the string representing the chemical bond includes the angle of the chemical bond. For example, the "N" atom is connected to two double bonds, facing the upper right and lower right respectively. In the sample symbol sequence, "[1]" and "[-1]" are used as "angle codes" to represent the approximate direction of the chemical bond "=". The angle information of the chemical bond can more accurately depict the graphic visual form of the sample molecular formula, providing richer visual annotation information.
[0038] In one embodiment, the sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents the direction of the branch. That is, the sample symbol sequence includes a string representing an atomic group in the sample molecular formula, a string representing a chemical bond in the sample molecular formula, a string representing a chemical bond including at least the angle of the chemical bond, and a branch symbol representing a branch in the sample molecular formula.
[0039] In one embodiment, when the sample symbol sequence includes branch symbols representing branches in the sample molecular formula, the sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each branch, the sample first subsequence includes branch symbols representing each branch respectively, and the branch symbols also represent the identification of the branch, and the sample second subsequence includes an order symbol, and the order symbol represents the identification of the branch.
[0040] Step S12: based on the symbol sequence, recover the target molecular formula in the image to be identified.
[0041] In this embodiment, the target molecular formula in the image to be recognized is restored based on the symbol sequence obtained in the above manner. Specifically, the characters in the second subsequence of the sample representing each branch are connected to the position of the branch symbol in the first subsequence through the chemical bond in the second subsequence, and the connection direction of the chemical bond and the first subsequence is adjusted according to the angle of the chemical bond to restore the target molecular formula.
[0042] In one embodiment, the symbol sequence can be restored to the DFS (Depth First Search) form of the target molecular formula. The specific meaning of the DFS form can be found in the following related descriptions and will not be repeated here. It can be understood that in other embodiments, the symbol sequence can also be restored to the graphical data structure form of the target molecular formula.
[0043] In the above implementation, the molecular formula recognition model is used to recognize the image to be recognized, and a symbol sequence is obtained, and based on the symbol sequence, the target molecular formula in the image to be recognized is restored; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. Therefore, the symbol sequence obtained by the formula recognition model can accurately reflect the graphic visual state of the target formula, that is, it can truthfully reflect the content of the molecular formula in the image to be recognized, and fully retain the original information of the molecular formula in the image to be recognized, thereby improving the accuracy and generalization ability of recognizing the target molecular formula.
[0044] See also Figure 6 , Figure 6 yes Figure 1 It should be noted that if there is substantially the same result, this embodiment does not use Figure 6 The process sequence shown is limited. Figure 6 As shown, this embodiment includes:
[0045] Step S111: Perform structural analysis based on the original tag sequence to obtain graphic data of the sample molecular formula.
[0046] The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language. The preset molecular formula markup language is not limited and can be specifically set according to actual use needs. For example, the preset molecular formula markup language is Chemfig language.
[0047] For the image of the same sample molecular formula, there will be multiple different but correct annotation results depending on the selected starting point and traversal order, that is, for the image of the same sample molecular formula, there will be multiple different original label sequences, and the annotator will only randomly select one of them during annotation, and it is impossible to make clear rules to restrict the annotation process. Therefore, the original label sequence obtained by Chemfig language annotation will be ambiguous, which will increase the learning difficulty of the molecular formula recognition model. In addition, in order to maintain the simplicity of the original label sequence obtained by Chemfig language annotation, a large number of prior rules will be set up in the grammar to help the annotator save the number of characters entered during annotation. For example, for the benzene ring, it can be simply represented by "*6(-----)", in which the 6 single bonds do not need to specify the direction, but the default direction is automatically inferred according to the rules of the benzene ring. It can be seen that these set prior rules are relatively complex, and the molecular formula recognition model is difficult to learn, which will cause the generalization performance to decline.
[0048] Therefore, in this embodiment, the structure analysis is performed based on the original tag sequence to obtain the graphic data of the sample molecular formula, so as to eliminate the ambiguity and complexity of the original tag sequence and try to unify the expression form of the same sample molecular formula. Figure 7 As shown, Figure 7 It is a flow chart of an embodiment of processing the original label sequence to obtain the sample symbol sequence provided by the present application. The "Original Chemfig annotation string" column has three different Chemfig annotation strings, that is, there are three different original label sequences, but they actually express the same sample molecular formula; therefore, in this embodiment, the original label sequence will be structurally analyzed according to the original Chemfig grammar rules to obtain the graphic data of the sample molecular formula.
[0049] The graph data consists of several data elements, including nodes and edges connecting nodes. Nodes represent atomic groups, and edges represent chemical bonds. Specifically, by using the Chemfig syntax rules and manual programming, all the atomic groups and chemical bonds behind the original label sequence are parsed and restored, and they are connected to obtain the graph data. Figure 7 As shown in the figure, the "Annotated Graph Representation" column is the graphic data of the sample molecular formula obtained after structural analysis, where the node, i.e., the rectangular box, refers to the atomic group. The atomic group can be a string, such as "HO" or "COOH". The atomic group can also be empty, such as the vertices on the benzene ring. For the benzene ring with a circle, the middle circle is also treated as a special atomic group. The edges other than the nodes, i.e., the line segments other than the atomic group, are chemical bonds. The chemical bonds can be single bonds, double bonds, triple bonds, etc. Figure 7It only contains single bonds. The two ends of a chemical bond must be connected to an atomic group, and an atomic group can be connected to multiple chemical bonds. Through this connection relationship, with the atomic group as the node and the chemical bond as the edge, the graphical data of the sample molecular formula can be obtained. Figure 7 It can be seen that even if the original label sequences are diverse, as long as they are the same atom, the graphic data behind them is the same, which eliminates the problem of ambiguity in the original label sequence obtained by annotating with the Chemfig language.
[0050] Among them, each data element in the graphic data is marked with a data attribute, that is, by performing structural analysis on the original label sequence, the attributes of each data element in the graphic data of the sample molecular formula can also be analyzed. The data attributes marked on each data element are not limited and can be specifically set according to actual use needs.
[0051] In one embodiment, the data attributes of the node include characters representing the atomic group, i.e., what atomic group the node represents. In one embodiment, the data attributes of the edge include at least the angle of the chemical bond, i.e., what the angle of the chemical bond is. It is understandable that in other embodiments, the data attributes of the edge may also include what chemical bond the edge represents, or may also include whether the chemical bond is a single bond, a double bond, or a triple bond, etc., i.e., the type of covalent bond of the chemical bond.
[0052] In order to ensure that the graphical data of the sample molecular formula obtained by structural analysis based on the original tag sequence is correct, it is necessary to ensure that the original tag sequence is correctly labeled. In one embodiment, Figure 8 As shown, Figure 8 This is a flow chart of an embodiment of determining whether an original label sequence is correct provided by the present application. Before performing structural analysis based on the original label sequence to obtain graphic data of the sample molecular formula, it is determined whether the original label sequence is correctly labeled by comparing the rendered molecular formula with the sample molecular formula, which specifically includes the following sub-steps:
[0053] Step S81: Rendering the original tag sequence using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula.
[0054] In this embodiment, the original tag sequence is rendered using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula. The preset molecular formula markup language is not limited and can be specifically set according to actual use needs. For example, the preset molecular formula markup language is Chemfig language.
[0055] In a specific embodiment, Figure 3As shown, the preset molecular formula markup language is Chemfig language. Chemfig language has an existing rendering engine. The original label sequence is rendered by the rendering engine of Chemfig language to obtain the rendered molecular formula, that is, Figure 3 Rendering results are marked in the figure.
[0056] Step S82: Based on the difference check result between the rendered molecular formula and the sample molecular formula, determine whether the original label sequence is correctly labeled.
[0057] In this embodiment, based on the difference detection result between the rendered molecular formula and the sample molecular formula, it is determined whether the original label sequence is correctly labeled. Figure 3 As shown, Figure 3 The annotated rendering result is consistent with its corresponding input image, that is, there is no difference between the rendered molecular formula and its corresponding sample molecular formula, so it is determined that the annotation of the original label sequence is correct.
[0058] Step S112: Traverse based on the graphic data to obtain a sample symbol sequence.
[0059] Since the data structure in the graphic data of the sample molecular formula is relatively complex, it is also necessary to convert it into a form that can be trained by the molecular formula recognition model. Therefore, in this embodiment, based on traversing the graphic data, a sample symbol sequence is obtained. Among them, the method of traversing the graphic data is not limited and can be specifically set according to actual use needs. For example, the graphic data is traversed using depth-first traversal DFS to obtain a sample symbol sequence; or the graphic data is traversed using breadth-first traversal BFS (Breadth First Search) to obtain a sample symbol sequence.
[0060] Specifically, a point is selected according to a fixed rule, and the graph data is traversed according to the fixed rule, and the visited nodes and edges are appended to the end of the string, thereby completing the traversal of the graph data and obtaining a sample symbol sequence.
[0061] In a specific embodiment, Figure 7 and Fig. 9 As shown, Fig. 9 is a schematic diagram of an embodiment of a sample symbol sequence provided by the present application, using a depth-first traversal to Figure 7 Traverse the graph data shown in Figure 7 Upper right corner and Fig. 9The sample symbol sequence shown is the regular label. From this sample symbol sequence, it can be seen that a pair of parentheses are used to indicate the occurrence of branches. The sample symbol sequence contains four elements: "atomic group", "chemical bond", "angle of chemical bond" and "virtual body". All chemical bonds are followed by the angle descriptor "[:<angle value>]". The angle value has been calculated when constructing the graphic data and can be directly output in the format; "virtual body" is used to indicate the nesting of branches. Therefore, when the sample molecular formula has many and deep branches, the sample symbol sequence obtained by traversing the graphic data using depth-first traversal will have more virtual body nesting. The molecular formula recognition model will pay too much attention to depth when learning, and ignore the global structure.
[0062] In other specific embodiments, Figure 7 As shown, the breadth-first traversal is used Figure 7 Traverse the graph data shown in Figure 7 The sample symbol sequence shown in the lower right corner is the regularized label. Fig.10 As shown, Fig.10 yes Figure 6 The flowchart of step S112 of an embodiment shown in FIG. 1 includes the following sub-steps:
[0063] Step S1001: traverse the data elements on the sample molecular formula trunk in the graphic data to obtain the sample first subsequence, and traverse the data elements on the sample molecular formula branches in the graphic data to obtain the sample second subsequence.
[0064] In this embodiment, first, the data elements on the sample molecular formula trunk are traversed in the graphic data to obtain the first subsequence of the sample, and the data elements on the sample molecular formula branches are traversed in the graphic data to obtain the second subsequence of the sample. Among them, the character string representing the data element in the sample symbol sequence includes the data attribute of the data element, the branch is represented by the branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains the sequence symbol, and the sequence symbol represents the identification of the branch. In other words, in this embodiment, the traversal strategy of first overall and then local is followed. The data elements on the sample molecular formula trunk will be traversed first on the graphic data, that is, the overall traversal will be performed first, and then the data elements on the sample molecular formula branches will be traversed on the graphic data, that is, the local traversal will be performed. Through the principle of traversal first overall and then local, when the sample symbol sequence obtained subsequently is used for the training of the molecular formula recognition model, the molecular formula recognition model can better grasp the overall structure of the sample molecular formula and not ignore the branch structure, that is, the detailed structure of the sample molecular formula.
[0065] There is no limitation on the manifestation of branching symbols and sequence symbols, which can be set according to actual use needs. For example, "\place{x}[:{θ}]" is represented as a branching symbol, that is, when traversing the data elements on the sample molecular formula trunk, when encountering a branch, the content of the branch is temporarily not output, and the branch is replaced by the branch symbol first; wherein {x} represents an integer serial number, and {θ} represents an angle and represents the direction of the branch. "\solveplace{x}" is represented as a sequence symbol, that is, it means starting to output the content of the corresponding branch symbol "\place{x}[:{θ}]", wherein {x} represents an integer serial number.
[0066] In one embodiment, after the content corresponding to a branch symbol is output, that is, after the graphic data traverses the data elements on any branch of the sample molecular formula, it ends with a preset end symbol, that is, the last symbol of the branch symbol is the preset end symbol. Among them, the embodiment of the preset end symbol is not limited, and it can be specifically set according to actual use needs. For example, the preset end symbol is "\eol" or "\eos".
[0067] For example, if Fig.11 As shown, Fig.11 is a schematic diagram of another embodiment of the sample symbol sequence provided by the present application, and the graphic data of the sample molecular formula is as follows Fig.11 As shown in (a), following the principle of traversing the whole first and then the local, the data elements on the sample molecular formula trunk are traversed in the graphic data to obtain the first subsequence of the sample. The specific details of the first subsequence of the sample are as follows: Fig.11 (b) is shown; then the data elements on the sample molecular formula branch are traversed in the graphic data. Since there are three branch structures, three second subsequences are obtained; among them, the second subsequence corresponding to the branch symbol "\place1[:300]" is "\solveplace1\eol", the second subsequence corresponding to the branch symbol "\place2[:0]" is "\solveplace2-[:0]COOH\eol", and the second subsequence corresponding to the branch symbol "\place1[:60]" is "\solveplace3--[:60]\circle\eol".
[0068] Specifically, in the breadth-first traversal process, the sample symbol sequence text can be first initialized to "\place[:0]\eol", and another queue Q = [(x,0)], where the element x is any element in the above-mentioned graphic data G, and the branch indicator ind is initialized to 1. On this basis, when the queue Q is not empty, the following steps can be executed cyclically: (1) Take out the element x and the current placeholder indicator ind' (initial 0, and then accumulate 1 each time this step is executed) from the head of the queue Q; (2) Extract the maximum connected subgraph G' containing the element x from the graphic data G; (3) Find the longest branch on the maximum connected subgraph G' as the main branch Bmain; (4) Delete all elements on Bmain from the graphic data G; (5) ) outputs "\solveplace{ind'}" to the sample symbol sequence text; (6) traverses each element y on Bmain in the connection order, and outputs y to the end of the sample symbol sequence text, and if y is an atomic group and y is connected to a chemical bond e that is not on Bmain, it can be determined that y is a branch, and the branch direction angle can be set to the angle of the chemical bond e, and outputs "\place{ind}[:{angle}]" to the sample symbol sequence text, and adds [(e,ind)] to the queue Q, and adds 1 to ind to update ind. After traversing Bmain, it can output "\eol" to the end of the sample symbol sequence text, thereby completing a loop execution operation. It should be noted that when the loop operation is executed for the first time, the first subsequence of the trunk can be obtained, and when the loop operation is continued to be executed subsequently, the second subsequence of the branch can be obtained.
[0069] Step S1002: Combine the first subsequence of samples and the second subsequence of samples to obtain a sample symbol sequence.
[0070] In this embodiment, the sample first subsequence and the sample second subsequence are combined to obtain a sample symbol sequence. That is, the first subsequence obtained by traversing the data elements on the sample molecular formula trunk and the second subsequence obtained by traversing the data elements on the sample molecular formula branches are combined to obtain a sequence corresponding to the data elements of the entire sample molecular formula, that is, to obtain a sample symbol sequence.
[0071] See also Fig.12 , Fig.12 is a flow chart of an embodiment of the training molecular formula recognition model provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not Fig.12 The process sequence shown is limited.
[0072] like Figure 4As shown in the figure, before training the molecular formula recognition model, a molecular formula recognition model is first built. The molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network. Specifically, the molecular formula encoding network E encodes the input image I and outputs a sample feature map F; the molecular formula decoding network D adopts an autoregressive iterative modeling method. At each decoding time t, the sample feature map F and the decoding result y at the previous decoding time are used. t-1 As the input of the molecular formula decoding network D, the decoding result y at the current decoding time is output t , and the decoding is repeated until the end signal is encountered. Assuming that there are a total of T decoding moments, the final output decoding result y=[y1,y2,…,y T ], where y t As the basic modeling unit, Fig.11 The sequence of sample symbols shown, separated by spaces, yields a sequence of modeling units.
[0073] Among them, the decoding process of the traditional molecular formula decoding network D is as follows:
[0074] c t =Attn(s t-1 ,y t-1 ,F)
[0075] o t ,s t =RNN(c t ,y t-1 ,s t-1 )
[0076] p t =MLP(o t )
[0077] y t = argmaxx i p t (i),i∈{0,1,...V-1}
[0078] Among them, c t is the feature that needs to be paid attention to at the current decoding moment; Attn is the attention mechanism network; s t-1 is the decoding status at the last decoding moment; y t-1 is the decoding result of the previous decoding moment; F is the sample feature map; s t is the hidden state of the recurrent neural network RNN; o t is the output feature of the recurrent neural network RNN; p t is the output probability distribution, p t ∈R V , V is the number of types of modeling units; MLP is a neural network composed of a fully connected layer and a softmax layer; y tThe decoding result corresponding to the current decoding time. Specifically, according to the decoding state s at the previous decoding time t-1 , the decoding result y of the last decoding time t-1 Calculate the correlation with the features at all positions in the sample feature map F, and perform weighted summation of the features at all positions in the sample feature map F through the correlation to obtain the feature c that needs to be paid attention to at the current decoding moment t ; Then, use the recurrent neural network RNN to associate the features c that need to be paid attention to at the current decoding moment t , the decoding state s at the last decoding moment t-1 and the decoding result y at the previous decoding time t-1 , get the decoding state o at the current decoding moment t ; Then, according to the decoding state at the current decoding moment o t By performing symbol prediction, we can obtain the probability distribution p of the preset symbol. t That is, the probability prediction value of the preset symbol. Specifically, a decoding symbol dictionary is provided, and the probability distribution includes the probability prediction value of each preset decoding symbol in the corresponding decoding symbol dictionary; the preset symbol corresponding to the maximum probability prediction value in the decoding symbol dictionary is used as the decoding result y at the current decoding moment t .
[0079] Since the sample symbol sequence includes a large number of branch symbols (e.g., \place{x}[:{θ}]) and sequence symbols (e.g., \solveplace{x}), the decoding process from the sequence symbol to the next preset end symbol (e.g., \eol) belongs to the extended content of the branch symbol. Therefore, the decoding state when decoding to the corresponding branch symbol is used when decoding the branch to be decoded. However, in the sample symbol sequence, the distance between the sequence symbol and the preset end symbol is far, such as Fig.11 As shown in (b), the distance between the sequence symbol "\solveplace2" and the next preset end symbol "\eol", that is, the distance between the sequence symbol "\solveplace2" and the placeholder "\place2[:0]" is 8 decoding moments, and the memory capacity of the RNN model in the traditional molecular formula decoding network D cannot associate two decoding moments with a long interval. Therefore, in order to make the molecular formula recognition model fully utilize the sample symbol sequence and learn the graphic visual morphological information of the sample molecular formula, in this embodiment, the "angle conditional decoding" mechanism is added to the molecular formula decoding network D in the constructed molecular formula recognition model, so that the molecular formula recognition model can effectively associate the features of placeholders and pseudo symbols that are far apart in time. Specifically, as Fig.12 As shown, this embodiment includes:
[0080] Step S1201: Randomly select a reference state that has not been selected.
[0081] In this embodiment, the "angle conditional decoding" mechanism added to the molecular formula decoding network D is specifically to add the conditional input z t ∈[0,t-1], which is used to tell the molecular formula recognition model which branch symbol is currently being expanded.
[0082] In one embodiment, the sample molecular formula is composed of a trunk and branches, and the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branches. Before decoding each branch to be decoded, it is necessary to first decode the predicted subsequence of the trunk of the sample molecular formula. Specifically, decoding is performed according to a preset state and a sample feature map to obtain a predicted subsequence of the trunk. Since decoding the predicted subsequence of the trunk of the sample molecular formula is the first decoding stage, there is no reference decoding state, so in the first decoding stage, the preset state can be set to a 0 vector.
[0083] In this embodiment, after decoding the predicted subsequence of the trunk of the sample molecular formula, it is necessary to decode the predicted subsequence of the branch of the sample molecular formula. First, a reference state that has not been selected is randomly selected, wherein the reference state is the decoding state when the branch symbol is decoded, so that the unexpanded branch symbol can be decoded subsequently. The selection strategy of randomly selecting a reference state that has not been selected can make certain disturbances to the decoding order of the branch symbol before training, thereby avoiding the molecular formula recognition model from overfitting to a certain fixed order of traversing branches during the training process, thereby forcing the molecular formula recognition model to fully consider the input conditional information, and use the conditional information to correctly decode, thereby improving the generalization ability of the molecular formula recognition model.
[0084] For example, if Fig.13 As shown, Fig.13 It is a schematic diagram of an embodiment of the molecular formula recognition model decoding process provided by the present application. After the molecular formula recognition model decodes an end symbol "eol" at the t-1 decoding time, a reference state that has not been selected will be randomly selected at the next decoding time.
[0085] Step S1202: taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded.
[0086] In this embodiment, the branch represented by the branch symbol corresponding to the reference state is used as the branch to be decoded. In other words, the branch symbol corresponding to the decoding state when the branch symbol is decoded is used as the branch to be decoded, so that the data content corresponding to the branch symbol is subsequently decoded.
[0087] Step S1203: Decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded.
[0088] In this implementation, decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded. Specifically, Fig.13 As shown, the molecular formula recognition model will take the branch represented by the branch symbol corresponding to the randomly selected reference state as the branch to be decoded and decode its content. The specific formula for obtaining the predicted subsequence of the branch to be decoded is as follows:
[0089] c t =Attn(s t-1 ,s zt ,y t-1 ,F)
[0090] o t ,s t =RNN(c t ,y t-1 ,s t-1 ,s zt )
[0091] p t =MLP(o t )
[0092] y t = argmaxx i p t (i),i∈{0,1,...V-1}
[0093] Among them, c t is the feature that needs to be paid attention to at the current decoding moment; Attn is the attention mechanism network; s t-1 is the decoding status at the last decoding moment; s zt The decoding status when decoding to the branch symbol; y t-1 is the decoding result of the previous decoding moment; F is the sample feature map; s t is the hidden state of the recurrent neural network RNN; o t is the output feature of the recurrent neural network RNN; p t is the output probability distribution, p t ∈R V , V is the number of types of modeling units; MLP is a neural network composed of a fully connected layer and a softmax layer; y t The decoding result corresponding to the current decoding time. Specifically, according to the decoding state s at the previous decoding time t-1 , the decoding result y of the last decoding time t-1 , decoding state s when decoding to the branch symbol zt Calculate the correlation with the features at all positions in the sample feature map F, and perform weighted summation of the features at all positions in the sample feature map F through the correlation to obtain the feature c that needs to be paid attention to at the current decoding momentt ; Then, use the recurrent neural network RNN to associate the features c that need to be paid attention to at the current decoding moment t , the decoding state s at the last decoding moment t-1 , the decoding result y of the last decoding time t-1 and the decoding state s when decoding to the branch symbol zt , get the decoding state o at the current decoding moment t ; Then, according to the decoding state at the current decoding moment o t By performing symbol prediction, we can obtain the probability distribution p of the preset symbol. t That is, the probability prediction value of the preset symbol. Specifically, a decoding symbol dictionary is provided, and the probability distribution includes the probability prediction value of each preset decoding symbol in the corresponding decoding symbol dictionary; the preset symbol corresponding to the maximum probability prediction value in the decoding symbol dictionary is used as the decoding result y at the current decoding moment t .
[0094] Step S1204: adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.
[0095] In this embodiment, according to the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted so that the molecular formula recognition network converges, thereby improving the recognition ability of the molecular formula recognition model and improving the generalization ability of the molecular formula recognition model.
[0096] In one embodiment, a sub-prediction loss value between each decoded character in a sample sub-sequence and each predicted character in a predicted sub-sequence belonging to the same branch is calculated by a loss function such as cross entropy, and then each sub-prediction loss value is counted to obtain a total loss value, so that the network parameters of the molecular formula recognition model can be adjusted based on the total loss value through optimization methods such as gradient descent, and the above process is repeated until the molecular formula recognition model is trained to convergence using a sample image containing the sample molecular formula.
[0097] In other embodiments, the sample molecular formula is composed of a trunk and branches, the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branches, and the network parameters of the molecular formula recognition model can be adjusted according to the difference between the predicted subsequence and the sample subsequence of the trunk and the difference between the sample subsequence and the predicted subsequence belonging to the same branch.
[0098] In one embodiment, if Fig.14 As shown, Fig.14 This is a flow chart of an embodiment of determining the end of decoding provided by the present application. Before adjusting the network parameters of the molecular formula recognition model, it is necessary to confirm whether the decoding of the entire sample image is completed. Specifically, it includes the following sub-steps:
[0099] Step S1401: Check whether the end character is decoded and all reference states have been selected.
[0100] In this embodiment, it is determined whether the end character is decoded (e.g., the end character is eol, etc.) and all reference states have been selected. If the end character has not been decoded or the end character is decoded and there is a reference state that has not been selected for use, it indicates that the decoding of the sample image has not been completed. If the end character is decoded and there is a reference state that has not been selected, step 1402 is executed; if the end character is decoded and all reference states have been selected for use, it indicates that the decoding of the sample image has been completed, and step S1403 is executed.
[0101] Step S1402: In response to decoding to an end character and there being reference states that have not been selected, executing a step of randomly selecting a reference state that has not been selected and subsequent steps.
[0102] In this embodiment, the sample subsequence ends with an end symbol, and in response to decoding to the end symbol and there are still reference states that have not been selected for use, the step of randomly selecting a reference state that has not been selected and the subsequent steps are re-executed. That is, in the case where there are still reference states that have not been selected for use, the step of randomly selecting a reference state that has not been selected and the subsequent steps are re-executed until all reference states have been selected for use, indicating that the decoding is completed.
[0103] Step S1403: In response to decoding to the end symbol and all reference states being selected, a step of adjusting network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed.
[0104] In this embodiment, in response to decoding to the end symbol and all reference states being selected for use, it is determined that decoding is finished, and a step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed. That is, after decoding to the end symbol (for example, the end symbol is "eol") and all reference states are selected for use in the current decoding stage, it indicates that decoding is finished; in addition, a step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed.
[0105] See also Fig.15 , Fig.151 is a schematic diagram of a framework of an embodiment of a molecular formula recognition device provided by the present application. The molecular formula recognition device 150 includes a sequence recognition module 151 and a formula recovery module 152; the sequence recognition module 151 is used to recognize the image to be recognized using the molecular formula recognition model to obtain a symbol sequence; the formula recovery module 152 is used to recover the target molecular formula in the image to be recognized based on the symbol sequence; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.
[0106] The sample symbol sequence includes a character string representing an atomic group in the sample molecular formula and a character string representing a chemical bond in the sample molecular formula, and the character string representing a chemical bond at least includes an angle of the chemical bond.
[0107] The sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents the direction of the branch.
[0108] The sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each branch. The sample first subsequence contains branch symbols representing each branch respectively, and the branch symbols also represent the identification of the branch. The sample second subsequence contains an order symbol, and the order symbol represents the identification of the branch.
[0109] Among them, the above-mentioned sample molecular formula is pre-annotated as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The molecular formula recognition device 150 also includes an acquisition module 153. The acquisition module 153 is used for the acquisition steps of the sample symbol sequence including: performing structural analysis based on the original label sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a number of data elements, and the several data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each data element in the graphic data is marked with data attributes; traversal is performed based on the graphic data to obtain the sample symbol sequence.
[0110] The data attributes of the nodes include characters representing atomic groups; and / or the data attributes of the edges include at least the angle of the chemical bond.
[0111] Among them, the acquisition module 153 is used to traverse based on the graphic data to obtain a sample symbol sequence, specifically including: traversing the data elements on the sample molecular formula trunk in the graphic data to obtain the sample first subsequence, and traversing the data elements on the sample molecular formula branches in the graphic data to obtain the sample second subsequence; combining the sample first subsequence and the sample second subsequence to obtain a sample symbol sequence; wherein the character string representing the data element in the sample symbol sequence includes the data attributes of the data element, the branch is represented by a branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains a sequence symbol, and the sequence symbol represents the identification of the branch.
[0112] Among them, before performing structural analysis based on the original label sequence to obtain graphic data of the sample molecular formula, the acquisition module 153 is also used to: render the original label sequence using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula; and determine whether the original label sequence is correctly labeled based on the difference check result between the rendered molecular formula and the sample molecular formula.
[0113] The sample molecular formula is composed of a trunk and branches, and the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branch, and the branch is represented by a branch symbol in the sample subsequence of the trunk; the molecular formula recognition device 150 also includes a training module 154, and the training module 154 is used for the training steps of the molecular formula recognition model, including: randomly selecting a reference state that has not been selected; wherein the reference state is a decoding state when decoding to a branch symbol; taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded; decoding based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded; adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.
[0114] The sample subsequence ends with a terminator; the training module 154 is used to randomly select a reference state that has not been selected, specifically including: in response to decoding to the terminator and there is still a reference state that has not been selected, executing the step of randomly selecting a reference state that has not been selected and subsequent steps.
[0115] Wherein, before adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the training module 154 is also used to: check whether the end symbol is decoded and all reference states have been selected; in response to decoding to the end symbol and all reference states have been selected, perform the step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.
[0116] Among them, before randomly selecting a reference state that has not been selected, the training module 154 is also used to: decode based on the preset state and the sample feature map to obtain a predicted subsequence of the trunk; the training module 154 is used to adjust the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, specifically including: adjusting the network parameters of the molecular formula recognition model based on the difference between the predicted subsequence and the sample subsequence of the trunk, and the difference between the sample subsequence and the predicted subsequence belonging to the same branch.
[0117] Among them, the above-mentioned molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model, and the image-text encoding model is trained using sample images containing at least one of sample text and sample formula.
[0118] See also Fig.16 , Fig.16 1 is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application. The electronic device 160 includes a memory 161 and a processor 162 coupled to each other, the memory 161 stores program instructions, and the processor 162 is used to execute the program instructions to implement the steps in any of the above-mentioned molecular formula recognition method embodiments. Specifically, the electronic device 160 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.
[0119] Specifically, the processor 162 is used to control itself and the memory 161 to implement the steps in any of the above-mentioned molecular formula recognition method embodiments. The processor 162 can also be referred to as a CPU (Central Processing Unit). The processor 162 may be an integrated circuit chip with signal processing capabilities. The processor 162 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 162 may be implemented by an integrated circuit chip.
[0120] See also Fig.17 , Fig.17The computer-readable storage medium 170 stores program instructions 171 that can be executed by a processor, and the program instructions 171 are used to implement the steps in any of the above molecular formula recognition method embodiments.
[0121] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0122] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0123] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0127] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A molecular formula recognition method, characterized in that: include: The image to be identified is identified using the molecular formula recognition model to obtain a symbol sequence; Based on the symbol sequence, recovering the target molecular formula in the image to be identified; The molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from a graphic visual form of the sample molecular formula; The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The step of obtaining the sample symbol sequence includes: Structural analysis is performed based on the original tag sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a plurality of data elements, the plurality of data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each of the data elements in the graphic data is marked with a data attribute; The sample symbol sequence is obtained by traversing based on the graphic data.
2. The method according to claim 1, characterized in that The sample symbol sequence includes a character string representing an atomic group in the sample molecular formula and a character string representing a chemical bond in the sample molecular formula, and the character string representing the chemical bond at least includes an angle of the chemical bond.
3. The method according to claim 2, characterized in that The sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents a direction of the branch.
4. The method according to claim 3, characterized in that The sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each of the branches, the sample first subsequence includes branch symbols representing each of the branches respectively, and the branch symbols also represent the identifiers of the branches, and the sample second subsequence includes an order symbol, and the order symbol represents the identifier of the branch.
5. The method according to claim 1, characterized in that The data attributes of the node include characters representing the atomic group; And / or, the data attributes of the edge include at least the angle of the chemical bond.
6. The method according to claim 1, characterized in that The traversing based on the graphic data to obtain the sample symbol sequence includes: Traversing the data elements on the trunk of the sample molecular formula in the graphic data to obtain a first subsequence of samples, and traversing the data elements on the branches of the sample molecular formula in the graphic data to obtain a second subsequence of samples; Combining the first subsequence of samples with the second subsequence of samples to obtain the sample symbol sequence; Among them, the character string representing the data element in the sample symbol sequence includes the data attribute of the data element, the branch is represented by a branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains an order symbol, and the order symbol represents the identification of the branch.
7. The method according to claim 1, characterized in that Before performing structural analysis based on the original tag sequence to obtain graphic data of the sample molecular formula, the method further includes: Rendering the original tag sequence using a rendering engine of the preset molecular formula markup language to obtain a rendered molecular formula; Based on the difference check result between the rendered molecular formula and the sample molecular formula, it is determined whether the original label sequence is correctly labeled.
8. The method according to claim 1, characterized in that The sample molecular formula is composed of a trunk and branches, the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branches, and the branches are represented by branch symbols in the sample subsequence of the trunk; The training steps of the molecular formula recognition model include: Randomly select a reference state that has not been selected; wherein the reference state is a decoding state when decoding to the branch symbol; Taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded; Decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded; Based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted.
9. The method according to claim 8, characterized in that The sample subsequence ends with a terminator; The randomly selecting a reference state that has not been selected includes: In response to decoding the end symbol and there being a reference state that has not been selected, the step of randomly selecting a reference state that has not been selected and subsequent steps are performed.
10. The method according to claim 9, characterized in that Before adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the method further includes: Check whether the terminator is decoded and all the reference states have been selected; In response to decoding the end symbol and all the reference states being selected, the step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed.
11. The method according to claim 8, characterized in that Before randomly selecting a reference state that has not been selected, the method further includes: Decoding is performed based on a preset state and the sample feature map to obtain a predicted subsequence of the backbone; The adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch comprises: Based on the difference between the predicted subsequence and the sample subsequence of the trunk, and the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted.
12. The method according to claim 1, characterized in that The molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model, and the image-text encoding model is trained using a sample image containing at least one of a sample text and a sample formula.
13. A molecular formula recognition device, characterized in that: include: A sequence recognition module is used to recognize the image to be recognized by using a molecular formula recognition model to obtain a symbol sequence; A formula recovery module, used for recovering the target molecular formula in the image to be identified based on the symbol sequence; The molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from a graphic visual form of the sample molecular formula; The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The step of obtaining the sample symbol sequence includes: Structural analysis is performed based on the original tag sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a plurality of data elements, the plurality of data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each of the data elements in the graphic data is marked with a data attribute; The sample symbol sequence is obtained by traversing based on the graphic data.
14. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the molecular formula recognition method according to any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the molecular formula recognition method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Formula recognition method and device, electronic equipment and storage medium
CN112200194A
device for encoding and / or rendering a curve visible.
NL7510843A
System and Method for Identifying Similar Molecules
US20080004810A1
Apparatus and method for optical recognition of chemical graphics
US5157736A
Molecular data storage systems and methods
WO2019224816A1
