Formula identification method and device, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a formula identification method, which comprises the following steps: training a formula classification model by using a single-row formula sample and a multi-row formula sample to obtain a trained formula classification model; and the trained formula classification model processes the trajectory image of the target formula to obtain the category of the target formula. And when the target formula is a single-row formula, processing a track image of the target formula by using the trained single-row formula identification model. And when the target formula is a multi-row formula, processing a track image of the target formula by using the trained multi-row formula identification model. And obtaining an identification result of the target formula. Wherein the category of the target formula is determined by utilizing the pre-trained formula classification model, and the target formula is identified by adopting the trained formula identification model corresponding to the formula category, so that the identification result of the target formula is obtained, and the identification of different categories of handwritten formulas is realized. The invention furthermore discloses a formula identification apparatus and device, and a storage medium.

Description

Technical Field

[0001] This application relates to the field of image processing technology, such as a method, apparatus, device, and storage medium for formula recognition. Background Technology

[0002] Currently, in fields such as academic research, education, automated office work, and intelligent conferencing, it is often necessary to intelligently recognize handwritten formulas in order to facilitate their use by users.

[0003] In related technologies, by inputting an image of a handwritten formula into a convolutional neural network model, the convolutional neural network model can recognize the handwritten formula in the image and output the character sequence corresponding to the handwritten formula.

[0004] However, in related technologies, the convolutional neural network model can only recognize handwritten formulas in a single line, that is, it can only recognize one type of handwritten formula and cannot recognize handwritten formulas of different types. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This application provides a method, apparatus, device, and storage medium for formula recognition, which can recognize different types of handwritten formulas.

[0007] In a first aspect, embodiments of this application provide a method for formula recognition, including:

[0008] A formula classification model is trained using a preset formula sample to obtain a trained formula classification model; wherein the preset formula sample includes at least single-line formula samples and multi-line formula samples.

[0009] The trajectory image of the target formula is processed using a trained formula classification model to obtain the category of the target formula;

[0010] When the target formula is a single-line formula, the trajectory image of the target formula is processed by the trained single-line formula recognition model to obtain the first recognition result of the target formula.

[0011] When the target formula is a multi-line formula, the trajectory image of the target formula is processed using a trained multi-line formula recognition model to obtain the second recognition result of the target formula.

[0012] Optionally, correspondingly, a formula classification model is trained using a preset formula sample to obtain a trained formula classification model, including: the preset formula sample further including organic chemical formula samples; and training the formula classification model using single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from the preset formula sample to obtain a trained formula classification model.

[0013] Optionally, a formula classification model is trained using single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from a preset formula sample to obtain a trained formula classification model. This includes: concatenating single-line formula samples and preset symbols according to a first preset rule to obtain expanded multi-line formula samples; concatenating monocyclic organic chemical formulas in organic chemical formula samples according to a second preset rule to obtain expanded organic chemical formula samples; and inputting the single-line formula samples, expanded multi-line formula samples, and expanded organic chemical formula samples into the formula classification model for training to obtain a trained formula classification model.

[0014] Optionally, after processing the trajectory image of the target formula using a trained formula classification model to obtain the category of the target formula, the method further includes: when the target formula is an organic chemical formula, processing the trajectory image of the target formula using a trained organic chemical formula recognition model to obtain a third recognition result of the target formula.

[0015] Optionally, training a single-line formula recognition model includes: obtaining multiple types of single-line formula samples from a preset formula sample set; inputting the multiple types of single-line formula samples into the single-line formula recognition model for training, so as to obtain a trained single-line formula recognition model.

[0016] Optionally, training a multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set, to obtain a trained multi-line formula recognition model; wherein the mixed formula sample set includes single-line formula samples and multi-line formula samples.

[0017] Optionally, training a multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set to obtain a first formula recognition model; wherein the mixed formula sample set includes single-line formula samples and multi-line formula samples; adjusting the dictionary of the first formula recognition model and inputting the mixed formula samples into the first formula recognition model with the adjusted dictionary for training to obtain a second formula recognition model; and inputting various types of multi-line formula samples into the second formula recognition model for training to obtain a trained multi-line formula recognition model.

[0018] Secondly, embodiments of this application provide a formula recognition apparatus, comprising:

[0019] The training module is used to train a formula classification model using a preset formula sample to obtain a trained formula classification model; wherein the preset formula sample includes at least single-line formula samples and multi-line formula samples.

[0020] The classification module is used to process the trajectory image of the target formula using a trained formula classification model in order to obtain the category of the target formula.

[0021] The recognition module is used to process the trajectory image of the target formula using a trained single-line formula recognition model when the target formula is a single-line formula, so as to obtain the first recognition result of the target formula; and to process the trajectory image of the target formula using a trained multi-line formula recognition model when the target formula is a multi-line formula, so as to obtain the second recognition result of the target formula.

[0022] Thirdly, embodiments of this application provide an electronic device including a processor and a memory storing program instructions, wherein the processor is configured to execute the formula recognition method as described in the first aspect when running the program instructions.

[0023] Fourthly, embodiments of this application provide a storage medium storing program instructions, wherein the program instructions, when executed, perform the formula recognition method as described in the first aspect.

[0024] The formula recognition method, apparatus, device, and storage medium provided in the embodiments of this application can achieve the following technical effects:

[0025] After training a formula classification model using single-line and multi-line formula samples from a preset formula sample library, the electronic device can classify both single-line and multi-line formulas. After recognizing the trajectory image of a target formula using the trained formula classification model, the electronic device outputs the category of the target formula. If the target formula is classified as a single-line formula, the electronic device uses a pre-trained single-line formula recognition model to process the trajectory image of the target formula, obtaining a first recognition result. Alternatively, if the target formula is classified as a multi-line formula, a pre-trained multi-line formula recognition model is used to process the trajectory image of the target formula, obtaining a second recognition result. In this embodiment, after recognizing the trajectory image of a target formula using a pre-trained formula classification model, the electronic device can determine the category of the target formula. Based on the category, it uses a trained formula recognition model corresponding to the category of the target formula to process the trajectory image of the target formula and outputs the recognition result. Therefore, it can recognize both single-line and multi-line formulas, achieving the recognition of handwritten formulas of different categories.

[0026] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0027] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0028] Figure 1 This is a schematic diagram of a formula recognition method;

[0029] Figure 2 This is a flowchart of a formula recognition method provided in an embodiment of this application;

[0030] Figure 3 This is an example diagram illustrating the expansion of an organic chemical formula sample provided in the embodiments of this application;

[0031] Figure 4 This is an example diagram illustrating how a target formula is converted into a character sequence, as provided in an embodiment of this application.

[0032] Figure 5 This is another example diagram illustrating how a target formula can be converted into a character sequence, as provided in an embodiment of this application.

[0033] Figure 6 This is a graph showing the overall line recognition accuracy of a multi-line formula recognition model provided in this application embodiment;

[0034] Figure 7 This is a schematic diagram of a formula recognition device provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0037] Unless otherwise stated, the term "multiple" means two or more. The term "various types" means two or more.

[0038] In this embodiment, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0039] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0040] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0041] To provide a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0042] Currently, users frequently utilize formulas in fields such as academic research, education, automated office work, and smart conferencing. Formulas can be categorized into two types: online formulas and offline formulas. Offline formulas generally refer to formula images captured from scanners or cameras. Online formulas are sequences of stroke coordinates recorded in real-time from devices such as touchpads or styluses that can record trajectory information; they are a series of trajectory points. To facilitate user use of formulas, intelligent recognition of handwritten online formulas is often required.

[0043] Combination Figure 1 As shown, in related technologies, a deep neural network model with an encoder-decoder network architecture is used to identify the character sequence corresponding to a handwritten formula from an image of the formula. Specifically, after the image of the handwritten formula is input into the encoder for processing, the encoding network in the encoder encodes the image of the handwritten formula to obtain the encoded features of the handwritten formula. After the encoded features are input into the decoder, the attention module in the decoder, based on historical information (i.e., y...),... t-1 and s t-1 ) and encoded features, perform attention mechanism calculations, and generate context vector features c t The recurrent neural network in the decoder is based on the context vector features c t It predicts the character sequence corresponding to the handwritten formula.

[0044] However, in related technologies, deep neural network models using encoder-decoder architectures to recognize handwritten formulas typically only identify single-line handwritten formulas. Since handwritten formulas include not only single-line formulas but also other categories such as multi-line formulas (e.g., piecewise function formulas and matrix formulas), the current approach of only recognizing single-line handwritten formulas has limitations and struggles to recognize different types of handwritten formulas.

[0045] In view of this, embodiments of this application provide a method, apparatus, device, and storage medium for formula recognition. In this solution, after determining the category of a formula using a pre-trained formula classification model, the formula can be recognized using a pre-trained formula recognition model corresponding to its category, whether it is a single-line or multi-line formula, and the recognition result is output. Thus, the recognition of handwritten formulas of different categories is achieved.

[0046] In this application embodiment, the execution entity is an electronic device, which can be a conference tablet, smart blackboard, or intelligent interactive whiteboard, or a server, desktop computer, or laptop. In this application embodiment, the electronic device can deploy a model for recognizing the target formula, and it can also deploy a model for classifying the target formula. In this application embodiment, the electronic device for model training and the electronic device for classifying and recognizing the target formula using the model can be the same electronic device or different electronic devices; this application embodiment does not specifically limit this.

[0047] The method for formula recognition in the embodiments of this application will be described in detail below.

[0048] Combination Figure 2 As shown in the figure, this application provides a method for formula recognition, the method including:

[0049] S21, use preset formula samples to train a formula classification model to obtain a trained formula classification model; wherein, the preset formula samples include at least single-line formula samples and multi-line formula samples.

[0050] The formula classification model can be a lightweight network model, MobileNetv3.

[0051] S22, use the trained formula classification model to process the trajectory image of the target formula in order to obtain the category of the target formula.

[0052] S231, when the target formula is a single-line formula, the trajectory image of the target formula is processed by the trained single-line formula recognition model to obtain the first recognition result of the target formula.

[0053] S232, when the target formula is a multi-line formula, the trajectory image of the target formula is processed by the trained multi-line formula recognition model to obtain the second recognition result of the target formula.

[0054] Using the formula recognition method provided in this application, the electronic device trains a formula classification model using single-line and multi-line formula samples from a preset formula sample library. This trained model can then classify both single-line and multi-line formulas. After recognizing the trajectory image of the target formula using the trained model, the electronic device outputs the category of the target formula. If the target formula is classified as a single-line formula, the electronic device uses a pre-trained single-line formula recognition model to process the trajectory image, obtaining a first recognition result. Alternatively, if the target formula is classified as a multi-line formula, a pre-trained multi-line formula recognition model is used to process the trajectory image, obtaining a second recognition result. In this application embodiment, after recognizing the trajectory image of the target formula using a pre-trained formula classification model, the electronic device can determine the category of the target formula. Based on the category, it uses a trained formula recognition model corresponding to that category to process the trajectory image and output the recognition result. Therefore, it can recognize both single-line and multi-line formulas, achieving recognition of handwritten formulas of different categories.

[0055] Optionally, the sample data for the preset formula samples in step S21 above can be obtained from a formula dataset. The formula dataset contains formulas of different types, such as mathematical formulas, physical formulas, and inorganic chemical formulas. Each type of formula contains both single-line and multi-line formulas; for example, trigonometric formulas in mathematical formulas are single-line formulas, while matrix formulas are multi-line formulas. Thus, single-line and multi-line formula samples can be obtained from the formula dataset to form the preset formula samples. Alternatively, the sample data for the preset formula samples can also be generated through random sampling. During random sampling, different types of formulas can be sampled separately to ensure the diversity of the sample data in the preset formula samples.

[0056] In step S21 above, a formula classification model is trained using preset formula samples to obtain a trained formula classification model. This includes training the formula classification model using single-line formula samples and multi-line formula samples obtained from the preset formula samples. After the formula classification model is trained, it can be used to identify the category of formulas under different formula types. For example, trigonometric formulas in mathematical formulas can be identified as single-line formulas, and matrix formulas in mathematical formulas can be identified as multi-line formulas.

[0057] Understandably, in addition to single-line inorganic chemical formulas, there are also organic chemical formulas, which can be the molecular formulas of organic compounds. Organic chemical formulas are a special category of formulas; they are neither single-line nor multi-line formulas. To identify different categories of formulas, in this embodiment, organic chemical formulas can be obtained from a formula dataset and added to a preset formula sample, so that the preset formula sample includes not only single-line and multi-line formula samples but also samples of organic chemical formulas.

[0058] When the preset formula sample also includes organic chemical formula samples, a formula classification model is trained using the preset formula sample to obtain a trained formula classification model. This includes training the formula classification model using single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from the preset formula sample. Specifically, the single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from the preset formula sample are input into the formula classification model for training. If the formula classification model converges, the converged formula classification model is determined as the trained formula classification model. Because organic chemical formula samples are included in the preset sample used to train the formula classification model, the trained formula classification model can recognize not only single-line and multi-line formulas but also organic chemical formulas. This improves the generalization ability of the formula classification model.

[0059] It should be noted that, traditionally, formulas are typically categorized by subject, such as physics, mathematics, and biology. In this implementation, formulas are categorized by their physical appearance, specifically as single-line formulas, multi-line formulas, and organic chemical formulas. The formula classification model is also trained using this categorization method. This is because neural network models, when classifying formulas, essentially rely on their images to identify their physical appearance and thus determine their category. This approach takes into account the possibility that formulas from different disciplines may have similar physical appearances; for example, the mathematical formula for the area of ​​a circle is often expressed as S = πr². 2 The mass-energy equivalence formula in physics is E = mc². 2 Because the formulas for the area of ​​a circle and the mass-energy equivalence formula have similar appearances, they are prone to misclassification when using neural network models to classify them. In this implementation, by classifying formulas according to their appearance and training the formula classification model based on this classification method, the model can more accurately identify the formula category, facilitating subsequent identification of formulas in different categories.

[0060] Furthermore, when training a formula classification model using single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from a preset formula sample set, the number of multi-line formulas and organic chemical formulas is less than that of single-line formulas. Therefore, whether the preset formula samples are generated by obtaining sample data from the dataset or by random sampling, an imbalance in the number of samples of different categories will occur within the preset formula samples. Thus, when training the formula classification model using single-line formula samples, multi-line formula samples, and organic chemical formula samples obtained from the preset formula sample set, the single-line formula samples and preset symbols can be concatenated according to the first preset rule to obtain expanded multi-line formula samples. Similarly, the single-ring organic chemical formulas in the organic chemical formula samples can be concatenated according to the second preset rule to obtain expanded organic chemical formula samples. Then, the single-line formula samples, expanded multi-line formula samples, and expanded organic chemical formula samples are respectively input into the formula classification model for training to obtain a trained formula classification model.

[0061] In expanding the multi-line formula sample, the first preset rule is the rule for concatenating single-line formula samples with preset symbols. Preset symbols can include curly braces, square brackets, parentheses, and absolute value symbols. According to the first preset rule, preset symbols can be concatenated with single-line formulas to obtain multi-line formula samples, thereby expanding the number of multi-line formula samples.

[0062] In expanding the organic chemical formula sample, the second pre-defined rule is a rule for splicing monocyclic organic chemical formulas within the sample. For example, in writing organic chemical formulas, regular polygons are used to represent organic compounds; for instance, a benzene ring is represented by a regular hexagon. According to the second pre-defined rule, monocyclic organic chemical formulas can be spliced ​​together using edges or vertices as a basis to obtain bicyclic or multicyclic organic chemical formulas, thereby expanding the number of organic chemical formula samples. Combined with... Figure 3 As an example, assuming that the expansion is based on the benzene ring and the organic chemical formula of the benzene ring is 31, after splicing the benzene ring 31 along the edge in a specific direction, the chemical formula of tetraphenylene 32 or the chemical formula of pentaphenylene 33 can be obtained.

[0063] This implementation method, by expanding the multi-line formula samples and organic chemical formula samples in the preset formula sample, enhances the sample data in the preset formula sample, ensuring a balance in the number of sample data of different categories, that is, maintaining a balance in the number of single-line formula samples, multi-line formula samples, and organic chemical formula samples in the preset formula sample. Therefore, after the electronic device uses the balanced sample data to train the formula classification model, the trained formula classification model can better classify different categories of formulas, thereby improving the accuracy of formula classification.

[0064] Optionally, in step S22 above, the user typically uses a stylus or smart pen to write formulas on the touchscreen of an electronic device. The electronic device can detect the coordinates of the trajectory points of the stylus or smart pen on the touchscreen, connect the trajectory point coordinates in chronological order to form the writing trajectory of the target formula, and the image corresponding to the writing trajectory of the target formula is the trajectory image of the target formula. The electronic device uses a trained formula classification model to identify the trajectory image of the target formula and determine the category of the target formula.

[0065] Optionally, in steps S231 and S232 above, after processing the trajectory image of the target formula using a trained formula classification model to obtain the category of the target formula, the electronic device selects a formula recognition model corresponding to the category of the target formula to process the trajectory image of the target formula and obtain the recognition result of the target formula. Specifically, when the target formula is a single-line formula, the trajectory image of the target formula is processed using a trained single-line formula recognition model to obtain the first recognition result of the target formula. When the target formula is a multi-line formula, the trajectory image of the target formula is processed using a trained multi-line formula recognition model to obtain the second recognition result of the target formula.

[0066] In this embodiment, to facilitate subsequent processing (such as formula restoration) by the electronic device based on the recognition result of the target formula, the recognition result of the target formula can be converted into a string format (such as LaTeX format). LaTeX can be regarded as a syntax rule that can convert the target formula into a one-dimensional string according to preset rules.

[0067] Combination Figure 4 For example, in the target formula, the example shown is... In the case of "\frac{\sqrt{3}}{3+2}+3^{2}", the LaTeX format recognition result is "\frac{\sqrt{3}}{3+2}+3^{2}". For example, when the target formula is... In this case, the LaTeX format recognition result is "H_{LC}=\int_{0}^{l}\mathcal{H}_{LC}".

[0068] Optionally, when the target formula is an organic chemical formula, the trajectory image of the target formula is processed using a trained organic chemical formula recognition model to obtain a third recognition result of the target formula.

[0069] In this implementation, when the target formula is an organic chemical formula, a corresponding organic chemical formula recognition model can be selected to process the trajectory image of the organic chemical formula and output the recognition result. This implementation not only achieves the recognition of single-line and multi-line handwritten formulas, but also the recognition of handwritten organic chemical formulas.

[0070] In this embodiment, to facilitate subsequent processing (e.g., formula reduction) of the identification results based on the target formula by the electronic device, the identification results of the organic chemical formula can be converted into a string format (e.g., SMILES format). SMILES can also be considered a syntax rule that can convert organic chemical formulas into one-dimensional ASCII strings according to preset rules. Combined with... Figure 5 For example, if the target formula is "cyclohexane", the recognition result in SMILES format is "C1CCCCC1".

[0071] The following sections will provide detailed explanations of the processes for training single-line formula recognition models, multi-line formula recognition models, and organic chemical formula recognition models. All three models are deep neural network models with an encoder-decoder architecture.

[0072] First, the training process of the single-line formula recognition model.

[0073] Obtain various types of single-line formula samples from a preset formula sample set. Input these various types of single-line formula samples into the single-line formula recognition model for training, resulting in a trained single-line formula recognition model.

[0074] During training, single-line formula samples of various types (e.g., mathematics, physics, chemistry, etc.) are used as input to iteratively train the single-line formula recognition model. The training is considered complete when the recognition result output by the single-line formula recognition model closely matches the reference result corresponding to the single-line formula sample.

[0075] This implementation method trains the single-line formula recognition model by inputting various types of single-line formula samples. The trained model can then recognize different types of single-line formula samples. This improves the generalization ability of the single-line formula recognition model and enhances the accuracy of electronic devices when using it to recognize single-line formulas.

[0076] Secondly, the training process of the multi-line formula recognition model.

[0077] Optionally, the training process of the multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set, to obtain a trained multi-line formula recognition model. The mixed formula samples include single-line formula samples and multi-line formula samples.

[0078] During training, a mixed set of formula samples, including single-line and multi-line formula samples, is used as input to iteratively train the multi-line formula recognition model. The multi-line formula recognition model is considered complete when its output closely matches the reference result corresponding to the multi-line formula sample.

[0079] Optionally, the training process of the multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set to obtain a first formula recognition model. The mixed formula samples include single-line formula samples and multi-line formula samples. Multiple types of multi-line formula samples are then input into the first formula recognition model for training to obtain a trained multi-line formula recognition model.

[0080] In this implementation, to further improve the recognition accuracy and generalization ability of the trained multi-line formula recognition model, the training process first uses a mixed formula sample, including single-line and multi-line formula samples, as input to coarsely train the initial multi-line formula recognition model, obtaining a first formula recognition model. Then, various types of multi-line formula samples are used as input to refine the first formula recognition model, fine-tuning it to obtain a trained multi-line formula recognition model. This implementation, by incorporating a fine-tuning process during training, improves the recognition accuracy and generalization ability of the trained multi-line formula recognition model.

[0081] Optionally, the training process of the multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set to obtain a first formula recognition model. The mixed formula samples include single-line formula samples and multi-line formula samples. The dictionary of the first formula recognition model is adjusted, and the mixed formula samples are input into the adjusted dictionary of the first formula recognition model for training to obtain a trained multi-line formula recognition model.

[0082] In this implementation, because the ratio of single-line formula samples to multi-line formula samples in the preset formula sample is unbalanced, the number of multi-line formula samples in the mixed formula sample obtained from the preset formula sample is relatively small. Therefore, the first formula recognition model obtained after training the initial multi-line formula recognition model using the mixed formula samples is prone to overfitting. Therefore, during the training process, an adjustment to the dictionary of the first formula recognition model is incorporated, and the first formula recognition model with the adjusted dictionary is retrained using the mixed formula samples. Because the dictionary of the first formula recognition model is adjusted, the multi-line formula recognition model obtained by retraining based on the first formula recognition model can reduce the problem of overfitting. This improves the recognition accuracy and generalization ability of the trained multi-line formula recognition model.

[0083] In this implementation, when retraining the first formula recognition model, the model parameters from before retraining need to be loaded to ensure that retraining is performed based on the first formula recognition model. Furthermore, when retraining the first formula recognition model, the hyperparameters used when training the initial multi-line formula classification model are still applied.

[0084] In this embodiment, adjusting the dictionary of the first formula recognition model can be achieved by reordering the dictionary. Here, the dictionary refers to the set of character types that the formula recognition model can recognize, equivalent to a character type library. During the training process of the multi-line formula recognition model, the model can predict each character in the organic chemical formula trajectory image based on the dictionary, and represent each character with numbers, ultimately obtaining a mapping relationship between multiple characters and multiple numbers. The sequence of numbers can be mapped to a token sequence. In this embodiment, reordering the dictionary of the first formula recognition model refers to remapping the mapping relationship between multiple characters and multiple numbers.

[0085] Optionally, the training process of the multi-line formula recognition model includes: training an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set to obtain a first formula recognition model. The mixed formula samples include single-line formula samples and multi-line formula samples. The dictionary of the first formula recognition model is adjusted, and the mixed formula samples are input into the first formula recognition model for training to obtain a second formula recognition model. Multiple types of multi-line formula samples are input into the second formula recognition model for training to obtain a trained multi-line formula recognition model.

[0086] In this implementation, to further improve the recognition accuracy and generalization ability of the trained multi-line formula recognition model, the dictionary of the first formula recognition model is first adjusted during training. Then, the first recognition model with the adjusted dictionary is retrained using mixed formula samples to obtain the second formula recognition model. Next, various types of multi-line formula samples are used as input to refine the training of the second formula recognition model, thereby fine-tuning it and obtaining the trained multi-line formula recognition model.

[0087] In this implementation, when retraining the first formula recognition model, it is necessary to load the model parameters from before retraining to ensure that retraining is performed based on the first formula recognition model. Similarly, when fine-tuning the training of the second formula recognition model, it is necessary to load the model parameters from before fine-tuning, i.e., load the parameters from the retrained second formula recognition model, thus ensuring that the formula recognition model is retrained based on the previously trained model, thereby maintaining training continuity. Furthermore, both retraining the first formula recognition model and fine-tuning the second formula recognition model are performed using the hyperparameters from the initial multi-line formula classification model.

[0088] It should be noted that in this embodiment, the method of adjusting the dictionary of the first formula recognition model can refer to the aforementioned embodiment, and the adjustment method is the same, so it will not be repeated here.

[0089] This implementation method, by adjusting the dictionary of the first formula recognition model and retraining the first formula recognition model with the adjusted dictionary, yields a second formula recognition model, which reduces the overfitting problem in the trained second formula recognition model. Furthermore, by fine-tuning the second formula recognition model, the recognition accuracy and generalization ability of the trained multi-line formula recognition model can be improved.

[0090] Furthermore, to further improve the recognition accuracy and generalization ability of the trained multi-line formula recognition model, the electronic device can increase the resolution of the multi-line formula samples within the mixed formula samples during training. This allows the multi-line formula recognition model to clearly identify the samples during training, thereby improving its recognition accuracy and generalization ability.

[0091] It should be noted that, in this application embodiment, the above four implementation methods are provided for the training process of the multi-line formula recognition model. Optionally, the above four implementation methods have been verified in this application embodiment, and the verification results are shown in Table 1 below:

[0092]

[0093]

[0094] Table 1

[0095] As shown in Table 1, mixed-sample training is the process of training the model using mixed formula samples, including single-line and multi-line formula samples. Dictionary adjustment training is the process of retraining the model by adjusting its dictionary and using mixed formula samples. Fine-tuning training is the process of refining the model's training using multi-line formula samples. The recognition accuracy of the multi-line formula recognition model is mainly reflected by verifying its character recognition accuracy and whole-line recognition accuracy. Referring to Table 1, it can be seen that in the training process of the multi-line formula recognition model, using only mixed-sample training results in low recognition accuracy. After mixed-sample training, adding dictionary adjustment training and / or fine-tuning training can appropriately improve the recognition accuracy of the multi-line formula recognition model. Among them, the multi-line formula recognition model achieves the highest recognition accuracy when both dictionary adjustment training and fine-tuning training are added after mixed-sample training.

[0096] Furthermore, in combination Figure 6 As shown, in the process of verifying multi-line formula recognition, this application embodiment provides a curve of the overall line recognition accuracy of the multi-line formula recognition model. The upper curve represents the training process of the multi-line formula recognition model with dictionary adjustment included. The lower curve represents the training process of the multi-line formula recognition model without dictionary adjustment included. (Reference) Figure 6 It can be seen that if a dictionary adjustment training process is added during the training of the multi-line formula recognition model, the overall recognition accuracy of the multi-line formula recognition model will be relatively stable.

[0097] Then, the training process of the organic chemical formula recognition model.

[0098] Organic chemical formula samples are obtained from a pre-defined dataset and input into an organic chemical formula recognition model for training to obtain the recognition features of the organic chemical formula samples. The trained feature extraction model is then used to process the organic chemical formula samples to obtain their predicted features. Based on the recognition and predicted features of the organic chemical formula samples, the contrastive learning loss of the organic chemical formula recognition model is calculated. The training of the organic chemical formula recognition model is complete when the contrastive learning loss meets a first pre-defined condition.

[0099] During training, the electronic device uses a pre-trained feature extraction model to identify organic chemical formula samples, obtaining predicted features of the samples. These predicted features are used as supervisory information. This supervisory information oversees the training process of the organic chemical formula recognition model, achieving self-supervised training. Specifically, during training, the model parameters are adjusted based on a contrastive learning loss between the supervisory information and the output features of the model, ensuring that the distribution of the output features matches the distribution in the supervisory information. Thus, training of the organic chemical formula recognition model is completed when the contrastive learning loss meets a first preset condition.

[0100] In the training process of the organic chemical formula recognition model, the predicted features output by the feature extraction model are used as supervisory information, thus achieving self-supervised training. Through supervised training, the organic chemical formula recognition model learns the detailed features of organic chemical formulas. This allows the trained model to better recognize organic chemical formulas, thereby improving its recognition performance.

[0101] The following sections will explain in detail the process of training the feature extraction model and the process of applying the feature extraction model. The feature extraction model can be a graph convolutional neural network model.

[0102] Optionally, the training process of the feature extraction model includes: constructing positive and negative samples based on a preset organic chemical formula sequence; inputting the positive and negative samples into the feature extraction model for training to obtain the feature distributions of the positive and negative samples; calculating the second contrastive learning loss of the feature extraction model based on the feature distributions of the positive and negative samples; and completing the training of the feature extraction model when the second contrastive learning loss satisfies a second preset condition.

[0103] The preset organic chemical formula sequence can be a string stored in SMILES format. The preset organic chemical formula sequence may include one or more organic chemical formula sequences. Based on the preset organic chemical formula sequence, the electronic device can reconstruct each organic chemical formula sequence to obtain a trajectory image of each organic chemical formula.

[0104] Furthermore, based on a preset organic chemical formula sequence, positive samples are constructed, including: obtaining a first organic chemical formula sequence from the preset organic chemical formula sequence and restoring the first organic chemical formula sequence to a trajectory image of the first organic chemical formula. The trajectory image of the first organic chemical formula is enhanced to construct a trajectory image of a second organic chemical formula that is similar to the trajectory image of the first organic chemical formula. The trajectory images of the first and second organic chemical formulas are used as positive samples.

[0105] When enhancing the trajectory image of a first organic chemical formula, nodes in the first organic chemical formula can be deleted to construct a trajectory image of a second organic chemical formula that is similar to the trajectory image of the first organic chemical formula. Here, nodes in the first organic chemical formula refer to atoms within the first organic chemical formula. Deleting a predetermined number of nodes or atoms from the first organic chemical formula converts it into a second organic chemical formula. The trajectory images of the first and second organic chemical formulas are similar, and these similar trajectory images are considered positive samples.

[0106] It should be noted that deleting a large number of nodes from the first organic chemical formula can easily lead to a lower similarity between the first organic chemical formula after node deletion and the first organic chemical formula without deleted nodes. Therefore, by limiting the number of nodes that can be deleted from the first organic chemical formula, a high similarity between the first and second organic chemical formulas can be ensured, that is, a high similarity between sample data in positive samples can be guaranteed. The preset number can be determined based on the total number of nodes in the first organic chemical formula. For example, in the first organic chemical formula, the preset number should be less than 1 / 4 of the total number of nodes.

[0107] It should also be noted that in the process of constructing the second organic chemical formula based on the first organic chemical formula, the corresponding second organic chemical formula can be obtained by deleting one node from the first organic chemical formula. Alternatively, the corresponding second organic chemical formula can be obtained by deleting multiple nodes from the first organic chemical formula. The specific configuration can be set according to requirements, and this application does not limit this.

[0108] Furthermore, based on a pre-defined organic chemical formula sequence, negative samples are constructed, including: obtaining a third organic chemical formula sequence from the pre-defined organic chemical formula sequence. The third organic chemical formula sequence is different from the first organic chemical formula sequence. The third organic chemical formula sequence is then restored to its trajectory image. The trajectory image of the third organic chemical formula is enhanced to construct a trajectory image of a fourth organic chemical formula that is similar to the trajectory image of the third organic chemical formula. Both the trajectory images of the third and fourth organic chemical formulas are used as negative samples.

[0109] Since the construction process for negative samples is similar to that for positive samples, it will not be described in detail here. Because the positive and negative samples are constructed based on different organic chemical formula sequences, the similarity between them is guaranteed to be low. Thus, after training the feature extraction model based on the positive and negative samples, the model's recognition ability can be ensured.

[0110] In this implementation, after constructing positive and negative samples, the electronic device can input the positive and negative samples into the feature extraction model for training, causing the feature extraction model to output the features of the positive and negative samples. During training, the goal is to maximize the contrastive learning loss between the feature distributions of positive and negative samples. A large contrastive learning loss between the feature distributions of positive and negative samples indicates that the trained feature extraction model can distinguish between positive and negative samples and classify them into different classes. Therefore, when the contrastive learning loss meets the second preset condition, the feature extraction model training can be considered complete.

[0111] After the feature extraction model is trained, the electronic device can use the predicted features generated by the feature extraction model as supervisory information and apply this supervisory information to the training process of the organic chemical formula recognition model. The application process of the feature extraction model includes: inputting organic chemical formula samples into the feature extraction model, causing the feature extraction model to output predicted features of the organic chemical formula samples, and using these predicted features as supervisory information. The reason for using the predicted features of the organic chemical samples output by the feature extraction model as supervisory information during the training process of the organic chemical formula recognition model is that the predicted features generated by the feature extraction model are relatively comprehensive. The predicted features of the organic chemical formula samples will be explained in detail below.

[0112] The predicted features of organic chemical formula samples include at least the chemical features of atoms and the topological features between atoms. The chemical features of atoms can be multi-dimensional, such as 64-dimensional features, as shown in Table 2 below:

[0113]

[0114] Table 2

[0115] In organic chemical formulas, chemical bonds between atoms can be classified into single bonds, double bonds, triple bonds, hybrid bonds, virtual ring bonds, and other types of bonds. The topological characteristics of atoms are used to characterize which of these chemical bonds connects the atoms. For example, the topological characteristics of atoms can be represented using an adjacency matrix.

[0116] This implementation method allows for the pre-training of a feature extraction model, and the comprehensive predictive features generated by the model are used as supervisory information in the training process of the organic chemical formula recognition model. By training the organic chemical formula recognition model with comprehensive and multi-dimensional predictive features, the trained model can learn the structural information of organic chemical formulas, thereby improving the performance of organic chemical formula recognition.

[0117] Combination Figure 7 As shown, this application provides a formula recognition device, which includes:

[0118] Training module 701 is used to train a formula classification model using preset formula samples to obtain a trained formula classification model. The preset formula samples include at least single-line formula samples and multi-line formula samples.

[0119] The classification module 702 is used to process the trajectory image of the target formula using a trained formula classification model to obtain the category of the target formula.

[0120] The recognition module 703 is used to process the trajectory image of the target formula using a trained single-line formula recognition model when the target formula is a single-line formula, so as to obtain the first recognition result of the target formula; and to process the trajectory image of the target formula using a trained multi-line formula recognition model when the target formula is a multi-line formula, so as to obtain the second recognition result of the target formula.

[0121] Optionally, the preset formula samples also include organic chemical formula samples; the training module 801 is specifically used to: train a formula classification model by obtaining single-line formula samples, multi-line formula samples and organic chemical formula samples from the preset formula samples, so as to obtain a trained formula classification model.

[0122] Optionally, the training module 701 is specifically used to: concatenate single-line formula samples and preset symbols according to a first preset rule to obtain expanded multi-line formula samples; concatenate monocyclic organic chemical formulas in organic chemical formula samples according to a second preset rule to obtain expanded organic chemical formula samples; and input the single-line formula samples, expanded multi-line formula samples, and expanded organic chemical formula samples into the formula classification model for training to obtain a trained formula classification model.

[0123] Optionally, the recognition module 703 is further configured to, after processing the trajectory image of the target formula using the trained formula classification model to obtain the category of the target formula, if the target formula is an organic chemical formula, process the trajectory image of the target formula using the trained organic chemical formula recognition model to obtain a third recognition result of the target formula.

[0124] Optionally, the training module 701 is also used to obtain various types of single-line formula samples from a preset formula sample. These various types of single-line formula samples are then input into the single-line formula recognition model for training, resulting in a trained single-line formula recognition model.

[0125] Optionally, the training module 701 is further configured to train an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set, so as to obtain a trained multi-line formula recognition model. The mixed formula sample set includes single-line formula samples and multi-line formula samples.

[0126] Optionally, the training module 701 is further configured to train an initial multi-line formula recognition model using mixed formula samples obtained from a preset formula sample set, to obtain a first formula recognition model; wherein the mixed formula sample set includes single-line formula samples and multi-line formula samples. The dictionary of the first formula recognition model is adjusted, and the mixed formula samples are input into the adjusted dictionary of the first formula recognition model for training, to obtain a second formula recognition model; multiple types of multi-line formula samples are input into the second formula recognition model for training, to obtain a trained multi-line formula recognition model.

[0127] Optionally, the training module 701 is further configured to acquire organic chemical formula samples from a preset dataset and input these samples into an organic chemical formula recognition model for training, thereby obtaining the recognition features of the organic chemical formula samples. The trained feature extraction model is then used to process the organic chemical formula samples to obtain their predicted features. Based on the recognition features and predicted features of the organic chemical formula samples, a first contrastive learning loss for the organic chemical formula recognition model is calculated. When the first contrastive learning loss satisfies a first preset condition, the training of the organic chemical formula recognition model is complete.

[0128] Optionally, the training module 701 is further used to construct positive and negative samples based on preset organic chemical formula sequences. The positive and negative samples are input into the feature extraction model for training to obtain the feature distributions of the positive and negative samples. Based on the feature distributions of the positive and negative samples, a second contrastive learning loss of the feature extraction model is calculated. The feature extraction model training is complete when the second contrastive learning loss satisfies a second preset condition.

[0129] The formula recognition device provided in this application embodiment can execute the formula recognition method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0130] Combination Figure 8As shown, this application embodiment provides an electronic device 800, including a processor 100 and a memory 101. Optionally, the electronic device 800 may further include a communication interface 102 and a bus 103. The processor 100, communication interface 102, and memory 101 can communicate with each other via the bus 103. The communication interface 102 can be used for information transmission. The processor 100 can call logical instructions in the memory 101 to execute the formula recognition method described in the above embodiment.

[0131] Furthermore, the logic instructions in the aforementioned memory 101 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0132] The memory 101, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 100 executes functional applications and data processing by running the program instructions / modules stored in the memory 101, that is, it implements the formula recognition method in the above embodiments.

[0133] The memory 101 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 101 may include high-speed random access memory and may also include non-volatile memory.

[0134] This application provides a storage medium storing computer-executable instructions configured to perform the formula recognition method described in the above embodiments.

[0135] The aforementioned storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0136] The technical solutions of this application embodiment can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this application embodiment. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0137] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for formula recognition, characterized in that, include: A formula classification model is trained using a preset formula sample to obtain a trained formula classification model; wherein the preset formula sample includes at least single-line formula samples and multi-line formula samples. The trained formula classification model is used to process the trajectory image of the target formula to obtain the category of the target formula; When the target formula is a single-line formula, the trajectory image of the target formula is processed using a trained single-line formula recognition model to obtain the first recognition result of the target formula; When the target formula is a multi-line formula, the trajectory image of the target formula is processed using a trained multi-line formula recognition model to obtain a second recognition result of the target formula.

2. The method according to claim 1, characterized in that, A formula classification model is trained using pre-defined formula samples to obtain a well-trained formula classification model, including: The preset formula sample includes organic chemical formula samples; The formula classification model is trained using the single-line formula samples, the multi-line formula samples, and the organic chemical formula samples obtained from the preset formula samples, so as to obtain the trained formula classification model.

3. The method according to claim 2, characterized in that, The formula classification model is trained using the single-line formula samples, the multi-line formula samples, and the organic chemical formula samples obtained from the preset formula samples, to obtain the trained formula classification model, including: According to the first preset rule, the single-line formula sample and the preset symbol are concatenated to obtain the expanded multi-line formula sample; According to the second preset rule, the monocyclic organic chemical formulas in the organic chemical formula sample are spliced ​​together to obtain an expanded organic chemical formula sample. The single-line formula sample, the expanded multi-line formula sample, and the expanded organic chemical formula sample are respectively input into the formula classification model for training to obtain the trained formula classification model.

4. The method according to claim 1, characterized in that, After processing the trajectory image of the target formula using the trained formula classification model to obtain the category of the target formula, the process further includes: When the target formula is an organic chemical formula, the trajectory image of the target formula is processed using a trained organic chemical formula recognition model to obtain the third recognition result of the target formula.

5. The method according to claim 1, characterized in that, Training the single-line formula recognition model includes: Obtain various types of single-line formula samples from the preset formula sample; The various types of single-line formula samples are input into the single-line formula recognition model for training to obtain the trained single-line formula recognition model.

6. The method according to claim 1, characterized in that, Training the multi-line formula recognition model includes: An initial multi-line formula recognition model is trained using mixed formula samples obtained from the preset formula samples, so as to obtain the trained multi-line formula recognition model; The mixed formula sample includes the single-line formula sample and the multi-line formula sample.

7. The method according to claim 1, characterized in that, Training the multi-line formula recognition model includes: An initial multi-line formula recognition model is trained using mixed formula samples obtained from the preset formula samples to obtain a first formula recognition model; wherein, the mixed formula samples include the single-line formula samples and the multi-line formula samples; The dictionary of the first formula recognition model is adjusted, and the mixed formula samples are input into the first formula recognition model with the adjusted dictionary for training to obtain the second formula recognition model; Multiple types of multi-line formula samples are input into the second formula recognition model for training to obtain the trained multi-line formula recognition model.

8. A formula recognition device, characterized in that, include: The training module is used to train a formula classification model using a preset formula sample to obtain a trained formula classification model; wherein the preset formula sample includes at least single-line formula samples and multi-line formula samples. The classification module is used to process the trajectory image of the target formula using the trained formula classification model to obtain the category of the target formula. The recognition module is used to process the trajectory image of the target formula using a trained single-line formula recognition model when the target formula is a single-line formula, so as to obtain a first recognition result of the target formula; and to process the trajectory image of the target formula using a trained multi-line formula recognition model when the target formula is a multi-line formula, so as to obtain a second recognition result of the target formula.

9. An electronic device comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the formula recognition method as described in any one of claims 1 to 7 when executing the program instructions.

10. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the formula recognition method as described in any one of claims 1 to 7.

Images