Chemical formula structure identification method, apparatus and device, and storage medium

Abstract

The invention discloses a chemical formula structure identification method and device, electronic equipment and a storage medium, the method inputs an image into a chemical formula structure identification model to output a chemical formula code, and the chemical formula structure identification model is utilized to improve the accuracy of identifying two chemical formula structures of handwriting and printing. The chemical formula structure model determination process comprises the following steps: obtaining a training sample; pre-training a preset model by using the synthetic training sample to obtain a basic model; training a basic model by using a sample obtained by processing the original training sample in the original training sample and the enhanced training sample to obtain a first model; training a basic model by using the original training sample, the enhanced training sample and the synthetic training sample to obtain a second model; based on preset model parameters, the first model and the second model are finely adjusted for multiple times through the original training samples; and according to the first model and the second model after each fine adjustment, a chemical formula structure identification model is determined through a model integration strategy.

Description

Technical Field

[0001] The present application relates to the technical field of identifying chemical formula structures, and in particular to a method, device, electronic device and storage medium for identifying chemical formula structures. Background Art

[0002] Recognizing chemical formula structures is crucial in fields such as education and biochemistry. Chemical formula structure recognition has been widely used in various fields such as drug development, human-computer interaction, and biochemistry, improving the efficiency and convenience of analyzing drug molecular structures. However, the important value of chemical molecular structure recognition in the field of education has not received enough attention, especially for handwritten chemical formula structures. Therefore, how to accurately recognize both handwritten and printed chemical formula structures has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] The present application provides a method, device, electronic device and storage medium for chemical formula structure recognition. The method inputs an image into a chemical formula structure recognition model determined based on training data including synthetic training samples using pre-training, transfer learning and fine-tuning steps to output a chemical formula code corresponding to the image. The chemical formula structure recognition model can improve the accuracy of recognizing both handwritten and printed chemical formula structures.

[0004] In a first aspect, the present invention provides a method for chemical formula structure recognition, comprising:

[0005] Inputting the image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image;

[0006] The process of determining the chemical formula structure model includes: obtaining training samples, the training samples including original training samples, synthetic training samples and enhanced training samples, the original training samples including a handwritten chemical formula image and a corresponding chemical formula code and a printed chemical formula image and a corresponding chemical formula code; the synthetic training samples are synthesized according to the chemical formula code in the original training samples, the enhanced training samples are samples processed according to the original training samples, and samples processed according to the synthetic training samples;

[0007] Pre-training a preset model using the synthetic training samples to obtain a basic model, wherein the preset model is a deep learning model;

[0008] Training a basic model using the original training samples and samples obtained by processing the original training samples in the enhanced training samples to obtain a first model;

[0009] Training the basic model using the original training samples, the enhanced training samples and the synthesized training samples to obtain a second model, wherein a learning rate corresponding to the second model is different from a learning rate corresponding to the first model;

[0010] Based on the preset model parameters, fine-tuning the first model and the second model multiple times respectively using the original training samples;

[0011] According to the first model and the second model after each fine-tuning, the chemical formula structure recognition model is determined through a model integration strategy.

[0012] In some embodiments, the process of determining the chemical formula structure model further includes:

[0013] Screening out original training samples whose chemical formula codes are target chemical formula codes from the original training data;

[0014] synthesizing a chemical formula synthetic image using the target chemical formula code;

[0015] Obtaining original training samples whose chemical formula codes are non-target chemical formula codes;

[0016] Converting the non-target chemical formula code into a target chemical formula code, and synthesizing a chemical formula synthetic image using the converted target chemical formula code;

[0017] The chemical formula synthetic image and the target chemical formula code corresponding to the chemical formula synthetic image are determined as synthetic training samples.

[0018] In some embodiments, the process of determining the chemical formula structure model further includes:

[0019] Processing the original training samples using data enhancement technology to generate samples among the trained samples that are processed according to the original training samples;

[0020] The synthetic training sample is processed using a data enhancement technique to generate a sample in the trained sample that is processed according to the synthetic training sample.

[0021] In some embodiments, the preset model is a Nougat model.

[0022] In some embodiments, the step of fine-tuning the first model and the second model multiple times respectively using the original training samples based on the preset model parameters includes:

[0023] Based on different preset model parameters, the first model and the second model are respectively fine-tuned for the first time using the original training samples to obtain the first model after the first fine-tuning and the second model after the first fine-tuning;

[0024] Select M models whose evaluation performance ranks in the top N from the first model after the first fine-tuning and the second model after the first fine-tuning, where M is less than N;

[0025] Based on different preset model parameters, the M models are fine-tuned using the original training samples to obtain a model after the second fine-tuning;

[0026] Selecting F models whose evaluation performance is in the top G from the models after the second fine-tuning, where F is smaller than G;

[0027] Based on different preset model parameters, the F models are fine-tuned for a third time using the original training samples to obtain a model after the third fine-tuning.

[0028] In some embodiments, the step of determining the chemical formula structure recognition model by a model integration strategy based on the first model and the second model after each fine-tuning includes:

[0029] Grouping the first model and the second model after each fine-tuning to obtain multiple groups of model sets; wherein the model set includes at least one of the first model and the second model after each fine-tuning;

[0030] Randomly select a preset number of data from the original training data as preset verification samples;

[0031] Inputting the image in the preset verification sample into each model in the model set, so that each model outputs a chemical formula code corresponding to the image;

[0032] Screening out chemical formula codes to be compared that meet preset rules from chemical formula codes corresponding to the images output by each model in the model set;

[0033] Determining the evaluation performance to be compared according to the chemical formula code to be compared and the preset chemical formula code corresponding to the image in the preset verification sample;

[0034] From the evaluation performances to be compared corresponding to the multiple groups of model sets, the best evaluation performance to be compared is screened out, and the model set corresponding to the best performance to be evaluated is determined as the chemical formula recognition result model.

[0035] In some embodiments, the arrangement order of the models in the model set is sorted according to the evaluation performance of the models; and the step of screening out the chemical formula codes to be compared that meet the preset rules from the chemical formula codes corresponding to the images output by each model in the model set comprises:

[0036] Filter out repeated chemical formula codes from the chemical formula codes corresponding to the images output by each model in the model set, and count the number of repeated occurrences;

[0037] Determine the number of chemical formula codes corresponding to the maximum number of repetitions;

[0038] If it is greater than 1, then determine the order of arrangement of the models in the model set used when generating the chemical formula code corresponding to the maximum number of repetitions; determine the chemical formula code corresponding to the model at the first position as the chemical formula code to be compared;

[0039] If it is 1, the chemical formula code corresponding to the greatest number of repetitions is determined as the chemical formula code to be compared.

[0040] In a second aspect, the present application also provides a device for chemical formula structure recognition, comprising:

[0041] determining a chemical formula code unit for inputting the image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image;

[0042] The process of determining the chemical formula structure model includes: obtaining training samples, the training samples include original training samples, synthetic training samples and enhanced training samples, the original training samples include chemical formula handwritten images and corresponding chemical formula codes and chemical formula printed images and corresponding chemical formula codes; the synthetic training samples are synthesized according to the chemical formula codes in the original training samples, the enhanced training samples are samples processed according to the original training samples, and samples processed according to the synthetic training samples; using the synthetic training samples to pre-train a preset model to obtain a basic model, the preset model is a deep learning model; using the original training samples and the samples obtained by processing the original training samples in the enhanced training samples to train the basic model to obtain a first model; using the original training samples, the enhanced training samples and the synthetic training samples to train the basic model to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; based on the preset model parameters, using the original training samples to fine-tune the first model and the second model for multiple times respectively; according to the first model and the second model after each fine-tuning, the chemical formula structure recognition model is determined through a model integration strategy.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for chemical formula structure recognition when executing the computer program.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for chemical formula structure recognition are implemented.

[0045] In the above embodiment, a method, device, electronic device and storage medium for chemical formula structure recognition are provided. The method inputs an image into a chemical formula structure recognition model determined by pre-training, transfer learning and fine-tuning based on training data including synthetic training samples to output a chemical formula code corresponding to the image. The chemical formula structure recognition model can improve the accuracy of recognizing both handwritten and printed chemical formula structures. The method includes: obtaining training samples; pre-training a preset model using the synthetic training samples to obtain a basic model, wherein the preset model is a deep learning model; training a basic model using samples obtained by processing the original training samples and the enhanced training samples according to the original training samples to obtain a first model; training the basic model using the original training samples, the enhanced training samples and the synthetic training samples to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; fine-tuning the first model and the second model multiple times respectively using the original training samples based on the preset model parameters; determining a chemical formula structure recognition model through a model integration strategy according to the first model and the second model after each fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for chemical formula structure recognition provided according to some embodiments is exemplified;

[0047] Figure 2 A schematic diagram of a process of synthesizing a chemical formula synthetic image using a non-target chemical formula code according to some embodiments is exemplarily shown;

[0048] Figure 3 A schematic diagram of a chemical formula handwritten image processed by the above four data enhancement techniques is exemplarily shown according to some embodiments;

[0049] Figure 4 A schematic structural diagram of a migration device based on a heterogeneous database provided according to some embodiments is exemplarily shown. DETAILED DESCRIPTION

[0050] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0051] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0052] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0053] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0054] Currently, the important value of chemical formula structure recognition in the field of education has not received enough attention, especially there are few studies on handwritten chemical formula structure recognition.

[0055] In the related technology, the recognition technology for chemical structures is called OCSR (Optical Chemical Structure Recognition), that is, optical chemical structure recognition. The technical routes of OCSR can be divided into rule-based and machine learning. The early OCSR technical route adopted the rule-based method, and its steps can be divided into image preprocessing, separation and segmentation, text recognition, vectorization, line segment classification, node recognition, assembly, post-processing, etc. However, the update and maintenance of rule-based OCSR is relatively complicated, and the optimization cost is high, and it is basically not universal. With the popularity of machine learning methods, OCSR technology has also incorporated machine learning. Compared with the rule-based method, the machine learning method of OCSR using machine learning, which manually designs features and then uses some models for classification, is more flexible and accurate. However, the recognition accuracy and generalization of the above methods are still low.

[0056] In order to solve the above technical problems, an embodiment of the present application provides a method for chemical formula structure recognition, which inputs an image into a chemical formula structure recognition model determined based on training data including synthetic training samples, using pre-training, transfer learning and fine-tuning steps to output a chemical formula code corresponding to the image. The chemical formula structure recognition model can improve the accuracy of recognizing both handwritten and printed chemical formula structures.

[0057] In an embodiment of the present application, the method includes: inputting an image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image.

[0058] Specifically, in fields such as education where there is a need to recognize handwritten chemical formulas and printed chemical formulas, an image including a chemical formula can be input into a chemical formula structure model, and the chemical formula structure model outputs a chemical formula code corresponding to the image.

[0059] The following is a detailed introduction to the process of determining the chemical formula structure recognition model.

[0060] Figure 1 A flowchart of a process for determining a chemical formula structure recognition model according to some embodiments is exemplarily shown. The method includes S100-S600.

[0061] S100. Acquire training samples, wherein the training samples include original training samples, synthetic training samples, and enhanced training samples. The original training samples include a handwritten chemical formula image and a corresponding chemical formula code and a printed chemical formula image and a corresponding chemical formula code. The synthetic training samples are synthesized based on the chemical formula codes in the original training samples. The enhanced training samples are samples processed based on the original training samples, and samples processed based on the synthetic training samples.

[0062] In the embodiment of the present application, in order to ensure that the chemical formula structure recognition model has good generalization and good recognition accuracy, it is necessary to ensure that the data scale and data category of the training sample are complete, and then the embodiment of the present application obtains the original training sample through various ways, and exemplary, the original training sample can be obtained from various ways such as open source data sets, competition data sets, public databases and digitally processed chemical data sets. In the embodiment of the present application, the chemical formula handwritten image can be understood as an image of an artificial handwritten chemical formula. The chemical formula code can be a common code format of at least one million levels such as SIMLES code (simplified molecular linear input and output system code), chemfig code (chemical graphics code) or ssml_sd code (speech synthesis markup language_speech description code). It should be noted that the number of training samples in the embodiment of the present application is not limited.

[0063] In some embodiments, the chemical formula codes in the original training samples include the categories shown in Table 1, and the original training samples include chemical formula codes of multiple categories.

[0064] Table 1

[0065]

[0066] In some embodiments, the method further comprises:

[0067] From the original training data, original training samples whose chemical formula codes are target chemical formula codes are selected; in one example, the target chemical formula code is a chemfig code.

[0068] A chemical formula synthetic image is synthesized using the target chemical formula code.

[0069] In this embodiment, the Chemistry Development Kit CDK tool can be used to synthesize the target chemical formula code with the chemical formula synthesis image. The chemical formula synthesis image can be understood as an image of a standard printed chemical formula.

[0070] Obtain original training samples whose chemical formula codes are non-target chemical formula codes.

[0071] Since the chemical formula codes in the original training samples obtained may be SIMLES codes or other codes, it is necessary to unify the data format, for example, converting SIMLES codes into chemfig codes.

[0072] The non-target chemical formula code is converted into a target chemical formula code, and the converted target chemical formula code is used to synthesize a chemical formula synthetic image.

[0073] For example, Figure 2 As shown, Figure 2 The following is a schematic diagram of a process for synthesizing a chemical formula image using a non-target chemical formula code according to some embodiments. Figure 2 The SIMLES code is on the left, the chemical formula synthesis image is in the middle, and the chemfig code is on the right.

[0074] The chemical formula synthetic image and the target chemical formula code corresponding to the chemical formula synthetic image are determined as synthetic training samples. In the embodiment of the present application, the target chemical formula code corresponding to the chemical formula synthetic image includes not only the target chemical formula code already in the original training sample, but also the target chemical formula code converted from the non-target chemical formula code.

[0075] In this embodiment, in order to expand the training samples as much as possible, the chemical formula codes in the original training samples are used to synthesize chemical formula synthetic images to form synthetic training samples. The target chemical formula code is used to reversely synthesize the chemical formula synthetic image. In this process, the target chemical formula code can be changed, for example, the font and font size can be modified, so that the chemical formula synthetic image synthesized by the changed target chemical formula code can enrich the training features, improve the accuracy of the chemical formula structure recognition model obtained by training, and enhance the generalization.

[0076] In some embodiments, in order to further expand the training samples and improve the generalization of the chemical formula structure recognition model, the method further includes:

[0077] Processing the original training samples using data enhancement technology to generate samples among the trained samples that are processed according to the original training samples;

[0078] The synthetic training sample is processed using a data enhancement technique to generate a sample in the trained sample that is processed according to the synthetic training sample.

[0079] In an embodiment of the present application, when the original training samples and the synthetic training samples are limited, data enhancement technology is used to expand the training samples by performing various transformations and expansions on the original training samples and the synthetic training samples, thereby increasing the generalization ability and robustness of the model.

[0080] In some embodiments, the data enhancement technique includes at least one of gamma transformation, adjusting brightness, adjusting contrast, adjusting hue, adjusting saturation, adjusting channel order, image scaling, and image translation.

[0081] In some embodiments, the data enhancement technology includes four technology combinations, and the four technology combinations can be selected to process the images in the original training samples and the synthetic training samples. The first technology combination is used to transform the picture color of the image. Specifically, the first technology combination includes three sub-combinations. When using the first technology combination to transform the image, one of the three sub-combinations can be selected. The first sub-combination includes random gamma transformation, brightness and contrast, the second sub-combination includes randomly adjusting the hue, saturation and brightness value of the image, and the third sub-combination includes randomly changing the order of the three RGB channels. The second technology combination includes random image size scaling and gamma transformation of the image. The third technology combination includes random translation, scaling, and rotation of the image. The fourth technology combination includes grid distortion, random brightness contrast, and adding Gaussian noise to the image.

[0082] For example, Figure 3 The schematic diagram of an image of a handwritten chemical formula processed by the above four data enhancement techniques according to some embodiments is exemplified. Figures a) to d) are images of a handwritten chemical formula image (original image) processed by the above four techniques.

[0083] S200. Pre-train a preset model using the synthetic training samples to obtain a basic model, where the preset model is a deep learning model.

[0084] In some embodiments, the preset model is a Nougat model, which can perform end-to-end chemical formula structure recognition based on deep learning.

[0085] In some embodiments, the synthetic training samples are divided into a first training set and a first validation set. In one example, a first training set including 800,000 synthetic training samples and a first validation set including 2,000 synthetic training samples are selected. The Nougat-latex-base model is trained and validated using the first training set and the first validation set to initialize the training model parameters and obtain a basic model. In one example, the model input layer is designed to have an input image size (image_size) of 560x224, and the model is designed to process sequence data with a max_length (maximum length of a labeled sequence) set to 800.

[0086] In some embodiments, since the chemical formula code in the synthetic training sample is the target chemical formula code, i.e., the chemfig code, the basic model will learn a large number of common features of the chemfig code during the training process, and has certain basic knowledge and representation capabilities, which can further accelerate the training to obtain a chemical formula structure recognition model.

[0087] In the embodiment of the present application, the Nougat model uses an encoder-decoder transformer architecture to provide the ability to directly parse chemical formula structures from images. The Nougat model, with its unique network architecture, does not require any OCR-related input modules during training and reasoning, and can capture subtle structural features and element relationships in complex chemical formulas, thereby achieving end-to-end accurate recognition. The Nougat model combines the image-to-text (or image description generation) model of the Swin Transformer (image encoder) and the mBART decoder (text decoder). This model architecture uses the Swin Transformer to process images in the image encoding stage, and then uses mBART (or a similar Transformer decoder) to generate text in the decoding stage.

[0088] The dimension of image x is R^(3×H0×W0), where 3 represents the RGB three channels, and H0 and W0 are the original height and width of the image. If the image size is smaller than the target size (H, W), the image size is adjusted by adding appropriate padding (such as black or white padding) to ensure that all images have the same dimensions when input to the model.

[0089] The image encoder processes the image by dividing it into non-overlapping windows of fixed size, each containing an image patch. These patches are fed into the Swin Transformer, which uses a series of self-attention layers to aggregate the information of these windows. A key feature of the Swin Transformer is its ability to process images at different scales, gradually increasing the perception by merging adjacent windows. The encoded image is represented as a sequence of embedded patches z with a dimension of R^(d×N), where d is the potential dimension (i.e., the dimension of the embedding) and N is the number of patches.

[0090] The text decoder processing includes: the decoder adopts the mBART (or similar Transformer decoder) architecture, which uses a Transformer with criss-cross attention to decode the encoded image z. The criss-cross attention mechanism allows the decoder to focus on different parts of the encoder output when generating each text token, thereby capturing key information in the image. Text tokens are generated in an autoregressive manner, that is, the generation of each token depends on the previously generated tokens. The decoder uses self-attention to focus on the input sequence (that is, the previously generated tokens) and uses criss-cross attention to focus on the image representation of the encoder output. The output of is projected to the size of the vocabulary v, producing logitsl, whose dimension is R^v. These logits can be interpreted as the predicted probability of each token in the vocabulary, usually normalized by a softmax function.

[0091] The Nougat model is used in the embodiments of the present application to perform end-to-end chemical formula structure recognition based on deep learning.

[0092] S300: Train a basic model using the original training samples and samples obtained by processing the original training samples in the enhanced training samples to obtain a first model.

[0093] Specifically, the original training samples and the samples obtained by processing the original training samples in the enhanced training samples can be split into a second training set and a second verification set, and the second training set and the second verification set are used to train and verify the basic model to obtain the first model.

[0094] S400: Train the basic model using the original training samples, the enhanced training samples and the synthetic training samples to obtain a second model, wherein a learning rate corresponding to the second model is different from a learning rate corresponding to the first model.

[0095] Specifically, the original training samples, the enhanced training samples and the synthesized training samples can be split into a third training set and a third validation set, and the third training set and the third validation set are used to train and validate the basic model to obtain the second model.

[0096] In the embodiment of the present application, in order to prevent overfitting during the model training process, the basic models are trained using data and learning rates with different distributions to obtain the first model (base_model_a) and the second model (base_model_b).

[0097] In model training, the learning rate is a hyperparameter that determines the magnitude of the model weight update in the optimization algorithm. The choice of learning rate is crucial for training deep learning models because it controls the speed at which the model updates weights at each step in the learning process.

[0098] In some embodiments, the basic model obtained by pre-training is trained by synthetic training samples whose target chemical formula code is chem fig code. At this time, the basic model only learns a large number of relevant features of chemfig code, and cannot recognize the features of other code formats, such as ssml_sd code format. Therefore, in the embodiment of the present application, the data of other codes are trained on the basic model by using the method of transfer learning. With the help of the idea of ​​transfer learning, the model parameters pre-trained on a large-scale data set are migrated to the tasks of the embodiment of the present application, so that the model makes full use of existing knowledge, better adapts to the new data distribution, and improves the output capacity of the model. Specifically, the basic model is continued to be trained using the samples obtained by processing the original training samples in the original training samples and the enhanced training samples to obtain the first model. The basic model is continued to be trained using the original training samples, the enhanced training samples and the synthetic training samples to obtain the second model. The samples obtained by processing the original training samples in the original training samples and the enhanced training samples all include chemical formula codes in other code formats, so that the basic model is continued to be trained using chemical formula codes in other code formats by using the method of transfer learning to improve training efficiency.

[0099] Exemplarily, as shown in Table 2, when training the base model to obtain the first model (base_model_a), the original training samples and the samples obtained by processing the enhanced training samples according to the original training samples are used, and the learning rate is set to 2.0e-04. When training the base model to obtain the second model (base_model_b), the original training samples, the enhanced training samples and the synthetic training samples are used for joint training, and the learning rate is set to 2.0e-05.

[0100] Table 2

[0101]

[0102] S500: Based on preset model parameters, fine-tune the first model and the second model multiple times using the original training samples.

[0103] In the embodiment of the present application, in order to ensure the generalization ability of the chemical formula structure recognition model obtained through training, the original training samples are used for fine-tuning on the basis of the first model and the second model, that is, two models with different initialization parameters are selected for fine-tuning, and data is randomly enhanced during the fine-tuning process. In order to further obtain a model with strong generalization ability, the input order of the training data during the training process is modified to ensure the randomness of the input of the training data, and then the second fine-tuning, third fine-tuning and multiple fine-tuning are performed on the model fine-tuned for the first time to obtain the first model and the second model after each fine-tuning.

[0104] In the embodiment of the present application, the preset model parameters include image size (size of the input image), max_length (maximum length of the label sequence), lr (learning rate), Epoch (training cycle / number of iterations) and Batch Size (batch size). Among them, Epoch refers to the process of forward propagation and back propagation of the entire data set through a neural network once. In an Epoch, each training sample has the opportunity to update the weight of the model, and Batch Size refers to the number of data samples used to train the model in each iteration (or each weight update).

[0105] In an embodiment of the present application, in order to further enhance the accuracy of the model prediction results and ensure the correctness of the test data recognition, the size of the input image and the maximum length of the label sequence can be expanded, and the learning rate, training cycle / iteration number and batch size can be adjusted, and then the original training samples can be used for fine-tuning on the first model and the second model.

[0106] Exemplarily, as shown in Table 3, Table 3 includes the first model and the second model after each fine-tuning. Figure 3 In the example, model_e and model_b are the first models after the first fine-tuning, and model_f and model_d are the second models after the first fine-tuning. model_c and model_single are the second models after the second fine-tuning obtained by performing the second fine-tuning based on the first model model_f after the first fine-tuning. model_a is the second model after the third fine-tuning obtained by performing the third fine-tuning based on the second model model_single after the second fine-tuning. Different preset model parameters can be used for each fine-tuning in Table 3.

[0107] Table 3

[0108]

[0109] In some embodiments, the step of fine-tuning the first model and the second model multiple times respectively using the original training samples based on the preset model parameters includes:

[0110] Based on different preset model parameters, the first model and the second model are respectively fine-tuned for the first time using the original training samples to obtain the first model after the first fine-tuning and the second model after the first fine-tuning;

[0111] M models whose evaluation performances are in the top N positions are selected from the first model after the first fine-tuning and the second model after the first fine-tuning, where M is less than N.

[0112] In the embodiment of the present application, most of the original training samples are used to fine-tune the first model and the second model, and a small part is used to determine the evaluation performance. Specifically, the images in the small part of the original training samples are input into the first model and the second model after the first fine-tuning, so that the model outputs the corresponding chemical formula code, and the evaluation performance is obtained according to the output chemical formula code and the chemical formula code in the small part of the original training samples.

[0113] In some embodiments, the evaluation performance is represented by two evaluation indicators, one is EM (exact match score) and the other is Structure_EM (structure exact match score).

[0114] Among them, EM indicates a complete match between the predicted string and the labeled string. Specifically, it is necessary to first convert the label and recognition string into a graph, and then determine if the graph form matches the graph matching tool. Assuming that T represents the total number of samples and R represents the number of matching prediction results, the calculation formula of EM is as follows:

[0115]

[0116] EM is used to evaluate whether the string of the output chemical formula code is the same as the string in the chemical formula code in a small part of the original training samples, and determine the number of identical original training samples, that is, the number of matching prediction results. The total number of samples is the total number of the small part of the original training samples.

[0117] Structure_EM is used to evaluate the model's ability to recognize molecular structures. For chemical formulas containing mixed molecular structures and regular formulas, when all molecular structure recognition results match the labeling graph, it means that the molecular structure of the chemical formula sample has been correctly recognized. Assuming that T represents the total number of samples and Rstruct represents the number of samples that have been correctly recognized, the calculation formula of Structure_EM is as follows:

[0118]

[0119] Structure EM is used to evaluate whether the molecular structure of the output chemical formula code is the same as the molecular structure of the chemical formula code in a small part of the original training samples, and to determine the number of identical original training samples, that is, the number of correctly identified samples. The total number of samples is the total number of the small part of the original training samples.

[0120] Based on different preset model parameters, the M models are fine-tuned using the original training samples to obtain a model after the second fine-tuning.

[0121] In this embodiment, a model with better evaluation performance may be selected for further fine-tuning to improve the recognition accuracy of the obtained chemical formula structure recognition model.

[0122] Selecting F models whose evaluation performance is in the top G from the models after the second fine-tuning, where F is smaller than G;

[0123] Based on different preset model parameters, the F models are fine-tuned for a third time using the original training samples to obtain a model after the third fine-tuning.

[0124] In this embodiment, three fine-tunings are generally performed to obtain a model after each fine-tuning. Subsequently, a chemical formula structure recognition model is determined for each fine-tuning model.

[0125] Exemplarily, as shown in Table 4, in Table 4, the M models with better evaluation performance selected from the models after the first fine-tuning refer to model_f. Then, model_f is fine-tuned to obtain the models after the second fine-tuning, including model_c and model_single. The F models with better evaluation performance selected from the models after the second fine-tuning refer to model_single, and the models after the third fine-tuning obtained by fine-tuning based on model_single include model_a.

[0126] Table 4

[0127]

[0128] S600: Determine a chemical formula structure recognition model through a model integration strategy based on the first model and the second model after each fine-tuning.

[0129] Among the single models in Table 4, model_single performs best in both tests. Because the distribution of the data identified by each model is not exactly the same, it is decided to adopt a model integration strategy to select the one with the highest accuracy from the output results of multiple models as the output result of the model.

[0130] A model integration strategy is adopted to combine multiple models with different parameters and obtain the recognition results by voting. This strategy can comprehensively utilize the advantages of each model, improve the overall recognition accuracy, and reduce the error rate.

[0131] In some embodiments, the step of determining the chemical formula structure recognition model by a model integration strategy based on the first model and the second model after each fine-tuning includes:

[0132] The first model and the second model after each fine-tuning are grouped to obtain multiple groups of model sets; wherein the model set includes at least one of the first model and the second model after each fine-tuning.

[0133] Exemplarily, as shown in Table 5, Table 5 includes 14 groups of model sets. Each horizontal row is a group of model sets.

[0134] Table 5

[0135]

[0136]

[0137] A preset number of data is randomly selected from the original training data as preset verification samples. Exemplarily, the preset number may be 1000.

[0138] Inputting the image in the preset verification sample into each model in the model set, so that each model outputs a chemical formula code corresponding to the image;

[0139] From the chemical formula codes corresponding to the images output by each model in the model set, the chemical formula codes to be compared that meet the preset rules are screened out.

[0140] In some embodiments, the arrangement order of the models in the model set is sorted according to the evaluation performance of the models; and the step of screening out the chemical formula codes to be compared that meet the preset rules from the chemical formula codes corresponding to the images output by each model in the model set comprises:

[0141] Filter out repeated chemical formula codes from the chemical formula codes corresponding to the images output by each model in the model set, and count the number of repeated occurrences;

[0142] Determine whether the number of chemical formula codes corresponding to the maximum number of repetitions is greater than 1;

[0143] If it is greater than 1, then determine the order of arrangement of the models in the model set used when generating the chemical formula code corresponding to the maximum number of repetitions; determine the chemical formula code corresponding to the model at the first position as the chemical formula code to be compared;

[0144] If it is 1, the chemical formula code corresponding to the greatest number of repetitions is determined as the chemical formula code to be compared.

[0145] In this embodiment, each model in a model set outputs a chemical formula code corresponding to an image. For example, images from 1,000 preset verification samples are input into 6 models in a model set. The 6 models include models AF that are sorted according to the evaluation performance of the models. The 6 models respectively output chemical formula codes corresponding to 1,000 images. The chemical formula code output by model A for the first image is w, the chemical formula code output by model B for the first image is Q, the chemical formula codes output by models C and D for the first image are both w, the chemical formula code output by model E for the first image is C, and the chemical formula code output by model F for the first image is C. At this time, the chemical formula codes that appear repeatedly are determined to be w and Q. The number of times the chemical formula code w appears repeatedly is counted as 2 times, and the number of times the chemical formula code Q appears repeatedly is counted as 2 times. It is determined whether the number of chemical formula codes corresponding to the most repeated times is greater than 1. Since the number of chemical formula codes corresponding to the most repeated times in this example is 2, namely w and Q, the number is greater than 1. At this time, the arrangement order of the models in the model set is used when determining the chemical formula code corresponding to the most repeated times, and the arrangement order is model A, model B, model C, and model D. The model located in the first position is model A. At this time, the chemical formula code corresponding to model A, namely chemical formula code w, is determined as the code to be compared.

[0146] The chemical formula code w is determined as the chemical formula code to be compared.

[0147] It can be understood that 1000 images are input into 6 models in the model set, and each image will determine the corresponding chemical formula code to be compared.

[0148] The evaluation performance to be compared is determined according to the chemical formula code to be compared and the preset chemical formula code corresponding to the image in the preset verification sample.

[0149] Exemplarily, images in 1000 preset verification samples are input into 6 models in the model set, and each image will determine the corresponding chemical formula code to be compared, that is, 1000 chemical formula codes to be compared. According to the 1000 chemical formula codes to be compared and the chemical formula codes corresponding to the images in the 1000 preset verification samples, the evaluation performance to be compared of the model set is determined. The specific calculation method of the evaluation performance to be compared is the same as the evaluation performance described above, which will not be repeated here.

[0150] From the evaluation performances to be compared corresponding to the multiple groups of model sets, the best evaluation performance to be compared is screened out, and the model set corresponding to the best performance to be evaluated is determined as the chemical formula recognition result model.

[0151] Exemplarily, the number of groups of model sets may be 14 groups. The evaluation performances to be compared corresponding to the 14 groups of model sets are compared, and the best evaluation performance to be compared is screened out. The model set corresponding to the best performance to be evaluated is determined as the chemical formula structure recognition model. Referring to Table 5 again, the first model set includes 7 models. Among the corresponding evaluation performances to be compared in the first model set, EM is 74.7 and Structure_EM is 84.90.

[0152] By comparison, it can be seen that the performance of the model set including multiple models is better than that of the model set including a single model. At the same time, it is found that if the top few models with the best performance of a single model are selected for voting, the results are not as good as the models obtained by selecting different basic models and fine-tuning different parameters. The sixth model set includes 5 models, specifically model_a to model_e. The performance of the model set to be compared is the best among the 14 model sets. At this time, the sixth model set is determined as the chemical formula recognition result model.

[0153] In one example, other preset validation samples are used to test the evaluation performance of the sixth model set. The test results are shown in Table 6. It can be seen that the sixth model set still has good evaluation performance.

[0154] Table 6

[0155]

[0156] The chemical formula structure model finally determined in the embodiments of the present application can not only identify handwritten chemical formulas but also print chemical formulas, while also improving the accuracy and generalization of the model.

[0157] In the above embodiment, a method, device, electronic device and storage medium for chemical formula structure recognition are provided. The method inputs an image into a chemical formula structure recognition model determined by pre-training, transfer learning and fine-tuning based on training data including synthetic training samples to output a chemical formula code corresponding to the image. The chemical formula structure recognition model can improve the accuracy of recognizing both handwritten and printed chemical formula structures. The method includes: obtaining training samples; pre-training a preset model using the synthetic training samples to obtain a basic model, wherein the preset model is a deep learning model; training a basic model using samples obtained by processing the original training samples and the enhanced training samples according to the original training samples to obtain a first model; training the basic model using the original training samples, the enhanced training samples and the synthetic training samples to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; fine-tuning the first model and the second model multiple times respectively using the original training samples based on the preset model parameters; determining a chemical formula structure recognition model through a model integration strategy according to the first model and the second model after each fine-tuning.

[0158] Further, as Figure 1 The specific implementation of the method, such as Figure 4 As shown, the embodiment of the present application provides a device for chemical formula structure recognition, comprising:

[0159] A chemical formula code determination unit 401 is used to input the image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image;

[0160] The process of determining the chemical formula structure model includes: obtaining training samples, the training samples include original training samples, synthetic training samples and enhanced training samples, the original training samples include chemical formula handwritten images and corresponding chemical formula codes and chemical formula printed images and corresponding chemical formula codes; the synthetic training samples are synthesized according to the chemical formula codes in the original training samples, the enhanced training samples are samples processed according to the original training samples, and samples processed according to the synthetic training samples; using the synthetic training samples to pre-train a preset model to obtain a basic model, the preset model is a deep learning model; using the original training samples and the samples obtained by processing the original training samples in the enhanced training samples to train the basic model to obtain a first model; using the original training samples, the enhanced training samples and the synthetic training samples to train the basic model to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; based on the preset model parameters, using the original training samples to fine-tune the first model and the second model for multiple times respectively; according to the first model and the second model after each fine-tuning, the chemical formula structure recognition model is determined through a model integration strategy.

[0161] It should be noted that for other corresponding descriptions of the functional units involved in the chemical formula structure recognition device provided in this embodiment, please refer to Figure 1 The corresponding description in will not be repeated here.

[0162] Based on the above Figure 1 The method shown in the embodiment of the present application accordingly provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned Figure 1 The method for structure recognition of the chemical formula shown.

[0163] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0164] Based on the above Figure 1 The method shown, and Figure 4In order to achieve the above-mentioned purpose, the embodiment of the present application also provides a physical device based on chemical formula structure recognition, which can be a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 The method of chemical formula-based structure recognition is shown.

[0165] Optionally, the physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0166] Those skilled in the art will appreciate that the electronic device structure provided in this embodiment does not limit the electronic device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0167] The storage medium may also include an operating device and a network communication module. The operating device is a program that manages and saves the hardware and software resources of the electronic device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the various controls inside the storage medium, and the communication with other hardware and software in the physical device.

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0169] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the units or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the units in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The units of the above-mentioned implementation scenarios can be combined into one unit, or can be further split into multiple sub-units.

[0170] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A method for chemical formula structure recognition, characterized in that: include: Inputting the image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image; The process of determining the chemical formula structure model includes: obtaining training samples, the training samples include original training samples, synthetic training samples and enhanced training samples, the original training samples include chemical formula handwritten images and corresponding chemical formula codes and chemical formula printed images and corresponding chemical formula codes; the synthetic training samples are synthesized according to the chemical formula codes in the original training samples, the enhanced training samples are samples processed according to the original training samples, and samples processed according to the synthetic training samples; using the synthetic training samples to pre-train a preset model to obtain a basic model, the preset model is a deep learning model; using the original training samples and the samples obtained by processing the original training samples in the enhanced training samples to train the basic model to obtain a first model; using the original training samples, the enhanced training samples and the synthetic training samples to train the basic model to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; based on the preset model parameters, using the original training samples to fine-tune the first model and the second model for multiple times respectively; according to the first model and the second model after each fine-tuning, the chemical formula structure recognition model is determined through a model integration strategy.

2. The method according to claim 1, characterized in that: The process of determining the chemical formula structure model also includes: Screening out original training samples whose chemical formula codes are target chemical formula codes from the original training data; synthesizing a chemical formula synthetic image using the target chemical formula code; Obtaining original training samples whose chemical formula codes are non-target chemical formula codes; Converting the non-target chemical formula code into a target chemical formula code, and synthesizing a chemical formula synthetic image using the converted target chemical formula code; The chemical formula synthetic image and the target chemical formula code corresponding to the chemical formula synthetic image are determined as synthetic training samples.

3. The method according to claim 1, characterized in that The process of determining the chemical formula structure model also includes: Processing the original training samples using data enhancement technology to generate samples among the trained samples that are processed according to the original training samples; The synthetic training sample is processed using a data enhancement technique to generate a sample in the trained sample that is processed according to the synthetic training sample.

4. The method according to claim 1, characterized in that: The preset model is the Nougat model.

5. The method according to claim 1, characterized in that The step of fine-tuning the first model and the second model multiple times respectively using the original training samples based on the preset model parameters comprises: Based on different preset model parameters, the first model and the second model are respectively fine-tuned for the first time using the original training samples to obtain the first model after the first fine-tuning and the second model after the first fine-tuning; Select M models whose evaluation performance ranks in the top N from the first model after the first fine-tuning and the second model after the first fine-tuning, where M is less than N; Based on different preset model parameters, the M models are fine-tuned using the original training samples to obtain a model after the second fine-tuning; Selecting F models whose evaluation performance is in the top G from the models after the second fine-tuning, where F is smaller than G; Based on different preset model parameters, the F models are fine-tuned for a third time using the original training samples to obtain a model after the third fine-tuning.

6. The method according to claim 1, characterized in that The step of determining the chemical formula structure recognition model by using a model integration strategy based on the first model and the second model after each fine-tuning includes: Grouping the first model and the second model after each fine-tuning to obtain multiple groups of model sets; wherein the model set includes at least one of the first model and the second model after each fine-tuning; Randomly select a preset number of data from the original training data as preset verification samples; Inputting the image in the preset verification sample into each model in the model set, so that each model outputs a chemical formula code corresponding to the image; Screening out chemical formula codes to be compared that meet preset rules from chemical formula codes corresponding to the images output by each model in the model set; Determining the evaluation performance to be compared according to the chemical formula code to be compared and the preset chemical formula code corresponding to the image in the preset verification sample; From the evaluation performances to be compared corresponding to the multiple groups of model sets, the best evaluation performance to be compared is screened out, and the model set corresponding to the best performance to be evaluated is determined as the chemical formula recognition result model.

7. The method according to claim 6, characterized in that The order of the models in the model set is sorted according to the evaluation performance of the models; The step of screening out chemical formula codes to be compared that meet preset rules from the chemical formula codes corresponding to the images output by each model in the model set comprises: Filter out repeated chemical formula codes from the chemical formula codes corresponding to the images output by each model in the model set, and count the number of repeated occurrences; Determine the number of chemical formula codes corresponding to the maximum number of repetitions; If it is greater than 1, then determine the order of arrangement of the models in the model set used when generating the chemical formula code corresponding to the maximum number of repetitions; determine the chemical formula code corresponding to the model at the first position as the chemical formula code to be compared; If it is 1, the chemical formula code corresponding to the greatest number of repetitions is determined as the chemical formula code to be compared.

8. A device for chemical formula structure recognition, characterized in that: include: determining a chemical formula code unit for inputting the image into a chemical formula structure recognition model to output a chemical formula code corresponding to the image; The process of determining the chemical formula structure model includes: obtaining training samples, the training samples include original training samples, synthetic training samples and enhanced training samples, the original training samples include chemical formula handwritten images and corresponding chemical formula codes and chemical formula printed images and corresponding chemical formula codes; the synthetic training samples are synthesized according to the chemical formula codes in the original training samples, the enhanced training samples are samples processed according to the original training samples, and samples processed according to the synthetic training samples; using the synthetic training samples to pre-train a preset model to obtain a basic model, the preset model is a deep learning model; using the original training samples and the samples obtained by processing the original training samples in the enhanced training samples to train the basic model to obtain a first model; using the original training samples, the enhanced training samples and the synthetic training samples to train the basic model to obtain a second model, wherein the learning rate corresponding to the second model is different from the learning rate corresponding to the first model; based on the preset model parameters, using the original training samples to fine-tune the first model and the second model for multiple times respectively; according to the first model and the second model after each fine-tuning, the chemical formula structure recognition model is determined through a model integration strategy.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for chemical formula structure recognition according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for chemical formula structure recognition according to any one of claims 1 to 7 are implemented.

Images