Formula identification model training and formula identification method and device

Abstract

The invention provides a formula recognition model training method and device, a formula recognition method and device, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence such as natural language processing, deep learning, large language models and cloud services. The formula recognition model training method comprises the following steps: constructing a first training sample set; training the source formula recognition model according to a first training sample randomly extracted from the first training sample set to obtain an initial formula recognition model; determining difficult evaluation samples in the evaluation sample set by using an initial formula identification model, and constructing a combined training sample set according to the difficult evaluation samples and the first training sample set; and training the initial formula recognition model by using the combined training sample set to obtain a target formula recognition model. The formula identification method comprises the steps of obtaining a to-be-processed image; and inputting the to-be-processed image into the target formula recognition model, and obtaining a formula recognition result according to an output result of the target formula recognition model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as natural language processing, deep learning, large language models, cloud services, etc. A method, device, electronic device, and readable storage medium for training and recognizing a formula recognition model are provided. Background Art

[0002] With the rapid development of science and technology, formula recognition technology is increasingly being used in various fields. The existing technology usually trains a model specifically for formula recognition to perform formula recognition. Therefore, how to improve the recognition performance of the formula recognition model is a technical problem that needs to be solved urgently. Summary of the invention

[0003] According to a first aspect of the present disclosure, a method for training a formula recognition model is provided, including: constructing a first training sample set, the first training sample set including a plurality of first training samples; training a source formula recognition model according to first training samples randomly selected from the first training sample set to obtain an initial formula recognition model; using the initial formula recognition model to determine difficult evaluation samples in an evaluation sample set, and constructing a merged training sample set according to the difficult evaluation samples and the first training sample set; and using the merged training sample set to train the initial formula recognition model to obtain a target formula recognition model.

[0004] According to a second aspect of the present disclosure, a formula recognition method is provided, comprising: acquiring an image to be processed; inputting the image to be processed into a target formula recognition model, and obtaining a formula recognition result corresponding to the image to be processed based on an output result of the target formula recognition model.

[0005] According to a third aspect of the present disclosure, a training device for a formula recognition model is provided, comprising: a first construction unit, used to construct a first training sample set, wherein the first training sample set includes multiple first training samples; a first training unit, used to train a source formula recognition model according to first training samples randomly selected from the first training sample set, to obtain an initial formula recognition model; a second construction unit, used to use the initial formula recognition model to determine difficult evaluation samples in an evaluation sample set, and to construct a merged training sample set according to the difficult evaluation samples and the first training sample set; and a second training unit, used to train the initial formula recognition model using the merged training sample set, to obtain a target formula recognition model.

[0006] According to a fourth aspect of the present disclosure, a formula recognition device is provided, comprising: an acquisition unit, used to acquire an image to be processed; and a recognition unit, used to input the image to be processed into a target formula recognition model, and obtain a formula recognition result corresponding to the image to be processed according to an output result of the target formula recognition model.

[0007] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0008] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0009] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0016] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0017] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0018] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0019] Figure 8 It is a block diagram of an electronic device used to implement the training of the formula recognition model or the formula recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0021] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the training method of the formula recognition model of this embodiment specifically includes the following steps:

[0022] S101, constructing a first training sample set, wherein the first training sample set includes a plurality of first training samples;

[0023] S102, training a source formula recognition model according to a first training sample randomly selected from the first training sample set to obtain an initial formula recognition model;

[0024] S103, using the initial formula recognition model to determine difficult evaluation samples in the evaluation sample set, and constructing a combined training sample set based on the difficult evaluation samples and the first training sample set;

[0025] S104: Use the combined training sample set to train the initial formula recognition model to obtain a target formula recognition model.

[0026] The training method of the formula recognition model of the present embodiment obtains the target formula recognition model by first using the first training sample set to obtain the initial formula recognition model, and then training the initial formula recognition model in combination with the first training sample set and the difficulty assessment samples determined by the initial formula recognition model, thereby achieving the purpose of more effectively utilizing the training samples in the first training sample set to train the target formula recognition model, which can improve the training efficiency of the target formula recognition model and enhance the recognition performance of the target formula recognition model.

[0027] In this embodiment, the target formula recognition model can output a LaTeX character (token) sequence corresponding to the formula in the image according to the input image; wherein the LaTeX character sequence is a character sequence in LaTeX format, and LaTeX is a typesetting system based on TEX.

[0028] In the first training sample set constructed by executing S101 in this embodiment, each first training sample includes a first formula image and its corresponding first formula annotation result; wherein the first formula annotation result corresponding to the first formula image is a real LaTeX character sequence of the formula included in the first formula image.

[0029] After executing S101 to construct the first training sample set, this embodiment executes S102 to train the source formula recognition model according to the first training sample randomly extracted from the first training sample set to obtain the initial formula recognition model.

[0030] In this embodiment, the source formula recognition model can be an existing formula recognition model in the prior art, or it can be a model after replacing the backbone network in the existing formula recognition model (for example, using a backbone network with stronger performance), or it can be a model after replacing the decoder in the existing formula recognition model (for example, using a decoder with stronger performance), etc.

[0031] When executing S102, this embodiment selects at least one first training sample from the first training sample set by random selection, and then uses the selected at least one first training sample to train the source formula recognition model, thereby adjusting the parameters of the source formula recognition model during the training process to obtain an initial formula recognition model.

[0032] The initial formula recognition model obtained by executing S102 in this embodiment is a model that can be applied to perform formula recognition on the first sample formula image in the first training sample set.

[0033] Specifically, when executing S102, the present embodiment trains the source formula recognition model according to the first training sample randomly selected from the first training sample set to obtain the initial formula recognition model. The implementation method that can be adopted is: inputting the first formula image in the randomly selected first training sample into the source formula recognition model to obtain the first formula prediction result corresponding to the first formula image output by the source formula recognition model. The first formula prediction result in the present embodiment is the predicted LaTeX character sequence obtained by the model performing formula recognition on the first formula image; calculating the loss function value according to the first formula prediction result corresponding to the first formula image and the first formula annotation result; adjusting the parameters of the source formula recognition model according to the calculated loss function value to obtain the initial formula recognition model.

[0034] After executing S102 to obtain the initial formula recognition model, this embodiment executes S103 to use the initial formula recognition model to determine difficult evaluation samples in the evaluation sample set, and constructs a merged training sample set based on the difficult evaluation samples and the first training sample set.

[0035] In this embodiment, the evaluation sample set includes multiple evaluation samples, and each evaluation sample includes a third formula image and its corresponding third formula annotation result.

[0036] In this embodiment, different evaluation samples in the preset evaluation sample set may correspond to different formula scenarios, such as formulas in papers, handwritten formulas, formulas in books, formulas in test papers, etc.

[0037] In this embodiment, the third formula annotation result corresponding to the third formula image is a real LaTeX character sequence of the formula included in the evaluation formula image.

[0038] In this embodiment, the difficult evaluation samples are evaluation samples that are difficult to accurately predict when the initial formula recognition model performs formula recognition.

[0039] When executing S103 and using the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set, the present embodiment may adopt the following implementation method: inputting the third formula image in the evaluation sample into the initial formula recognition model, and obtaining the third formula prediction result of the initial formula recognition model outputting the third formula image corresponding to the third formula image; and determining whether the evaluation sample is a difficult evaluation sample based on the third formula prediction result corresponding to the third formula image and the third formula annotation result.

[0040] When executing S103, this embodiment can calculate the similarity between the prediction result of the third formula and the annotation result of the third formula (that is, calculate the similarity between LaTeX character sequences), and then when it is determined that the calculated similarity is less than or equal to the preset similarity threshold, the current evaluation sample is determined as a difficult evaluation sample.

[0041] In addition, when executing S103, this embodiment can also determine whether the prediction result of the third formula is correct or wrong according to the marking result of the third formula, and then determine the current evaluation sample as a difficult evaluation sample when it is determined that the prediction result of the third formula is wrong.

[0042] After executing S103 to complete the determination of the difficult assessment samples, this embodiment can construct a merged training sample set based on the determined difficult assessment samples and the first training sample set; wherein the merged training sample set constructed in this embodiment includes at least one first training sample.

[0043] When executing S103 to construct a merged training sample set based on the difficulty assessment sample and the first training sample set, the present embodiment may adopt the following implementation methods: obtaining difficulty sample features based on the difficulty assessment sample; constructing a first training sample subset based on the first training sample corresponding to the difficulty sample features selected from the first training sample set; and constructing a merged training sample set based on the first training sample subset and the first training sample randomly selected from the first training sample set.

[0044] That is to say, the merged training sample set constructed in this embodiment includes, on the one hand, a first training sample subset constructed based on the difficulty assessment samples, and on the other hand, also includes a first training sample randomly selected from the first training sample set, thereby improving the type richness of the first training samples in the merged training sample set, that is, the first training sample in the merged training sample set reflects both the randomness of the sample and the specificity of the sample, which can improve the training effect when using the merged training sample set to train the initial formula recognition model, and effectively improve the recognition ability of the initial formula recognition model for difficult samples.

[0045] When executing S103 to obtain difficult sample features based on difficult evaluation samples, this embodiment can perform feature extraction on the evaluation formula image in the difficult evaluation sample, and use the extracted image feature vector as the difficult sample feature; it can also use the formula type of the formula included in the evaluation formula image in the difficult evaluation sample as the difficult sample feature; it can also perform feature extraction on the second formula annotation result in the difficult evaluation sample, and use the extracted sequence feature vector as the difficult sample feature.

[0046] When executing S103 to construct the first training sample subset, this embodiment can first obtain the first sample feature of the first training sample, then match the first sample feature with the difficult sample feature, and finally select the first training sample corresponding to the first sample feature that is similar or identical to the difficult sample feature to construct the first training sample subset.

[0047] That is to say, this embodiment mines specific first training samples from the first training sample set according to the difficult sample characteristics of the corresponding difficult evaluation samples, and then performs intensive training on the initial formula recognition model in combination with the mined specific training samples, so that the trained model can accurately recognize formulas for difficult evaluation samples.

[0048] After executing S103 to construct the combined training sample set, this embodiment executes S104 to train the initial formula recognition model using the combined training sample set to obtain the target formula recognition model.

[0049] In this embodiment, when executing S104 to train the initial formula recognition model using the merged training sample set to obtain the target formula recognition model, the implementation method that can be adopted is: inputting the first formula image in the merged training sample set into the initial formula recognition model to obtain the third formula prediction result corresponding to the first formula image output by the initial formula recognition model; calculating the loss function value according to the third formula prediction result corresponding to the first formula image and the first formula annotation result; adjusting the parameters of the initial formula recognition model according to the calculated loss function value to obtain the target formula recognition model.

[0050] In this embodiment, after executing S104 to obtain the target formula recognition model, the image to be processed can be input into the target formula recognition model to obtain a formula recognition result output by the target formula recognition model, which is specifically a character sequence in LaTeX format.

[0051] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown in , in this embodiment, when executing S101 "constructing a first training sample set", the implementation method that can be adopted is:

[0052] S201, extracting a LaTeX character sequence corresponding to a formula from a LaTeX document;

[0053] S202, using a rendering engine to render the LaTeX character sequence to obtain a first rendered image corresponding to the LaTeX character sequence;

[0054] S203, using the first rendered image as a first formula image, and using the LaTeX character sequence as a first formula annotation result corresponding to the first formula image;

[0055] S204: Use the first formula image and the first formula annotation result corresponding thereto as a first training sample to construct the first training sample set including a plurality of first training samples.

[0056] That is, this embodiment constructs the first training sample set according to the obtained LaTeX document, so that there is no need to manually mark formulas, which can reduce the construction cost of the first training sample set and improve the construction efficiency of the first training sample set.

[0057] In this embodiment, the LaTeX document is a document created using the LaTeX typesetting system.

[0058] When executing S201, this embodiment may first obtain multiple LaTeX documents from an open source database, and then extract LaTeX character sequences corresponding to formulas from the obtained multiple LaTeX documents by using a regular expression or the like.

[0059] When executing S202 , this embodiment may use an existing rendering engine (such as MathJax, KaTeX, etc.) to render the extracted LaTeX character sequence, thereby obtaining a first rendering image corresponding to different LaTeX character sequences.

[0060] When executing S203, this embodiment establishes a correspondence between different LaTeX character sequences and different first rendered images, and then uses the LaTeX character sequence and the first rendered image rendered according to the LaTeX character sequence as a first formula image in a first training sample and its corresponding first formula annotation result.

[0061] When executing S204, the present embodiment constructs a first training sample set according to the obtained multiple first training samples, and then the first training sample set can be used to train and obtain a target formula recognition model.

[0062] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 As shown in , when executing S104 “using the combined training sample set to train the initial formula recognition model to obtain a target formula recognition model”, the implementation method that can be adopted in this embodiment is:

[0063] S301, using the combined training sample set to train the initial formula recognition model to obtain a candidate formula recognition model;

[0064] S302, using the evaluation sample set to evaluate the candidate formula recognition model to obtain a model evaluation index;

[0065] S303, in response to determining that the model evaluation index does not meet the preset requirement, after replacing the initial formula recognition model with the candidate formula recognition model, proceeding to the step of using the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set, and repeating this process until determining that the model evaluation index meets the first preset requirement;

[0066] S304. Obtain the target formula recognition model according to the candidate formula recognition model when the model evaluation index meets the first preset requirement.

[0067] That is to say, this embodiment determines whether to obtain the target formula recognition model based on the current candidate formula recognition model according to the model evaluation index obtained by evaluating the candidate formula recognition model using the evaluation sample set, thereby achieving the purpose of obtaining the target formula recognition model by training the model multiple times using the first training sample set, and can improve the training effect of the target formula recognition model, so that the target formula recognition model has higher accuracy when performing formula recognition.

[0068] The model evaluation index obtained by executing S302 in this embodiment can be the recognition accuracy of the candidate formula recognition model, that is, the division result between (the number of evaluation samples accurately recognized by the candidate formula recognition model) and (the total number of evaluation samples) is used as the recognition accuracy.

[0069] When executing S303, this embodiment can determine that the model evaluation index does not meet the first preset requirement when it is determined that the recognition accuracy is less than the preset accuracy threshold, and then proceed to execute the step of determining difficult evaluation samples to retrain the initial formula recognition model; if it is determined that the recognition accuracy is greater than or equal to the preset accuracy threshold, it is determined that the model evaluation index meets the first preset requirement, and then obtain the target formula recognition model based on the current candidate formula recognition model.

[0070] When executing S304, this embodiment can directly use the candidate formula recognition model when the model evaluation index meets the preset requirements as the target formula recognition model.

[0071] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown in , when executing S304 "obtaining the target formula recognition model according to the candidate formula recognition model when the model evaluation index meets the first preset requirement", the implementation method that can be adopted is:

[0072] S401, obtaining a second training sample set, where the second training sample set includes a plurality of second formula images;

[0073] S402, constructing a second training sample according to the second formula image;

[0074] S403: Use the second training sample to train the candidate formula recognition model when the model evaluation index meets the first preset requirement to obtain the target formula recognition model.

[0075] That is to say, after using the first training sample set to train the initial formula recognition model multiple times, this embodiment uses the second training sample constructed according to the second training sample set to continue training the candidate formula recognition model when the model evaluation index meets the first preset requirement, thereby obtaining the target formula recognition model. This embodiment can further expand the number of training samples used in training the target formula recognition model, and can improve the training effect of the target formula recognition model and enhance the recognition performance of the target formula recognition model.

[0076] Specifically, when executing S402, the present embodiment constructs a second training sample according to the second formula image, the implementation method that can be adopted is: using the candidate formula recognition model when the model evaluation index meets the first preset requirement, to obtain the second formula prediction result corresponding to the second formula image; using the rendering engine to render the second formula prediction result to obtain a second rendered image corresponding to the second formula prediction result; in response to determining that the similarity between the second rendered image and the second formula image meets the second preset requirement, constructing a second training sample according to the second formula image and its corresponding second formula prediction result.

[0077] That is to say, this embodiment constructs the second training sample according to the second rendered image rendered by the prediction result of the second formula and the prediction result of the second formula, which can improve the accuracy of the constructed second training sample.

[0078] The second training sample constructed by executing S402 in this embodiment includes the second formula image and its corresponding formula annotation result (the formula annotation result is the second formula prediction result).

[0079] When executing S402 , this embodiment may use an existing rendering engine (such as MathJax, KaTeX, etc.) to render the second formula prediction result, thereby obtaining a second rendered image corresponding to the second formula prediction result.

[0080] When executing S402 to determine whether the similarity between the second rendered image and the second formula image meets the second preset requirement, this embodiment can first obtain the image feature vector of the second rendered image and the image feature vector of the second formula image, and then calculate the similarity (for example, cosine similarity) between the two images based on the obtained image feature vectors. Finally, if it is determined that the calculated similarity is greater than or equal to a similarity threshold, it is determined that the similarity between the second rendered image and the second formula image meets the second preset requirement; otherwise, it is determined that the second preset requirement is not met.

[0081] It can be understood that this embodiment can use a pre-trained formula feature model to obtain the image feature vector of the image; the formula feature model in this embodiment can be pre-trained using a formula image set that is a mixture of a first formula image in a first training sample set and a second formula image in a second training sample set.

[0082] It can be understood that the number of the second training samples constructed by executing S402 in this embodiment may be one or more.

[0083] In this embodiment, when executing S403, the second training sample is used to train the candidate formula recognition model when the model evaluation index meets the first preset requirement, and when the target formula recognition model is obtained, the implementation method that can be adopted is: input the second formula image in the second training sample into the candidate formula recognition model when the model evaluation index meets the first preset requirement, and obtain the fourth formula prediction result corresponding to the second formula image output by the candidate formula recognition model; calculate the loss function value according to the fourth formula prediction result corresponding to the second formula image and the second formula prediction result; adjust the parameters of the candidate formula recognition model when the model evaluation index meets the first preset requirement according to the calculated loss function value, and obtain the target formula recognition model.

[0084] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 As shown in , the formula recognition method of this embodiment specifically includes the following steps:

[0085] S501, obtaining an image to be recognized;

[0086] S502: Input the image to be recognized into a target formula recognition model, and obtain a formula recognition result corresponding to the image to be recognized according to an output result of the target formula recognition model.

[0087] That is to say, this embodiment uses the pre-trained target formula recognition model to recognize the formula in the image to be recognized to obtain the formula recognition result, which can improve the accuracy of formula recognition.

[0088] The formula recognition result obtained by executing S502 in this embodiment is a character sequence in LaTeX format.

[0089] After executing S502 to obtain the formula recognition result corresponding to the image to be recognized, this embodiment can perform processing such as formula editing, formula input, and formula semantic understanding based on the formula recognition result.

[0090] For example, in this embodiment, the formula recognition result can be input into a LaTeX document to complete the formula input; the characters in the formula recognition result can also be replaced, deleted, etc. to complete the formula editing.

[0091] After executing S502 to obtain the formula recognition result, this embodiment may also include the following contents: using a large language model (LLM) to obtain a semantic understanding result of the formula in the image to be processed, such as the formula type, the formula meaning, etc., based on the obtained formula recognition result.

[0092] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure. Figure 6 As shown, the training device 600 of the formula recognition model of this embodiment includes:

[0093] A first constructing unit 601 is used to construct a first training sample set, where the first training sample set includes a plurality of first training samples;

[0094] A first training unit 602 is used to train a source formula recognition model according to a first training sample randomly selected from the first training sample set to obtain an initial formula recognition model;

[0095] A second construction unit 603 is configured to use the initial formula recognition model to determine difficult evaluation samples in the evaluation sample set, and to construct a combined training sample set based on the difficult evaluation samples and the first training sample set;

[0096] The second training unit 604 is used to train the initial formula recognition model using the combined training sample set to obtain a target formula recognition model.

[0097] In this embodiment, the target formula recognition model can output a LaTeX character (token) sequence corresponding to the formula in the image according to the input image; wherein the LaTeX character sequence is a character sequence in LaTeX format, and LaTeX is a typesetting system based on TEX.

[0098] In the first training sample set constructed by the first construction unit 601, each first training sample includes a first formula image and its corresponding first formula annotation result; wherein the first formula annotation result corresponding to the first formula image is a real LaTeX character sequence of the formula included in the first formula image.

[0099] When constructing the first training sample set, the first construction unit 601 can be implemented in the following manner: extracting a LaTeX character sequence corresponding to a formula from a LaTeX document; rendering the LaTeX character sequence using a rendering engine to obtain a first rendered image corresponding to the LaTeX character sequence; using the first rendered image as a first formula image, and using the LaTeX character sequence as a first formula annotation result corresponding to the first formula image; using the first formula image and its corresponding first formula annotation result as a first training sample to construct a first training sample set including multiple first training samples.

[0100] That is, the first construction unit 601 constructs the first training sample set according to the obtained LaTeX document, thereby eliminating the need for manual formula annotation, thereby reducing the construction cost of the first training sample set and improving the construction efficiency of the first training sample set.

[0101] In this embodiment, after the first construction unit 601 constructs the first training sample set, the first training unit 602 trains the source formula recognition model according to the first training sample randomly selected from the first training sample set to obtain the initial formula recognition model.

[0102] The first training unit 602 selects at least one first training sample from the first training sample set by random selection, and then uses the selected at least one first training sample to train the source formula recognition model, thereby adjusting the parameters of the source formula recognition model during the training process to obtain an initial formula recognition model.

[0103] The initial formula recognition model obtained by the first training unit 602 is a model that can be applied to perform formula recognition on the first sample formula image in the first training sample set.

[0104] Specifically, when the first training unit 602 trains the source formula recognition model according to the first training sample randomly selected from the first training sample set to obtain the initial formula recognition model, the implementation method that can be adopted is: inputting the first formula image in the randomly selected first training sample into the source formula recognition model to obtain the first formula prediction result corresponding to the first formula image output by the source formula recognition model; calculating the loss function value according to the first formula prediction result corresponding to the first formula image and the first formula labeling result; adjusting the parameters of the source formula recognition model according to the calculated loss function value to obtain the initial formula recognition model.

[0105] In this embodiment, after the first training unit 602 obtains the initial formula recognition model, the second construction unit 603 uses the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set, and constructs a combined training sample set based on the difficult evaluation samples and the first training sample set.

[0106] When the second construction unit 603 uses the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set, the implementation method that can be adopted is: input the third formula image in the evaluation sample into the initial formula recognition model, and obtain the third formula prediction result of the initial formula recognition model output corresponding to the third formula image; determine whether the evaluation sample is a difficult evaluation sample based on the third formula prediction result corresponding to the third formula image and the third formula annotation result.

[0107] The second construction unit 603 can calculate the similarity between the third formula prediction result and the third formula annotation result (i.e., calculate the similarity between LaTeX character sequences), and then when it is determined that the calculated similarity is less than or equal to a preset similarity threshold, determine the current evaluation sample as a difficult evaluation sample.

[0108] In addition, the second construction unit 603 can also determine whether the prediction result of the third formula is correct or wrong according to the annotation result of the third formula, and then determine the current evaluation sample as a difficult evaluation sample when it is determined that the prediction result of the third formula is wrong.

[0109] After determining the difficulty assessment samples, the second construction unit 603 may construct a combined training sample set according to the determined difficulty assessment samples and the first training sample set; wherein the combined training sample set constructed in this embodiment includes at least one first training sample.

[0110] When the second construction unit 603 constructs a merged training sample set based on the difficulty assessment sample and the first training sample set, the implementation method that can be adopted is: obtaining difficulty sample features based on the difficulty assessment sample; constructing a first training sample subset based on the first training sample corresponding to the difficulty sample features selected from the first training sample set; constructing a merged training sample set based on the first training sample subset and the first training sample randomly selected from the first training sample set.

[0111] That is to say, the merged training sample set constructed by the second construction unit 603 includes, on the one hand, a first training sample subset constructed based on the difficulty assessment sample, and on the other hand, also includes a first training sample randomly selected from the first training sample set, thereby improving the type richness of the first training samples in the merged training sample set, that is, the first training sample in the merged training sample set reflects both the randomness of the sample and the specificity of the sample, which can improve the training effect when the initial formula recognition model is trained using the merged training sample set, and effectively improve the recognition ability of the initial formula recognition model for difficult samples.

[0112] When obtaining difficult sample features based on difficult evaluation samples, the second construction unit 603 can perform feature extraction on the evaluation formula image in the difficult evaluation sample, and use the extracted image feature vector as the difficult sample feature; it can also use the formula type of the formula included in the evaluation formula image in the difficult evaluation sample as the difficult sample feature; it can also perform feature extraction on the second formula annotation result in the difficult evaluation sample, and use the extracted sequence feature vector as the difficult sample feature.

[0113] When constructing the first training sample subset, the second construction unit 603 can first obtain the first sample feature of the first training sample, then match the first sample feature with the difficult sample feature, and finally select the first training sample corresponding to the first sample feature that is similar or identical to the difficult sample feature to construct the first training sample subset.

[0114] That is to say, the second construction unit 603 mines specific first training samples from the first training sample set according to the difficult sample characteristics of the corresponding difficult evaluation samples, and then performs intensive training on the initial formula recognition model in combination with the mined specific training samples, so that the trained model can accurately recognize formulas for difficult evaluation samples.

[0115] In this embodiment, after the second construction unit 603 constructs the combined training sample set, the second training unit 604 uses the combined training sample set to train the initial formula recognition model to obtain the target formula recognition model.

[0116] When the second training unit 604 uses the merged training sample set to train the initial formula recognition model to obtain the target formula recognition model, the implementation method that can be adopted is: input the first formula image in the merged training sample set into the initial formula recognition model to obtain the third formula prediction result corresponding to the first formula image output by the initial formula recognition model; calculate the loss function value according to the third formula prediction result corresponding to the first formula image and the first formula annotation result; adjust the parameters of the initial formula recognition model according to the calculated loss function value to obtain the target formula recognition model.

[0117] After obtaining the target formula recognition model, the second training unit 604 can input the image to be processed into the target formula recognition model to obtain a formula recognition result output by the target formula recognition model, which is specifically a character sequence in LaTeX format.

[0118] When the second training unit 604 uses the merged training sample set to train the initial formula recognition model to obtain the target formula recognition model, the implementation method that can be adopted is: use the merged training sample set to train the initial formula recognition model to obtain a candidate formula recognition model; use the evaluation sample set to evaluate the candidate formula recognition model to obtain a model evaluation index; in response to determining that the model evaluation index does not meet the preset requirements, replace the initial formula recognition model with the candidate formula recognition model, and then proceed to execute the step of using the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set, and repeat this process until it is determined that the model evaluation index meets the first preset requirement; obtain the target formula recognition model according to the candidate formula recognition model when the model evaluation index meets the first preset requirement.

[0119] That is to say, the second training unit 604 determines whether to obtain the target formula recognition model based on the current candidate formula recognition model according to the model evaluation index obtained by evaluating the candidate formula recognition model using the evaluation sample set, thereby achieving the purpose of obtaining the target formula recognition model by training the model multiple times using the first training sample set, which can improve the training effect of the target formula recognition model and make the target formula recognition model have higher accuracy when performing formula recognition.

[0120] The second training unit 604 can determine that the model evaluation index does not meet the first preset requirement when it is determined that the recognition accuracy is less than the preset accuracy threshold, and then proceed to execute the step of determining the difficult evaluation sample to retrain the initial formula recognition model; if it is determined that the recognition accuracy is greater than or equal to the preset accuracy threshold, it is determined that the model evaluation index meets the first preset requirement, and then obtain the target formula recognition model based on the current candidate formula recognition model.

[0121] The second training unit 604 may directly use the candidate formula recognition model when the model evaluation index meets the preset requirements as the target formula recognition model.

[0122] When the second training unit 604 obtains the target formula recognition model based on the candidate formula recognition model when the model evaluation index meets the first preset requirement, the implementation method that can be adopted is: obtaining a second training sample set, the second training sample set including multiple second formula images; constructing a second training sample based on the second formula image; using the second training sample to train the candidate formula recognition model when the model evaluation index meets the first preset requirement, to obtain the target formula recognition model.

[0123] That is to say, after using the first training sample set to train the initial formula recognition model for multiple times, the second training unit 604 uses the second training samples constructed according to the second training sample set to continue training the candidate formula recognition model when the model evaluation index meets the first preset requirement, thereby obtaining the target formula recognition model. This embodiment can further expand the number of training samples used in training the target formula recognition model, and can improve the training effect of the target formula recognition model and enhance the recognition performance of the target formula recognition model.

[0124] Specifically, when the second training unit 604 constructs a second training sample based on the second formula image, the implementation method that can be adopted is: using the candidate formula recognition model when the model evaluation index meets the first preset requirement, to obtain the second formula prediction result corresponding to the second formula image; using a rendering engine to render the second formula prediction result to obtain a second rendered image corresponding to the second formula prediction result; in response to determining that the similarity between the second rendered image and the second formula image meets the second preset requirement, constructing a second training sample according to the second formula image and its corresponding second formula prediction result.

[0125] That is, the second training unit 604 constructs the second training sample according to the second rendered image rendered by the prediction result of the second formula and the prediction result of the second formula, which can improve the accuracy of the constructed second training sample.

[0126] When determining whether the similarity between the second rendered image and the second formula image meets the second preset requirement, the second training unit 604 can first obtain the image feature vector of the second rendered image and the image feature vector of the second formula image, and then calculate the similarity (for example, cosine similarity) between the two images based on the obtained image feature vectors. Finally, if it is determined that the calculated similarity is greater than or equal to the similarity threshold, it is determined that the similarity between the second rendered image and the second formula image meets the second preset requirement; otherwise, it is determined that the second preset requirement is not met.

[0127] The second training unit 604 uses the second training sample to train the candidate formula recognition model when the model evaluation index meets the first preset requirement, and when obtaining the target formula recognition model, the implementation method that can be adopted is: inputting the second formula image in the second training sample into the candidate formula recognition model when the model evaluation index meets the first preset requirement, and obtaining the fourth formula prediction result corresponding to the second formula image output by the candidate formula recognition model; calculating the loss function value according to the fourth formula prediction result corresponding to the second formula image and the second formula prediction result; adjusting the parameters of the candidate formula recognition model when the model evaluation index meets the first preset requirement according to the calculated loss function value, and obtaining the target formula recognition model.

[0128] Figure 7 is a schematic diagram according to the seventh embodiment of the present disclosure. Figure 7 As shown in , the formula recognition device 700 of this embodiment includes:

[0129] An acquisition unit 701 is used to acquire an image to be recognized;

[0130] The recognition unit 702 is used to input the image to be recognized into a target formula recognition model, and obtain a formula recognition result corresponding to the image to be recognized according to an output result of the target formula recognition model.

[0131] That is to say, this embodiment uses the pre-trained target formula recognition model to recognize the formula in the image to be recognized to obtain the formula recognition result, which can improve the accuracy of formula recognition.

[0132] The formula recognition result obtained by the recognition unit 702 is a character sequence in LaTeX format.

[0133] After obtaining the formula recognition result corresponding to the image to be recognized, the recognition unit 702 can perform processing such as formula editing, formula input, and formula semantic understanding according to the formula recognition result.

[0134] The formula recognition device 700 of this embodiment may further include a processing unit 703 for executing the following contents: using a large language model, and obtaining a semantic understanding result of the formula in the image to be processed according to the obtained formula recognition result.

[0135] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0137] like Figure 8 , is a block diagram of an electronic device for training a formula recognition model and a formula recognition method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0138] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0139] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the training of the formula recognition model and the formula recognition method. For example, in some embodiments, the training of the formula recognition model and the formula recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808.

[0141] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training of the formula recognition model and the formula recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the training of the formula recognition model and the formula recognition method by any other appropriate means (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable formula recognition model training and formula recognition device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a formula recognition model, comprising: Constructing a first training sample set, wherein the first training sample set includes a plurality of first training samples; Training the source formula recognition model according to a first training sample randomly selected from the first training sample set to obtain an initial formula recognition model; Using the initial formula recognition model to determine difficult evaluation samples in the evaluation sample set, and constructing a combined training sample set based on the difficult evaluation samples and the first training sample set; The initial formula recognition model is trained using the combined training sample set to obtain a target formula recognition model.

2. The method according to claim 1, wherein: The constructing of the first training sample set comprises: Extract the LaTeX character sequence corresponding to the formula from the LaTeX document; Rendering the LaTeX character sequence using a rendering engine to obtain a first rendered image corresponding to the LaTeX character sequence; Using the first rendered image as a first formula image, and using the LaTeX character sequence as a first formula annotation result corresponding to the first formula image; The first formula image and the corresponding first formula annotation result are used as first training samples to construct the first training sample set including multiple first training samples.

3. The method according to claim 1, wherein: The constructing a combined training sample set according to the difficulty assessment sample and the first training sample set includes: Obtaining difficult sample features according to the difficult assessment sample; Constructing a first training sample subset according to the first training samples corresponding to the difficult sample features selected from the first training sample set; The merged training sample set is constructed according to the first training sample subset and a first training sample randomly selected from the first training sample set.

4. The method according to claim 1, wherein: The using the combined training sample set to train the initial formula recognition model to obtain a target formula recognition model comprises: Using the combined training sample set to train the initial formula recognition model to obtain a candidate formula recognition model; Using the evaluation sample set to evaluate the candidate formula recognition model, to obtain a model evaluation index; In response to determining that the model evaluation index does not meet the preset requirement, after replacing the initial formula recognition model with the candidate formula recognition model, the step of using the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set is performed, and this is repeated until it is determined that the model evaluation index meets the first preset requirement; The target formula recognition model is obtained according to the candidate formula recognition model when the model evaluation index meets the first preset requirement.

5. The method according to claim 4, wherein: The step of obtaining the target formula recognition model according to the candidate formula recognition model when the model evaluation index satisfies the first preset requirement comprises: Acquire a second training sample set, wherein the second training sample set includes a plurality of second formula images; Constructing a second training sample according to the second formula image; The candidate formula recognition model is trained using the second training sample when the model evaluation index meets the first preset requirement to obtain the target formula recognition model.

6. The method according to claim 5, wherein: The constructing a second training sample according to the second formula image comprises: Using the candidate formula recognition model when the model evaluation indicator meets the first preset requirement, obtaining a second formula prediction result corresponding to the second formula image; Rendering the prediction result of the second formula using a rendering engine to obtain a second rendered image corresponding to the prediction result of the second formula; In response to determining that the similarity between the second rendered image and the second formula image meets a second preset requirement, a second training sample is constructed according to the second formula image and its corresponding second formula prediction result.

7. A formula recognition method, comprising: Get the image to be processed; Inputting the image to be processed into a target formula recognition model, and obtaining a formula recognition result corresponding to the image to be processed according to an output result of the target formula recognition model; Wherein, the target formula recognition model is trained according to any one of the methods in claims 1-6.

8. The method according to claim 7, further comprising: Using a large language model, according to the formula recognition result, a semantic understanding result of the formula in the image to be processed is obtained.

9. A training device for a formula recognition model, comprising: A first constructing unit, configured to construct a first training sample set, wherein the first training sample set includes a plurality of first training samples; A first training unit, configured to train a source formula recognition model according to a first training sample randomly selected from the first training sample set to obtain an initial formula recognition model; A second construction unit is configured to use the initial formula recognition model to determine difficult evaluation samples in the evaluation sample set, and to construct a combined training sample set based on the difficult evaluation samples and the first training sample set; The second training unit is used to train the initial formula recognition model using the combined training sample set to obtain a target formula recognition model.

10. The device according to claim 9, wherein: When constructing the first training sample set, the first construction unit specifically performs: Extract the LaTeX character sequence corresponding to the formula from the LaTeX document; Rendering the LaTeX character sequence using a rendering engine to obtain a first rendered image corresponding to the LaTeX character sequence; Using the first rendered image as a first formula image, and using the LaTeX character sequence as a first formula annotation result corresponding to the first formula image; The first formula image and the corresponding first formula annotation result are used as first training samples to construct the first training sample set including multiple first training samples.

11. The device according to claim 9, wherein: When the second construction unit constructs the combined training sample set according to the difficulty assessment sample and the first training sample set, the second construction unit specifically performs: Obtaining difficult sample features according to the difficult assessment sample; Constructing a first training sample subset according to the first training samples corresponding to the difficult sample features selected from the first training sample set; The merged training sample set is constructed according to the first training sample subset and a first training sample randomly selected from the first training sample set.

12. The device according to claim 9, wherein: When the second training unit trains the initial formula recognition model using the combined training sample set to obtain a target formula recognition model, specifically: Using the combined training sample set to train the initial formula recognition model to obtain a candidate formula recognition model; Using the evaluation sample set to evaluate the candidate formula recognition model, to obtain a model evaluation index; In response to determining that the model evaluation index does not meet the preset requirement, after replacing the initial formula recognition model with the candidate formula recognition model, the step of using the initial formula recognition model to determine the difficult evaluation samples in the evaluation sample set is performed, and this is repeated until it is determined that the model evaluation index meets the first preset requirement; The target formula recognition model is obtained according to the candidate formula recognition model when the model evaluation index meets the first preset requirement.

13. The device according to claim 12, wherein: When the second training unit obtains the target formula recognition model according to the candidate formula recognition model when the model evaluation index meets the first preset requirement, the second training unit specifically performs: Acquire a second training sample set, wherein the second training sample set includes a plurality of second formula images; Constructing a second training sample according to the second formula image; The candidate formula recognition model is trained using the second training sample when the model evaluation index meets the first preset requirement to obtain the target formula recognition model.

14. The device according to claim 13, wherein: When the second training unit constructs the second training sample according to the second formula image, specifically: Using the candidate formula recognition model when the model evaluation indicator meets the first preset requirement, obtaining a second formula prediction result corresponding to the second formula image; Rendering the prediction result of the second formula using a rendering engine to obtain a second rendered image corresponding to the prediction result of the second formula; In response to determining that the similarity between the second rendered image and the second formula image meets a second preset requirement, a second training sample is constructed according to the second formula image and its corresponding second formula prediction result.

15. A formula recognition device, comprising: An acquisition unit, used for acquiring an image to be processed; A recognition unit, used for inputting the image to be processed into a target formula recognition model, and obtaining a formula recognition result corresponding to the image to be processed according to an output result of the target formula recognition model; Wherein, the target formula recognition model is trained according to any one of the devices in claims 9-14.

16. The apparatus according to claim 15, further comprising a processing unit configured to execute: Using a large language model, according to the formula recognition result, a semantic understanding result of the formula in the image to be processed is obtained.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images