A handwritten formula image recognition method based on rule injection and a related device

Abstract

The application provides a handwritten formula image recognition method based on rule injection and a related device, and belongs to the technical field of artificial intelligence and computer vision, and comprises the following steps: performing formula region positioning and segmentation on an obtained handwritten formula image to obtain a region image; performing recognition on a formula in the obtained region image to obtain a LaTeX format formula; performing semantic detection and correction on the recognized LaTeX format formula by using a preset rule library to obtain a corrected LaTeX format formula; and comparing the obtained corrected LaTeX format formula with a preset answer pool to obtain a handwritten answer condition; the application can not only avoid interference with the logic of student handwritten answers, but also can accurately output the handwritten answer condition through double comparison of the symbol level and the structure level with the answer pool, thereby greatly improving the accuracy, interpretability and efficiency of formula recognition in an intelligent education automatic correction scene.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, specifically relating to a method and related apparatus for recognizing handwritten formula images based on rule injection. Background Technology

[0002] In intelligent education and automated grading scenarios, handwritten formula recognition and parsing have become core components for achieving "human-computer collaborative grading" and "formula semantic understanding." However, most existing formula recognition models are based on a direct mapping method between Optical Character Recognition (OCR) and Convolutional Neural Networks (CNN), which can only complete recognition at the character level and struggles to capture the formula's structural hierarchy, symbolic grammatical relationships, and mathematical logical consistency. This purely visual recognition method is highly susceptible to structural mismatches and semantic ambiguities when dealing with linear algebra-specific structures such as overlapping symbols, matrix indentation, and nested fractions, leading to discrepancies between the recognized LaTeX output and the actual calculated expression.

[0003] Traditional deep learning methods typically rely on large-scale labeled data for training. However, in educational settings, the number of standard answers and student assignments is often unbalanced and significantly different. Due to a lack of sufficient high-quality labeled samples, the models have insufficient generalization ability when faced with complex handwritten symbols. Furthermore, existing OCR systems lack constraints on the syntactic consistency and symbolic dependency structure of mathematical expressions, making it impossible to perform logical verification and error correction during the recognition stage, severely impacting the accuracy and interpretability of the final output.

[0004] Furthermore, although recognition methods based on Transformer and Large Language Model (LLM) have made progress in visual semantic understanding in recent years, these models lack a rule-based prior injection mechanism, making it difficult to achieve effective self-correction at the symbolic logic and mathematical semantic levels. Purely supervised training also struggles to cover differences in teachers' handwriting styles and students' assignments, and is insufficiently adaptable to complex formulas. Summary of the Invention

[0005] The purpose of this invention is to provide a method and related apparatus for recognizing handwritten formula images based on rule injection, which solves the above-mentioned shortcomings in the prior art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for recognizing handwritten formula images based on rule injection, comprising the following steps:

[0008] The acquired handwritten formula image is processed by formula region localization and segmentation to obtain a region image;

[0009] Formulas in the obtained region image are identified to obtain LaTeX format formulas;

[0010] The identified LaTeX format formulas are semantically detected and corrected using a pre-defined rule base to obtain the corrected LaTeX format formulas.

[0011] The corrected LaTeX formulas are compared with a pre-set answer pool to obtain handwritten responses, where:

[0012] The preset rule base includes syntax correction functions and formula expression structure rules. The syntax correction functions identify the correspondence between error descriptions and corrected target results. The formula expression structure rules are formed by the calculation steps, symbol patterns, and fixed solution methods in the standard answer.

[0013] Preferably, the error description includes ambiguous symbols, missing parentheses, incorrect lines, misspelled LaTeX symbols, consecutive operators, and non-standard variable naming.

[0014] Preferably, the method for obtaining the syntax correction function is as follows:

[0015] Formula recognition is performed on the obtained standard answer printed image to obtain the predicted LaTeX format formula;

[0016] The predicted LaTeX format formula is manually corrected to obtain the corrected LaTeX format formula and the syntax correction function.

[0017] The method for obtaining the structure rules of formula expressions is as follows:

[0018] Based on the revised LaTeX format formulas, the calculation steps, symbol patterns, and fixed solution methods in the standard answers are extracted to form formula expression structure rules.

[0019] Preferably, the specific method for constructing the preset answer pool is as follows:

[0020] The obtained standard answer printed image is identified to obtain the predicted LaTeX format formula;

[0021] The obtained predicted LaTeX formulas are manually corrected and labeled with knowledge point categories to obtain the corrected LaTeX formulas.

[0022] The corrected LaTeX format formulas are used as the preset answer pool.

[0023] Preferably, the Unimernet network is used to identify the formulas in the obtained regional image to obtain the LaTeX format formulas.

[0024] Secondly, the present invention provides a handwritten formula image recognition system based on rule injection, comprising:

[0025] The image segmentation unit is used to locate and segment the formula region in the acquired handwritten formula image to obtain the region image;

[0026] The formula recognition unit is used to recognize the formulas in the obtained region image and obtain the LaTeX format formulas;

[0027] The formula correction unit is used to perform semantic detection and correction on the identified LaTeX format formulas using a preset rule base, resulting in corrected LaTeX format formulas, where:

[0028] The pre-defined rule base includes syntax correction functions and formula expression structure rules. The syntax correction functions identify the correspondence between error descriptions and corrected target results. The formula expression structure rules are formed from the calculation steps, symbol patterns, and fixed solution methods in the standard answer.

[0029] The formula comparison unit is used to compare the corrected LaTeX formula with the preset answer pool to obtain the handwritten answer.

[0030] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the method described thereon.

[0031] Fourthly, the present invention provides a computing device cluster, comprising at least one computing device, each computing device including a processor and a memory;

[0032] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to the method.

[0033] Fifthly, the present invention provides a computer program product, the computer program product including computer-executable instructions, which, when executed, implement the method described.

[0034] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described herein.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This invention provides a rule-injection-based method for recognizing handwritten formulas. It converts handwritten formulas into LaTeX format formulas, then uses a pre-defined rule base to perform semantic detection and correction on the recognized LaTeX formulas. This effectively solves the recognition bias problems unique to linear algebra structures, such as symbol ambiguity, missing brackets, and structural mismatches, ensuring the syntactic consistency and logical rationality of the LaTeX formulas. The corrected LaTeX formulas are then compared with a pre-defined answer pool. This invention avoids interfering with students' handwritten answers and accurately outputs the handwritten answers through symbol-level and structure-level comparisons with the answer pool, significantly improving the accuracy, interpretability, and efficiency of formula recognition in intelligent education automatic grading scenarios. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the image recognition process according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the image segmentation process according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the handwritten formula recognition process according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the rule base construction process according to an embodiment of the present invention. Detailed Implementation

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] It should be understood that, when used in this application specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0046] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] Example 1

[0049] This embodiment provides a rule-injected handwritten formula image recognition method, which includes the following steps:

[0050] Step 1: Locate and segment the formula region in the acquired handwritten formula image to obtain the region image;

[0051] Step 2: Identify the formulas in the obtained region image to obtain the LaTeX format formulas;

[0052] Step 3: Use a preset rule base to perform semantic detection and correction on the identified LaTeX format formulas to obtain the corrected LaTeX format formulas;

[0053] Step 4: Compare the corrected LaTeX format formula with the preset answer pool to obtain the handwritten answer.

[0054] Example 2

[0055] Based on Example 1, this example provides a handwritten formula image recognition method based on rule injection, which performs formula region localization and segmentation on the acquired handwritten formula image. The specific method is as follows:

[0056] The YOLOv8 algorithm is used to locate and segment the formula regions in the acquired handwritten formula images to obtain region images.

[0057] Example 3

[0058] Based on Example 1, this example provides a rule-injection-based handwritten formula image recognition method to recognize formulas in the obtained region image. The specific method is as follows:

[0059] The Unimernet network is used to identify formulas in the obtained regional image, resulting in LaTeX format formulas.

[0060] Example 4

[0061] Based on Example 1, this example provides a handwritten formula image recognition method based on rule injection. The specific method for constructing the preset rule base is as follows:

[0062] S1, Recognize the obtained standard answer printed image to obtain the LaTeX format formula;

[0063] S2, manually correct the obtained LaTeX format to obtain the corrected LaTeX formulas and syntax correction functions, where:

[0064] The syntax correction function identifies the correspondence between error descriptions and corrected target results. The error descriptions include, but are not limited to, ambiguous symbols, missing parentheses, incorrect line identification, misspelled LaTeX symbols, consecutive operators, and non-standard variable naming.

[0065] The corrected target result is a correct expression or step that conforms to mathematical norms and format requirements, including but not limited to the correct formula after parentheses are completed, the corrected LaTeX symbols after spelling correction, and the variables after standardized naming.

[0066] Based on the revised LaTeX format, the calculation steps, symbol patterns, and fixed solution methods in the standard answer are extracted to form formula expression structure rules;

[0067] Among them, the rules of symbols include the precedence and position rules of operators in the standard answer, the rules of capitalization and naming of variable symbols, the rules of nesting parentheses, the syntax rules of LaTeX symbols, and the rules of symbol typesetting and alignment.

[0068] Fixed solution methods are the general problem-solving steps, logical processes, and standard solutions for a certain type of question in the standard answer. They are part of the structural rules of formula expressions, such as fixed steps for solving equations, fixed logic for geometric proofs, general problem-solving processes for function problems, and fixed formula application methods for specific question types.

[0069] The syntax correction function and formula expression structure rules are combined to form a preset rule base, which is then stored in the agent.

[0070] Example 5

[0071] Based on Example 1, this example provides a handwritten formula image recognition method based on rule injection. The pre-set answer pool is constructed as follows:

[0072] S1, Recognize the obtained standard answer printed image to obtain the predicted LaTeX format formula;

[0073] S2, manually correct the obtained predicted LaTeX format formula and mark the knowledge point categories to obtain the corrected LaTeX format formula;

[0074] S3 uses the corrected LaTeX format formulas as the preset answer pool.

[0075] Example 6

[0076] This embodiment provides a handwritten formula image recognition method based on rule injection. This method achieves high-precision recognition, structured representation, and adaptive error correction of linear algebra formulas by introducing a multi-level rule-driven mechanism, symbolic syntax parsing, and a pseudo-supervised learning closed-loop structure. The method comprehensively utilizes deep convolutional feature extraction, symbolic syntax modeling, rule-injected intelligent agents, and a pseudo-supervised model iterative optimization mechanism to construct an intelligent learning system from standard answers to student homework recognition feedback. It can continuously optimize recognition performance and grammatical consistency in small sample scenarios. Specifically, it includes the following steps:

[0077] S100: Extraction of Standard Answer Rules and Preparation for Pre-training.

[0078] Formula recognition is performed on the printed image of the standard answer to obtain the predicted LaTeX format formula;

[0079] The predicted LaTeX format is manually corrected to obtain the corrected LaTeX format formula and syntax correction function.

[0080] Based on the revised LaTeX format formulas, the calculation steps, symbol patterns, and fixed solution methods in the standard answers are extracted to form formula expression structure rules;

[0081] The syntax correction function and formula expression structure rules are combined into a pre-defined rule base;

[0082] The corrected LaTeX format formulas will be used as the answer pool.

[0083] S200: Large model identification and preliminary analysis.

[0084] The handwritten formula image is input into a pre-trained recognition model (such as one based on the Unimernet network) for formula recognition, and the formula is output in LaTeX format.

[0085] A preliminary grammatical and structural check is performed on the identification results to assess their reasonableness.

[0086] S300: Rule-injected agent detection and error correction.

[0087] Perform semantic detection on the LaTeX format formulas output by S200;

[0088] Based on the rule base and answer pool obtained from S100, a rule injection agent is constructed. Through the rule injection mechanism, symbol errors, structural errors, etc. are corrected to obtain the corrected LaTeX format formula.

[0089] The score of the corrected LaTeX format formula is calculated using context-free grammar (CFG) to evaluate the structural rationality of the corrected LaTeX format formula, and the large model is optimized based on the structural rationality.

[0090] S400: Pseudo-supervised learning and model iterative optimization.

[0091] Collect multiple handwritten formula images for the same problem, and generate pseudo-label samples using high-confidence recognition results;

[0092] Using pseudo-labeled samples to retrain and fine-tune the recognition model improves recognition and error correction accuracy.

[0093] S500: Loop optimization.

[0094] The processed results in S300 and S400 are fed back to S100 to update the rule base and answer pool;

[0095] Through multiple iterations, the generalization ability and error correction accuracy of the recognition model are continuously improved.

[0096] Example 7

[0097] Building upon Example 6, this example provides a rule-injection-based handwritten formula image recognition method. This method recognizes the input standard answer printed image to obtain a predicted LaTeX format formula. The specific method is as follows:

[0098] S101, perform grayscale and binarization preprocessing on the printed image of the standard answer, and use a convolutional neural network (CNN) model to extract local symbol features and positional relationships.

[0099] Let the input image be ,in, The real number field is the input image. The collection to which the data type belongs; The height of the input image; The width of the input image.

[0100] The expression for convolutional feature extraction is:

[0101]

[0102] in, For convolution kernel weights, For bias terms, It is a non-linear activation function; The convolutional feature map is obtained after convolutional feature extraction from the input image I, and it is the feature encoding result of the original image.

[0103] S102: LaTeX Structure Parsing and Syntax Tree Generation

[0104] The standard LaTeX expression is parsed into a syntax tree, and its computation order and nested structure are extracted to form a symbol-level structural template. In this embodiment, the standard LaTeX expression is a matrix inversion expression:

[0105]

[0106] in, Let A be the determinant of matrix A; Let A be the adjoint matrix of matrix A; A is a matrix. Let be the inverse of matrix A.

[0107] The symbol-level structural template is transformed into syntax rules to obtain the predicted LaTeX format formula. In this embodiment, the transformed syntax rules are as follows:

[0108]

[0109] in, This is the expression for the inverse matrix; <frac>It is a fraction; <det>It is a determinant; <adj>The adjoint matrix; <var>For variables.

[0110] S103: Comparison and Error Calculation

[0111] Calculate the prediction expression output by the recognition model With real LaTeX Sign level error :

[0112]

[0113] in, It is the total number of samples; It is an index for a single sample.

[0114] The gradient is updated based on the error to obtain the preliminary symbol pattern recognition capability of the syntax correction function.

[0115] S104: Knowledge Point Injection and Symbol Vectorization

[0116] Based on the formula type, the knowledge point category corresponding to the LaTeX format formula is predicted. In this embodiment, the knowledge point categories include "matrix multiplication", "determinant properties", and "eigenvalue calculation", forming a knowledge mapping set. :

[0117]

[0118] in, The m-th knowledge point in the knowledge mapping set;

[0119] Establish symbol vector embedding matrix :

[0120]

[0121] in, For the j-th symbol, and related knowledge points Combine the elements of the generated symbol vector embedding matrix;

[0122] Used for pre-training and feature alignment of rule-injection agents.

[0123] In step S200, the large model is identified and initially analyzed. The specific method includes the following sub-steps:

[0124] S201: Image Data Extraction

[0125] Recognize formulas in handwritten formula images captured or scanned from student assignments and generate LaTeX format formulas:

[0126]

[0127] The image feature extraction module uses the Vision Encoder network. This is a function for extracting features from handwritten images; These are the feature vectors in the symbol embedding space;

[0128] S202: CFG Syntax Verification Calculation

[0129] Input image Projected into symbol embedding space The semantic decoder outputs the LaTeX structure according to the context-free grammar (CFG) rules:

[0130]

[0131] in, For LaTeX structured expressions; This is the optimal prediction LaTeX structure expression; Optimize operators for mathematics; The probability distribution function of the LaTeX structure;

[0132] Simultaneously, the structural accuracy score is calculated using CFG syntax verification. :

[0133]

[0134] in, The number of valid nodes; This represents the total number of nodes.

[0135] Generate structured representation results and confidence distribution.

[0136] If the recognition confidence level is lower than the set threshold If a sample is found to be low confidence, it will be marked as a "low confidence sample" and added to the pseudo-supervised sample pool.

[0137] Step S300 includes the following sub-steps:

[0138] Based on the pre-defined rule base established in the S100 stage, a rule-injection agent is constructed to perform symbol-level detection and structural correction during the model inference stage.

[0139] S301: Rule Injection Mechanism

[0140] The syntax correction function is defined as follows:

[0141]

[0142] in, This is a syntax correction function; This is the nth semantic correction rule;

[0143] When generating each level of symbols, a corresponding syntax correction function is called to perform logical checks:

[0144]

[0145] in, This is the rule influence coefficient; Output the symbol for the t-th layer; Let be the feature vector of the t-th layer.

[0146] S302: Error Detection and Correction Algorithm

[0147] When errors such as missing symbols or unclosed brackets are detected, the node replacement algorithm is executed:

[0148]

[0149] in, The corrected node; This is an error node; For rule templates; This is a replacement algorithm.

[0150] Further fill in missing or identified errors to achieve error detection and rule correction functions.

[0151] S303: Feedback Update Mechanism

[0152] The corrected samples are used to adjust the weights of the recognition layer via the error backpropagation module.

[0153]

[0154] in, This refers to the weight update amount; The learning rate; The loss function; To identify layer weights.

[0155] Simultaneously, the syntax parsing subnetwork is updated, enabling the model to gradually acquire structural self-correction capabilities.

[0156] Step S400 includes the following sub-steps:

[0157] Based on the recognition results and correction data of S200 and S300, a pseudo-label sample set is generated, and the model is subjected to multiple rounds of pseudo-supervised training and self-learning optimization.

[0158] S401: Pseudo-tag generation and filtering

[0159] High-confidence recognition results are used as pseudo-label samples. :

[0160]

[0161] in The threshold for pseudo-labels; Let i be the input image for the i-th pseudo-label sample; This is the optimal label.

[0162] S402: Retraining and Incremental Optimization

[0163] Using pseudo-labeled samples as the new training set, continuous incremental updates are performed:

[0164]

[0165] in, For the first Parameters after one round of training; These are the parameters after the t-th round of training; For the loss function with respect to the parameters The gradient.

[0166] Continuously increase the amount of data in the dataset to compensate for the lack of sufficient data samples.

[0167] Step S500 includes the following sub-steps:

[0168] After completing one recognition cycle, the symbol patterns and structural statistics of the current model are fed back to S100, and the recognition accuracy is then improved after subsequent pseudo-supervised training. With error rate Also serves as a cyclic feedback signal:

[0169]

[0170] in, This refers to the amount of rule updates. For the first Wheel recognition accuracy; For the first Error rate per cycle.

[0171] This updates the rule template and symbol embedding matrix, forming a closed-loop optimization structure and achieving adaptive evolution.

[0172] Example 8

[0173] This embodiment provides a handwritten formula image recognition system based on rule injection, including:

[0174] The image segmentation unit is used to locate and segment the formula region in the acquired handwritten formula image to obtain the region image;

[0175] The formula recognition unit is used to recognize the formulas in the obtained region image and obtain the LaTeX format formulas;

[0176] The formula correction unit is used to perform semantic detection and correction on the identified LaTeX format formulas using a preset rule base, so as to obtain the corrected LaTeX format formulas.

[0177] The formula comparison unit is used to compare the corrected LaTeX formula with the preset answer pool to obtain the handwritten answer.

[0178] Example 9

[0179] This embodiment also provides a computing device. The computing device includes a bus, a processor, a memory, and a communication interface. The processor, memory, and communication interface communicate with each other via the bus. The computing device can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device.

[0180] A bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, a bus can include a path for transmitting information between various components of a computing device (e.g., memory, processor, communication interfaces).

[0181] The processor may include any one or more of the following: central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), microprocessor (MP), or digital signal processor (DSP).

[0182] Memory can include volatile memory, such as random access memory (RAM). Processors can also include non-volatile memory. volatile memory, such as read-only memory (ROM). ROM (memory only), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0183] The memory stores executable program code, which the processor executes to implement the functions of the aforementioned units, thereby achieving, for example, the method described in Embodiment 1. That is, the memory may store instructions for the methods and functions relating to the computing device in any of the above embodiments.

[0184] The communication interface uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between computing devices and other devices or communication networks.

[0185] Example 10

[0186] This embodiment also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0187] The computing device cluster includes at least one computing device. The memory of one or more computing devices in the computing device cluster may store the same instructions for performing the methods and functions related to the computing devices in any of the above embodiments.

[0188] In some possible implementations, the memory of one or more computing devices in the computing device cluster may also store partial instructions for performing the methods and functions of the computing devices involved in any of the above embodiments. In other words, a combination of one or more computing devices can jointly execute the instructions for performing the methods and functions of the computing devices.

[0189] It should be noted that the memory in different computing devices within a computing device cluster can store different instructions, which are used to execute parts of the device's functions.

[0190] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Two computing devices are connected to each other via the network. Specifically, they connect to the network through communication interfaces in each computing device.

[0191] Embodiments of this disclosure also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions related to a computing device in any of the above embodiments.

[0192] Example 11

[0193] This embodiment also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, cause the processor to perform the methods and functions of the computing device involved in any of the above embodiments.

[0194] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0195] Example 12

[0196] This embodiment provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods as described above with reference to the accompanying drawings. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0197] Computer program code used to implement the methods of this disclosure may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the computer or other programmable data processing apparatus, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be performed. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0198] In the context of this disclosure, computer program code or related data may be carried on any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0199] Computer-readable media can be any tangible medium that contains or stores programs for or relating to an instruction execution system, apparatus, or device, or a data storage device such as a data center containing one or more available media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0200] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.< / var> < / adj> < / det> < / frac>

Claims

1. A method for recognizing handwritten formula images based on rule injection, characterized in that, Includes the following steps: The acquired handwritten formula image is processed by formula region localization and segmentation to obtain a region image; Formulas in the obtained region image are identified to obtain LaTeX format formulas; The identified LaTeX format formulas are semantically detected and corrected using a pre-defined rule base to obtain the corrected LaTeX format formulas. The corrected LaTeX formulas are compared with a pre-set answer pool to obtain handwritten responses, where: The pre-defined rule base includes syntax correction functions and formula expression structure rules. The syntax correction functions identify the correspondence between error descriptions and corrected target results. The formula expression structure rules are formed from the calculation steps, symbol patterns, and fixed solution methods in the standard answer. The method to obtain the syntax correction function is: The obtained standard answer printed image is identified to obtain the predicted LaTeX format formula; The predicted LaTeX format formula is manually corrected to obtain the corrected LaTeX format formula and the syntax correction function. The method for obtaining the structure rules of formula expressions is as follows: Based on the revised LaTeX format formulas, the calculation steps, symbol patterns, and fixed solution methods in the standard answers are extracted to form formula expression structure rules; The specific method for constructing the preset answer pool is as follows: The obtained standard answer printed image is identified to obtain the predicted LaTeX format formula; The obtained predicted LaTeX formulas are manually corrected and labeled with knowledge point categories to obtain the corrected LaTeX formulas. The revised LaTeX format formulas are used as the preset answer pool; The identified LaTeX formulas are semantically detected and corrected using a pre-defined rule base to obtain corrected LaTeX formulas. The specific method is as follows: A rule-injection agent is constructed based on a pre-defined rule base and a pre-defined answer pool. The rule-injection agent is then used to perform semantic detection and correction on the identified LaTeX format formulas, resulting in corrected LaTeX format formulas.

2. The handwritten formula image recognition method based on rule injection according to claim 1, characterized in that, The identification errors include ambiguous symbols, missing parentheses, incorrect lines, misspelled LaTeX symbols, consecutive operators, and non-standard variable naming.

3. The handwritten formula image recognition method based on rule injection according to claim 1, characterized in that, The process of identifying formulas in the obtained region image to obtain LaTeX format formulas includes: The Unimernet network is used to identify formulas in the obtained regional image, resulting in LaTeX format formulas.

4. A handwritten formula image recognition system based on rule injection, characterized in that, The identification method based on claim 1 includes: The image segmentation unit is used to locate and segment the formula region in the acquired handwritten formula image to obtain the region image; The formula recognition unit is used to recognize the formulas in the obtained region image and obtain the LaTeX format formulas; The formula correction unit is used to perform semantic detection and correction on the identified LaTeX format formulas using a preset rule base, resulting in corrected LaTeX format formulas, where: The pre-defined rule base includes syntax correction functions and formula expression structure rules. The syntax correction functions identify the correspondence between error descriptions and corrected target results. The formula expression structure rules are formed from the calculation steps, symbol patterns, and fixed solution methods in the standard answer. The formula comparison unit is used to compare the corrected LaTeX formula with a preset answer pool to obtain the handwritten answer.

5. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 3.

6. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method according to any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes computer-executable instructions that, when executed, implement the method of any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method of any one of claims 1 to 3.

Images