Mathematical ancient book illustration identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a mathematical ancient book illustration identification method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a mathematical ancient book illustration; dividing the mathematical ancient book illustration into a character area, a formula area and an image area, and recording respective position information of the three parts as first position information; carrying out OCR (Optical Character Recognition) on the character region, carrying out formula restoration on the formula region, and restoring the image region; and performing layout reconstruction on the math ancient book illustration according to the text identified by the OCR, the restored formula, the restored image and the first position information to obtain a math ancient book illustration identification result, thereby realizing effective identification and digitization of the math ancient book illustration.

Description

Technical Field

[0001] The present invention belongs to the field of ancient book digitization, and specifically relates to an ancient mathematics book illustration recognition method, device, electronic equipment and storage medium. Background Art

[0002] Ancient books are an important part of cultural heritage. The application of digital technology can achieve the long-term preservation and inheritance of these precious documents, avoid physical damage to ancient books such as wear and tear, creases, etc., and reduce the loss caused by borrowing and reading.

[0003] Among the related technologies, the intelligent OCR (image text recognition) system for ancient books developed based on machine learning technology can process images of ancient books of various versions such as engravings and manuscripts. It has the ability to perform intelligent layout analysis, intelligent component annotation and output annotation results. It can handle conventional ancient book layouts as well as special ancient book layouts including double-line notes, marginal notes, interline annotations, tables, etc.

[0004] Different from ancient book images with text, illustrations in ancient mathematics books contain a large number of mathematical symbols, formulas and diagrams. Although the existing ancient book OCR technology can recognize ordinary text, it often makes mistakes or fails to recognize these complex illustrations containing mathematical elements. Since the digital processing and analysis of ancient mathematics books are of great significance for in-depth understanding of the development of ancient mathematics, the evolution of mathematical ideas, and the cross-integration of mathematics with other disciplines, there is an urgent need for a solution that can effectively recognize and digitize illustrations in ancient mathematics books. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method, device, electronic device and storage medium for recognizing illustrations in ancient mathematics books.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a method for identifying illustrations in ancient mathematics books, comprising:

[0008] Obtain illustrations from ancient mathematics books;

[0009] Segmenting the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area, and recording position information of each of the three parts as first position information;

[0010] Performing OCR recognition on the text area, restoring the formula on the formula area, and repairing the image area;

[0011] The layout of the ancient mathematics book illustration is reconstructed according to the OCR-recognized text, the restored formula, the repaired image and the first position information as the ancient mathematics book illustration recognition result.

[0012] Optionally, dividing the ancient mathematics book illustration into three parts: a text area, a formula area and an image area, comprises:

[0013] Inputting the ancient mathematics book illustration into a pre-trained ancient mathematics book illustration segmentation model, and using the ancient mathematics book illustration segmentation model to segment the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area;

[0014] Among them, the ancient mathematics book illustration segmentation model is trained based on the ancient mathematics book illustration library and the annotation information of each sample ancient mathematics book illustration contained in the ancient mathematics book illustration library, and the annotation information is information indicating the text area, formula area and image area in the sample ancient mathematics book illustration.

[0015] Optionally, restoring the formula in the formula area includes:

[0016] Identify the digital code corresponding to each character in the formula, and determine the position information of each character relative to the reference symbol in the formula as the second position information;

[0017] Constructing an initial restoration formula according to the digital code and the second position information;

[0018] The initial restoration formula is modified in accordance with mathematical semantics to obtain a restored formula.

[0019] Optionally, performing mathematical semantics-related corrections on the initial restoration formula to obtain a restored formula includes:

[0020] Determine the formula template that the initial recovery formula complies with;

[0021] Using a formula checker corresponding to the formula template to check whether the initial recovery formula contains errors and the location of the errors, to obtain a check result;

[0022] The initial restoration formula is modified according to the inspection result and the context information included in the initial restoration formula.

[0023] Optionally, determining the formula template that the initial restoration formula complies with includes:

[0024] Constructing a formula coding vector according to the digital code and the second position information;

[0025] The formula template that the initial restoration formula conforms to is determined according to the formula encoding vector.

[0026] Optionally, the using a formula checker corresponding to the formula template to check whether the initial recovery formula contains an error and the location of the error to obtain a check result includes:

[0027] Inputting the formula encoding vector into a formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains an error and the position of the error as a check result;

[0028] The formula checker is trained based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding sample formulas that conform to the formula template, and the negative samples are obtained by encoding sample formulas that conform to the formula template after introducing local errors.

[0029] Optionally, the context information includes context information extracted according to the formula encoding vector.

[0030] In a second aspect, the present invention provides a device for recognizing illustrations in ancient mathematics books, comprising:

[0031] The acquisition module is used to obtain illustrations of ancient mathematics books;

[0032] A segmentation module, used for segmenting the illustration of the ancient mathematics book into three parts: a text area, a formula area and an image area, and recording the position information of each of the three parts as the first position information;

[0033] An OCR recognition module, used for performing OCR recognition on the text area;

[0034] A formula recovery module, used for recovering the formula in the formula area;

[0035] An image restoration module, used for restoring the image area;

[0036] The layout reconstruction module is used to reconstruct the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image and the first position information as the ancient mathematics book illustration recognition result.

[0037] In a third aspect, the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0038] Memory, used to store computer programs;

[0039] The processor is used to implement any of the steps of the above-mentioned method for recognizing illustrations in ancient mathematics books when executing the computer program stored in the memory.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for recognizing illustrations in ancient mathematics books are implemented.

[0041] The method for recognizing illustrations in ancient mathematics books provided by the present invention performs image segmentation on the illustrations in ancient mathematics books, divides them into three parts: a text area, a formula area and an image area, and processes the three parts separately, especially restores the formulas contained in the formula area separately, thereby improving the recognition ability of the illustrations in ancient mathematics books and realizing the effective recognition and digitization of the illustrations in ancient mathematics books.

[0042] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for recognizing illustrations in ancient mathematics books provided by an embodiment of the present invention;

[0044] Figure 2 It is a structural schematic diagram of a mathematical ancient book illustration segmentation model provided by an embodiment of the present invention;

[0045] Figure 3 It is a schematic diagram of the structure of a device for recognizing illustrations in ancient mathematics books provided by an embodiment of the present invention;

[0046] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0048] In order to realize the effective recognition and digitization of illustrations in ancient mathematics books, an embodiment of the present invention provides a method for recognizing illustrations in ancient mathematics books, such as Figure 1 As shown, the method comprises the following steps:

[0049] S10. Obtain illustrations from ancient mathematics books.

[0050] S20, dividing the illustrations of the ancient mathematics book into three parts: a text area, a formula area and an image area, and recording the position information of each of the three parts as the first position information.

[0051] Here, the illustrations of ancient mathematics books are divided into three parts: text area, formula area and image area, which can be achieved through image segmentation.

[0052] For example, the ancient mathematics book illustrations can be input into a pre-trained ancient mathematics book illustration segmentation model, and the ancient mathematics book illustration segmentation model is used to segment the ancient mathematics book illustrations into three parts: a text area, a formula area and an image area.

[0053] The mathematical ancient book illustration segmentation model is trained based on the mathematical ancient book illustration library and the annotation information of each sample mathematical ancient book illustration contained in the mathematical ancient book illustration library. The annotation information is information indicating the text area, formula area and image area in the sample mathematical ancient book illustration, specifically, the information for annotating the category (text, formula, image or background) corresponding to each pixel point in the sample mathematical ancient book illustration. When actually training the mathematical ancient book illustration segmentation model, the data set used can be composed of real data and synthetic data, wherein the real data is real mathematical ancient book illustrations, and the synthetic data can be synthesized by text corpus and target scene images, for example, the text corpus is transferred to a specified background template through text style.

[0054] The above-mentioned mathematical ancient book illustration segmentation model can be implemented using the neural network of the yolo framework. For example, using yolov8 as the baseline model, replace the Conv2d in the Convmodule convolution module in its skeleton with the linear mapping layer GhostNeT to obtain the mathematical ancient book illustration segmentation model; here, GhostNeT can generate more feature maps with fewer parameters. Alternatively, you can also use Figure 2 The neural network model shown is used as a segmentation model for ancient mathematics book illustrations. Figure 2 In the figure, ACNet is an attention convolution binary neural tree model for fine-grained image classification, Conv means convolution, Conv2d means two-dimensional convolution, and FC means fully connected.

[0055] When training the mathematical ancient book illustration segmentation model, the sample mathematical ancient book illustration is input into the mathematical ancient book illustration segmentation model being trained. The model predicts the location and category of the text area, formula area and image area in the sample mathematical ancient book illustration by performing image segmentation on the sample mathematical ancient book illustration. Then, the loss function is used to calculate the loss value to evaluate the difference between the current predicted segmentation result and the annotation information. The training is terminated until the loss value converges or a sufficient number of training times are reached, and the trained mathematical ancient book illustration segmentation model is obtained.

[0056] Regarding the above loss function, CLS Loss (Classification Loss) can be used to evaluate the loss of category prediction accuracy for the three regions, and DFL (Distribution Focal Loss) loss can be combined to evaluate the loss of segmentation accuracy for the three regions. Alternatively, ALRP Loss (Average Localization Recall Precision Loss) can be directly used as the above loss function.

[0057] S30, performing OCR recognition on the text area, restoring the formula on the formula area, and repairing the image area.

[0058] In step S30, PaddleOCR can be used to perform OCR recognition on the text area. PaddleOCR is an open source OCR (Optical Character Recognition) tool based on the PaddlePaddle deep learning framework. In addition, the DM-Router (DomainMLP Router) module can be added to the EAST text detection module of PaddleOCR, so that the chapters in the illustrations of ancient mathematics books or the text parts attached to some images can be divided by DM-Router to improve the recognition ability of ancient texts.

[0059] In step S30, restoring the formula in the formula area may include the following steps:

[0060] (1) Identify the digital code corresponding to each character in the formula, and determine the position information of each character relative to the reference symbol in the formula as the second position information.

[0061] Specifically, an image of each character is extracted from the formula area, and the digital code corresponding to the character is determined based on the image using a character recognition method. For common symbols, their digital codes are the unique codes corresponding to them in GB2312; for symbols unique to ancient mathematical books, they can be pre-added to uncoded positions in GB2312 so that these symbols can also have unique corresponding digital codes.

[0062] Regarding the extraction of the second position information, first determine the reference symbol from the formula. Specifically, you can take the symbol on the far left of the formula, or the symbol in the middle, or the symbol with the largest image area. All are possible. You only need to keep the rule for determining the reference symbol consistent each time. Then, based on the absolute position of each symbol, determine its relative position relationship with the reference symbol, and use the relative position relationship to represent the relative position relationship between the symbol and the reference symbol, and use the relative position relationship to represent the contextual relationship between the symbols, so as to facilitate the subsequent use of this relative position relationship to obtain the logical order between the symbols in the formula. Among them, the absolute position information of the symbol is also the area where the image of the symbol is located. For example, the absolute position information of the symbol can be represented by the pixel coordinates at the upper left corner and the lower right corner of the image. The second position information of the symbol refers to the absolute position of the symbol relative to the absolute position of the reference symbol. For example, assuming that the absolute position information of the reference symbol includes [x1, y1, x2, y2], that is, in the image of the reference symbol, the pixel coordinates at the upper left corner and the lower right corner are (x1, y1) and (x2, y2) respectively. If the absolute position information of a symbol includes [x3, y3, x4, y4], then the second position information of the symbol can be expressed as [D1=x3-x1, D2=y3-y1, D3=x3-x2, D4=y4-y2].

[0063] (2) Construct an initial restoration formula based on the digital code of each symbol and the second position information.

[0064] Specifically, according to the second position information of each character, the characters corresponding to the digital codes are combined together to obtain the initial restoration formula.

[0065] It is understandable that for well-preserved and clear ancient mathematics illustrations, in the step of identifying the digital codes corresponding to the characters in the formulas contained therein, the recognition accuracy can be basically ensured, and the initial restoration formula can be directly used as the restoration formula. For ancient mathematics illustrations with unclear images, the incorrect digital codes may be recognized due to unclear images, so the initial restoration formula needs to be checked and corrected, and the details are shown in the subsequent steps.

[0066] (3) Make mathematical and semantically relevant corrections to the initial restoration formula to obtain the restored formula.

[0067] Here, the initial restoration formula is modified with mathematical semantics to obtain the restored formula, including:

[0068] (3-1) Determine the formula template that the initial recovery formula conforms to.

[0069] Specifically, a formula coding vector is constructed according to the digital code and the second position information; for example, the arrangement order of each symbol can be determined by referring to the second position information of each symbol, so that the digital code and the second position information of each symbol can be combined according to a certain sorting rule to obtain a formula coding vector. For example, in the formula coding vector, the first half is the digital code of each character in turn, and the second half is the second position information of the character corresponding to each digital code in the first half. Then, the formula template that the initial restoration formula conforms to is determined according to the formula coding vector, which can be specifically achieved by calculating the vector similarity. It can be understood that it is necessary to construct the formula coding vector in advance using the same coding method for various formula templates.

[0070] (3-2) Use the formula checker corresponding to the formula template to check whether the initial recovery formula contains errors and the location of the errors to obtain the inspection results.

[0071] Specifically, the formula encoding vector is input into the formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains errors and the location of the errors as the checking result.

[0072] Among them, the formula checker can be obtained by training a large NLP (natural language processing) model. Specifically, it can be obtained by training based on multiple positive samples and multiple negative samples; the positive sample is obtained by encoding the sample formula that conforms to the formula template, and the negative sample is obtained by encoding the sample formula that conforms to the formula template after introducing a local error. Here, the positive sample and the negative sample are also provided with annotation information, specifically including whether the sample contains an error, and the location of the error when it contains an error. For the negative sample, the location of the error in the second part is known, while for the positive sample, the location of the error in the second part is empty.

[0073] It is worth mentioning that in the embodiment of the present invention, different formula checkers are trained for different formula templates, which reduces the difficulty of training on the one hand, and can also improve the detection accuracy of the formula checker corresponding to each formula template on the other hand.

[0074] (3-3) The initial recovery formula is modified based on the inspection results and the context information contained in the initial recovery formula.

[0075] Here, the context information contained in the initial restoration formula includes context information extracted according to the formula encoding vector. For example, according to the inspection result, the position of the error in the initial restoration formula can be determined, and the characters similar to the characters with recognition errors can be determined, thereby correcting the initial restoration formula according to the digital codes at adjacent positions, the digital codes of similar characters, and the formula semantics of the formula template itself to obtain the restored formula.

[0076] In step S30, the image area is repaired, specifically, the SRCNN (Super Resolution CNN) method based on interpolation super-resolution reconstruction can be used to enlarge the image, fit it using convolution, reduce the difference between the enlarged image and the original image, and improve the image resolution. Alternatively, the image area can be repaired by referring to the existing old photo repair tools to perform high-definition repair on old photos, which will not be described here.

[0077] S40, reconstructing the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image and the first position information to obtain the ancient mathematics book illustration recognition result.

[0078] Specifically, according to the first position information of the text recognized by OCR, the restored formula, and the repaired image, they are combined into a new ancient mathematics book illustration to ensure that the layout of the new ancient mathematics book illustration is consistent with the recognized ancient mathematics book illustration, thereby completing the effective recognition of the ancient mathematics book illustration.

[0079] The method for recognizing illustrations in ancient mathematics books provided by the embodiment of the present invention performs image segmentation on the illustrations in ancient mathematics books, divides them into three parts: a text area, a formula area, and an image area, and processes the three parts separately, especially restores the formulas contained in the formula area separately, thereby improving the recognition ability of illustrations in ancient mathematics books and realizing effective recognition and digitization of illustrations in ancient mathematics books.

[0080] After the illustrations in ancient mathematics books are digitized using the ancient mathematics book illustration recognition method provided by the embodiment of the present invention, scholars, students and ordinary readers can conveniently access and study them at any time and any place, avoiding the risk of ancient books being damaged or lost due to time and natural factors, helping to promote academic research, educational inheritance and cultural exchanges in the field of mathematics, and also facilitating the understanding of formulas in ancient books and searching for specific types of formulas.

[0081] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0082] Corresponding to the above-mentioned method for recognizing illustrations in ancient mathematics books, an embodiment of the present invention further provides a device for recognizing illustrations in ancient mathematics books, see Figure 3 As shown, the device comprises:

[0083] An acquisition module 301 is used to acquire illustrations of ancient mathematics books;

[0084] The segmentation module 302 is used to segment the illustration of the ancient mathematics book into three parts: a text area, a formula area and an image area, and record the position information of each of the three parts as the first position information;

[0085] OCR recognition module 303, used for performing OCR recognition on the text area;

[0086] A formula recovery module 304 is used to recover the formula in the formula area;

[0087] An image restoration module 305 is used to restore the image area;

[0088] The layout reconstruction module 306 is used to reconstruct the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image and the first position information to obtain the ancient mathematics book illustration recognition result.

[0089] Optionally, the segmentation module 302 is specifically configured to:

[0090] Inputting the ancient mathematics book illustrations into a pre-trained ancient mathematics book illustration segmentation model, and using the ancient mathematics book illustration segmentation model to segment the ancient mathematics book illustrations into three parts: a text area, a formula area, and an image area;

[0091] Among them, the ancient mathematics book illustration segmentation model is trained based on the ancient mathematics book illustration library and the annotation information of each sample ancient mathematics book illustration contained in the ancient mathematics book illustration library. The annotation information is information indicating the text area, formula area and image area in the sample ancient mathematics book illustration.

[0092] Optionally, the formula recovery module 304 is specifically used for:

[0093] Identify the digital code corresponding to each character in the formula, and determine the position information of each character relative to the reference symbol in the formula as the second position information;

[0094] Constructing an initial restoration formula according to the digital code and the second position information;

[0095] The initial restoration formula is modified with mathematical semantics to obtain the restored formula.

[0096] Optionally, the formula restoration module 304 performs mathematical semantics-related corrections on the initial restoration formula to obtain a restored formula, including:

[0097] Determine the formula template that the initial recovery formula complies with;

[0098] Use the formula checker corresponding to the formula template to check whether the initial recovery formula contains errors and the location of the errors to obtain the check result;

[0099] The initial restoration formula is modified according to the inspection result and the context information contained in the initial restoration formula.

[0100] Optionally, the formula recovery module 304 determines a formula template that the initial recovery formula complies with, including:

[0101] Constructing a formula encoding vector according to the digital encoding and the second position information;

[0102] The formula template that the initial recovery formula conforms to is determined according to the formula encoding vector.

[0103] Optionally, the formula recovery module 304 uses a formula checker corresponding to the formula template to check whether the initial recovery formula contains errors and the location of the errors, and obtains a check result, including:

[0104] Inputting the formula encoding vector into the formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains errors and the location of the errors as the check result;

[0105] The formula checker is trained based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding sample formulas that conform to the formula template, and the negative samples are obtained by encoding sample formulas that conform to the formula template after introducing local errors.

[0106] Optionally, the above context information includes context information extracted according to a formula encoding vector.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0108] Memory 603, used for storing computer programs;

[0109] The processor 601 is used to implement the method steps of any of the above-mentioned methods for recognizing illustrations in ancient mathematics books when executing the program stored in the memory 603.

[0110] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not represent that there is only one bus or one type of bus.

[0111] The communication interface is used for communication between the above electronic device and other devices.

[0112] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0113] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0114] The present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the method steps described in any of the above-mentioned methods for recognizing illustrations in ancient mathematics books are implemented.

[0115] Optionally, the computer-readable storage medium may be a non-volatile memory (NVM), such as at least one disk memory.

[0116] Optionally, the computer-readable memory may also be at least one storage device located away from the aforementioned processor.

[0117] It should be noted that, for the device / electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0118] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0120] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0121] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses (devices) and computer program products of the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for identifying illustrations in ancient mathematics books, characterized in that: include: Obtain illustrations from ancient mathematics books; Segmenting the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area, and recording position information of each of the three parts as first position information; Performing OCR recognition on the text area, restoring the formula on the formula area, and repairing the image area; The layout of the ancient mathematics book illustrations is reconstructed according to the OCR-recognized text, the restored formula, the repaired image and the first position information to obtain the ancient mathematics book illustration recognition result.

2. The method for recognizing illustrations in ancient mathematics books according to claim 1, characterized in that: The step of dividing the ancient mathematics book illustration into three parts: a text area, a formula area and an image area includes: Inputting the ancient mathematics book illustration into a pre-trained ancient mathematics book illustration segmentation model, and using the ancient mathematics book illustration segmentation model to segment the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area; Among them, the ancient mathematics book illustration segmentation model is trained based on the ancient mathematics book illustration library and the annotation information of each sample ancient mathematics book illustration contained in the ancient mathematics book illustration library, and the annotation information is information indicating the text area, formula area and image area in the sample ancient mathematics book illustration.

3. The method for recognizing illustrations in ancient mathematics books according to claim 1, characterized in that: The restoring the formula in the formula area includes: Identify the digital code corresponding to each character in the formula, and determine the position information of each character relative to the reference symbol in the formula as the second position information; Constructing an initial restoration formula according to the digital code and the second position information; The initial restoration formula is modified in accordance with mathematical semantics to obtain a restored formula.

4. The method for recognizing illustrations in ancient mathematics books according to claim 3, characterized in that: The initial restoration formula is modified with mathematical semantics to obtain a restored formula, including: Determine the formula template that the initial recovery formula complies with; Using a formula checker corresponding to the formula template to check whether the initial recovery formula contains errors and the location of the errors, to obtain a check result; The initial restoration formula is modified according to the inspection result and the context information included in the initial restoration formula.

5. The method for recognizing illustrations in ancient mathematics books according to claim 4, characterized in that: The step of determining the formula template that the initial restoration formula complies with includes: Constructing a formula coding vector according to the digital code and the second position information; The formula template that the initial restoration formula conforms to is determined according to the formula encoding vector.

6. The method for recognizing illustrations in ancient mathematics books according to claim 5, characterized in that: The method of using a formula checker corresponding to the formula template to check whether the initial recovery formula contains an error and the location of the error to obtain a check result includes: Inputting the formula encoding vector into a formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains an error and the position of the error as a check result; The formula checker is trained based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding sample formulas that conform to the formula template, and the negative samples are obtained by encoding sample formulas that conform to the formula template after introducing local errors.

7. The method for recognizing illustrations in ancient mathematics books according to claim 4, characterized in that: The context information includes context information extracted according to the formula encoding vector.

8. A device for recognizing illustrations in ancient mathematics books, characterized in that: include: The acquisition module is used to obtain illustrations of ancient mathematics books; A segmentation module, used for segmenting the illustration of the ancient mathematics book into three parts: a text area, a formula area and an image area, and recording the position information of each of the three parts as the first position information; An OCR recognition module, used for performing OCR recognition on the text area; A formula recovery module, used for recovering the formula in the formula area; An image restoration module, used for restoring the image area; The layout reconstruction module is used to reconstruct the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image and the first position information to obtain the ancient mathematics book illustration recognition result.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the method for recognizing illustrations in ancient mathematics books as described in any one of claims 1 to 8 when executing the computer program stored in the memory.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for recognizing illustrations in ancient mathematics books according to any one of claims 1 to 8 are implemented.

Images