A mathematical ancient book illustration identification method and device, electronic equipment and storage medium

Abstract

The application discloses a mathematical ancient book illustration identification method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: obtaining a mathematical ancient book illustration; dividing the mathematical ancient book illustration into three parts of a text area, a formula area and an image area, and recording position information of the three parts as first position information; performing OCR identification on the text area, formula restoration on the formula area, and image repair on the image area; performing page reconstruction on the mathematical ancient book illustration according to the OCR-identified text, the restored formula, the repaired image and the first position information, to obtain a mathematical ancient book illustration identification result, and effectively identify and digitize the mathematical ancient book illustration.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of digitalization of ancient books, and particularly relates to a mathematical ancient book illustration identification method and device, electronic equipment and a storage medium. BACKGROUND

[0002] As an important part of cultural heritage, ancient books can be long-term preserved and inherited by applying digitalization technology, and physical damage such as wear and tear, creases, etc. can be avoided, and the loss caused by borrowing and reading can be reduced.

[0003] In related technologies, an intelligent OCR (image text recognition) system for ancient books based on machine learning technology can process ancient book images of various types of versions such as engraved edition and handwritten edition, has the ability of intelligent page analysis, intelligent component labeling and output of labeling results, and can better process regular ancient book pages and special ancient book pages containing double-line marginal notes, marginal notes, interlineal notes and tables, etc.

[0004] Different from ancient book images of text type, mathematical ancient book illustrations contain a large number of mathematical symbols, formulas and charts. Although existing ancient book OCR technology can recognize ordinary text, it often causes recognition errors or cannot recognize when processing these complex illustrations containing mathematical elements. Since the digitalization processing and analysis of mathematical ancient books are of great significance for in-depth understanding of the development history of ancient mathematics, the evolution of mathematical thought and the cross-fusion of mathematics and other disciplines, there is an urgent need for a scheme capable of effectively identifying and digitizing mathematical ancient book illustrations. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the application provides a mathematical ancient book illustration identification method, device, electronic equipment and storage medium.

[0006] The technical problem to be solved by the application is solved by the following technical scheme:

[0007] In a first aspect, the application provides a mathematical ancient book illustration identification method, comprising:

[0008] Obtaining a mathematical ancient book illustration;

[0009] Dividing the mathematical ancient book illustration into three parts of a text region, a formula region and an image region, and recording the position information of the three parts as first position information;

[0010] Performing OCR identification on the text region, formula restoration on the formula region, and repair on the image region;

[0011] According to the text recognized by the OCR, the restored formula, the repaired image, and the first position information, a layout reconstruction is performed on the mathematical ancient book illustration, as a mathematical ancient book illustration recognition result.

[0012] Optionally, the splitting the mathematical ancient book illustration into three parts of a text region, a formula region, and an image region comprises:

[0013] The mathematical ancient book illustration is input into a pre-trained mathematical ancient book illustration splitting model, and the mathematical ancient book illustration is split into three parts of a text region, a formula region, and an image region by using the mathematical ancient book illustration splitting model.

[0014] The mathematical ancient book illustration splitting model is trained based on a mathematical ancient book illustration library and annotation information of each sample mathematical ancient book illustration contained in the mathematical ancient book illustration library. The annotation information is information indicating a text region, a formula region, and an image region in a sample mathematical ancient book illustration.

[0015] Optionally, the formula restoration in the formula region comprises:

[0016] Each character in the formula is identified to correspond to a digital code, and position information of each character relative to a reference symbol in the formula is determined as second position information.

[0017] An initial restored formula is constructed according to the digital code and the second position information.

[0018] The initial restored formula is corrected in relation to mathematical semantics to obtain a restored formula.

[0019] Optionally, the correction of the initial restored formula in relation to mathematical semantics to obtain a restored formula comprises:

[0020] A formula template to which the initial restored formula conforms is determined.

[0021] A formula checker corresponding to the formula template is used to check whether an error is contained in the initial restored formula and a position of the error, to obtain a checking result.

[0022] The initial restored formula is corrected according to the checking result and context information contained in the initial restored formula.

[0023] Optionally, the determination of the formula template to which the initial restored formula conforms comprises:

[0024] A formula code vector is constructed according to the digital code and the second position information.

[0025] The formula template to which the initial restored formula conforms is determined according to the formula code vector.

[0026] Optionally, the formula corresponding to the formula template is used to check whether the initial restored formula contains errors and the positions of the errors, and a checking result is obtained, including:

[0027] The formula encoding vector is input into the formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains errors and the positions of the errors as a checking result.

[0028] The formula checker is trained based on a plurality of positive samples and a plurality of negative samples; the positive samples are obtained by encoding sample formulas conforming to the formula template, and the negative samples are obtained by encoding sample formulas conforming to the formula template after introducing local errors.

[0029] Optionally, the context information includes context information extracted according to the formula encoding vector.

[0030] In a second aspect, the present application provides a mathematical ancient book illustration recognition device, comprising:

[0031] An acquisition module is configured to acquire a mathematical ancient book illustration.

[0032] A segmentation module is configured to segment the mathematical ancient book illustration into three parts of a text region, a formula region and an image region, and record position information of the three parts as first position information.

[0033] An OCR recognition module is configured to perform OCR recognition on the text region.

[0034] A formula restoration module is configured to perform formula restoration on the formula region.

[0035] An image repair module is configured to repair the image region.

[0036] A layout reconstruction module is configured to perform layout reconstruction on the mathematical ancient book illustration according to the recognized text, the restored formula, the repaired image and the first position information, as a mathematical ancient book illustration recognition result.

[0037] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0038] The memory is configured to store a computer program.

[0039] The processor is configured to execute the computer program stored on the memory, so as to realize the steps of any one of the mathematical ancient book illustration recognition methods.

[0040] In a fourth aspect, the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program, when executed by a processor, implements the steps of any one of the above-mentioned mathematical ancient book illustration recognition methods.

[0041] The mathematical ancient book illustration recognition method provided by the present application improves the recognition ability for mathematical ancient book illustrations by segmenting the mathematical ancient book illustrations into three parts of a text region, a formula region and an image region, and processing the three parts respectively, and in particular, the formula contained in the formula region is separately restored, thereby realizing effective recognition and digitization of mathematical ancient book illustrations.

[0042] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of a mathematical ancient book illustration recognition method provided by an embodiment of the present application;

[0044] Figure 2 is a structural schematic diagram of a mathematical ancient book illustration segmentation model provided by an embodiment of the present application;

[0045] Figure 3 is a structural schematic diagram of a mathematical ancient book illustration recognition device provided by an embodiment of the present application;

[0046] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0048] In order to realize effective recognition and digitization of mathematical ancient book illustrations, an embodiment of the present application provides a mathematical ancient book illustration recognition method, as shown in Figure 1 The method comprises the following steps:

[0049] S10, obtaining a mathematical ancient book illustration.

[0050] S20, segmenting the mathematical ancient book illustration into three parts of a text region, a formula region and an image region, and recording the position information of the three parts respectively as first position information.

[0051] Here, the mathematical ancient book illustration is segmented into three parts of a text region, a formula region and an image region, which can be realized by image segmentation.

[0052] Exemplarily, the mathematical ancient book illustration can be input into a pre-trained mathematical ancient book illustration segmentation model, and the mathematical ancient book illustration is segmented into three parts of a text region, a formula region and an image region by using the mathematical ancient book illustration segmentation model.

[0053] The mathematical ancient book illustration segmentation model is trained based on a mathematical ancient book illustration library and annotation information of each sample mathematical ancient book illustration contained in the mathematical ancient book illustration library. The annotation information is information indicating the text region, the formula region and the image region in the sample mathematical ancient book illustration, and specifically is information of marking a category (text, formula, image or background) corresponding to each pixel point in the sample mathematical ancient book illustration. In actual training of the mathematical ancient book illustration segmentation model, a data set used can be composed of real data and synthetic data. The real data is a real mathematical ancient book illustration, and the synthetic data can be obtained by synthesizing a text corpus and a target scene picture, for example, by migrating the text corpus to a specified background template through text style migration.

[0054] The mathematical ancient book illustration segmentation model can be implemented by using a neural network of a yolo framework. For example, yolov8 is taken as a baseline model, and Conv2d in a Conv module in a skeleton part of the yolov8 is replaced by a linear mapping layer GhostNeT to obtain a mathematical ancient book illustration segmentation model. Here, the GhostNeT can generate more feature maps with fewer parameters. Alternatively, a neural network model as shown in the figure can be used as the mathematical ancient book illustration segmentation model. Figure 2 Figure 2 In the formula, ACNet is an attention convolution binary neural tree model for fine-grained image classification, Conv represents convolution, Conv2d represents two-dimensional convolution, and FC represents full connection.

[0055] In training the mathematical ancient book illustration segmentation model, the sample mathematical ancient book illustration is input into the mathematical ancient book illustration segmentation model in training, and the model predicts the positions and categories of the text region, the formula region and the image region in the sample mathematical ancient book illustration by performing image segmentation on the sample mathematical ancient book illustration. Then, a loss value is calculated by using a loss function to evaluate the difference between the current predicted segmentation result and the annotation information, and the training is continued until the loss value converges or a sufficient number of training times is reached, that is, the training is ended, and the trained mathematical ancient book illustration segmentation model is obtained.

[0056] ​Regarding the loss function described above, the loss of the classification prediction accuracy of the three regions can be evaluated using CLS Loss (Classification Loss), and the loss of the segmentation accuracy of the three regions can be evaluated in combination with DFL (Distribution Focal Loss) loss. Alternatively, ALRP Loss (Average Localization Recall Precision Loss) can also be directly used as the loss function described above.

[0057] S30, performing OCR recognition on the text region, formula restoration on the formula region, and image repair on the image region.

[0058] In this step S30, the OCR recognition of the text region can be implemented using PaddleOCR. PaddleOCR is an open-source OCR (Optical Character Recognition) tool based on the PaddlePaddle deep learning framework. In addition, a DM-Router (DomainMLP Router) module can be added to the EAST text detection module of PaddleOCR, so as to divide the chapter or some image-attached text part in the mathematical ancient book illustration using DM-Router, thereby improving the recognition ability of the ancient book text.

[0059] In this step S30, the formula restoration of the formula region can include the following steps:

[0060] (1) identifying the digital code corresponding to each character in the formula, and determining the position information of each character relative to the reference symbol in the formula as the second position information.

[0061] Specifically, the image of each character is extracted from the formula region, and the digital code corresponding to the character is determined using a character recognition method according to the image. For common symbols, their digital codes are the codes uniquely corresponding to them in GB2312; for symbols unique to mathematical ancient books, they can be added to the positions not coded in GB2312 in advance, so that these symbols also have unique digital codes.

[0062] As to the second position information extraction, first, the reference symbol is determined from the formula, which can be the leftmost symbol, or the middle symbol, or the symbol with the largest image area, which is all acceptable, as long as the rule of determining the reference symbol is consistent each time. Then, according to the absolute position of each symbol, the relative position relationship with the reference symbol is determined, so as to represent the relative position relationship between the symbols and the reference symbol by using the relative position relationship, so as to facilitate the subsequent use of the relative position relationship to obtain the logical order of the symbols in the formula. Among them, the absolute position information of the symbol is also the area where the image of the symbol is located, which can be represented by the pixel coordinates of the upper left corner and the lower right corner of the image. The second position information of the symbol is the absolute position of the symbol relative to the absolute position of the reference symbol. For example, assuming that the absolute position information of the reference symbol includes [x1, y1, x2, y2], that is, the pixel coordinates of the upper left corner and the lower right corner of the image of the reference symbol are (x1, y1) and (x2, y2) respectively, if the absolute position information of a symbol includes [x3, y3, x4, y4], the second position information of the symbol can be represented as [D1 = x3 - x1, D2 = y3 - y1, D3 = x3 - x2, D4 = y4 - y2].

[0063] (2) Constructing an initial restored formula according to the digital encoding of each symbol and the second position information.

[0064] Specifically, according to the second position information of each character, the characters corresponding to the digital encoding are combined together, and the initial restored formula is obtained.

[0065] It can be understood that for the well-preserved and clear image of the mathematical ancient book illustration, the identification accuracy can be basically ensured in the step of identifying the digital encoding corresponding to the characters in the formula, so that the initial restored formula can be directly used as the restored formula. For the mathematical ancient book illustration with unclear image, the incorrect digital encoding may be identified due to unclear image, so that the initial restored formula needs to be checked and corrected, which will be described in the subsequent steps.

[0066] (3) Correcting the initial restored formula according to the mathematical semantics to obtain the restored formula.

[0067] Here, the initial restored formula is corrected according to the mathematical semantics to obtain the restored formula, which includes:

[0068] (3-1) Determining the formula template to which the initial restored formula conforms.

[0069] Specifically, a formula coding vector is constructed according to the digital coding and the second position information; for example, the arrangement order of each symbol can be determined by referring to the second position information of each symbol, so that the digital coding and the second position information of each symbol are combined together according to a certain ordering rule, and the formula coding vector is obtained. For example, in the formula coding vector, the first half is the digital coding of each character in turn, and the second half is the second position information of the character corresponding to each digital coding in the first half. Then, the initial restored formula is determined to conform to the formula template according to the formula coding vector, which can be specifically realized by a method of calculating vector similarity. It can be understood that the formula coding vector of each formula template needs to be constructed by using the same coding mode in advance.

[0070] (3-2) The initial restored formula is checked by using the formula checker corresponding to the formula template to determine whether the initial restored formula contains errors and the position of the errors, and a checking result is obtained.

[0071] Specifically, the formula coding vector is input into the formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains errors and the position of the errors as the checking result.

[0072] The formula checker can be obtained by training an NLP (natural language processing) large model. The formula checker can be trained based on a plurality of positive samples and a plurality of negative samples; the positive samples are obtained by encoding sample formulas conforming to the formula template, and the negative samples are obtained by encoding sample formulas conforming to the formula template after introducing local errors. Here, the positive samples and the negative samples are also provided with annotation information, which specifically includes indication information of whether the sample contains errors, and the position of the errors when the sample contains errors. For the negative samples, the position of the second part of the errors is known, and for the positive samples, the position of the second part of the errors is empty.

[0073] It is worth mentioning that, in the embodiment of the application, different formula checkers are trained for different formula templates, which reduces the training difficulty on the one hand, and also improves the detection accuracy of the formula checker corresponding to each formula template on the other hand.

[0074] (3-3) The initial restored formula is corrected according to the checking result and the context information contained in the initial restored formula.

[0075] Here, the context information contained in the initial restored formula includes the context information extracted according to the formula coding vector. For example, according to the checking result, the position of the error in the initial restored formula can be determined, and the character similar to the recognized error character can be determined, so that the initial restored formula is corrected according to the digital coding of the adjacent position, the digital coding of the similar character, and the formula semantics of the formula template itself, and a restored formula is obtained.

[0076] In step S30, the image region is repaired. Specifically, a SRCNN (Super Resolution CNN) method based on interpolation super-resolution reconstruction can be used to magnify the image, fit by convolution, reduce the difference between the magnified image and the original image, and improve the resolution of the image. Alternatively, the image region can be repaired by referring to the existing old photo repair tool for high-definition repair of old photo images, which is not described here.

[0077] In S40, the mathematical ancient book illustration is page reconstructed according to the text recognized by OCR, the restored formula, the repaired image, and the first position information, to obtain a mathematical ancient book illustration recognition result.

[0078] Specifically, the text recognized by OCR, the restored formula, and the repaired image are combined into a new mathematical ancient book illustration according to the first position information of each of them, to ensure that the new mathematical ancient book illustration is consistent with the layout of the recognized mathematical ancient book illustration, thereby completing effective recognition of the mathematical ancient book illustration.

[0079] The mathematical ancient book illustration recognition method provided by the embodiment of the present application divides the mathematical ancient book illustration into three parts, i.e., a text region, a formula region, and an image region, and processes the three parts respectively, especially restores the formula contained in the formula region separately, thereby improving the recognition ability of the mathematical ancient book illustration and realizing effective recognition and digitization of the mathematical ancient book illustration.

[0080] After the mathematical ancient book illustration is digitized by using the mathematical ancient book illustration recognition method provided by the embodiment of the present application, scholars, students, and ordinary readers can conveniently access and study the mathematical ancient book illustration at any time and in any place, the risk of damage or loss of the ancient book due to time and natural factors can be avoided, the method is helpful for promoting academic research, education inheritance, and cultural exchange in the field of mathematics, and the formula in the ancient book can be understood and specific types of formulas can be found.

[0081] The method provided by the embodiment of the present application can be applied to an electronic device. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, or the like. Herein, any electronic device that can implement the present application belongs to the protection scope of the present application.

[0082] Corresponding to the above-mentioned mathematical ancient book illustration recognition method, the embodiment of the present application further provides a mathematical ancient book illustration recognition device, as shown in Figure 3 The device comprises:

[0083] The acquisition module 301 is configured to acquire a mathematical ancient book illustration.

[0084] The segmentation module 302 is configured to segment the mathematical ancient book illustration into a text region, a formula region and an image region, and record position information of the three regions as first position information.

[0085] The OCR recognition module 303 is configured to perform OCR recognition on the text region.

[0086] The formula restoration module 304 is configured to perform formula restoration on the formula region.

[0087] The image restoration module 305 is configured to perform image restoration on the image region.

[0088] The layout reconstruction module 306 is configured to perform layout reconstruction on the mathematical ancient book illustration according to the recognized text, the restored formula, the restored image and the first position information, to obtain a mathematical ancient book illustration recognition result.

[0089] Optionally, the segmentation module 302 is specifically configured to:

[0090] input the mathematical ancient book illustration into a pre-trained mathematical ancient book illustration segmentation model, and segment the mathematical ancient book illustration into the text region, the formula region and the image region by using the mathematical ancient book illustration segmentation model.

[0091] The mathematical ancient book illustration segmentation model is trained based on a mathematical ancient book illustration library and annotation information of each sample mathematical ancient book illustration in the mathematical ancient book illustration library, and the annotation information is information indicating the text region, the formula region and the image region in the sample mathematical ancient book illustration.

[0092] Optionally, the formula restoration module 304 is specifically configured to:

[0093] identify a digital code corresponding to each character in the formula, and determine position information of each character relative to a reference symbol in the formula as second position information.

[0094] construct an initial restored formula according to the digital code and the second position information.

[0095] perform mathematical semantic related correction on the initial restored formula to obtain the restored formula.

[0096] Optionally, the formula restoration module 304 performs mathematical semantic related correction on the initial restored formula to obtain the restored formula, including:

[0097] determining a formula template to which the initial restored formula conforms;

[0098] checking, by using a formula checker corresponding to the formula template, whether the initial restored formula contains an error and a position of the error to obtain a checking result;

[0099] The initial restoration formula is modified according to the inspection results and the context information contained in the initial restoration formula.

[0100] Optionally, the formula restoration module 304 determines the formula template that the initial restoration formula complies with, including:

[0101] constructing a formula coding vector according to the digital coding and the second position information;

[0102] The formula template that the initial restoration formula conforms to is determined according to the formula encoding vector.

[0103] Optionally, the formula restoration module 304 uses a formula checker corresponding to the formula template to check whether the initial restoration formula contains errors and the location of the errors, and obtains a check result including:

[0104] Inputting the formula encoding vector into the formula checker corresponding to the formula template, so that the formula checker outputs whether the initial restored formula contains errors and the location of the errors as a check result;

[0105] Among them, the formula checker is trained based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding the sample formulas that conform to the formula template, and the negative samples are obtained by encoding the sample formulas that conform to the formula template after introducing local errors.

[0106] Optionally, the above context information includes context information extracted according to the formula encoding vector.

[0107] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0108] Memory 603, used for storing computer programs;

[0109] The processor 601 is configured to implement any of the aforementioned methods for recognizing illustrations in ancient mathematics books when executing the program stored in the memory 603 .

[0110] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used to represent in the figure, but it does not represent that there is only one bus or only one type of bus.

[0111] The communication interface is used for communication between the above electronic device and other devices.

[0112] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0113] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0114] The application further provides a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the method steps of any one of the above-mentioned mathematical ancient book illustration recognition methods are implemented.

[0115] Optionally, the computer readable storage medium can be a Non-Volatile Memory (NVM), for example, at least one disk memory.

[0116] Optionally, the computer readable storage medium can also be at least one storage device located away from the aforementioned processor.

[0117] It should be noted that, for the device / electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0118] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0119] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0120] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "a" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0121] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (devices) and computer program products of the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1means for performing the function specified in the block or blocks.

[0122] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 flow or flows and / or blocks Figure 1 means for performing the function specified in the block or blocks.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 flow or flows and / or blocks Figure 1 means for performing the function specified in the block or blocks.

[0124] The above description is further to combine the specific preferred embodiments of the present application, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can also be made, which should be considered as belonging to the protection scope of the present application.

Claims

1. A method for identifying illustrations in ancient mathematics books, characterized in that: include: Obtain illustrations from ancient mathematics books; Segmenting the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area, and recording position information of each of the three parts as first position information; Performing OCR recognition on the text area, restoring the formula on the formula area, and repairing the image area; Reconstructing the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image, and the first position information to obtain a recognition result of the ancient mathematics book illustrations; The restoring the formula in the formula area includes: Identifying the digital code corresponding to each character in the formula, and determining the position information of each character relative to the reference symbol in the formula as second position information; wherein the digital code of a common symbol is a unique code corresponding thereto in GB2312; the digital code of a symbol unique to ancient mathematical texts is obtained by pre-adding the symbol to a position not encoded in GB2312; Constructing an initial restoration formula according to the digital code and the second position information; The initial restoration formula is modified with mathematical semantics to obtain a restored formula, including: Constructing a formula code vector according to the digital code and the second position information; According to the formula coding vector, the formula template corresponding to the initial restoration formula is determined by calculating the vector similarity; wherein the corresponding formula coding vector is constructed in advance for various formula templates using the same coding method; Inputting a formula encoding vector constructed according to the digital code and the second position information into a formula checker matched to the corresponding formula template, so that the formula checker outputs whether the initial restored formula contains erroneous characters and the positions of the erroneous characters as a check result; the formula checker is obtained by training a large NLP model based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding sample formulas that conform to the formula template, and the negative samples are obtained by encoding sample formulas that conform to the formula template after introducing local errors; wherein different formula checkers are pre-trained for different formula templates so that each formula template has a matching formula checker; The initial restoration formula is modified based on the inspection result and the context information contained in the initial restoration formula; the context information includes: the digital codes of the adjacent positions of the erroneous characters in the initial restoration formula determined based on the inspection result, the digital codes of characters similar to the erroneous characters, and the formula semantics of the formula template itself.

2. The method for recognizing illustrations in ancient mathematics books according to claim 1, characterized in that: The method of dividing the illustrations of the ancient mathematics book into three parts: a text area, a formula area, and an image area includes: Inputting the ancient mathematics book illustration into a pre-trained ancient mathematics book illustration segmentation model, and using the ancient mathematics book illustration segmentation model to segment the ancient mathematics book illustration into three parts: a text area, a formula area, and an image area; Among them, the ancient mathematics book illustration segmentation model is trained based on the ancient mathematics book illustration library and the annotation information of each sample ancient mathematics book illustration contained in the ancient mathematics book illustration library, and the annotation information is information indicating the text area, formula area and image area in the sample ancient mathematics book illustration.

3. A device for recognizing illustrations in ancient mathematics books, characterized in that: include: Acquisition module, used to obtain illustrations from ancient mathematics books; a segmentation module, configured to segment the illustration of the ancient mathematics book into three parts: a text area, a formula area, and an image area, and record position information of each of the three parts as first position information; An OCR recognition module, used for performing OCR recognition on the text area; A formula recovery module, used for recovering the formula in the formula area; An image restoration module, configured to restore the image area; a layout reconstruction module, configured to reconstruct the layout of the ancient mathematics book illustrations according to the OCR-recognized text, the restored formula, the repaired image, and the first position information, to obtain a recognition result of the ancient mathematics book illustrations; The formula recovery module is specifically used to: Identifying the digital code corresponding to each character in the formula, and determining the position information of each character relative to the reference symbol in the formula as second position information; wherein the digital code of a common symbol is a unique code corresponding thereto in GB2312; the digital code of a symbol unique to ancient mathematical texts is obtained by pre-adding the symbol to a position not encoded in GB2312; Constructing an initial restoration formula according to the digital code and the second position information; The initial restoration formula is modified with mathematical semantics to obtain a restored formula, including: Constructing a formula code vector according to the digital code and the second position information; According to the formula coding vector, the formula template corresponding to the initial restoration formula is determined by calculating the vector similarity; wherein the corresponding formula coding vector is constructed in advance for various formula templates using the same coding method; Inputting a formula encoding vector constructed based on the digital code and the second position information into a formula checker matched to the corresponding formula template, so that the formula checker outputs whether the initial restored formula contains errors and the locations of the errors as a check result; the formula checker is obtained by training a large NLP model based on multiple positive samples and multiple negative samples; the positive samples are obtained by encoding sample formulas that conform to the formula template, and the negative samples are obtained by encoding sample formulas that conform to the formula template after introducing local errors; wherein different formula checkers are pre-trained for different formula templates so that each formula template has a matching formula checker; The initial restoration formula is modified based on the inspection result and the context information contained in the initial restoration formula; the context information includes: the digital codes of the adjacent positions of the erroneous characters in the initial restoration formula determined based on the inspection result, the digital codes of characters similar to the erroneous characters, and the formula semantics of the formula template itself.

4. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, communication interface, and memory communicate with each other through a communication bus; Memory for storing computer programs; The processor is configured to implement the steps of the method for recognizing illustrations in ancient mathematics books as described in any one of claims 1 to 2 when executing a computer program stored in a memory.

5. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for recognizing illustrations in ancient mathematical books according to any one of claims 1 to 2 are implemented.

Images