A method and apparatus for knowledge point extraction based on formula recognition

Abstract

This invention provides a method and apparatus for knowledge point extraction based on formula recognition, relating to the field of computer information processing technology. The method includes: detecting formulas in a target image to extract formula regions from the target image; using OCR technology to recognize and integrate the formula regions and text regions outside the formula regions to obtain a text document corresponding to the target image; and searching for knowledge points corresponding to the formulas in the context of the formula regions in the text document to establish a relationship between formulas and knowledge points. This invention improves the efficiency of extracting formula-related knowledge points and significantly reduces labor costs, even if subsequent manual calibration is required.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and in particular to a method and apparatus for extracting knowledge points based on formula recognition. Background Technology

[0002] Mathematical formulas play a crucial role in education, especially in STEM disciplines. Applications such as question bank construction, knowledge graph building, and document digitization all present demands for the collection, processing, and display of formula information. Currently, a common solution for formula processing is to convert them to LaTeX (a typesetting system based on TeX) format strings, which is convenient for storage and processing and supported by many rendering frameworks. However, most educational resources, such as e-books and exam papers, are obtained through scanning and stored as images on computers. Most commercial OCR software currently cannot effectively extract formula information from these images and further convert them to LaTeX format. Traditional formula localization methods analyze document layout, but this only solves certain scenarios and is less effective for inline formulas with more subtle differences.

[0003] Meanwhile, due to a lack of readily available formula data for analysis, the use of formulas in the computer field is still relatively rudimentary, mostly limited to rendering them as images. However, formulas actually possess specific meanings. In education, for example, a single formula may correspond to several knowledge points, and sometimes several paragraphs can be summarized using several formulas. Currently, however, these interconnected knowledge points are mainly annotated manually by experts, resulting in high labor costs and low efficiency. Summary of the Invention

[0004] This invention provides a method and apparatus for extracting knowledge points based on formula recognition, which solves the problem that in the prior art, the related knowledge points in formulas are mainly manually annotated by experts, which is labor-intensive and inefficient.

[0005] This invention provides a knowledge point extraction method based on formula recognition, comprising:

[0006] Detect formulas in the target image to extract the formula region in the target image;

[0007] The formula region and the text region outside the formula region are identified and integrated using OCR technology to obtain the text document corresponding to the target image;

[0008] Search for the knowledge points corresponding to the formulas in the context of the formula area of ​​the text document to establish the association between the formulas and the knowledge points.

[0009] According to the knowledge point extraction method based on formula recognition provided by the present invention, the step of detecting formulas in a target image to extract formula regions in the target image includes:

[0010] The target image is segmented using a multi-scale sliding window.

[0011] Each segmented window subgraph is input into a pre-trained formula detection model and filtered using a non-maximum prediction algorithm to obtain the quasi-formula region of each window subgraph.

[0012] Based on the statistical results of whether the quasi-formula region of each window sub-image and the region where the pixel is located is a quasi-formula region, the formula region of the target image is determined.

[0013] According to the knowledge point extraction method based on formula recognition provided by the present invention, the step of using OCR technology to recognize and integrate the formula region and the text region outside the formula region to obtain the text document corresponding to the target image includes:

[0014] The formula regions in the target image are covered with marker characters that can uniquely identify different formula regions to obtain text regions;

[0015] OCR technology is used to identify the formula region and the text region respectively to obtain the formula string and the text content;

[0016] The formula string and text content are integrated based on the marked characters to obtain the text document.

[0017] According to the knowledge point extraction method based on formula recognition provided by the present invention, the step of integrating the formula string and text content according to the marked characters to obtain the text document includes:

[0018] Locate the position of the marked character in the text content;

[0019] The formula string corresponding to the marked character is inserted at the specified position to obtain the text document.

[0020] According to a knowledge point extraction method based on formula recognition provided by the present invention, the method searches for knowledge points corresponding to formulas in the context of the formula area of ​​a text document to establish a relationship between formulas and knowledge points, including:

[0021] Traverse the text document to obtain document keywords that can represent the content of the text document;

[0022] Use regular expression matching to retrieve pre-formatted statements within the context of a formula;

[0023] The document keywords appearing in the predefined format statements are identified as the knowledge points of the corresponding formulas;

[0024] Establish the relationship between the knowledge points and the formulas.

[0025] According to a knowledge point extraction method based on formula recognition provided by the present invention, the method searches for knowledge points corresponding to formulas in the context of the formula area of ​​a text document to establish a relationship between formulas and knowledge points, including:

[0026] Traverse the text document to obtain document keywords that can represent the content of the text document;

[0027] Based on their distance from the formula, the document keywords appearing within the formula's predetermined context are sorted from closest to furthest.

[0028] The keywords from the first n documents are identified as the knowledge points corresponding to the formulas.

[0029] Establish the relationship between the knowledge points and the formulas.

[0030] According to the knowledge point extraction method based on formula recognition provided by the present invention, after establishing the association between formulas and knowledge points, the method further includes:

[0031] Extract all formulas related to a given knowledge point;

[0032] The formulas were modified according to the unified standards defined by LaTeX rules;

[0033] Extract the multi-layered structure and local features of the formula by using the parentheses within the formula itself;

[0034] Based on character similarity, multi-level structure similarity, and the similarity score of local structure calculation formulas, the judgment result that the formulas belong to the same knowledge point is obtained. The similarity score is calculated as follows:

[0035]

[0036] Where G represents the set of formulas associated with the same knowledge point, Sim s (i,j) is the function for calculating character similarity; Sim c (i,j) is the function for calculating the similarity of multi-layer structures; Sim f (i,j) is the function for calculating local feature similarity, Size(G) is the number of formulas in G, i and j represent different formulas in G, and score i,i∈G This represents the average similarity score between formula i and other formulas in G.

[0037] The present invention also provides a knowledge point extraction device based on formula recognition, comprising:

[0038] The formula detection module is used to detect formulas in the target image in order to extract the formula region in the target image;

[0039] The OCR recognition module is used to recognize and integrate the formula region and the text region outside the formula region using OCR technology to obtain the text document corresponding to the target image.

[0040] The knowledge point search module is used to search for knowledge points corresponding to formulas in the context of the formula area of ​​the text document, so as to establish the association between formulas and knowledge points.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the knowledge point extraction method based on formula recognition as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge point extraction method based on formula recognition as described above.

[0043] The knowledge point extraction method and apparatus based on formula recognition provided by this invention detects the formula region in the target image, identifies the text region and the formula region respectively, and then integrates the two parts into a text document. The knowledge points corresponding to the formula are searched in the context of the formula region in the text document, and the formula is automatically associated with the corresponding knowledge points, thereby improving the extraction efficiency of formula-associated knowledge points. Even if manual calibration is required later, it greatly reduces the labor cost. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the knowledge point extraction method based on formula recognition provided by the present invention;

[0046] Figure 2 This is a schematic diagram of the knowledge point extraction device based on formula recognition provided by the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] The knowledge point extraction method based on formula recognition in this embodiment of the invention is as follows: Figure 1 As shown, it includes:

[0050] Step S110: Detect formulas in the target image to extract formula regions. The target image only requires clear text; it can be an electronic textbook page or exam paper, a screenshot of online exam questions, or a scanned image of a printed textbook page, etc. This step extracts all regions containing formulas from the target image; each formula region is the image of the area containing the formula.

[0051] Step S120: OCR technology is used to recognize and integrate the formula region and the text region outside the formula region to obtain the text document corresponding to the target image. After the formula region is determined, the remaining region is the text region. OCR is used to recognize these two regions to obtain the text document corresponding to the target image.

[0052] Step S130: Search for knowledge points corresponding to the formula in the context of the formula area of ​​the text document to establish the association between the formula and the knowledge points. After the target image is recognized as a text document by OCR, the formula can be treated as a special character and traditional text processing methods can be applied. Since the knowledge points associated with the formula are usually in the context of the formula, the knowledge points associated with the formula can be found by performing contextual retrieval in the text document to establish the association between the formula and the knowledge points, thereby extracting the knowledge information of the formula.

[0053] The knowledge point extraction method based on formula recognition provided in this embodiment detects formula regions in the target image, identifies the text area and formula area separately, and then integrates the two parts into a text document. It then searches for knowledge points corresponding to the formulas within the context of the formula areas in the text document, automatically associating formulas with their corresponding knowledge points. This improves the extraction efficiency of formula-associated knowledge points and significantly reduces labor costs, even if subsequent manual calibration is required. This method is particularly applicable to text images in the educational field.

[0054] In this embodiment, step S110 includes:

[0055] The target image is segmented using a multi-scale sliding window. Specifically, the target image can be sampled using sliding windows of various scales, such as 800×800 pixels and 1200×1200 pixels. The sliding distance is 1 / 10 of the side length each time. The window size can be adjusted according to the actual size of the image. Because the sliding window has multiple scales, there will be overlap between the resulting window sub-images.

[0056] Each segmented window sub-image is input into a pre-trained formula detection model, and a non-maximum suppression algorithm is used for filtering to obtain the quasi-formula region for each window sub-image. Both the quasi-formula region and the formula region are rectangular areas defined by the coordinates of their top-left and bottom-right corners. The images of the quasi-formula region and the formula region can be obtained using the coordinates of these two corners. Specifically, the formula detection model is the SSD (Single Shot MultiBox Detector) model, trained using over 400 pages of manually annotated textbook images containing formula location markers. This formula detection model is used to calculate the IOU (Intersection over Union) of adjacent candidate boxes, and a non-maximum suppression algorithm is used to filter the candidate boxes to obtain the quasi-formula region. Based on the characteristics of formulas, in this embodiment, the default box of the SSD architecture is set to an aspect ratio of 3:1 or higher.

[0057] The formula region of the target image is determined based on the statistical results of whether the quasi-formula region of each window sub-image and the region where a pixel is located is a quasi-formula region. Because there are multiple scale windows, there is overlap between the generated window sub-images, and the formula detection model will also output multiple detection boxes, i.e., multiple quasi-formula regions, which will also overlap. In addition to filtering by non-maximum suppression algorithm, the formula region is also optimized by the statistical results of whether the region where a pixel is located is a quasi-formula region. Specifically, the number of times the region where a pixel is located is considered to be a quasi-formula region in different window sub-images is counted. When the number exceeds a preset threshold (e.g., 30), the pixel is retained. Finally, the retained pixel is corrected into a rectangular formula region, thereby determining the formula region of the target image.

[0058] The defined formula region of the target image may exhibit issues such as: a small formula region with edges extending beyond the characters in the formula; or a large formula region with edges too far removed from the characters in the formula, or even overlap or enclosing problems. Therefore, fine-tuning of the formula region is necessary. Specifically, the fine-tuning steps include:

[0059] Based on the connectivity of the characters, the formula area is expanded or contracted. When the edge of the formula area passes through a character in the formula, the formula area is expanded to the nearest blank space to the character that was passed through. When the distance between the edge of the formula area and the character in the formula is too large, the formula area is contracted.

[0060] To obtain a more accurate formula region, split the formula region in blank spaces exceeding the blank threshold, or merge adjacent formula regions on adjacent text lines.

[0061] For example, in science and engineering disciplines, there may be two formulas separated by only a small amount of text, that is, the text does not run through the entire line. Formulas are generally centered, and there will be blank spaces between the two formulas. In this case, the formula detection model may only be able to detect one formula area. It can be split into two formula areas by judging the larger line spacing in the middle compared to a typical multi-line formula.

[0062] Whitespace thresholds are divided into interline whitespace thresholds and intraline whitespace thresholds. The appropriate threshold values ​​are adjusted based on the specific circumstances, taking into account factors such as multi-line formulas and line spacing. For example, in commonly used textbooks, the line spacing for multi-line formulas is typically within 25 pixels, and a single line of text is 35 pixels high. Therefore, the interline whitespace threshold can be chosen between 30 and 55 pixels, such as 40. The spacing between characters within the same line needs to be less than the distance between the formula and its conditions, usually 80 pixels. In this case, the intraline whitespace threshold can be set to 90 pixels to avoid incorrectly separating the formula and its conditions into two separate formula areas.

[0063] In this embodiment, step S120 includes:

[0064] The formula regions in the target image are covered with marker characters that can uniquely identify different formula regions to obtain the text regions. In this step, the formula regions and text regions are separated based on the formula detection results. After extracting the formula regions, a covering process needs to be performed on the target image to facilitate the subsequent integration of the two parts. Markers need to be added to the formula regions. The requirements for the marker characters are as follows:

[0065] 1) It can be successfully recognized by text OCR software;

[0066] 2) It will not cause confusion with characters in the target image;

[0067] 3) It can correspond one-to-one with the formula range.

[0068] For example, choosing to use the "$number$" format as the marker character is less common in general documents; the number in the middle serves as a unique identifier for the formula area. During merging, regular expressions can be used to quickly search for text containing the "$number$" character, thus determining the correct insertion point for the formula.

[0069] OCR technology is used to identify the formula area and the text area respectively to obtain the formula string and the text content. The formula string conforms to the LaTeX format.

[0070] The formula string and text content are integrated based on the marked characters to obtain the text document. Specifically, the position of the marked character in the text content is located; the formula string corresponding to the marked character is inserted into the position to obtain the text document.

[0071] In this embodiment, the formula region and the text region are separated, and OCR technology can be used to recognize them separately. Compared with the overall recognition of the original image, a more appropriate recognition scheme with a higher recognition rate can be selected for different regions, thereby improving the overall recognition rate.

[0072] In this embodiment, step S130 can be implemented in the following two ways:

[0073] Method 1, step S130 includes:

[0074] The text document is traversed to obtain document keywords that represent its content. This can be achieved using methods such as TF-IDF (term frequency–inverse document frequency) and NER (Named Entity Recognition), or by importing keywords from a pre-acquired knowledge base. TF-IDF and NER automatically extract keywords as candidate knowledge points based on term frequency and part-of-speech, allowing for manual noise reduction and selection of the final knowledge points. Manually acquired knowledge bases obtain keywords related to the subject from existing knowledge bases, primarily through manual extraction of keywords from chapter titles, PPT slides, outlines, and other materials.

[0075] The method uses regular expression matching to search for statements in a predefined format within the context of the formula. These predefined statements can be in the format of "xx is defined as", "xx, that is", etc., where "xx" may refer to the specific meaning of the formula or related knowledge points. The regular expression used in this embodiment is "key+'[is defined as::]*$|'+key+'[,,]that is?[::]?$'", where "key" represents the knowledge point. This regular expression can match most potentially related sentence structures without generating too many interfering terms.

[0076] The document keywords appearing in the predefined format statements are identified as the knowledge points of the corresponding formulas.

[0077] Establish the relationship between the knowledge points and the formulas.

[0078] Method 2, step S130 includes:

[0079] Traverse the text document to obtain document keywords that can represent the content of the text document. Specifically, document keywords can be obtained using methods such as TF-IDF and NER, or they can be imported into a knowledge base that has been manually obtained in advance.

[0080] Based on their distance from the formula, the document keywords appearing within the formula's predetermined context are sorted from closest to furthest. For a formula associated with multiple keywords, keywords closer to the formula are more likely to be relevant and therefore ranked higher. "Distance from the formula" can be understood as: within the formula's context, the line containing the formula is the closest to it; the closer the line is to the formula, the closer it is to the formula.

[0081] The keywords from the first n documents are identified as the knowledge points corresponding to the formulas, for example, n can be 1 to 3;

[0082] Establish the relationship between the knowledge points and the formulas.

[0083] In some fields, formulas and keywords have a one-to-many relationship. Using multiple keywords to qualify the formula can effectively improve the accuracy of the formula expression and the success rate of retrieval, and can also reduce noise interference.

[0084] Method 1 is more accurate and easily identifies well-defined knowledge points, but the number is relatively small and it is limited by sentence structure and regular expressions. Method 2 theoretically allows one formula to be associated with multiple keywords, and in practice, many formulas do involve multiple knowledge points; associating multiple keywords can also reduce the impact of noise to some extent. However, current applications primarily consider the accurate one-to-one association between formulas and knowledge points, so only one knowledge point is retained after manual verification. Compared to Method 1, Method 2 has the advantage of obtaining more candidate knowledge points and is simpler to operate.

[0085] Since the main considerations are the accurate one-to-one association between formulas and knowledge points and ensuring the accuracy of the association, both of the above methods require manual verification after establishing the association between formulas and knowledge points, especially the second method. However, compared with the existing technology where the association between formulas and knowledge points mainly relies on manual annotation by experts, this method greatly reduces labor costs and improves efficiency.

[0086] In Method 2, judging the distance between formulas and related knowledge points can introduce noise, in addition to potential errors in the manually acquired knowledge base. Before manual verification, automated methods can be used for initial screening to further reduce labor costs and improve efficiency. The core idea of ​​automatic screening is to compare all formulas related to the same knowledge point, retaining commonalities and eliminating formulas with significant differences. The definition and actual usage of a formula can have significant character differences, and their LaTeX characters are not unique; therefore, simple string matching results cannot accurately reflect the degree of association between them. Based on this, Method 2 has been optimized. After establishing the association between the knowledge points and formulas in Method 2, it also includes:

[0087] Extract all formulas related to a given knowledge point.

[0088] The formulas were modified to conform to the unified standards defined by LaTeX rules. The modifications include: 1. Removing modifiers used to define fonts and character widths, such as "\mathrm{}". 2. Standardizing character formats, removing spaces; for example, parentheses are no longer modified with "\left", and LaTeX-defined characters like "cos" are now uniformly represented as '\cos'.

[0089] Because formulas, in practical use, involve specific values ​​that lead to differences between characters, they should still possess a similar overall structure. Therefore, it is necessary to extract the structural information of the formulas, specifically by examining the formula's parentheses to obtain its multi-layered structure and local features. The outermost layer represents the overall structure, while the structure within the innermost parentheses represents the local features. For example: Formula The overall structure of the formula is as follows: Local features include: t, 2πf c t, this formula has only one layer of structure.

[0090] Based on character similarity, multi-level structure similarity, and the similarity score of local structure calculation formulas, the judgment result that the formulas belong to the same knowledge point is obtained. The similarity score is calculated as follows:

[0091]

[0092] Where G represents the set of formulas associated with the same knowledge point, Sim s (i,j) is the function for calculating character similarity; Sim c (i,j) is the function for calculating the similarity of multi-layer structures; Sim f (i,j) is the function for calculating local feature similarity, Size(G) is the number of formulas in G, i and j represent different formulas in G, and score i,i∈GThis represents the average similarity score between formula i and other formulas in G.

[0093] In this embodiment, for multiple formulas associated with the same knowledge point, each formula is compared with all other formulas, and the average of the results yields a score. Finally, formulas are sorted based on this score; those with a score above 0.5 are highly likely to be correctly associated. Further manual verification is then performed based on this score, with formulas scoring below 0.5 receiving focused verification, which further reduces labor costs.

[0094] In practical applications, the target image contains multiple formulas, and not every formula has a pre-defined format in its context. Therefore, method one is used to establish associations between some knowledge points and formulas, and then method two is used to establish associations between the remaining formulas and unassociated knowledge points as a supplement. Specifically, step S130 includes:

[0095] Traverse the text document to obtain document keywords that can represent the content of the text document. Specifically, document keywords can be obtained using methods such as TF-IDF and NER, or they can be imported into a knowledge base that has been manually obtained in advance.

[0096] The method uses regular expression matching to search for statements in a predefined format within the context of the formula. These predefined statements can be in the format of "xx is defined as", "xx, that is", etc., where "xx" may refer to the specific meaning of the formula or related knowledge points. The regular expression used in this embodiment is "key+'[is defined as::]*$|'+key+'[,,]that is?[::]?$'", where "key" represents the knowledge point. This regular expression can match most potentially related sentence structures without generating too many interfering terms.

[0097] The document keywords appearing in the predefined format statements are identified as the knowledge points of the corresponding formulas.

[0098] For formulas for which no knowledge points were found, the document keywords appearing within the formula's predetermined context are sorted according to their distance from the formula.

[0099] The keywords of the first n documents are identified as the knowledge points of the corresponding formulas, where n ranges from 1 to 3.

[0100] Establish the relationship between the knowledge points and the formulas.

[0101] In the above schemes of Method 1, Method 2, and the combination of Method 1 and Method 2, the manual denoising process involves manually denoising the collected formula association data (which is the noisy knowledge points automatically collected by Method 1, Method 2, or a combination of both) to obtain the formula knowledge association. The steps are as follows:

[0102] Data reading and formula display: The system reads the formula-related data collected in the above steps from the Excel spreadsheet, displays the formulas and knowledge points on the interface, and simultaneously renders LaTeX format formulas in real time, making it convenient for operators to judge the correctness of the data.

[0103] Data modification allows you to change formulas or knowledge points, or add or delete data.

[0104] Save the data and export the modified results to an Excel spreadsheet to obtain the verified formulas and knowledge point association results.

[0105] The knowledge point extraction device based on formula recognition provided by the present invention is described below. The knowledge point extraction device based on formula recognition described below can be referred to in correspondence with the knowledge point extraction method based on formula recognition described above.

[0106] like Figure 2 As shown, the knowledge point extraction device based on formula recognition of the present invention includes:

[0107] Formula detection module 210 is used to detect formulas in the target image in order to extract the formula region in the target image.

[0108] The OCR recognition module 220 is used to recognize and integrate the formula region and the text region outside the formula region using OCR technology to obtain the text document corresponding to the target image.

[0109] The knowledge point search module 230 is used to search for knowledge points corresponding to formulas in the context of the formula area of ​​the text document, so as to establish the association between formulas and knowledge points.

[0110] The knowledge point extraction device based on formula recognition provided by this invention detects formula regions in a target image, identifies both the text area and the formula area separately, and then integrates the two parts into a text document. It then searches for knowledge points corresponding to the formulas within the context of the formula regions in the text document, automatically associating formulas with their corresponding knowledge points. This improves the extraction efficiency of formula-associated knowledge points and significantly reduces labor costs, even if subsequent manual calibration is required. This method is particularly applicable to text images in the educational field.

[0111] Optionally, the formula detection module 210 includes:

[0112] The image segmentation module is used to segment the target image using a multi-scale sliding window.

[0113] The formula detection module is used to input each segmented window subgraph into a pre-trained formula detection model, and filter it using a non-maximum prediction algorithm to obtain the quasi-formula region of each window subgraph.

[0114] The formula region determination module is used to determine the formula region of the target image based on the statistical results of whether the quasi-formula region of each window sub-image and the region where the pixel is located is a quasi-formula region.

[0115] Optionally, the OCR recognition module 220 includes:

[0116] The formula region overlay module is used to overlay the formula region in the target image with a marker character that can uniquely identify different formula regions to obtain a text region.

[0117] The region recognition module is used to identify the formula region and the text region respectively using OCR technology to obtain the formula string and the text content.

[0118] The document integration module is used to integrate the formula string and text content according to the marked characters to obtain the text document.

[0119] Specifically, the document integration module is used to: locate the position of the marked character in the text content; and insert the formula string corresponding to the marked character into the position to obtain the text document.

[0120] Optionally, the knowledge point search module 230 includes:

[0121] The document keyword acquisition module is used to traverse the text document and obtain document keywords that can represent the content of the text document.

[0122] The pre-formatted statement retrieval module is used to retrieve pre-formatted statements within the context of formulas using string regular expression matching.

[0123] The knowledge point identification module is used to identify the document keywords appearing in statements with a predefined format as the knowledge points of the corresponding formulas.

[0124] The association establishment module is used to establish the association between the knowledge points and the formulas.

[0125] Optionally, the knowledge point search module 230 includes:

[0126] The document keyword acquisition module is used to traverse the text document and obtain document keywords that can represent the content of the text document.

[0127] The document keyword sorting module is used to sort the document keywords that appear in the predetermined context of the formula from closest to furthest based on their distance from the formula.

[0128] The knowledge point identification module is used to identify the keywords in the first n documents as the knowledge points corresponding to the formulas.

[0129] The association establishment module is used to establish the association between the knowledge points and the formulas.

[0130] Optionally, the knowledge point search module 230 also includes:

[0131] The related formula extraction module is used to extract all formulas related to a knowledge point.

[0132] The formula correction module is used to correct formulas according to the unified standards defined by LaTeX rules.

[0133] The formula structure extraction module is used to extract the multi-layered structure and local features of a formula by using the parentheses within the formula itself.

[0134] The similarity calculation module is used to calculate the similarity score of formulas based on character similarity, multi-level structure similarity, and local structure, in order to determine whether the formulas belong to the same knowledge point. The similarity score is calculated as follows:

[0135]

[0136] Where G represents the set of formulas associated with the same knowledge point, Sim s (i,j) is the function for calculating character similarity; Sim c (i,j) is the function for calculating the similarity of multi-layer structures; Sim f (i,j) is the similarity function for calculating the structure (i.e., local features), Size(G) is the number of formulas in G, i,j represent different formulas in G, and score i,i∈G This represents the average similarity score between formula i and other formulas in G.

[0137] Figure 3 A schematic diagram of the physical structure of an electronic device is provided. This electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a knowledge point extraction method based on formula recognition. This method includes:

[0138] Detect formulas in the target image to extract the formula region in the target image;

[0139] The formula region and the text region outside the formula region are identified and integrated using OCR technology to obtain the text document corresponding to the target image;

[0140] Search for the knowledge points corresponding to the formulas in the context of the formula area of ​​the text document to establish the association between the formulas and the knowledge points.

[0141] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the knowledge point extraction method based on formula recognition provided by the above methods, the method comprising:

[0143] Detect formulas in the target image to extract the formula region in the target image;

[0144] The formula region and the text region outside the formula region are identified and integrated using OCR technology to obtain the text document corresponding to the target image;

[0145] Search for the knowledge points corresponding to the formulas in the context of the formula area of ​​the text document to establish the association between the formulas and the knowledge points.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the knowledge point extraction method based on formula recognition provided by the above methods, the method comprising:

[0147] Detect formulas in the target image to extract the formula region in the target image;

[0148] The formula region and the text region outside the formula region are identified and integrated using OCR technology to obtain the text document corresponding to the target image;

[0149] Search for the knowledge points corresponding to the formulas in the context of the formula area of ​​the text document to establish the association between the formulas and the knowledge points.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge point extraction method based on formula recognition, characterized in that, include: Detect formulas in the target image to extract the formula region in the target image; The formula region and the text region outside the formula region are identified and integrated using OCR technology to obtain the text document corresponding to the target image; Search for the knowledge points corresponding to the formulas in the context of the formula area of ​​the text document to establish the association between the formulas and the knowledge points; After establishing the relationship between formulas and knowledge points, the following is also included: Extract all formulas related to a given knowledge point; The formulas were modified according to the unified standards defined by LaTeX rules; Extract the multi-layered structure and local features of the formula by using the parentheses within the formula itself. The structure of the outermost parentheses is the overall structure, and the structure in the innermost parentheses is the local feature. The similarity score is calculated based on character similarity, multi-layer structure similarity, and local feature similarity to determine whether the formulas belong to the same knowledge point. The similarity score is calculated as follows: ； in, G This represents a set of formulas that are associated with the same knowledge point. A function for calculating character similarity; To calculate the similarity function for multi-layer structures; To calculate the local feature similarity function, for G The number of formulas in i , j express G Different formulas in China Formula i and G The average of the similarity scores of other formulas.

2. The knowledge point extraction method based on formula recognition according to claim 1, characterized in that, The process of detecting formulas in the target image to extract formula regions from the target image includes: A multi-scale sliding window is used to segment the target image; Each segmented window subgraph is input into a pre-trained formula detection model and filtered using a non-maximum prediction algorithm to obtain the quasi-formula region of each window subgraph. Based on the statistical results of whether the quasi-formula region of each window sub-image and the region where the pixel is located is a quasi-formula region, the formula region of the target image is determined.

3. The knowledge point extraction method based on formula recognition according to claim 1, characterized in that, The step of using OCR technology to recognize and integrate the formula region and the text region outside the formula region to obtain the text document corresponding to the target image includes: The formula regions in the target image are covered with marker characters that can uniquely identify different formula regions to obtain text regions; OCR technology is used to identify the formula region and the text region respectively to obtain the formula string and the text content; The formula string and text content are integrated based on the marked characters to obtain the text document.

4. The knowledge point extraction method based on formula recognition according to claim 3, characterized in that, The process of integrating the formula string and text content based on the marked characters to obtain the text document includes: Locate the position of the marked character in the text content; The formula string corresponding to the marked character is inserted at the specified position to obtain the text document.

5. The knowledge point extraction method based on formula recognition according to any one of claims 1 to 4, characterized in that, In the context of the formula area of ​​the text document, search for the knowledge points corresponding to the formulas to establish the association between formulas and knowledge points, including: Traverse the text document to obtain document keywords that can represent the content of the text document; Use regular expression matching to retrieve pre-formatted statements within the context of a formula; The document keywords appearing in the predefined format statements are identified as the knowledge points of the corresponding formulas; Establish the relationship between the knowledge points and the formulas.

6. The knowledge point extraction method based on formula recognition according to any one of claims 1 to 4, characterized in that, In the context of the formula area of ​​the text document, search for the knowledge points corresponding to the formulas to establish the association between formulas and knowledge points, including: Traverse the text document to obtain document keywords that can represent the content of the text document; Based on their distance from the formula, the document keywords appearing within the formula's predetermined context are sorted from closest to furthest. The keywords from the first n documents are identified as the knowledge points corresponding to the formulas. Establish the relationship between the knowledge points and the formulas.

7. A knowledge point extraction device based on formula recognition, characterized in that, include: The formula detection module is used to detect formulas in the target image in order to extract the formula region in the target image; The OCR recognition module is used to recognize and integrate the formula region and the text region outside the formula region using OCR technology to obtain the text document corresponding to the target image; The knowledge point search module is used to search for knowledge points corresponding to formulas in the context of the formula area of ​​the text document, so as to establish the association between formulas and knowledge points. The knowledge point search module also includes: The related formula extraction module is used to extract all formulas related to a knowledge point. The formula correction module is used to correct formulas according to the unified standards defined by LaTeX rules; The formula structure extraction module is used to extract the multi-layer structure and local features of the formula through its own brackets. The structure of the outermost bracket is the overall structure, and the structure in the innermost bracket is the local feature. The similarity calculation module is used to calculate the similarity score of formulas based on character similarity, multi-level structure similarity, and local features, in order to determine whether the formulas belong to the same knowledge point. The similarity score is calculated as follows: ； in, G This represents a set of formulas that are associated with the same knowledge point. A function for calculating character similarity; To calculate the similarity function for multi-layer structures; To calculate the similarity function of structures (i.e., local features), for G The number of formulas in i , j express G Different formulas in China Formula i and G The average of the similarity scores of other formulas.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the knowledge point extraction method based on formula recognition as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the knowledge point extraction method based on formula recognition as described in any one of claims 1 to 6.

Images