Formula analysis and retrieval method and system based on large model

Abstract

The invention discloses a formula analysis and retrieval method and system based on a large model. The method comprises the steps of S10, formula coding analysis, S20, formula analysis large model training and S30, coding formula retrieval output. According to the method, the formula of the picture format, the annotation content of the formula and the function description content of the formula can be subjected to code analysis, and the comprehensiveness and the accuracy of analysis of the formula of the picture format, the annotation content of the formula and the function description content are improved; coded retrieval of formulas of picture formats, annotations of the formulas and function description contents of the formulas can be achieved. The method has the advantage that the retrieval result is convenient to copy and extract.

Description

Technical Field

[0001] The present invention relates to the field of building large models, and particularly relates to a method and system for formula parsing and retrieval based on large models. Background Art

[0002] Large language models, abbreviated as large models, have good effects in text retrieval, but have poor effects in formula parsing and retrieval. Especially in the construction industry, there are a large number of long, difficult, and complex calculation formulas, and existing large models cannot accurately parse and retrieve them. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for formula parsing and retrieval based on large models to solve the problem that large models cannot parse and retrieve formulas.

[0004] To solve the above technical problems, the present invention provides a method for formula parsing and retrieval based on large models, including:

[0005] Step S10, formula coding parsing, including:

[0006] Step S11, detecting formulas and their annotation areas and function description areas in the text object;

[0007] Step S12, recognizing and parsing the text content of the formula and its annotation area and function description area through text recognition technology;

[0008] Step S13, converting the formula and its annotation content in picture format into character code format to form the parsing of the coded formula and its annotation content;

[0009] Step S14, summarizing and extracting the function description content of the formula, and establishing a one-to-one matching relationship parsing between the coded formula and its annotation content and the function description content;

[0010] Step S20, formula parsing large model training:

[0011] Through the large model, summarize, split, or extract the function description content of each coded formula to form questions for querying the corresponding formula, use each coded formula and its annotation content and function description content as answers, and splice the questions for querying each coded formula and their answers to form more than one set of question-and-answer pairs; use the whole set of question-and-answer pairs as training data to train the large model to form a formula parsing large model;

[0012] Step S30, coded formula retrieval output:

[0013] Output the coded formula and its annotation content and function description content corresponding to the user's retrieved formula question through the formula parsing large model.

[0014] Furthermore, the method for formula parsing and retrieval based on a large model provided by the present invention further includes:

[0015] Step S40, inferring and outputting the retrieval result, including:

[0016] Step S41, when the user conducts a retrieval formula question, the formula parsing large model associates and paraphrases the retrieval formula question to form multiple questions and outputs the coded formula corresponding to the retrieval formula question and its annotation content and function description content.

[0017] Furthermore, in the method for formula parsing and retrieval based on a large model provided by the present invention, in the step S40, it further includes:

[0018] Step S42, the formula parsing large model summarizes the multiple questions and outputs the coded formula corresponding to the retrieval formula question and its annotation content and function description content.

[0019] Furthermore, the method for formula parsing and retrieval based on a large model provided by the present invention further includes:

[0020] Step S50, rendering the output result: performing vector rendering on the coded formula output by the formula parsing large model and its annotation content and function description content.

[0021] Furthermore, the method for formula parsing and retrieval based on a large model provided by the present invention performs low-rank fine-tuning on the formula parsing large model.

[0022] To solve the above technical problems, the present invention also provides a formula parsing and retrieval system based on a large model, including:

[0023] A server, which has a formula parsing subsystem and a large model training subsystem, wherein:

[0024] The formula parsing subsystem is used to perform text content recognition and parsing on the formula, its annotation area, and function description area in the text object containing the formula, convert the formula and its annotation content in the picture format into the coded formula and its annotation content, and summarize and extract the function description content of the formula, and establish a one-to-one matching relationship between the coded formula and its annotation content and the function description content;

[0025] The large model training subsystem is used to summarize, split, or extract the function description content of each coded formula through a general large model to form questions for asking about the corresponding formula, use each coded formula and its annotation content and function description content as answers, splice the questions for asking about each coded formula and their answers to form more than one group of question-and-answer pairs; use the whole group of question-and-answer pairs as training data to train the large model to form a formula parsing large model;

[0026] A retrieval terminal, which is network - connected to the server, is used to retrieve formula problems. When retrieving formula problems, it connects to the server through the network, calls the formula parsing large model therein, outputs the coded formula corresponding to the retrieved formula problem, its annotation content, and function description content, and downloads them to the retrieval terminal for display.

[0027] Furthermore, in the formula parsing and retrieval system based on a large model provided by the present invention, the server also establishes a formula parsing large model inference subsystem. Through the large model inference subsystem, the retrieved formula problem of the retrieval terminal is associated and paraphrased to form multiple problems, and the multiple problems are summarized to output the coded formula corresponding to the retrieved formula problem, its annotation content, and function description content.

[0028] Furthermore, in the formula parsing and retrieval system based on a large model provided by the present invention, the server includes a first server and a second server that are network - interconnected. The formula parsing subsystem and the large model training subsystem are established on the second server. The formula parsing large model and the formula parsing large model inference subsystem formed through the large model training subsystem are deployed on the first server, and the retrieval terminal is network - connected to the first server.

[0029] Furthermore, in the formula parsing and retrieval system based on a large model provided by the present invention, the retrieval terminal is a computer or a smart phone.

[0030] Furthermore, in the formula parsing and retrieval system based on a large model provided by the present invention, the retrieval terminal performs vector rendering on the displayed coded formula, its annotation content, and function description content.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] The formula parsing and retrieval method and system based on a large model provided by the present invention perform image format to codified format and text content recognition on the formulas, their annotations, and functional description content of text objects, establish a codified parsing of one-to-one matching relationships between each codified formula and its annotation content and functional description content, thereby performing codified parsing on the formulas, their annotation content, and functional description content in image format, improving the parsing accuracy of the formulas, their annotations, and functional description content in image format; form a question-and-answer for the codified parsed formulas, their annotations, and functional description content through the large model as the training data of the formula parsing large model; output the codified formula and its annotation content and functional description content corresponding to the user's retrieved formula question through the formula parsing large model, thereby realizing the codified retrieval of the formulas, their annotation content, and functional description content in image format. It overcomes the problem that general large models cannot parse and retrieve formulas in image format, especially complex formulas and formulas in the construction industry, and can output the formula in a codified manner, having the advantages of being convenient for copying, extracting the codified formula and its annotation content and functional description content. Brief Description of the Drawings

[0033] Figure 1 is a flowchart of the formula parsing and retrieval method based on a large model;

[0034] Figure 2 is a schematic structural diagram of the formula parsing and retrieval system based on a large model;

[0035] As shown in the figure:

[0036] 100. Formula parsing and retrieval system based on a large model; 110. First server; 120. Second server; 130. Retrieval terminal. Detailed Embodiments

[0037] The present invention will be described in detail below with reference to the accompanying drawings: According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise scales, only for the purpose of facilitating and clearly assisting in explaining the embodiments of the present invention.

[0038] Please refer to Figure 1 , an embodiment of the present invention provides a formula parsing and retrieval method based on a large model, which may include:

[0039] Step S10, codified parsing of the formula. Specifically, it includes:

[0040] Step S11, detect the formula, its annotation area, and functional description area in the text object. The annotation area includes but is not limited to the annotations of constants and variables in the formula.

[0041] Step S12: Use text recognition technology to recognize and parse the text content of the formula, its annotation area, and the function description area. In this step, the main task is to parse the text content.

[0042] Step S13: Convert the formula and its annotation content in picture format into character code format to form the parsing of the coded formula and its annotation content. Since the meanings of letters such as formula constants and variables in the formulas and annotation content in construction manuals, specification documents, professional textbooks and other document materials are embedded in the form of pictures, it is necessary to perform coded parsing on the formula and its annotation content in picture format.

[0043] Step S14: Summarize and extract the function description content of the formula, and establish a one-to-one matching relationship parsing between the coded formula and its annotation content and the function description content.

[0044] Step S20: Train the formula parsing large model. Specifically: Use the large model to summarize, split, or extract the function description content of each coded formula to form questions for querying the corresponding formula, use each coded formula and its annotation content, and function description content as answers, and splice the questions for querying each coded formula and their answers to form more than one set of question-answer pairs; Use the whole set of question-answer pairs as training data to train the large model to form a formula parsing large model. The function description content of the coded formula can be split into several paragraphs, extract keywords, summarize ideas, etc. to form questions, and the answers are uniformly: each coded formula and its annotation content, and function description content. In step S20, it can also include low-rank fine-tuning of the formula parsing large model to improve the accuracy of the coded formula and its annotation content, and function description content output by the formula parsing large model for the retrieved formula question.

[0045] Step S30: Retrieve and output the coded formula. Specifically: Use the formula parsing large model to output the coded formula and its annotation content, and function description content corresponding to the user's retrieved formula question. The user's retrieved formula question can be the coded formula and at least part of its function description content.

[0046] To improve the comprehensiveness and accuracy of the retrieval results, it can also include:

[0047] Step S40: Inference and output of the retrieval results. Specifically including:

[0048] Step S41: When the user conducts a retrieved formula question, the formula parsing large model associates and paraphrases the retrieved formula question to form multiple questions and outputs the coded formula and its annotation content, and function description content corresponding to the retrieved formula question. This step S41 can improve the comprehensiveness of the output results, increase the probability of outputting retrieval results, and avoid no result output.

[0049] Step S42: The formula parsing large model summarizes multiple questions and outputs the coded formula corresponding to the retrieved formula question, its annotation content, and function description content. This step S42 can improve the accuracy of the output result.

[0050] To improve the clarity of the coded formula, its annotation content, and function description content output by the formula parsing large model, it may further include:

[0051] Step S50: Output result rendering. Specifically, it performs vector rendering on the coded formula, its annotation content, and function description content output by the formula parsing large model.

[0052] Please refer to Figure 2 , The embodiment of the present invention also provides a formula parsing and retrieval system 100 based on a large model, which adopts the method of the above embodiment and includes:

[0053] A server, which has a formula parsing subsystem and a large model training subsystem, where:

[0054] The formula parsing subsystem is used to identify and parse the text content of the formula, its annotation area, and function description area in the text object containing the formula, convert the formula and its annotation content in the picture format into the coded formula and its annotation content, and summarize and extract the function description content of the formula to establish a one-to-one matching relationship between the coded formula and its annotation content and the function description content. The text object includes, but is not limited to, document materials such as construction manuals, specification documents, professional textbooks, periodicals, and papers.

[0055] The large model training subsystem is used to summarize, split, or extract the function description content of each coded formula through a general large model to form questions for asking about the corresponding formula, use each coded formula, its annotation content, and function description content as answers, splice the questions for asking about each coded formula and their answers to form more than one group of question and answer pairs; use the whole group of question and answer pairs as training data to train the large model to form a formula parsing large model.

[0056] A retrieval terminal 130, which is network-connected to the server and is used to, when retrieving a formula question, connect to the server through the network, call the formula parsing large model in it to output the coded formula corresponding to the retrieved formula question, its annotation content, and function description content, and download them to the retrieval terminal 130 for display. Among them, the retrieval terminal 130 includes, but is not limited to, a computer, a smart phone, and may also be other handheld terminals.

[0057] To improve the comprehensiveness and accuracy of retrieval results, in the formula parsing and retrieval system 100 based on a large model provided by an embodiment of the present invention, the server further establishes a formula parsing large model inference subsystem. Through the large model inference subsystem, the retrieval formula problem of the retrieval terminal 130 is associated and paraphrased to form multiple problems, and the multiple problems are summarized to output the coded formula corresponding to the retrieval formula problem and its annotation content and function description content.

[0058] To avoid mutual influence and interference between the formula parsing large models before and after large model training, in the formula parsing and retrieval system 100 based on a large model provided by an embodiment of the present invention, the server includes a first server 110 and a second server 120 interconnected by a network. The formula parsing subsystem and the large model training subsystem are established on the second server 120. The formula parsing large model and the formula parsing large model inference subsystem formed through the large model training subsystem are deployed on the first server 110, and the retrieval terminal 130 is network-connected to the first server 110.

[0059] To improve the clarity of the display result, in the formula parsing and retrieval system 100 based on a large model provided by an embodiment of the present invention, the retrieval terminal 130 performs vector rendering on the coded formula and its annotation content and function description content displayed by it.

[0060] The formula parsing and retrieval method and system 100 based on a large model provided by an embodiment of the present invention perform recognition on the formula, its annotations, and function description content of a text object in a coded format and text content with a picture format, establish a coded parsing of a one-to-one matching relationship between each coded formula and its annotation content and function description content, so as to perform coded parsing on the formula, its annotation content, and function description content in a picture format, improving the parsing accuracy of the formula, its annotations, and function description content in a picture format; form questions and answers for the coded-parsed formula, its annotations, and function description content through a large model as training data for the large model to form a formula parsing large model; output the coded formula corresponding to the user's retrieval formula problem and its annotation content and function description content through the formula parsing large model, thereby realizing the coded retrieval of the formula, its annotation content, and function description content in a picture format. It overcomes the problem that a general large model cannot parse and retrieve formulas in a picture format, especially complex formulas and formulas in the construction industry, and can output the formula in a coded form, having the advantages of being convenient for copying and extracting the coded formula and its annotation content and function description content.

[0061] The present invention is not limited to the above specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention fall within the scope of protection of the present invention. Those skilled in the art can make other levels of modifications and changes to the present invention. Thus, if these modifications and changes of the present invention are within the scope of the claims of the present invention, the present invention also intends to include these modifications and changes.

Claims

1. A formula parsing and retrieval method based on a large model, characterized in that: include: Step S10, formula coding analysis, includes: Step S11, detecting the formula and its annotation area and function description area in the text object; Step S12, identifying and parsing the text content of the formula and its annotation area and function description area by using text recognition technology; Step S13, converting the formula in picture format and its annotation content into character code format to form a coded formula and its annotation content analysis; Step S14, summarizing and extracting the functional description content of the formula, and establishing a one-to-one matching relationship between the coded formula and its annotation content and the functional description content; Step S20, formula parsing large model training: The function description content of each coded formula is summarized, split or extracted through the big model to form questions about the corresponding formula, and each coded formula and its annotation content and function description content are used as answers. The questions and answers of each coded formula are spliced ​​to form one or more question-answer pairs; the big model is trained by using the whole set of question-answer pairs as training data to form a formula parsing big model; Step S30, coded formula retrieval output: The formula parsing model outputs the coded formula corresponding to the user's search formula question, as well as its annotation content and functional description content.

2. The formula parsing and retrieval method based on a large model according to claim 1, characterized in that: Also includes: Step S40, retrieval result reasoning output, including: Step S41, when the user performs a search formula question, the formula parsing model associates and rephrases the search formula question to form multiple questions and outputs the coded formula corresponding to the search formula question and its annotation content and functional description content.

3. The formula parsing and retrieval method based on a large model according to claim 2 is characterized in that: In the step S40, it also includes: Step S42, the formula parsing model summarizes the multiple questions and outputs the coded formula corresponding to the search formula question and its annotation content and functional description content.

4. The formula parsing and retrieval method based on a large model according to claim 2 is characterized in that: Also includes: Step S50, output result rendering: Vector rendering is performed on the coded formulas and their annotations and functional descriptions output by the formula parsing model.

5. The formula parsing and retrieval method based on a large model according to claim 1 is characterized in that: Low-rank fine-tuning of large models for formula parsing.

6. A formula parsing and retrieval system based on a large model, characterized in that: include: The server has a formula parsing subsystem and a large model training subsystem, including: The formula parsing subsystem is used to identify and parse the text content of the formula and its annotation area and function description area in the text object containing the formula, convert the formula and its annotation content in the image format into the coded formula and its annotation content, summarize and extract the function description content of the formula, and establish a one-to-one matching relationship between the coded formula and its annotation content and the function description content; The large model training subsystem is used to summarize, split or extract the functional description content of each coded formula through the general large model to form questions about the corresponding formula, and use each coded formula and its annotation content and functional description content as answers, and splice the questions and answers of each coded formula to form one or more question-answer pairs; the large model is trained using the entire set of question-answer pairs as training data to form a formula parsing large model; The retrieval terminal is connected to the server network and is used to retrieve formula problems. When the retrieval formula problem is connected to the server through the network and the formula parsing model therein is called, the coded formula corresponding to the retrieval formula problem and its annotation content and functional description content are output and downloaded to the retrieval terminal for display.

7. The large model-based formula parsing and retrieval system according to claim 6, characterized in that: The server also has a formula parsing large model reasoning subsystem, which associates and rephrases the search formula questions of the search terminal to form multiple questions, summarizes the multiple questions and outputs the coded formula corresponding to the search formula question and its annotation content and functional description content.

8. The large model-based formula parsing and retrieval system according to claim 7, characterized in that: The server includes a first server and a second server interconnected by a network. The formula parsing subsystem and the large model training subsystem are established on the second server. The formula parsing large model and the formula parsing large model reasoning subsystem formed by the large model training subsystem are deployed on the first server, and the retrieval terminal is connected to the first server network.

9. The large model-based formula parsing and retrieval system according to claim 6, characterized in that: The retrieval terminal is a computer or a smart phone.

10. The large model-based formula parsing and retrieval system according to claim 6, characterized in that: The retrieval terminal performs vector rendering on the displayed coded formula and its annotation content and function description content.

Images