Receipt element analysis method and device based on RAG, equipment and medium
Through the RAG-based document element analysis method, using the knowledge base to obtain similar documents and enhance the propt, the problems of poor generalization ability, low accuracy and high cost of document element analysis in the prior art are solved, and higher analytical accuracy and lower cost are achieved.
Patent Information
- Application Number
- CN202411697707.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art has poor generalization ability, low accuracy and high cost in document factor analysis, making it difficult to deal with variable document analysis scenarios.
The document feature analysis method based on RAG is used to obtain the target similar documents of the target document from the pre-constructed knowledge base through text embedding vectors and image embedding vectors, and enhance the propt based on similar documents, and call the large language model for feature analysis.
It improves the accuracy and generalization ability of document analysis, reduces manpower, time and technical costs, and can better deal with changing document scenarios.
Smart Images
Figure CN119649376A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a RAG-based document element parsing method, device, equipment and medium. Background Art
[0002] In the prior art, document elements are usually parsed using a large language model or according to traditional rules.
[0003] However, when using large language models to parse document elements, hallucination problems often occur, that is, the model extracts document elements that do not exist, or the large language model cannot correctly understand the semantics of the elements and leads to extraction errors; traditional rules are only applicable to scenarios with relatively fixed document formats. Obviously, in the face of documents with changing customer types and formats in financial scenarios, the above two document parsing methods cannot adapt to the high-frequency changes in documents, especially for documents that have never been seen, which is more difficult to handle and has poor generalization ability.
[0004] In addition, if large model fine-tuning technology is used, higher computing power and time costs will be consumed. If algorithm engineers intervene and adjust interpretations or write rules for new documents, higher labor costs will be consumed.
[0005] It can be seen that the existing technology is difficult to cope with the changing document parsing scenarios with relatively low manpower, time and technical costs. Summary of the invention
[0006] The embodiments of the present application provide a RAG-based document element parsing method, apparatus, device and medium, which aim to solve the problems of poor generalization ability, low accuracy and high cost of the document element parsing method.
[0007] In a first aspect, an embodiment of the present application provides a method for parsing document elements based on RAG, which includes:
[0008] In response to an element parsing instruction for a target document, a target document image of the target document is acquired, and text recognition is performed on the target document image to obtain a target text;
[0009] Using an embedding model, the target text is converted into a text embedding vector, and the target document image is converted into an image embedding vector;
[0010] Acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector;
[0011] Enhance the prompt according to the target similar documents;
[0012] An element parsing request is generated according to the enhanced prompt and the element parsing instruction, and a large language model is called to execute the element parsing request to obtain the target element of the target document.
[0013] In a second aspect, an embodiment of the present application provides a document element parsing device based on RAG, which includes:
[0014] a recognition unit, configured to obtain a target document image of the target document in response to an element parsing instruction for the target document, and perform text recognition on the target document image to obtain a target text;
[0015] A conversion unit, configured to convert the target text into a text embedding vector and convert the target document image into an image embedding vector by using an embedding model;
[0016] An acquisition unit, configured to acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector;
[0017] An enhancement unit, used for enhancing the prompt according to the target similar document;
[0018] The execution unit is used to generate an element parsing request according to the enhanced prompt and the element parsing instruction, and call the large language model to execute the element parsing request to obtain the target element of the target document.
[0019] In the third aspect, an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the RAG-based document element parsing method described in the first aspect above is implemented.
[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the RAG-based document element parsing method described in the first aspect above.
[0021] The embodiments of the present application provide a document element parsing method, apparatus, device and medium based on RAG. On the one hand, based on the RAG principle, target similar documents of the target document are obtained from a pre-built knowledge base according to text embedding vectors and image embedding vectors, and the prompt is enhanced according to the target similar documents. Not only can the knowledge base be used to pre-store typical document templates for enhancing prompts, making the document parsing of the large language model more accurate, but more accurate similar documents can also be obtained from multiple dimensions to further improve the accuracy of model parsing. On the other hand, the knowledge base and the large language model are combined to extract document elements, which can cope with changing document parsing scenarios with relatively low manpower, time and technical costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 A schematic diagram of an application scenario of the RAG-based document element parsing method provided in an embodiment of the present application;
[0024] Figure 2 A flowchart of a RAG-based document element parsing method provided in an embodiment of the present application;
[0025] Figure 3 A schematic block diagram of a document element parsing device based on RAG provided in an embodiment of the present application;
[0026] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0029] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0030] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0031] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the RAG-based document element parsing method provided in an embodiment of the present application; Figure 2 A flow chart of a RAG-based document element parsing method provided in an embodiment of the present application. The RAG-based document element parsing method is applied in a server, and the method is executed by application software installed in the server.
[0032] like Figure 2 As shown, the method includes steps S101 to S105.
[0033] S101 . In response to an element parsing instruction for a target document, a target document image of the target document is acquired, and text recognition is performed on the target document image to obtain a target text.
[0034] In this embodiment, the technical solution is described with the server as the execution subject. The user terminal (such as a smart phone, tablet computer and other smart terminals) used by the user can interact with the server for data. Specifically, the server provides a document element parsing platform, and the user can log in to the platform using the user terminal. The user interaction interface of the platform is displayed on the terminal interface of the user terminal, and there is at least one image upload interface in the user interaction interface. When the user selects an image as the image to be identified and uploads it to the server through the image upload interface, the subsequent document element parsing can be performed in the server.
[0035] In this embodiment, the element parsing instruction can be triggered by relevant staff, or it can be automatically triggered after detecting that the image to be identified is uploaded to the server from the image upload interface. This application does not limit the triggering method of the element parsing instruction.
[0036] In this embodiment, the target document may be various documents in the financial field, such as invoices.
[0037] In this embodiment, the target document image may be a scanned copy or a photo of the target document.
[0038] S102: Using an embedding model, convert the target text into a text embedding vector, and convert the target document image into an image embedding vector.
[0039] In this embodiment, the converting the target text into a text embedding vector and the converting the target document image into an image embedding vector by using an embedding model includes:
[0040] Perform sliding window segmentation on the target text according to the configured overlap ratio to obtain multiple text blocks;
[0041] Using the m3e-large model, the multiple text blocks are converted into embedding vectors to obtain the text embedding vectors;
[0042] The target document image is converted into an embedding vector using a CLIP (Contrastive Language-Image Pre-Training) model to obtain the image embedding vector.
[0043] The configuration overlap ratio can be configured according to actual needs. For example, when there are 1000 tokens of content, the window size is 500 tokens, and the configuration overlap ratio is 20%, then the 1000 tokens will be divided into 3 blocks, the first block is 1 to 500 tokens, the second block is 400 to 900 tokens, and the third block is 800 to 1000 tokens.
[0044] By configuring the overlap ratio and dividing the blocks according to the sliding window, it is possible to effectively prevent semantic truncation during block division.
[0045] In addition, by using different embedding models to perform targeted vectorization processing on text data and image data, corresponding embedding vectors can be generated more accurately based on semantic understanding.
[0046] S103: Acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector.
[0047] In this embodiment, before obtaining a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector, the method further includes:
[0048] Acquire historical parsing data, and acquire a document whose parsing accuracy is less than or equal to the configuration accuracy from the historical parsing data as a first candidate document;
[0049] Get the second candidate document uploaded by the user;
[0050] Combining the first candidate document and the second candidate document to obtain a document template;
[0051] The document template is annotated with elements, and the annotated document template is stored in the knowledge base.
[0052] The knowledge base may be a local knowledge base or a cloud knowledge base, and the specific choice may be made according to the actual application scenario.
[0053] The configuration accuracy can be configured according to actual scenarios, such as 95%.
[0054] For example, if the analysis of historical data shows that the parsing effect of document type X is poor, the corresponding document can be stored in the knowledge base as a document template; when the user finds that the parsing effect of document type Y is poor and there are many errors in actual use, the corresponding document can also be stored in the knowledge base as a document template. In this way, the user can input the documents with poor parsing effect into the knowledge base, and the subsequent parsing effect of such documents will be effectively improved. The user only needs to input the content of the element to realize the iterative upgrade of the document element parsing solution on the customer side. There is no technical threshold, and it can quickly adapt to different document scenarios faced by different customers, and no additional computing resources are required.
[0055] It can be seen that combining with the knowledge base can adapt to new documents more quickly, without the need for algorithm engineers to operate, without technical barriers, and with good generalization for documents that have never been seen.
[0056] In the above embodiment, documents with poor parsing effects (including new documents that have not been parsed or poor parsing effects caused by other reasons) can be used as document templates based on historical parsing effects and actual user usage effects, and added to the knowledge base after marking, so as to provide accurate parsing basis for subsequent document parsing, make the parsing more accurate, and be able to cope with various styles of documents, and have stronger generalization capabilities.
[0057] In this embodiment, obtaining a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector includes:
[0058] Calculating the similarity between the text embedding vector and the text vector of each document template in the knowledge base to obtain each text similarity;
[0059] Calculating the similarity between the image embedding vector and the image vector of each document template in the knowledge base to obtain the similarity of each image;
[0060] Obtaining a first weight corresponding to the text similarity, and obtaining a second weight corresponding to the image similarity;
[0061] Calculate the similarity between the target document and each document template according to the first weight, the second weight, each text similarity and each image similarity;
[0062] Acquire the document templates whose similarity is greater than or equal to the configured similarity threshold to construct a candidate document template set;
[0063] Each candidate document template in the candidate document template set is sorted in descending order according to the similarity, and the candidate document template ranked in a front preset position is obtained as the target similar document of the target document.
[0064] The configuration similarity threshold may be configured according to actual usage scenarios to prevent erroneous document templates from being retrieved and polluting the prompt.
[0065] The first weight and the second weight can be configured according to actual needs. For example, the first weight and the second weight can both be configured to 0.5.
[0066] Among them, the pre-set positions can be configured according to actual needs, such as the first 10 digits, the first 3 digits, etc.
[0067] For example, for a document template M, when the text similarity corresponding to the document template M is a, the image similarity is b, and the first weight and the second weight are both 0.5, the similarity between the document template M and the target document is 0.5 (a+b). The similarity between each document template and the target document is compared, and they are sorted in descending order of similarity to obtain a specified number of similar documents.
[0068] Through the above embodiments, weight calculation can be performed in combination with multi-dimensional similarities, so that the calculated similarity is more accurate.
[0069] S104. Enhance the prompt according to the target similar document.
[0070] In this embodiment, the prompt is enhanced according to the target similar document, including:
[0071] Convert the target similar document into one-shot or few-shot;
[0072] The prompt is enhanced using the one-shot or few-shot.
[0073] Specifically, the one-shot or few-shot may be embedded into the original prompt to enhance the prompt.
[0074] S105. Generate an element parsing request according to the enhanced prompt and the element parsing instruction, and call the large language model to execute the element parsing request to obtain the target element of the target document.
[0075] In this embodiment, the enhanced prompt is used to update the element parsing request, which can effectively guide the large language model, thereby making the parsing result of the large language model more accurate.
[0076] In this embodiment, after obtaining the target element of the target document, the method further includes:
[0077] Constructing a test set, and dividing the test set according to the document template type to obtain each sub-test set;
[0078] Randomly select a to-be-recalled document from each sub-test set and add it to the knowledge base;
[0079] Obtaining similar documents corresponding to each sub-test set from the knowledge base;
[0080] Obtaining, for each to-be-recalled document, the total number of times similar documents in the corresponding subtest set are recalled as a first value;
[0081] Obtaining the total number of times similar documents corresponding to each subtest set are obtained from the knowledge base as a second value;
[0082] Calculating the quotient of the first value and the second value to obtain a recall rate;
[0083] When the recall rate is greater than or equal to the configured recall rate, it is determined that the test has passed; or
[0084] When the recall rate is less than the configuration recall rate, it is determined that the test has not passed.
[0085] The configuration recall rate may be configured according to actual needs, such as 90%, 95%, etc.
[0086] Through the above embodiments, the recall rate of the document element parsing method can be tested to determine whether the document retrieved from the knowledge base has a high similarity with the input document, thereby verifying the accuracy of the document element parsing method.
[0087] In this embodiment, after determining that the test fails, the method further includes:
[0088] Reasons for failure of the test;
[0089] When the reason for failing the test is that the corresponding similar document is missing in the knowledge base, adding the missing document template to the knowledge base;
[0090] When the reason for failing the test is that a corresponding similar document exists in the knowledge base but has not been successfully recalled, the embedding model and / or the first weight and the second weight are adjusted.
[0091] Through the above embodiments, the document element parsing method can be adjusted specifically according to the reasons for failing the test to optimize the parsing effect.
[0092] In addition, in order to further verify the accuracy of this solution, this embodiment takes transport documents, free-format documents, and STS data sets as examples to construct self-made data sets and training sets, and respectively uses the native large language model, the large language model fine-tuned by algorithm engineers, this solution, this solution fine-tuned by algorithm engineers, and the existing technology to perform element analysis on the documents in each data set. It can be found that the accuracy of this solution can reach 92.2% before fine-tuning, and the accuracy after fine-tuning is much higher than other solutions, and can reach up to 92.9%.
[0093] It can be seen from the above technical solutions that, on the one hand, based on the RAG (Retrieval-Augmented Generation) principle, target similar documents of the target document are obtained from the pre-built knowledge base according to the text embedding vector and the image embedding vector, and the prompt is enhanced according to the target similar documents. Not only can the knowledge base be used to pre-store typical document templates for enhancing the prompt, making the document parsing of the large language model more accurate, but more accurate similar documents can also be obtained from multiple dimensions to further improve the accuracy of model parsing; on the other hand, combining the knowledge base with the large language model to extract document elements can cope with changing document parsing scenarios with relatively low manpower, time and technical costs.
[0094] The present application also provides a RAG-based document element parsing device, which is used to execute any embodiment of the RAG-based document element parsing method. Figure 3 , Figure 3 It is a schematic block diagram of the RAG-based document element parsing device 100 provided in an embodiment of the present application.
[0095] Among them, Figure 3 As shown, the RAG-based document element parsing device 100 includes an identification unit 101, a conversion unit 102, an acquisition unit 103, an enhancement unit 104, and an execution unit 105.
[0096] The recognition unit 101 is used to obtain a target document image of the target document in response to an element parsing instruction for the target document, and perform text recognition on the target document image to obtain a target text.
[0097] In this embodiment, the technical solution is described with the server as the execution subject. The user terminal (such as a smart phone, tablet computer and other smart terminals) used by the user can interact with the server for data. Specifically, the server provides a document element parsing platform, and the user can log in to the platform using the user terminal. The user interaction interface of the platform is displayed on the terminal interface of the user terminal, and there is at least one image upload interface in the user interaction interface. When the user selects an image as the image to be identified and uploads it to the server through the image upload interface, the subsequent document element parsing can be performed in the server.
[0098] In this embodiment, the element parsing instruction can be triggered by relevant staff, or it can be automatically triggered after detecting that the image to be identified is uploaded to the server from the image upload interface. This application does not limit the triggering method of the element parsing instruction.
[0099] In this embodiment, the target document may be various documents in the financial field, such as invoices.
[0100] In this embodiment, the target document image may be a scanned copy or a photo of the target document.
[0101] The conversion unit 102 is used to convert the target text into a text embedding vector and convert the target document image into an image embedding vector by using an embedding model.
[0102] In this embodiment, the conversion unit 102 converts the target text into a text embedding vector and converts the target document image into an image embedding vector by using an embedding model, including:
[0103] Perform sliding window segmentation on the target text according to the configured overlap ratio to obtain multiple text blocks;
[0104] Using the m3e-large model, the multiple text blocks are converted into embedding vectors to obtain the text embedding vectors;
[0105] The target document image is converted into an embedding vector using a CLIP (Contrastive Language-Image Pre-Training) model to obtain the image embedding vector.
[0106] The configuration overlap ratio can be configured according to actual needs. For example, when there are 1000 tokens of content, the window size is 500 tokens, and the configuration overlap ratio is 20%, then the 1000 tokens will be divided into 3 blocks, the first block is 1 to 500 tokens, the second block is 400 to 900 tokens, and the third block is 800 to 1000 tokens.
[0107] By configuring the overlap ratio and dividing the blocks according to the sliding window, it is possible to effectively prevent semantic truncation during block division.
[0108] In addition, by using different embedding models to perform targeted vectorization processing on text data and image data, corresponding embedding vectors can be generated more accurately based on semantic understanding.
[0109] The acquisition unit 103 is used to acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector.
[0110] In this embodiment, the acquisition unit 103 acquires historical parsing data before acquiring a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector, and acquires a document whose parsing accuracy is less than or equal to the configured accuracy from the historical parsing data as a first candidate document;
[0111] Get the second candidate document uploaded by the user;
[0112] Combining the first candidate document and the second candidate document to obtain a document template;
[0113] The document template is annotated with elements, and the annotated document template is stored in the knowledge base.
[0114] The knowledge base may be a local knowledge base or a cloud knowledge base, and the specific choice may be made according to the actual application scenario.
[0115] The configuration accuracy can be configured according to actual scenarios, such as 95%.
[0116] For example, if the analysis of historical data shows that the parsing effect of document type X is poor, the corresponding document can be stored in the knowledge base as a document template; when the user finds that the parsing effect of document type Y is poor and there are many errors in actual use, the corresponding document can also be stored in the knowledge base as a document template. In this way, the user can input the documents with poor parsing effect into the knowledge base, and the subsequent parsing effect of such documents will be effectively improved. The user only needs to input the content of the element to realize the iterative upgrade of the document element parsing solution on the customer side. There is no technical threshold, and it can quickly adapt to different document scenarios faced by different customers, and no additional computing resources are required.
[0117] It can be seen that combining with the knowledge base can adapt to new documents more quickly, without the need for algorithm engineers to operate, without technical barriers, and with good generalization for documents that have never been seen.
[0118] In the above embodiment, documents with poor parsing effects (including new documents that have not been parsed or poor parsing effects caused by other reasons) can be used as document templates based on historical parsing effects and actual user usage effects, and added to the knowledge base after marking, so as to provide accurate parsing basis for subsequent document parsing, make the parsing more accurate, and be able to cope with various styles of documents, and have stronger generalization capabilities.
[0119] In this embodiment, the acquisition unit 103 acquires a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector, including:
[0120] Calculating the similarity between the text embedding vector and the text vector of each document template in the knowledge base to obtain each text similarity;
[0121] Calculating the similarity between the image embedding vector and the image vector of each document template in the knowledge base to obtain the similarity of each image;
[0122] Obtaining a first weight corresponding to the text similarity, and obtaining a second weight corresponding to the image similarity;
[0123] Calculate the similarity between the target document and each document template according to the first weight, the second weight, each text similarity and each image similarity;
[0124] Acquire the document templates whose similarity is greater than or equal to the configured similarity threshold to construct a candidate document template set;
[0125] Each candidate document template in the candidate document template set is sorted in descending order according to the similarity, and the candidate document template ranked in a front preset position is obtained as the target similar document of the target document.
[0126] The configuration similarity threshold may be configured according to actual usage scenarios to prevent erroneous document templates from being retrieved and polluting the prompt.
[0127] The first weight and the second weight can be configured according to actual needs. For example, the first weight and the second weight can both be configured to 0.5.
[0128] Among them, the pre-set positions can be configured according to actual needs, such as the first 10 digits, the first 3 digits, etc.
[0129] For example, for a document template M, when the text similarity corresponding to the document template M is a, the image similarity is b, and the first weight and the second weight are both 0.5, the similarity between the document template M and the target document is 0.5 (a+b). The similarity between each document template and the target document is compared, and they are sorted in descending order of similarity to obtain a specified number of similar documents.
[0130] Through the above embodiments, weight calculation can be performed in combination with multi-dimensional similarities, so that the calculated similarity is more accurate.
[0131] The enhancement unit 104 is used to enhance the prompt according to the target similar document.
[0132] In this embodiment, the enhancement unit 104 enhances the prompt according to the target similar document, including:
[0133] Convert the target similar document into one-shot or few-shot;
[0134] The prompt is enhanced using the one-shot or few-shot.
[0135] Specifically, the one-shot or few-shot may be embedded into the original prompt to enhance the prompt.
[0136] The execution unit 105 is used to generate an element parsing request according to the enhanced prompt and the element parsing instruction, and call the large language model to execute the element parsing request to obtain the target element of the target document.
[0137] In this embodiment, the enhanced prompt is used to update the element parsing request, which can effectively guide the large language model, thereby making the parsing result of the large language model more accurate.
[0138] In this embodiment, after obtaining the target elements of the target document, a test set is constructed, and the test set is divided according to the document template type to obtain each sub-test set;
[0139] Randomly select a to-be-recalled document from each sub-test set and add it to the knowledge base;
[0140] Obtaining similar documents corresponding to each sub-test set from the knowledge base;
[0141] Obtaining, for each to-be-recalled document, the total number of times similar documents in the corresponding subtest set are recalled as a first value;
[0142] Obtaining the total number of times similar documents corresponding to each subtest set are obtained from the knowledge base as a second value;
[0143] Calculating the quotient of the first value and the second value to obtain a recall rate;
[0144] When the recall rate is greater than or equal to the configured recall rate, it is determined that the test has passed; or
[0145] When the recall rate is less than the configuration recall rate, it is determined that the test has not passed.
[0146] The configuration recall rate may be configured according to actual needs, such as 90%, 95%, etc.
[0147] Through the above embodiments, the recall rate of the document element parsing method can be tested to determine whether the document retrieved from the knowledge base has a high similarity with the input document, thereby verifying the accuracy of the document element parsing method.
[0148] In this embodiment, after determining that the test failed, detecting the reason for the test failure;
[0149] When the reason for failing the test is that the corresponding similar document is missing in the knowledge base, adding the missing document template to the knowledge base;
[0150] When the reason for failing the test is that a corresponding similar document exists in the knowledge base but has not been successfully recalled, the embedding model and / or the first weight and the second weight are adjusted.
[0151] Through the above embodiments, the document element parsing method can be adjusted specifically according to the reasons for failing the test to optimize the parsing effect.
[0152] In addition, in order to further verify the accuracy of this solution, this embodiment takes transport documents, free-format documents, and STS data sets as examples to construct self-made data sets and training sets, and respectively uses the native large language model, the large language model fine-tuned by algorithm engineers, this solution, this solution fine-tuned by algorithm engineers, and the existing technology to perform element analysis on the documents in each data set. It can be found that the accuracy of this solution can reach 92.2% before fine-tuning, and the accuracy after fine-tuning is much higher than other solutions, and can reach up to 92.9%.
[0153] It can be seen from the above technical solutions that, on the one hand, based on the RAG (Retrieval-Augmented Generation) principle, target similar documents of the target document are obtained from the pre-built knowledge base according to the text embedding vector and the image embedding vector, and the prompt is enhanced according to the target similar documents. Not only can the knowledge base be used to pre-store typical document templates for enhancing the prompt, making the document parsing of the large language model more accurate, but more accurate similar documents can also be obtained from multiple dimensions to further improve the accuracy of model parsing; on the other hand, combining the knowledge base with the large language model to extract document elements can cope with changing document parsing scenarios with relatively low manpower, time and technical costs.
[0154] The RAG-based document element parsing device can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.
[0155] See also Figure 4 , Figure 4 4 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 400 is a server, or a server cluster. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0156] See also Figure 4 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a device bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .
[0157] The storage medium 403 may store an operating system 4031 and a computer program 4032. When the computer program 4032 is executed, the processor 402 may execute a document element parsing method based on RAG.
[0158] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device 400 .
[0159] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the RAG-based document element parsing method.
[0160] The network interface 405 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 400 to which the solution of the present application is applied. The specific computer device 400 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0161] The processor 402 is used to run a computer program 4032 stored in the memory to implement the RAG-based document element parsing method disclosed in the embodiment of the present application.
[0162] Those skilled in the art will understand that Figure 4 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 4 The embodiments shown are consistent and will not be described again here.
[0163] It should be understood that in the embodiment of the present application, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0164] In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the RAG-based document element parsing method disclosed in the embodiment of the present application is implemented.
[0165] It should be noted that the data involved in this case were all obtained legally.
[0166] It will be clear to those skilled in the art that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It will be appreciated by those of ordinary skill in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0167] In the several embodiments provided in the present application, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, and units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.
[0168] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.
[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a background server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a disk or an optical disk.
[0171] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A document element parsing method based on RAG, characterized in that: The RAG-based document element parsing method includes: In response to an element parsing instruction for a target document, a target document image of the target document is acquired, and text recognition is performed on the target document image to obtain a target text; Using an embedding model, the target text is converted into a text embedding vector, and the target document image is converted into an image embedding vector; Acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector; Enhance the prompt according to the target similar documents; An element parsing request is generated according to the enhanced prompt and the element parsing instruction, and a large language model is called to execute the element parsing request to obtain the target element of the target document.
2. The RAG-based document element parsing method according to claim 1, characterized in that: The method of converting the target text into a text embedding vector by using an embedding model, and converting the target document image into an image embedding vector, comprises: Perform sliding window segmentation on the target text according to the configured overlap ratio to obtain multiple text blocks; Using the m3e-large model, the multiple text blocks are converted into embedding vectors to obtain the text embedding vectors; The target document image is converted into an embedding vector using the CLIP model to obtain the image embedding vector.
3. The RAG-based document element parsing method according to claim 1, characterized in that: Before acquiring a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector, the method further includes: Acquire historical parsing data, and acquire a document whose parsing accuracy is less than or equal to the configuration accuracy from the historical parsing data as a first candidate document; Get the second candidate document uploaded by the user; Combining the first candidate document and the second candidate document to obtain a document template; The document template is annotated with elements, and the annotated document template is stored in the knowledge base.
4. The RAG-based document element parsing method according to claim 3, characterized in that: The step of acquiring a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector includes: Calculating the similarity between the text embedding vector and the text vector of each document template in the knowledge base to obtain each text similarity; Calculating the similarity between the image embedding vector and the image vector of each document template in the knowledge base to obtain the similarity of each image; Obtaining a first weight corresponding to the text similarity, and obtaining a second weight corresponding to the image similarity; Calculate the similarity between the target document and each document template according to the first weight, the second weight, each text similarity and each image similarity; Acquire the document templates whose similarity is greater than or equal to the configured similarity threshold to construct a candidate document template set; Each candidate document template in the candidate document template set is sorted in descending order according to the similarity, and the candidate document template ranked in a front preset position is obtained as the target similar document of the target document.
5. The RAG-based document element parsing method according to claim 1, characterized in that: The step of enhancing the prompt according to the target similar document includes: Convert the target similar document into one-shot or few-shot; The prompt is enhanced using the one-shot or few-shot.
6. The RAG-based document element parsing method according to claim 4, characterized in that: After obtaining the target element of the target document, the method further includes: Constructing a test set, and dividing the test set according to the document template type to obtain each sub-test set; Randomly select a to-be-recalled document from each sub-test set and add it to the knowledge base; Obtaining similar documents corresponding to each sub-test set from the knowledge base; Obtaining, for each to-be-recalled document, the total number of times similar documents in the corresponding subtest set are recalled as a first value; Obtaining the total number of times similar documents corresponding to each subtest set are obtained from the knowledge base as a second value; Calculating the quotient of the first value and the second value to obtain a recall rate; When the recall rate is greater than or equal to the configured recall rate, it is determined that the test has passed; or When the recall rate is less than the configuration recall rate, it is determined that the test has not passed.
7. The RAG-based document element parsing method according to claim 6, characterized in that: After the determination that the test has not been passed, the method further comprises: Reasons for failure of the test; When the reason for failing the test is that the corresponding similar document is missing in the knowledge base, adding the missing document template to the knowledge base; When the reason for failing the test is that a corresponding similar document exists in the knowledge base but has not been successfully recalled, the embedding model and / or the first weight and the second weight are adjusted.
8. A document element parsing device based on RAG, characterized in that: The RAG-based document element parsing device includes: a recognition unit, configured to obtain a target document image of the target document in response to an element parsing instruction for the target document, and perform text recognition on the target document image to obtain a target text; A conversion unit, configured to convert the target text into a text embedding vector and convert the target document image into an image embedding vector by using an embedding model; An acquisition unit, configured to acquire a target similar document of the target document from a pre-built knowledge base according to the text embedding vector and the image embedding vector; An enhancement unit, used for enhancing the prompt according to the target similar document; The execution unit is used to generate an element parsing request according to the enhanced prompt and the element parsing instruction, and call the large language model to execute the element parsing request to obtain the target element of the target document.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the RAG-based document element parsing method as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the RAG-based document element parsing method as described in any one of claims 1 to 7.