Intelligent agent-driven file content extraction method and device

Abstract

The embodiment of the invention provides an agent-driven file content extraction method and device, and particularly relates to the technical field of character recognition. The method comprises the following steps: performing paging processing on an obtained file to obtain a paging processing result; aiming at each page in the paging processing result, generating first question information aiming at page contents; based on the generated first question information, the intelligent agent selects and calls a corresponding model to process the page, and a page processing result is obtained; the page processing result at least comprises a first character recognition result for the printed body page, a file knowledge extraction result extracted based on the first character recognition result, and a second character recognition result for the handwritten body page; the second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the file knowledge extraction result. And the effects of more accurately identifying and extracting the character information in the file and reducing errors and omissions are achieved.

Description

Technical Field

[0001] This application relates to the field of text recognition technology, and in particular to a method and apparatus for extracting case file content driven by an intelligent agent, as well as related equipment, media and program products. Background Technology

[0002] With the rapid development of case investigation and detection technologies, there is a need to extract information from a large number of existing cases to support the investigation of subsequent complex cases. However, existing case files are mostly stored in scanned form, and images are difficult to transform into digitized information with reasoning support capabilities. Furthermore, the text recognition of a large number of handwritten scanned images suffers from problems such as low recognition accuracy, difficulty in extracting key information, and high training and annotation costs. Even when using specialized OCR (Optical Character Recognition) software to recognize handwritten text in professional materials, ideal results are still difficult to achieve, often resulting in grammatical inconsistencies and extremely low recognition rates. Summary of the Invention

[0003] This application provides intelligent agent-driven methods, apparatus, devices, media, and program products for extracting case file content, in order to more accurately identify and extract text information in case files and reduce errors and omissions.

[0004] In a first aspect, embodiments of this application provide an agent-driven method for extracting case file content, including:

[0005] The acquired case files are paginated to obtain the pagination results;

[0006] For each page in the pagination result, generate a first question about the page content;

[0007] Based on the generated first question information, the intelligent agent selects and calls the corresponding model to process the page and obtain the page processing result;

[0008] The page processing result includes at least a first character recognition result for the printed page, a case file knowledge extraction result extracted based on the first character recognition result, and a second character recognition result for the handwritten page.

[0009] The second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the case file knowledge extraction result.

[0010] In one possible implementation, based on the generated first question information, the agent selects and invokes a corresponding model to process the page, obtaining a page processing result, including:

[0011] For the first question generated, the intelligent agent generates a corresponding first answer.

[0012] When the first response information indicates that the current page is a printed page, the printed page is subjected to text recognition to obtain a first text recognition result, and a pre-trained large language model is called to extract entities and entity relationships from the first text recognition result to obtain the case file knowledge extraction result.

[0013] When the first response indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined, and a pre-trained multimodal large model is called to perform character recognition on the handwritten page to obtain a second character recognition result.

[0014] In one possible implementation, the pre-trained multimodal large model includes an image encoder, which is used for:

[0015] The page to be processed is divided into multiple tiles, each tile corresponding to a local area of ​​the page to be processed;

[0016] For any given image patch, determine its location code, convert its classification information into first embedding information, and convert its image content into second embedding information. Based on the location code, the first embedding information, and the second embedding information, generate the original image code of the image patch.

[0017] In one possible implementation, the step of performing text recognition on the printed page to obtain a first text recognition result when the first response information indicates that the current page is a printed page includes:

[0018] When the first answer information indicates that the current page is a printed page, and the text recognition accuracy for the printed page meets the recognition accuracy requirements in the first question information, a first prompt information is generated, and the first prompt information includes the text recognition requirements for the printed page.

[0019] The multimodal large model is invoked to perform text recognition on the printed page, and the first text recognition result is output in combination with the first prompt information.

[0020] In one possible implementation, the step of calling a pre-trained large language model to perform entity extraction and entity relation extraction on the first character recognition result to obtain the case file knowledge extraction result includes:

[0021] A second question is generated based on the first character recognition result. The second question is used to determine the case type to which the case content in the printed page belongs.

[0022] In response to the generated second question, the intelligent agent generates a corresponding second answer.

[0023] Based on the case type answered in the second response information, a second prompt information is generated. The second prompt information includes one or more of the following: entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first text recognition result.

[0024] The large language model is invoked to extract entities and entity relationships from the first text recognition result, and the case file knowledge extraction result is output in combination with the second prompt information.

[0025] In one possible implementation, when the first response information indicates that the current page is a handwritten page, combining the first character recognition result and the case file knowledge extraction result, and calling a pre-trained multimodal large model to perform character recognition on the handwritten page to obtain a second character recognition result includes:

[0026] When the first answer indicates that the current page is a handwritten page, and the text recognition accuracy of the handwritten page does not meet the recognition accuracy requirements in the first question, a third prompt is generated based on the handwritten page type to which the handwritten page belongs, as indicated in the first answer. The third prompt includes page content extraction requirements for the handwritten page.

[0027] A fourth prompt message is generated based on the case file knowledge extraction results. The fourth prompt message includes entity and entity relationship reference information for the handwritten page.

[0028] A fifth prompt message is generated based on the first character recognition result, and the fifth prompt message includes contextual reference information for the handwritten page;

[0029] The multimodal large model is invoked to perform text recognition on the handwritten page, and the second text recognition result is output by combining the third prompt information, the fourth prompt information and the fifth prompt information.

[0030] Secondly, embodiments of this application provide an agent-driven file content extraction device, comprising:

[0031] The pagination processing module is used to paginate the acquired case file files and obtain the pagination results.

[0032] The generation module is used to generate a first question about the content of each page in the pagination process result.

[0033] The page processing module is used to process the page based on the generated first question information, and the agent selects and calls the corresponding model to obtain the page processing result.

[0034] The page processing result includes at least a first character recognition result for the printed page, a case file knowledge extraction result extracted based on the first character recognition result, and a second character recognition result for the handwritten page.

[0035] The second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the case file knowledge extraction result.

[0036] In one possible implementation, the page processing module is specifically used for:

[0037] For the first question generated, the intelligent agent generates a corresponding first answer.

[0038] When the first response information indicates that the current page is a printed page, the printed page is subjected to text recognition to obtain a first text recognition result, and a pre-trained large language model is called to extract entities and entity relationships from the first text recognition result to obtain the case file knowledge extraction result.

[0039] When the first response indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined, and a pre-trained multimodal large model is called to perform character recognition on the handwritten page to obtain a second character recognition result.

[0040] In one possible implementation, the pre-trained multimodal large model includes an image encoder, which is specifically used for:

[0041] The page to be processed is divided into multiple tiles, each tile corresponding to a local area of ​​the page to be processed;

[0042] For any given image patch, determine its location code, convert its classification information into first embedding information, and convert its image content into second embedding information. Based on the location code, the first embedding information, and the second embedding information, generate the original image code of the image patch.

[0043] In one possible implementation, the page processing module is specifically used for:

[0044] When the first answer information indicates that the current page is a printed page, and the text recognition accuracy for the printed page meets the recognition accuracy requirements in the first question information, a first prompt information is generated, and the first prompt information includes the text recognition requirements for the printed page.

[0045] The multimodal large model is invoked to perform text recognition on the printed page, and the first text recognition result is output in combination with the first prompt information.

[0046] In one possible implementation, the page processing module is specifically used for:

[0047] A second question is generated based on the first character recognition result. The second question is used to determine the case type to which the case content in the printed page belongs.

[0048] In response to the generated second question, the intelligent agent generates a corresponding second answer.

[0049] Based on the case type answered in the second response information, a second prompt information is generated. The second prompt information includes one or more of the following: entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first text recognition result.

[0050] The large language model is invoked to extract entities and entity relationships from the first text recognition result, and the case file knowledge extraction result is output in combination with the second prompt information.

[0051] In one possible implementation, the page processing module is specifically used for:

[0052] When the first answer indicates that the current page is a handwritten page, and the text recognition accuracy of the handwritten page does not meet the recognition accuracy requirements in the first question, a third prompt is generated based on the handwritten page type to which the handwritten page belongs, as indicated in the first answer. The third prompt includes page content extraction requirements for the handwritten page.

[0053] A fourth prompt message is generated based on the case file knowledge extraction results. The fourth prompt message includes entity and entity relationship reference information for the handwritten page.

[0054] A fifth prompt message is generated based on the first character recognition result, and the fifth prompt message includes contextual reference information for the handwritten page;

[0055] The multimodal large model is invoked to perform text recognition on the handwritten page, and the second text recognition result is output by combining the third prompt information, the fourth prompt information and the fifth prompt information.

[0056] Thirdly, embodiments of this application provide an agent-driven case file content extraction device, including: a memory and a processor;

[0057] The memory stores computer-executed instructions;

[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0060] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0061] The intelligent agent-driven case file content extraction method, apparatus, device, medium, and program product provided in this application firstly performs pagination processing on the acquired case file to divide it into multiple pages, obtaining a pagination processing result. For the content of each page, a first question is generated. The first question is used to guide subsequent processing steps, ensuring the relevance and effectiveness of the processing. Based on the generated first question, the intelligent agent selects and calls an appropriate model to process each page, obtaining a page processing result. The page processing result includes at least: a first text recognition result: text recognition result for printed pages; a case file knowledge extraction result: case file-related knowledge extracted based on the first text recognition result; and a second text recognition result: text recognition result for handwritten pages. The second text recognition result is obtained by combining the first text recognition result and the case file knowledge extraction result to recognize the content of the handwritten pages. By utilizing the results of printed text recognition and case file knowledge extraction to enhance the accuracy of handwritten recognition, the accuracy of text information in the case file is more accurately recognized and extracted, reducing errors and omissions. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 A schematic diagram of a scenario for the agent-driven case file content extraction method provided in this application;

[0064] Figure 2 Flowchart of the agent-driven case file content extraction method provided in this application Figure 1 ;

[0065] Figure 3Flowchart of the agent-driven case file content extraction method provided in this application Figure 2 ;

[0066] Figure 4 A schematic diagram illustrating part of the content extraction process from case file documents provided for this application;

[0067] Figure 5 A schematic diagram of the intelligent agent-driven case file content extraction device provided in this application;

[0068] Figure 6 A schematic diagram of the structure of the agent-driven case file content extraction device provided in this application.

[0069] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0071] The extraction of case file information can be divided into two levels: text-based recognition and information extraction based on case knowledge.

[0072] In text recognition, image OCR technology and multimodal VLM (Vision-Language Model) large-scale model technology can be used. The former improves the accuracy of image recognition in OCR by using a large number of labeled samples, while the latter improves the character recognition accuracy of the VLM large-scale model by automatically generating features of individual characters in different fonts.

[0073] Traditional image OCR extraction methods suffer from high annotation costs and high error rates. Furthermore, significant differences in handwriting styles between investigators and other document writers in case files make handwriting recognition difficult and annotation costs prohibitively high. Additionally, it is difficult to annotate samples from all writers before system and model operation, making handwriting OCR recognition impractical. Moreover, image OCR cannot generate usable knowledge triples during the information extraction stage, limiting its application in case file information extraction.

[0074] Information extraction based on multimodal large model technology faces three significant challenges. First, current multimodal technologies tend towards zero-shot / few-shot annotation, making it difficult to effectively fine-tune these methods directly for OCR problems. Even with fine-tuning methods (similar to image OCR), multimodal large models struggle to simultaneously recognize handwriting features from all writers. Second, multimodal large models integrate features from language large models, often referencing text features from the large model when handling single tasks. For example, when handwriting recognition accuracy is low, to ensure a sufficient number of tokens output for the recognition task, the multimodal large model introduces existing textual context from the model, leading to significant deviations between the recognition results and the original text information in the image. Finally, signature information in handwriting is often very illegible, and some crucial information related to case investigation may significantly deviate from the original handwritten content when lacking background knowledge.

[0075] This application provides an agent-driven method for extracting case file content. This method introduces agent technology to drive a large language model and a multimodal large model, focusing on the automated extraction of case file content. Driven by the agent, a zero-shot recognition method is provided for handwritten text that is difficult to recognize in paper case files, significantly improving the efficiency and accuracy of text recognition and knowledge extraction. This method utilizes an agent-based multimodal large model to extract easily recognizable printed text content and generate corresponding knowledge triples. Based on this, entity information and key case information are generated for the printed pages, and prompts and recognition are provided for the handwritten pages, thereby effectively improving the accuracy of handwritten information recognition and solving the technical problem of low accuracy in current case file text recognition.

[0076] Figure 1 A schematic diagram illustrating a scenario for the agent-driven case file content extraction method provided in this application, such as... Figure 1 As shown, an intelligent agent, a large language model, and a multimodal large model can be deployed in the terminal device 101. The acquired case file can be paginated on the terminal device 101 to obtain pagination results. For each page in the pagination results, a first question about the page content is generated. Based on the generated first question, the intelligent agent selects and calls the corresponding model to process the page, obtaining the page processing result. The terminal device 101 can be various types of computing devices, including desktop computers, laptops, and tablets.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0078] Figure 2 Flowchart of the agent-driven case file content extraction method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0079] S201. Perform pagination on the obtained case file to obtain the pagination result.

[0080] The acquired case files are paginated to break them down into multiple independently processable pages, ensuring that each page can be identified and analyzed individually.

[0081] S202. For each page in the pagination results, generate a first question about the page content.

[0082] Generate an initial query for each page. The purpose of this step is to generate specific questions or queries for the content of each page. These questions are used to guide subsequent processing steps, such as determining the content type of the page or the processing methods to be applied.

[0083] S203. Based on the generated first question information, the agent selects and calls the corresponding model to process the page and obtains the page processing result. The page processing result includes at least the first character recognition result for the printed page, the case file knowledge extraction result extracted based on the first character recognition result, and the second character recognition result for the handwritten page. The second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the case file knowledge extraction result.

[0084] An intelligent agent is an entity that has the ability to make autonomous decisions, interact, and execute. It can perceive the environment, understand needs, and dynamically drive the generation of content or complete complex tasks through multiple rounds of interaction or task execution.

[0085] Based on the generated initial question information, the agent selects and invokes the appropriate model to process the page. The processing results include:

[0086] First character recognition result: This is the character recognition result for the printed page.

[0087] Case file knowledge extraction results: Case knowledge extracted from the case file based on the first character recognition results. This may involve factual descriptions of the case and related evidentiary information. This may include the course of events, witness testimonies, descriptions of physical evidence, etc.

[0088] Second text recognition result: This is the text recognition result for the handwritten page.

[0089] The second character recognition result is obtained by combining the first character recognition result and the case file knowledge extraction result to recognize the page content of the handwritten page.

[0090] The agent-driven case file content extraction method provided in this application first performs pagination on the acquired case file to divide it into multiple pages, resulting in a pagination result. For each page's content, a first query is generated. This first query guides subsequent processing steps, ensuring the relevance and effectiveness of the processing. Based on the generated first query, the agent selects and invokes an appropriate model to process each page, obtaining a page processing result. The page processing result includes at least: a first character recognition result: character recognition results for printed pages; a case file knowledge extraction result: case file-related knowledge extracted based on the first character recognition result; and a second character recognition result: character recognition results for handwritten pages. The second character recognition result is obtained by combining the first character recognition result and the case file knowledge extraction result to recognize the page content of the handwritten pages. By utilizing the results of printed character recognition and case file knowledge extraction to enhance the accuracy of handwritten recognition, the method achieves more accurate recognition and extraction of text information in the case file, reducing errors and omissions.

[0091] Figure 3 Flowchart of the agent-driven case file content extraction method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the method for extracting case file content driven by intelligent agents is described in detail. The method includes:

[0092] S301. Perform pagination on the obtained case file to obtain the pagination result.

[0093] In this embodiment of the application, the acquired case file can be paginated to divide long documents into separate pages for subsequent processing.

[0094] Specifically, the pagination process may include: determining the target pagination tool based on the file format of the case file, and using the target pagination tool to paginate the case file.

[0095] In one implementation, it is necessary to first identify the file format of the case file (such as PDF, TIFF, etc.), and then select an appropriate pagination tool for pagination processing.

[0096] Taking PDF document segmentation in Python as an example, the process includes the following three stages:

[0097] Use the "import" statement to import libraries used for reading and writing PDF files, such as "PdfReader" and "PdfWriter". These libraries provide functionality for reading and writing PDF files.

[0098] Use "PdfReader" to read the page numbers from the original file. This step is to determine the total number of pages in the document so that page-by-page processing can be performed.

[0099] Each page is generated sequentially, and "PdfWriter" is called to output the file. This means that each page is extracted from the original file and saved as a separate page file for subsequent text recognition and knowledge extraction processing. The output page files can be named using page numbers. This naming convention ensures that the page files are arranged sequentially and are easily identifiable.

[0100] S302. For each page in the pagination results, generate a first question about the page content.

[0101] The first question can be generated by the agent based on predefined rules, or it can be generated through interaction between the agent and the user.

[0102] When processing a single page, the page type can be identified first. Page type identification can be achieved by driving a multimodal large model through dialogue and question-and-answer methods.

[0103] In one example, a large multimodal model can be driven by the following initial question information:

[0104] 1a. "Is the text in the given image handwritten or printed?"

[0105] 1b. "When recognizing text in an image, can the text recognition rate reach 90% and the recall rate reach 80%?"

[0106] 1c. "If the text is handwritten and the character recognition accuracy cannot meet the above recognition accuracy requirements, is the image a record of XX materials or a form of XX?"

[0107] Question 1a aims to determine the basic category of the text. This step helps in selecting an appropriate text recognition technology, as handwritten and printed text require different processing methods. Question 1b evaluates the effectiveness of the current recognition technology on the image. If both the recognition rate and recall rate reach the set thresholds, the current method is sufficiently effective; otherwise, it may be necessary to adjust the strategy or use a more advanced model. Question 1c further categorizes the types of handwritten content. This helps in selecting a more specialized recognition process. For example, XX material records and XX tables may have different formats and content structures, requiring different processing strategies for recognition.

[0108] S303. For the generated first question information, the agent generates the corresponding first answer information.

[0109] Based on the generated initial question information, the agent can invoke a multimodal large model to perform page type recognition and text recognition accuracy analysis, and generate corresponding initial response information. The initial response information may include page type recognition results and text recognition accuracy evaluation results, etc.

[0110] S304. When the first response information indicates that the current page is a printed page, perform text recognition on the printed page to obtain the first text recognition result, and call the pre-trained large language model to perform entity extraction and entity relationship extraction on the first text recognition result to obtain the case file knowledge extraction result.

[0111] In one possible implementation, when the first response information indicates that the current page is a printed page, text recognition is performed on the printed page to obtain a first text recognition result, which may specifically include:

[0112] When the first response indicates that the current page is a printed page, and the text recognition accuracy for the printed page meets the recognition accuracy requirements in the first question, a first prompt is generated. The first prompt includes the text recognition requirements for the printed page.

[0113] The multimodal large model is invoked to perform text recognition on the printed page, and the first text recognition result is output in combination with the first prompt information.

[0114] In this embodiment, if the first response information confirms that the current page is a printed page, and confirms that the text recognition accuracy of the current recognition of the printed page meets the recognition accuracy requirements in the first question information, then a first prompt information will be generated. The purpose of the first prompt information is to ensure the recognition quality in subsequent processing and to provide specific operational guidance, including but not limited to:

[0115] Format consistency: Ensure that the recognition results maintain the original format of the page.

[0116] Text accuracy: Emphasizing the accuracy of text recognition to avoid misspellings and omissions.

[0117] Date validity: Pay special attention to the format and validity of dates to prevent misidentification.

[0118] In one example, a large multimodal model can be driven by the following initial cue message:

[0119] 2a. “Extract all text from the page and output it according to the page's format.”

[0120] 2b. "Pay attention to the consistency of the text in the page recognition results, and pay attention to the legality of the dates in the recognition results."

[0121] Tip 2a instructs the model to extract all text on the page while maintaining the integrity of the page format. Tip 2b reminds the model to pay attention to text consistency and date validity during the recognition process to improve the accuracy of the recognition results.

[0122] Based on the initial prompt, the multimodal large model, when processing printed pages, first performs image analysis to identify all text content and attempts to preserve the page's formatting. In the output, the model checks text consistency, ensuring that the same words are recognized consistently in different positions, and validates dates to ensure their legitimacy. This method not only improves the accuracy of text recognition but also enhances the readability and reliability of the output.

[0123] Finally, the output of the first character recognition result is associated with the page number (provided by the page file name) and stored in the corresponding text set to ensure that the recognition result of each page can be accurately recorded and retrieved for subsequent processing and analysis.

[0124] Based on the first character recognition result, a pre-trained large language model can be used for entity extraction and entity relation extraction to generate case file knowledge extraction results. Entity extraction refers to identifying key entities in the text, such as names of people, places, and dates. Entity relation extraction refers to identifying the relationships between entities to construct a case file knowledge graph.

[0125] In one possible implementation, a pre-trained large language model is invoked to perform entity extraction and entity relation extraction on the first character recognition result to obtain the case file knowledge extraction result, which may specifically include:

[0126] The second question information is generated based on the first character recognition result. The second question information is used to determine the case type to which the case content in the printed page belongs.

[0127] For the generated second question, the agent generates a corresponding second answer.

[0128] Based on the case type answered in the second response information, a second prompt information is generated. The second prompt information includes one or more of the following: entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first character recognition result.

[0129] The large language model is invoked to extract entities and entity relationships from the first character recognition result, and the case file knowledge extraction result is output in combination with the second prompt information.

[0130] In this embodiment, in response to obtaining a first text recognition result by performing text recognition on the printed page, a second question is generated. The second question can be generated by the intelligent agent based on predefined rules, or it can be generated through interaction between the intelligent agent and the user. The purpose of the second question is to determine the case type to which the case content on the page belongs.

[0131] The beginning of a case file usually contains a description of the case, which can effectively establish the type and nature of the case. Therefore, the second question can be as follows:

[0132] 3a. Does this page specifically describe the nature of the case?

[0133] If the answer to the above question is "yes", then continue with question 3b. "What type of case does this case belong to? Category I, Category II, Category III, Category IV, Category V, Category VI, or other types of cases at the same level. If it is not among the above types, please specify the case type."

[0134] If the answer to the above question is "no", proceed to the next page and mark the current page number for further processing.

[0135] The agent generates a second answer based on the second question information. Based on the case type answered in the second answer information, a second prompt information is generated. The second prompt information includes one or more of the following: entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first character recognition result.

[0136] In one example, the following second prompt information can drive a large speech model to extract case knowledge:

[0137] 4a. “Extract all entities and their relationships from the text, and output the triples in JSON format.”

[0138] 4b. “Pay attention to the common perpetrators and steps involved in the crime, depending on the type of case.”

[0139] Hint 4a instructs the model to extract all entities and their relationships from the text and output knowledge triples in JSON format. A knowledge triple is a structured knowledge representation used to describe entities and their relationships, typically presented as a (subject, relation, object) triple. It breaks down complex knowledge into understandable and computable units through a concise semantic structure.

[0140] Tip 4b reminds users to pay attention to common perpetrators and modus operandi in this type of case during knowledge extraction. This means that when processing text, the model needs to combine background knowledge of specific case types in order to more accurately identify and understand relevant entities and relationships.

[0141] Based on the second cue information, a large language model is used to extract case file knowledge. In practice, if a page can indicate the case type, it can be used as an identifier page. All pages are divided into multiple page sets, and pages belonging to the same set use the same second cue information during case file knowledge extraction. The identifier page contains key information that indicates or explains the case type. Classification based on the identifier pages yields page sets corresponding to different case types. The purpose of this is to allow for the application of specific processing strategies based on case type during subsequent knowledge extraction.

[0142] The triples extracted from pages that can describe the case type are called "background triples", and the triples extracted from pages that cannot describe the case type are called "sequence triples". The output triples (case knowledge extraction results) are associated with the pages, and the output triple results are recorded in the triple set corresponding to the page number (given by the page file name).

[0143] S305. When the first response indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined, and a pre-trained multimodal large model is called to perform character recognition on the handwritten page to obtain the second character recognition result.

[0144] In one possible implementation, when the first response indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined, and a pre-trained multimodal large model is invoked to perform character recognition on the handwritten page to obtain a second character recognition result. Specifically, this may include:

[0145] When the first response indicates that the current page is a handwritten page and the text recognition accuracy of the handwritten page does not meet the recognition accuracy requirements in the first question, a third prompt is generated based on the handwritten page type to which the handwritten page belongs, as indicated in the first response. The third prompt includes the page content extraction requirements for the handwritten page.

[0146] The fourth prompt information is generated based on the knowledge extraction results from the case file. The fourth prompt information includes entity and entity relationship reference information for the handwritten page.

[0147] The fifth prompt message is generated based on the first character recognition result. The fifth prompt message includes contextual reference information for the handwritten page.

[0148] The multimodal large model is invoked to perform text recognition on the handwritten page, and the second text recognition result is output by combining the third, fourth and fifth prompt information.

[0149] In this embodiment, if the first response confirms that the current page is a handwritten page, and confirms that the current character recognition accuracy for recognizing the handwritten page cannot meet the recognition accuracy requirements in the first question, then a third prompt can be generated based on the handwritten page type indicated in the first response. The purpose of the third prompt is to ensure the recognition quality in subsequent processing and to provide specific operational guidance.

[0150] In one example, when the handwritten page mentioned in the first response is a transcript-type handwritten page, the generated third prompt message may look like this:

[0151] 5a-1. “Extract the text from the image, paying attention to the correspondence between the number of lines of text in the image and the questions and answers in the transcript.”

[0152] 5a-2. "Text lines need to maintain their place value, and the text needs to be fully recognized."

[0153] Hint 5a-1 instructs you to extract text from images, paying attention to the correspondence between the number of text lines and the question-and-answer format. Hint 5a-2 emphasizes maintaining the placeholder text lines and ensuring complete recognition.

[0154] In one example, when the handwritten page mentioned in the first response is of type XX table-type handwritten page, the generated third prompt message may look like this:

[0155] 6b-1. “Extract text from an image and output it in an image table format”.

[0156] 6b-2. "When printing all the printed text, placeholders should be provided for characters that cannot be recognized by handwriting."

[0157] Hint 6b-1 instructs you to extract the text from the image and output it in a table format. Hint 6b-2 requires you to print all printed text and use placeholders for unrecognizable handwritten characters.

[0158] The fourth prompt message is generated based on the knowledge extraction results from the case file. Specifically, the text on the page can be extracted based on the background triples and sequence triples in the case file knowledge extraction results. The relevance can be determined based on the page number.

[0159] In one example, the generated fourth prompt message could look like this:

[0160] 7a. “Extract the text from the image again, paying attention to the correctness and consistency of the entities.”

[0161] 7b. “The entities and their relations are as follows: [{background triples}, {sequence triples}]”

[0162] The fourth prompt provides background and sequence triple information to help identify entities and their relationships.

[0163] Furthermore, a fifth prompt message is generated based on the first character recognition result. Specifically, related text content can be extracted from the first character recognition result to extract the text on the page.

[0164] In one example, the generated fifth prompt message might look like this:

[0165] 8a. “Extract the text from the image again, paying attention to the context and reference text.”

[0166] 8b. “Contextual reference text includes: {text on the page corresponding to the background triple} {text on the page corresponding to the sequence triple}”

[0167] The fifth prompt provides contextual text for reference, including the page text corresponding to the background triplet and the page text corresponding to the sequence triplet.

[0168] Reference Figure 4 The diagram shown illustrates part of the content extraction process for case file documents provided in this application. As shown, the correlation between handwritten and printed pages can be determined based on page numbers, and then the handwritten text can be recognized by combining the prompts (text recognition prompts and knowledge extraction prompts) given on the printed page.

[0169] In one possible implementation, the pre-trained multimodal large model includes an image encoder, which is specifically used for:

[0170] The page to be processed is divided into multiple tiles, each tile corresponding to a local area of ​​the page to be processed.

[0171] For any given image patch, determine its location code, convert its classification information into first embedding information, and convert its image content into second embedding information. Generate the original image code of the image patch based on the location code, the first embedding information, and the second embedding information.

[0172] The image encoder of this application can extract information from images of different scales, ensuring that the model can handle image data of various resolutions and details. This image encoder can achieve segmentation and recognition of large-scale images while ensuring that the positional information of the segmented image is not lost. By adjusting the hyperparameter of the positional encoding bit depth, encoding of different resolutions (such as 4K images) is achieved.

[0173] In existing technologies, image encoders often struggle to balance the details of high-resolution images with the overall integrity of low-resolution images when processing images of different resolutions, resulting in insufficient accuracy in image text recognition. The image encoder of this application, however, can effectively improve the accuracy of text recognition in images of different resolutions and is suitable for multimodal, large-scale image processing tasks.

[0174] It should be noted that the large language model mentioned above can be the DeepSeek model, and the multimodal large model mentioned above can be the Qwen model.

[0175] The agent-driven case file content extraction method provided in this application first performs pagination on the acquired case file to divide it into multiple pages, resulting in a pagination result. For each page's content, a first query is generated. This first query guides subsequent processing steps, ensuring the relevance and effectiveness of the processing. Based on the generated first query, the agent selects and invokes an appropriate model to process each page, obtaining a page processing result. The page processing result includes at least: a first character recognition result: character recognition results for printed pages; a case file knowledge extraction result: case file-related knowledge extracted based on the first character recognition result; and a second character recognition result: character recognition results for handwritten pages. The second character recognition result is obtained by combining the first character recognition result and the case file knowledge extraction result to recognize the page content of the handwritten pages. By utilizing the results of printed character recognition and case file knowledge extraction to enhance the accuracy of handwritten recognition, the method achieves more accurate recognition and extraction of text information in the case file, reducing errors and omissions.

[0176] Figure 5 This is a schematic diagram of the intelligent agent-driven case file content extraction device provided in this application, as shown below. Figure 5 As shown, the agent-driven case file content extraction device 50 provided in this embodiment includes:

[0177] The pagination processing module 501 is used to perform pagination processing on the acquired case file and obtain the pagination processing result.

[0178] The generation module 502 is used to generate a first question about the content of each page in the pagination processing result;

[0179] The page processing module 503 is used to select and call the corresponding model to process the page based on the generated first question information, and obtain the page processing result;

[0180] The page processing result includes at least the first character recognition result for the printed page, the case file knowledge extraction result extracted based on the first character recognition result, and the second character recognition result for the handwritten page;

[0181] The second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the case file knowledge extraction result.

[0182] In one possible implementation, the page processing module is specifically used for:

[0183] For the first question generated, the agent generates the corresponding first answer.

[0184] When the first response indicates that the current page is a printed page, the text recognition of the printed page is performed to obtain the first text recognition result. Then, a pre-trained large language model is called to extract entities and entity relationships from the first text recognition result to obtain the case file knowledge extraction result.

[0185] When the first response indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined, and a pre-trained multimodal large model is called to perform character recognition on the handwritten page to obtain the second character recognition result.

[0186] In one possible implementation, the pre-trained multimodal large model includes an image encoder, which is specifically used for:

[0187] The page to be processed is divided into multiple tiles, each tile corresponding to a local area of ​​the page to be processed.

[0188] For any given image patch, determine its location code, convert its classification information into first embedding information, and convert its image content into second embedding information. Generate the original image code of the image patch based on the location code, the first embedding information, and the second embedding information.

[0189] In one possible implementation, the page processing module is specifically used for:

[0190] When the first response indicates that the current page is a printed page, and the text recognition accuracy for the printed page meets the recognition accuracy requirements in the first question, a first prompt is generated. The first prompt includes the text recognition requirements for the printed page.

[0191] The multimodal large model is invoked to perform text recognition on the printed page, and the first text recognition result is output in combination with the first prompt information.

[0192] In one possible implementation, the page processing module is specifically used for:

[0193] The second question information is generated based on the first character recognition result. The second question information is used to determine the case type to which the case content in the printed page belongs.

[0194] For the generated second question, the agent generates a corresponding second answer.

[0195] Based on the case type answered in the second response information, a second prompt information is generated. The second prompt information includes one or more of the following: entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first character recognition result.

[0196] The large language model is invoked to extract entities and entity relationships from the first character recognition result, and the case file knowledge extraction result is output in combination with the second prompt information.

[0197] In one possible implementation, the page processing module is specifically used for:

[0198] When the first response indicates that the current page is a handwritten page and the text recognition accuracy of the handwritten page does not meet the recognition accuracy requirements in the first question, a third prompt is generated based on the handwritten page type to which the handwritten page belongs, as indicated in the first response. The third prompt includes the page content extraction requirements for the handwritten page.

[0199] The fourth prompt information is generated based on the knowledge extraction results from the case file. The fourth prompt information includes entity and entity relationship reference information for the handwritten page.

[0200] The fifth prompt message is generated based on the first character recognition result. The fifth prompt message includes contextual reference information for the handwritten page.

[0201] The multimodal large model is invoked to perform text recognition on the handwritten page, and the second text recognition result is output by combining the third, fourth and fifth prompt information.

[0202] The intelligent agent-driven case file content extraction device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0203] Figure 6 A schematic diagram of the intelligent agent-driven case file content extraction device provided in this application. Figure 6 As shown, the agent-driven case file content extraction device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0204] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0205] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0206] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0207] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0208] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0209] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0210] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0211] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0212] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0213] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0216] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0218] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An agent-driven docket content extraction method, comprising: The method comprises the following steps: performing pagination processing on the obtained case file to obtain a pagination processing result; for each page in the pagination processing result, generating first questioning information for the content of the page; based on the generated first questioning information, the intelligent agent selects and calls a corresponding model to process the page to obtain a page processing result; the page processing result at least contains a first character recognition result for a printed page, a case knowledge extraction result extracted based on the first character recognition result, and a second character recognition result for a handwritten page; wherein the second character recognition result is obtained by recognizing the content of the handwritten page based on the first character recognition result and the case knowledge extraction result.

2. The method of claim 1, wherein, based on the generated first questioning information, the intelligent agent selects and calls a corresponding model to process the page to obtain a page processing result, comprising: for the generated first questioning information, the intelligent agent generates corresponding first answer information; when the first answer information indicates that the current page is a printed page, performing character recognition on the printed page to obtain a first character recognition result, and calling a pre-trained large language model to perform entity extraction and entity relationship extraction on the first character recognition result to obtain a case knowledge extraction result; when the first answer information indicates that the current page is a handwritten page, combining the first character recognition result and the case knowledge extraction result, and calling a pre-trained multi-modal large model to perform character recognition on the handwritten page to obtain a second character recognition result.

3. The method of claim 2, wherein, the pre-trained multi-modal large model contains an image encoder, which is used to: segment the to-be-processed page into multiple tiles, each tile corresponding to a local area of the to-be-processed page; for any tile, determine its position code, convert its classification information into first embedding information, and convert its image content into second embedding information, and generate the original image code of the tile based on the position code, the first embedding information and the second embedding information.

4. The method of claim 2, wherein, when the first answer information indicates that the current page is a printed page, performing character recognition on the printed page to obtain a first character recognition result, comprising: when the first answer information indicates that the current page is a printed page, and the character recognition accuracy for the printed page meets the recognition accuracy requirement in the first questioning information, generate a first prompt information, the first prompt information contains the character recognition requirement for the printed page; call the multi-modal large model to perform character recognition on the printed page, and output the first character recognition result combined with the first prompt information.

5. The method of claim 2, wherein, calling the pre-trained large language model to perform entity extraction and entity relationship extraction on the first character recognition result to obtain a case knowledge extraction result, comprising: generating second questioning information based on the first character recognition result, the second questioning information is used to determine the case type to which the case content in the printed page belongs; for the generated second questioning information, the intelligent agent generates corresponding second answer information; generate second prompt information based on the case type answered in the second answer information, the second prompt information including one or more of entity extraction requirements, entity relationship extraction requirements, and output format requirements for the first character recognition result; invoke the large language model to perform entity extraction and entity relationship extraction on the first character recognition result, and output the case file knowledge extraction result in combination with the second prompt information.

6. The method of claim 2, wherein, When the first answer information indicates that the current page is a handwritten page, the first character recognition result and the case file knowledge extraction result are combined to invoke a pre-trained multi-modal large model to perform character recognition on the handwritten page to obtain a second character recognition result, including: When the first answer information indicates that the current page is a handwritten page, and the character recognition accuracy for the handwritten page does not meet the recognition accuracy requirement in the first question information, generate third prompt information based on the handwritten page type of the handwritten page answered in the first answer information, the third prompt information including page content extraction requirements for the handwritten page; generate fourth prompt information based on the case file knowledge extraction result, the fourth prompt information including entity and entity relationship reference information for the handwritten page; generate fifth prompt information based on the first character recognition result, the fifth prompt information including context reference information for the handwritten page; invoke the multi-modal large model to perform character recognition on the handwritten page, and output the second character recognition result in combination with the third prompt information, the fourth prompt information, and the fifth prompt information.

7. An agent-driven docket content extraction apparatus, comprising: including: a paging processing module configured to perform paging processing on the obtained case file to obtain a paging processing result; a generation module configured to generate, for each page in the paging processing result, first question information for page content; a page processing module configured to, based on the generated first question information, select and invoke a corresponding model by the agent to process the page to obtain a page processing result; the page processing result at least includes a first character recognition result for a printed page, a case file knowledge extraction result extracted based on the first character recognition result, and a second character recognition result for a handwritten page; wherein the second character recognition result is obtained by recognizing the page content of the handwritten page based on the first character recognition result and the case file knowledge extraction result.

8. An agent-driven docket content extraction apparatus, comprising: including: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method of any one of claims 1-6.

9. A computer readable storage medium, characterized in that, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images