A method, device and medium for generating recognition results using OCR

Abstract

The application provides a method, device and medium for generating an identification result by using OCR, applied to the technical field of archive digitization, generates an identification control instruction according to a predefined extended mark specification, guides a multi-modal large model to identify page elements in a to-be-identified document, obtains page element identification information, determines a corresponding extended label according to a label mapping rule, and structurally encapsulates the page element identification information by using the extended label according to a format requirement, realizes unified mapping between the page element identification information and the extended label, and standardized expression and bearing of different page element identification information, generates a standardized OCR identification result convenient for unified analysis and utilization, and solves the problem that the existing OCR identification result is mainly expressed in pure text, unified label definition and encapsulation specification are lacking in a general format, and page element identification information is difficult to be uniformly expressed, completely borne and effectively utilized.

Description

Technical Field

[0001] This application relates to the field of archival digitization technology, specifically to a method, device, and medium for generating recognition results using OCR. Background Technology

[0002] Optical Character Recognition (OCR) technology is used to recognize text in document images and convert the text content into editable and searchable text. Traditional OCR result generation schemes typically output the recognized text as plain text in the order it appears in the document; some structured recognition result generation schemes use formats such as Markdown to organize the recognized content. However, existing schemes still have the following problems: Information dimension missing: The plain text recognition results are mainly used to carry the recognized text content, and it is difficult to retain the page element categories, page attributes and their correspondence with the recognized content at the same time, which makes it difficult to uniformly express and structurally carry the page element recognition information in the document; Semantic boundaries cannot be distinguished: Existing recognition results usually lack differentiated markers for different page elements, and cannot effectively distinguish different layout areas such as headers, footers, annotations and body text, making it difficult to accurately distinguish and utilize the recognition information corresponding to different page elements. The expressive capabilities of general Markdown are insufficient: native Markdown lacks standardized tag definitions, cannot describe document physical attributes such as page numbers, stamps, stains, handwriting, and images, and lacks a unified standard to carry the full metadata of OCR.

[0003] Therefore, a new OCR recognition result generation scheme is needed. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide a method, device and medium for generating recognition results using OCR, which can uniformly express and structurally carry all physical information and regional semantics of a document, and provide standardized recognition results for scenarios such as large model knowledge bases, intelligent retrieval and archive compliance verification.

[0005] The embodiments in this specification provide the following technical solutions: This specification provides an embodiment of a method for generating recognition results using OCR, the method comprising: Obtain a predefined extended tag specification, which includes tag mapping rules and format requirements. The tag mapping rules are used to specify the correspondence between page element identification information and extended tags. Generate identification control instructions according to the extended marking specification; Input the document to be identified and the identification control command into the multimodal large model; The page elements in the document to be identified are identified using the multimodal large model to obtain the page element identification information; Based on the page element identification information and the tag mapping rules, determine the extended tag corresponding to the page element identification information; According to the format requirements, the page element identification information is structured and encapsulated using the extended tags to generate standardized OCR recognition results.

[0006] Further, the step of identifying page elements in the document to be identified using the multimodal large model to obtain page element identification information includes: The identification content, location information, and visual information of each page element are extracted using the multimodal large model. Based on the positional and visual information of each page element, determine the page element category and page attributes of each page element; The page element identification information is obtained by associating the identified content, page element category, and page attribute of each page element.

[0007] Furthermore, the page elements include entity elements and text elements, and determining the page attributes of each page element based on its positional and visual information includes: Based on the positional and visual information of the entity element and the text element, determine the entity region corresponding to the entity element and the text region corresponding to the text element; The occlusion relationship between the entity element and the text element is determined based on the overlap between the entity region and the text region. The occlusion relationship is written as a page attribute into the page element identification information; Based on the page element identification information written into the occlusion relationship and the tag mapping rules, determine the extended tag corresponding to the occlusion relationship; According to the format requirements, the entity element, the text element, and the occlusion relationship are structurally encapsulated using the extended tags corresponding to the occlusion relationship, and then written into the standardized OCR recognition result.

[0008] Further, the step of generating identification control instructions according to the extended tag specification includes: The tag selection constraint is determined according to the tag mapping rule. The tag selection constraint is used to constrain the multimodal large model to determine the corresponding extended tags according to each of the page elements. The output structure constraints are determined according to the format requirements. The output structure constraints are used to constrain the multimodal large model to use the extended tags corresponding to each page element to structurally encapsulate the identification information of the corresponding page elements. The label selection constraint and the output structure constraint are combined to generate the recognition control instruction.

[0009] Furthermore, the extended tags are used to structurally encapsulate the page element identification information to generate standardized OCR recognition results, including: The page element identification information is structured and encapsulated using the extended tags to obtain candidate identification results; The candidate recognition results are analyzed to obtain start and end markers, required extended labels, label data, and recognition content; According to the format requirements, the pairing relationship of the start and end identifiers, the integrity of the required extended labels, and the data format of the label data are verified for compliance, and the format verification result is obtained. Detect whether the identified content is repeated continuously to obtain the content detection result; Based on the format verification result and the content detection result, the verification result of the candidate recognition result is determined; In response to the verification result indicating that the candidate recognition result has failed the verification, the generation parameters of the multimodal large model are automatically adjusted, and the document to be recognized and the recognition control command are re-inputted into the adjusted multimodal large model to obtain the updated candidate recognition result; The updated candidate recognition result is used as the candidate recognition result, and the process of parsing the candidate recognition result is returned until the verification is passed or the preset number of retries is reached. In response to the verification result indicating that the candidate recognition result has passed the verification, the candidate recognition result is determined as the standardized OCR recognition result.

[0010] Furthermore, adjusting the generation parameters of the multimodal large model includes: Anomaly analysis is performed on the candidate recognition results that fail the verification to determine the output anomaly type corresponding to the candidate recognition results; In response to the output anomaly type being start / end identifier mismatch, missing required extended labels, or label data format not conforming to the format requirements, the generation temperature parameter of the multimodal large model is reduced; In response to the output anomaly type being continuous repetition of identified content, the generation temperature parameter of the multimodal large model is increased.

[0011] Furthermore, the method also includes: Obtain the verification result, the generated temperature parameter, and the number of retries for each compliance verification; Based on the verification results, the generated temperature parameters, and the number of retries, a processing log is generated; The processing log is associated with the corresponding standardized OCR recognition result to generate an audit record of the standardized OCR recognition result, and the audit record is stored.

[0012] Furthermore, the method also includes: Obtain the document source information of the document to be identified; According to the extended tag specification, the extended tags corresponding to the document source information are determined, and the document source information is structured and encapsulated using the extended tags corresponding to the document source information. The encapsulated document source information is then written into the standardized OCR recognition result. The standardized OCR recognition results are stored in a knowledge base; The extended tags in the standardized OCR recognition results are analyzed to obtain the document source information, page identifier, and recognition content, as well as the correspondence between the document source information, page identifier, and recognition content and the corresponding extended tags; A tag index is established based on the extended tags and the corresponding relationships; The tag index is queried based on the received query conditions to obtain the identified content, document source information and page identifier that match the query conditions; Based on the document source information and the page identifier, the source location of the identified content in the document to be identified is determined.

[0013] This specification also provides an electronic device, including: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform: the aforementioned method for generating recognition results using OCR.

[0014] This specification also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, perform the aforementioned method for generating recognition results using OCR.

[0015] Compared with the prior art, the technical effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: This application generates recognition control instructions based on predefined extended tag specifications, guiding a multimodal large model to recognize page elements in the document to be recognized, obtaining page element recognition information, and determining corresponding extended tags according to tag mapping rules. Following format requirements, the extended tags are used to structurally encapsulate the page element recognition information, achieving a unified mapping between page element recognition information and extended tags, as well as standardized expression and carrying of different page element recognition information. This generates standardized OCR recognition results that are easy to uniformly parse and utilize, solving the problem that existing OCR recognition results are mainly expressed in plain text, lacking unified tag definitions and encapsulation specifications in general formats, which makes it difficult to uniformly express, fully carry, and effectively utilize page element recognition information. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a method for generating recognition results using OCR in a specific embodiment of this application; Figure 2 This is a schematic diagram of the structure of an electronic device according to a specific embodiment of this application. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0023] Regarding existing OCR recognition result generation schemes, such as the prior art submitted in the background section, through analysis of the problems and exploration of solutions, it was found that: multimodal large models usually output recognition results in a free-generating manner; existing OCR recognition results mainly carry text content and are difficult to retain page element categories, page attributes and their correspondence with the recognized content; different page elements lack differentiated markers, and the labels and data formats used by different structured recognition schemes are not uniform, making it difficult to uniformly express and structurally carry page element recognition information.

[0024] In the process of solving the above-mentioned technical problems, the inventors further discovered that existing OCR recognition results cannot be bound to the original document and page position, making it difficult to accurately locate the original page number and page block during intelligent question answering, and also making it difficult to support original page jump to trace the source, as well as to filter and query according to page elements, page attributes or specified page numbers.

[0025] Based on this, the embodiments of this specification propose a processing scheme. The overall idea is as follows: An extended tag specification including tag mapping rules and format requirements is pre-constructed, and recognition control instructions are generated based on the extended tag specification. The document to be recognized and the recognition control instructions are input into a multimodal large model to obtain page element recognition information of the document to be recognized. Based on the tag mapping rules, corresponding extended tags are determined for the page element recognition information. The page element recognition information is then structured and encapsulated using the extended tags according to the format requirements, generating a standardized OCR recognition result with a unified tag mapping relationship and encapsulation form. This allows for the complete preservation of multi-dimensional information such as document page numbers, layout blocks, image types, stains, stamps, handwriting, occlusion relationships, and original image parameters. The output content is compatible with ordinary Markdown parsing, improving the accuracy of document AI parsing, knowledge base construction, and archive retrieval.

[0026] The technical solutions provided by the various embodiments of this application are described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown in the embodiments of this specification, a method for generating recognition results using OCR is provided, including: Step S10: Obtain the predefined extended tag specification and generate identification control instructions according to the extended tag specification.

[0028] The extended tag specification defines the tagging and structured representation of page element identification information. It includes tag mapping rules and formatting requirements. The tag mapping rules define the correspondence between page element identification information and extended tags, while the formatting requirements define the organization and encapsulation of extended tags and page element identification information.

[0029] Specifically, the extended tagging specification can be pre-stored in the electronic device executing this method, or it can be retrieved by the electronic device from a configuration file, database, or specification management service. Before performing the document recognition task, the electronic device reads the extended tagging specification and converts the content in the extended tagging specification related to the current recognition task into recognition control instructions that can be recognized and executed by the multimodal large model.

[0030] For example, tag mapping rules and format requirements can be incorporated into recognition control instructions, enabling multimodal large models to process recognition results according to extended tag specifications when recognizing documents.

[0031] For example, a recognition instruction template can be pre-set, and the label mapping rules and format requirements can be filled into the corresponding positions in the recognition instruction template to generate the recognition control instructions corresponding to the current recognition task.

[0032] It should be noted that the specific form of the identification control command can be determined based on the multimodal large model used and its command input method.

[0033] Step S20: Obtain the document to be identified, input the document to be identified and the identification control command into the multimodal large model, and use the multimodal large model to identify the page elements in the document to be identified, thereby obtaining the page element identification information.

[0034] Specifically, the document to be identified is a document containing the content of the page to be identified, which can be a single-page document or a multi-page document. The document to be identified can be uploaded by the user or obtained from a preset storage location or a document management system. For the document to be identified in the form of a document file, the corresponding document image can be extracted according to the page and used as the visual input of the multimodal large model.

[0035] The document to be recognized and the recognition control instructions generated in step S10 are input into the multimodal large model. The multimodal large model performs visual analysis and content recognition on the document to be recognized according to the recognition control instructions, identifies page elements in the document that can form recognition results, and generates corresponding page element recognition information for the identified page elements. The page element recognition information is used to describe the recognition results of the page elements and serves as the processing object for subsequent determination of extended tags and structured encapsulation.

[0036] It should be noted that for multi-page documents to be recognized, the multimodal model can be entered page by page, or the multimodal model can be entered according to a preset combination of page numbers. Alternatively, the recognition information of page elements corresponding to different pages can be saved separately.

[0037] Step S30: Determine the extended tag corresponding to the page element identification information based on the page element identification information and the tag mapping rule.

[0038] Specifically, after obtaining the page element identification information, the page element described by the page element identification information is matched with the extended tag according to the tag mapping rules. The tag mapping rules pre-establish the correspondence between different page element identification information and extended tags. The matching correspondence is searched in the tag mapping rules according to the page element identification information, thereby determining the extended tag used to mark the page element identification information.

[0039] For example, under the constraints of recognition control instructions, the multimodal large model can directly determine extended labels for page element recognition information according to label mapping rules, and output the correspondence between page element recognition information and extended labels; For example, the multimodal large model first outputs page element identification information, and then the electronic device matches the page element identification information according to the label mapping rules to determine the corresponding extended labels; For example, a multimodal large model outputs some extended labels, and an electronic device supplements and determines the remaining extended labels according to the label mapping rules.

[0040] When the document to be identified contains multiple page elements, the corresponding extended tags are determined according to the page element identification information of each page element, and the correspondence between each extended tag and the corresponding page element identification information is maintained so that the corresponding page element identification information can be structurally encapsulated using each extended tag in accordance with the format requirements.

[0041] Step S40: According to the format requirements, the page element recognition information is structurally encapsulated using the extended tags to generate standardized OCR recognition results.

[0042] Specifically, after determining the extended tags corresponding to the page element identification information, the extended tags are combined with the corresponding page element identification information according to the format requirements, so that each page element identification information is written into the recognition result according to the structure specified by the extended tag specification. For multiple page element identification information, the corresponding extended tags are used for structured encapsulation, and the encapsulation results are organized according to the format requirements to form a standardized OCR recognition result.

[0043] This embodiment is applicable to the full-dimensional structured parsing of materials such as PDF archives, scanned paper documents, and on-site photographed documents, providing standardized recognition results with complete metadata and regional semantic tags for knowledge base construction, intelligent question answering, and archive retrieval.

[0044] In some implementations, the extended markup specification is the OCR Result Markup Language (ORML), which adds exclusive semantic markup on top of native Markdown. This allows for the unified annotation of full-dimensional document information that traditional OCR ignores, including: Basic page metadata tags used to mark the original document size, image resolution, and current page number; Layout area labels used to mark header blocks, footer blocks, body text, and comments; Content type tags used to mark printed text, handwritten content, regular images, and map / graph blocks; Used to mark page entities such as stains and stamps, and to record physical defects and entity labels related to the occlusion and overlay relationship between stamps and text.

[0045] This embodiment is compatible with Markdown at the underlying level and can be directly reused in regular text scenarios without modifying existing storage and parsing systems. While retaining the regular text expression form of Markdown, it uses uniformly formatted extended semantic tags to carry page metadata, layout areas, content types, and physical attribute information. This makes it easy for programs and users to identify the page element recognition information represented by different extended tags, and enables standardized OCR recognition results to fully cover all the auxiliary metadata required for archives and government documents, adapting to knowledge base and knowledge graph construction scenarios.

[0046] In some implementations, after the document to be identified and the identification control command are input into the multimodal large model, the multimodal large model performs visual analysis on the page image of the document to be identified, identifies each page element contained in the page image, and extracts the identification content, position information and visual information of each page element respectively.

[0047] For text-based page elements, the recognized content includes the text content identified from the page element; for image-based page elements, the recognized content includes the recognition result obtained based on the image content of the page element. Therefore, the recognized content of page elements includes not only printed text content, but also the recognition results of handwritten content, ordinary images, and map / chart content.

[0048] Location information indicates the position of a corresponding page element within the document page to be identified and the page area it occupies. For example, based on whether a page element is located at the top, bottom, or main body of the page, it can be determined whether the page element corresponds to a header block, footer block, or body text block.

[0049] Visual information is used to represent the visual representation of corresponding page elements in page images. Visual information includes information such as the text form, handwriting style, color, image form, and physical appearance of page elements. Based on the visual information of page elements, the multimodal large model distinguishes between printed text and handwritten content, distinguishes between text-based page elements and image-based page elements such as ordinary pictures, maps, and charts, and identifies page entities such as stamps and stains in page images.

[0050] The multimodal large model also identifies page-level information of the document to be identified, including the original document size, page image resolution, and current page number. This page-level information serves as the page attributes of each page element within the corresponding page, establishing a connection between the page elements and the page elements to preserve the basic metadata of the page containing each page element.

[0051] Based on the location and visual information of each page element, determine the page element category for each page element. Page element categories include layout area categories and content type categories. Layout area categories indicate whether a page element belongs to a header, footer, body text, or annotation / notes section; content type categories indicate whether a page element belongs to printed text, handwritten content, a regular image, or a map / chart.

[0052] Based on the positional and visual information of each page element, the corresponding page attributes are also determined. Page attributes include the current page number, text color, physical state of page elements, and the relationship between page elements. For page entities such as stamps and stains, page attributes record the visual appearance of the corresponding page entity and its position on the page; for cases where stamps and text overlap, page attributes also record the occlusion relationship between the corresponding page elements.

[0053] In some implementations, entity elements refer to non-text page elements in the document page to be identified that have corresponding physical forms or visual boundaries, such as stamps, stains, etc.; text elements refer to page elements in the document page to be identified that contain text recognition content, such as printed text elements and handwritten text elements.

[0054] The multimodal large model identifies the document by extracting the positional and visual information of entity elements and text elements. Based on the positional and visual information of entity elements, it determines the entity region occupied by the entity element on the document page; similarly, based on the positional and visual information of text elements, it determines the text region occupied by the text element on the document page. Entity regions and text regions each correspond to specific areas within the same document page, maintaining the correspondence between entity regions and their corresponding entity elements, as well as between text regions and their corresponding text elements.

[0055] After identifying the entity region and the text region, the entity region and the text region are compared to determine whether they overlap in the document page to be identified. In response to the existence of overlap between the entity region and the text region, an occlusion relationship is determined based on the overlapping entity elements and text elements, and a correspondence is established between the occlusion relationship and the corresponding entity elements and text elements.

[0056] For example, when the physical area corresponding to the seal overlaps with the text area corresponding to the text content, it is determined that there is an occlusion relationship between the seal and the text content. If the same seal occludes multiple text elements, the occlusion relationship between the seal and each text element is established separately; if multiple physical elements occlude the same text element, the occlusion relationship between each physical element and the text element is established separately.

[0057] After determining the occlusion relationship, the occlusion relationship is written as a page attribute into the page element identification information. The page element identification information that writes the occlusion relationship also records the entity elements and text elements involved in the occlusion relationship, as well as the association state between them, in order to maintain the correspondence between the occlusion relationship and the corresponding page elements.

[0058] The page element identification information of the occlusion relationship is matched with the tag mapping rule. The tag mapping rule specifies the correspondence between the occlusion relationship and the corresponding extended tag. Based on the correspondence, the extended tag corresponding to the current occlusion relationship is determined, so that the extended tag is used to mark the occlusion relationship between entity elements and text elements.

[0059] After determining the extended tags, following the format requirements in the extended tag specification, the entity elements, text elements, and their occlusion relationships involved in the occlusion relationship are structurally encapsulated using the extended tags corresponding to the occlusion relationships. The structured encapsulation result maintains the correspondence between entity elements, text elements, and occlusion relationships, and the encapsulation result is written into the standardized OCR recognition result.

[0060] For example, in cases where there is an occlusion relationship between the seal and the text content, the standardized OCR recognition result retains the page element recognition information corresponding to the seal, the page element recognition information corresponding to the text content, and the occlusion relationship between the two, and uses corresponding extended tags to mark the occlusion relationship. Therefore, when parsing the standardized OCR recognition result, the occlusion relationship between the seal and the text content is determined based on the extended tags.

[0061] This embodiment can solve the problem that traditional OCR recognition results are difficult to express the occlusion relationship between physical elements such as seals and stains and text elements.

[0062] In some implementations, generating recognition control instructions based on the extended tagging specification includes: the system pre-loading the complete ORML syntax specification and tag definitions as recognition control instructions; inputting the image to be recognized and the complete ORML specification (recognition control instructions) into the multimodal large model, forcing the output content to strictly follow the ORML format, and fully annotating all page-related information and metadata; finally, the multimodal large model, based on image visual information, synchronously extracts all information such as text, layout, images, seals, stains, and handwriting distinctions, and encapsulates them into ORML text output according to the specification.

[0063] In some implementations, the multimodal large model identifies the document to be identified according to recognition control instructions and uses corresponding extended tags to structurally encapsulate the recognition information of page elements. After completing the current round of recognition, the multimodal large model uses the generated structured text as a candidate recognition result.

[0064] After obtaining the candidate recognition results, the verification module parses them. The verification module identifies the start and end markers, extended tags, tag data, and recognition content in the candidate recognition results according to the extended tag specification, and determines the objects that need to be verified based on the parsing results.

[0065] Start and end markers are used to define the start and end positions of standardized OCR recognition results or corresponding structured blocks. When verifying the pairing relationship of start and end markers, it is determined whether the candidate recognition results contain corresponding start and end markers, and whether the corresponding start and end markers are organized according to the format requirements.

[0066] Required extension tags are extension tags that the extension tag specification stipulates must be included in the candidate recognition results. When verifying the completeness of required extension tags, the extension tags parsed from the candidate recognition results are compared with the required extension tags specified in the extension tag specification to determine whether the candidate recognition results are missing any corresponding required extension tags.

[0067] Tag data is the data carried or defined by extended tags. When validating the data format of tag data, the format requirements are used to determine whether the expression of the tag data conforms to the extended tag specification. For example, for tag data representing the current page number, the format requirements stipulate that page numbers should be expressed in Arabic numerals. If the candidate recognition result uses Chinese numerals to express the page number, the corresponding tag data is determined to be inconsistent with the format requirements.

[0068] After completing the format compliance check, the format verification result is obtained. The format verification result is used to indicate whether the start and end markers, required extended labels, and label data in the candidate recognition results meet the format requirements.

[0069] Furthermore, the system detects whether there are consecutive repetitions of the identified content in the candidate recognition results. Specifically, it checks adjacent content according to the order of the identified content. When the same character or the same content is detected to appear consecutively and form an abnormal repetition, it is determined that there is consecutive repetition of the identified content, and a content detection result indicating abnormal content is obtained; when no consecutive repetition of the identified content is detected, a content detection result indicating normal content is obtained.

[0070] The verification results of the candidate recognition results are determined based on the format verification results and the content detection results. When the pairing relationship of the start and end markers, the completeness of the required extended tags, and the data format of the tag data all meet the format requirements, and the recognition content does not have continuous repetition, the candidate recognition result is determined to pass the verification, and the candidate recognition result is determined as the standardized OCR recognition result.

[0071] Conversely, if a candidate recognition result fails verification, the generation parameters of the multimodal large model are adjusted in response. The document to be recognized and the recognition control instructions are re-inputted into the adjusted multimodal large model to re-initiate the recognition request, resulting in an updated candidate recognition result. This updated candidate recognition result is then used as the current candidate recognition result. The start and end markers, necessary extended tags, tag data, and recognition content are parsed again, and format compliance verification and content detection are performed again. This recognition, parsing, and verification process is repeated until a candidate recognition result that passes verification is obtained or the preset maximum number of retries is reached.

[0072] In some implementations, the maximum number of retries is set to 6, and the correction is performed cyclically until the output conforms to the ORML specification, thus solving the format disorder problems caused by image blurring, excessive size, and abnormal model generation degrees of freedom.

[0073] In some implementations, anomaly analysis is performed on candidate recognition results that fail verification. Based on the format verification results and content detection results, the specific reasons that caused the current candidate recognition result to fail verification are determined, and the specific reasons are classified into the corresponding output anomaly types.

[0074] When the format validation result indicates that the start and end identifiers in the candidate recognition results are mismatched, necessary extended labels are missing, or the label data format does not meet the requirements, the output anomaly type is determined to be a format anomaly. In response to this, the generation temperature parameter of the multimodal large model is reduced, thus reducing the randomness of the output when the multimodal large model generates recognition results according to the label mapping rules and format requirements. Re-identification is then performed using the reduced generation temperature parameter to reduce the recurrence of start and end identifier mismatches, missing necessary extended labels, or label data format non-compliance issues.

[0075] When the content detection result indicates that there are a large number of consecutive repeated characters or consecutive repeated content in the candidate recognition result, the output anomaly type is determined to be consecutive repetition of recognition content. In response to the output anomaly type being consecutive repetition of recognition content, the generation temperature parameter of the multimodal large model is increased to adjust the output state of the multimodal large model when generating recognition content, and recognition is re-performed based on the increased generation temperature parameter.

[0076] In some implementations, an initial generation temperature parameter can be selected based on the content type of the document to be recognized before performing the document recognition task. For documents with relatively fixed formats and textual expressions, such as legal documents, a relatively low initial generation temperature parameter is selected; for poems, picture albums, or art documents, a relatively high initial generation temperature parameter is selected. The initial generation temperature parameter is used to perform the first round of recognition. If the first round of recognition fails the verification, the generation temperature parameter is adjusted according to the type of output anomaly.

[0077] When document images contain easily confused characters, the multimodal large model combines the text's location, adjacent recognition content, and page visual information to determine the corresponding recognition content. For example, for the visually similar letter "o" and number "0", the multimodal large model determines the recognition result based on the visual information of adjacent text content and page elements, and adjusts the generated temperature parameters according to the corresponding output anomaly type when the recognition result shows format anomalies or continuous repetitions.

[0078] This embodiment analyzes candidate recognition results that fail verification to determine the type of output anomaly. For format anomalies, the generation temperature parameter is lowered; for continuous and repeated recognition of content, the generation temperature parameter is raised. This ensures that the adjustment direction of the generation temperature parameter corresponds to the anomaly state of the candidate recognition result, reducing repeated recognition caused by using the same parameter tuning method to handle different output anomalies, and increasing the possibility of obtaining candidate recognition results that conform to the extended label specification during the retry process.

[0079] In some implementations, after each round of candidate recognition result verification, the verification result, the corresponding generation temperature parameter, and the number of retries are recorded, and a processing log is generated according to the execution order of each round of recognition and verification. After obtaining the standardized OCR recognition result that has passed verification, the processing log is associated with the standardized OCR recognition result, and an audit record is generated and stored to determine the verification process and parameter adjustments that the standardized OCR recognition result has undergone based on the audit record.

[0080] This embodiment records the verification and retry logs for each round throughout the entire process, making the identification process auditable.

[0081] In some implementations, the knowledge base storage does not require additional cleaning and transformation; ORML-specific tags can be directly stored in the database without interfering with regular text retrieval.

[0082] Furthermore, in intelligent question-answering scenarios, standardized OCR recognition results are provided to the question-answering model as document content in the knowledge base. The question-answering model identifies the content of different page elements based on extended tags and reads the corresponding document source information and page identifiers. When generating answer content, the question-answering model outputs the document source information and page identifiers corresponding to the answer content as source information to determine the original file and specific page corresponding to the answer content, and supports one-click jump to the corresponding page's original text on the front end.

[0083] In some implementations, after the standardized OCR recognition results are stored in a knowledge base, the extended tags are parsed to determine the document source information, page identifier, and recognition content corresponding to each extended tag, and to obtain the correspondence between the extended tags and the aforementioned information. A tag index is then established based on the extended tags and the correspondence, so that each extended tag corresponds to the corresponding recognition content, its document source information, and page identifier.

[0084] After receiving the query conditions, the system determines the extended tags or page information to be queried based on the query conditions, and queries the tag index to obtain the identified content, document source information and page identifier that match the query conditions.

[0085] For example, search all pages marked with a stamp, query the tag index based on the extended tags corresponding to the stamp, and obtain the identified content and page identifier with the corresponding extended tags; For example, to retrieve the content of a specific header chapter, the tag index can be queried based on the extended tags corresponding to the header chapter; For example, when you need to quickly retrieve the recognition results for a specific page number, you can query the tag index based on the corresponding page identifier to obtain the recognition content on the corresponding page.

[0086] The system determines the original document to which the content to be identified belongs based on the document source information obtained from the query, and identifies the specific page where the content is located based on the page identifier, thus obtaining the source location of the content within the document to be identified. After obtaining the source location, the front end can open the corresponding original document based on the document source information and jump to the page where the content to be identified is located based on the page identifier, thereby achieving a fine-grained filtering that traditional plain text OCR cannot achieve.

[0087] This embodiment obtains the document source information of the document to be identified and uses corresponding extended tags to write the document source information into the standardized OCR recognition result, so that the identified content, document source information and page identifier remain consistent; further, a tag index is established based on the above correspondence, which can obtain the identified content and its source location according to the query conditions, thereby providing a data foundation for the filtering and querying of identified content and the tracing of the original text.

[0088] Based on the same inventive concept, this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: a method for generating recognition results using OCR as described in any embodiment of this application.

[0089] like Figure 2As shown in the figure, the present application also provides a schematic diagram of the structure of an electronic device 500. The electronic device 500 is merely an example and should not be construed as limiting the function and scope of use of the embodiments of the present invention.

[0090] The electronic device 500 may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein the memory 520 stores instructions executable by the at least one processor 510, the instructions being executed by the at least one processor 510 to enable the at least one processor 510 to perform: the method for generating recognition results using OCR as described in any embodiment of this application.

[0091] It should be noted that the electronic device 500 can be represented in the form of a general-purpose computing device, such as a server device.

[0092] In practice, the components of electronic device 500 may include, but are not limited to: at least one processor 510, at least one memory 520, and a bus 530 connecting different system components (including memory 520 and processor 510), wherein bus 530 may include a data bus, an address bus, and a control bus.

[0093] In implementation, memory 520 may include volatile memory, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include read-only memory (ROM) 5203.

[0094] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] The processor 510 performs various functional applications and data processing by running computer programs stored in the memory 520.

[0096] Electronic device 500 can also communicate with one or more external devices 540 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 560, which communicates with other modules in electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0097] Based on the same inventive concept, embodiments of this application also relate to a computer-readable medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the method for generating recognition results using OCR as described in any embodiment of this application. This computer-readable medium may be a memory included in an electronic device, or it may be a standalone storage medium not assembled into the device.

[0098] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium, a computer-readable storage medium, or a combination of both. Examples include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. Specific examples of storage media may include, but are not limited to, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, optical fibers, portable CD-ROMs, optical storage devices, magnetic storage devices, etc., or any suitable combination thereof.

[0099] Computer-readable media may store one or more programs that can be used by or in conjunction with an instruction execution system. The media may be permanent or non-permanent, removable or non-removable, and may store information by any method or technology, including computer-readable instructions, data structures, program modules, or other data.

[0100] The computer program code used to implement the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, and C++) and conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. The remote computer can be connected to the user's computer via any network (including a local area network or a wide area network) or can be connected to an external computer.

[0101] In the above embodiments, the functions can be implemented in whole or in part by software, hardware, firmware, or any combination thereof, for example, by using application-specific integrated circuits, general-purpose computers, or other similar hardware devices. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions; it can also be implemented by hardware, for example, as a circuit that works in conjunction with the processor to execute the steps or functions.

[0102] Based on the same inventive concept, embodiments of this application also provide a computer program product, including one or more computer programs / instructions, which, when executed by a processor, generate, in whole or in part, a method for generating recognition results using OCR as described in any embodiment of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one storage medium to another via wired (e.g., DSL) or wireless (e.g., Wireless, Microwave) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0103] In this specification, the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the descriptions of the embodiments described later are relatively simple, and relevant parts can be referred to the descriptions of the foregoing embodiments.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating recognition results using OCR, characterized in that, The method includes: Obtain a predefined extended tag specification, which includes tag mapping rules and format requirements. The tag mapping rules are used to specify the correspondence between page element identification information and extended tags. Generate identification control instructions according to the extended marking specification; Input the document to be identified and the identification control command into the multimodal large model; The page elements in the document to be identified are identified using the multimodal large model to obtain the page element identification information; Based on the page element identification information and the tag mapping rules, determine the extended tag corresponding to the page element identification information; According to the format requirements, the page element identification information is structured and encapsulated using the extended tags to generate standardized OCR recognition results.

2. The method for generating recognition results using OCR according to claim 1, characterized in that, The step of identifying page elements in the document to be identified using the multimodal large model to obtain page element identification information includes: The identification content, location information, and visual information of each page element are extracted using the multimodal large model. Based on the positional and visual information of each page element, determine the page element category and page attributes of each page element; The page element identification information is obtained by associating the identified content, page element category, and page attribute of each page element.

3. The method for generating recognition results using OCR according to claim 2, characterized in that, The page elements include physical elements and text elements. Determining the page attributes of each page element based on its position and visual information includes: Based on the positional and visual information of the entity element and the text element, determine the entity region corresponding to the entity element and the text region corresponding to the text element; The occlusion relationship between the entity element and the text element is determined based on the overlap between the entity region and the text region. The occlusion relationship is written as a page attribute into the page element identification information; Based on the page element identification information written into the occlusion relationship and the tag mapping rules, determine the extended tag corresponding to the occlusion relationship; According to the format requirements, the entity element, the text element, and the occlusion relationship are structurally encapsulated using the extended tags corresponding to the occlusion relationship, and then written into the standardized OCR recognition result.

4. The method for generating recognition results using OCR according to claim 1, characterized in that, The step of generating identification control instructions according to the extended tagging specification includes: The tag selection constraint is determined according to the tag mapping rule. The tag selection constraint is used to constrain the multimodal large model to determine the corresponding extended tags according to each of the page elements. The output structure constraints are determined according to the format requirements. The output structure constraints are used to constrain the multimodal large model to use the extended tags corresponding to each page element to structurally encapsulate the identification information of the corresponding page elements. The label selection constraint and the output structure constraint are combined to generate the recognition control instruction.

5. The method for generating recognition results using OCR according to any one of claims 1-4, characterized in that, The page element recognition information is structured and encapsulated using the extended tags to generate standardized OCR recognition results, including: The page element identification information is structured and encapsulated using the extended tags to obtain candidate identification results; The candidate recognition results are analyzed to obtain the start and end markers, required extended labels, label data, and recognition content; According to the format requirements, the pairing relationship of the start and end identifiers, the integrity of the required extended labels, and the data format of the label data are verified for compliance, and the format verification result is obtained. Detect whether the identified content is repeated continuously to obtain the content detection result; Based on the format verification result and the content detection result, the verification result of the candidate recognition result is determined; In response to the verification result indicating that the candidate recognition result has failed the verification, the generation parameters of the multimodal large model are automatically adjusted, and the document to be recognized and the recognition control command are re-inputted into the adjusted multimodal large model to obtain the updated candidate recognition result; The updated candidate recognition result is used as the candidate recognition result, and the process of parsing the candidate recognition result is returned until the verification is passed or the preset number of retries is reached. In response to the verification result indicating that the candidate recognition result has passed the verification, the candidate recognition result is determined as the standardized OCR recognition result.

6. The method for generating recognition results using OCR according to claim 5, characterized in that, The adjustment of the generation parameters of the multimodal large model includes: Anomaly analysis is performed on the candidate recognition results that fail the verification to determine the output anomaly type corresponding to the candidate recognition results; In response to the output anomaly type being start / end identifier mismatch, missing required extended labels, or label data format not conforming to the format requirements, the generation temperature parameter of the multimodal large model is reduced; In response to the output anomaly type being continuous repetition of identified content, the generation temperature parameter of the multimodal large model is increased.

7. The method for generating recognition results using OCR according to claim 6, characterized in that, The method further includes: Obtain the verification result, the generated temperature parameter, and the number of retries for each compliance verification; Based on the verification results, the generated temperature parameters, and the number of retries, a processing log is generated; The processing log is associated with the corresponding standardized OCR recognition result to generate an audit record of the standardized OCR recognition result, and the audit record is stored.

8. The method for generating recognition results using OCR according to claim 1, characterized in that, The method further includes: Obtain the document source information of the document to be identified; According to the extended tag specification, the extended tags corresponding to the document source information are determined, and the document source information is structured and encapsulated using the extended tags corresponding to the document source information. The encapsulated document source information is then written into the standardized OCR recognition result. The standardized OCR recognition results are stored in a knowledge base; The extended tags in the standardized OCR recognition results are analyzed to obtain the document source information, page identifier, and recognition content, as well as the correspondence between the document source information, page identifier, and recognition content and the corresponding extended tags; A tag index is established based on the extended tags and the corresponding relationships; The tag index is queried based on the received query conditions to obtain the identified content, document source information and page identifier that match the query conditions; Based on the document source information and the page identifier, the source location of the identified content in the document to be identified is determined.

9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: the method for generating recognition results using OCR as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which, when executed by a processor, perform the following method for generating recognition results using OCR as described in any one of claims 1-8.

Images