Intelligent document recognition method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to the technical field of document recognition, and provides an intelligent document recognition method and apparatus, an electronic device and a storage medium. The method comprises: extracting text information and layout information from a document to be recognized; acquiring a template containing a specified field; on the basis of an industry knowledge base corresponding to the document, adding a label to the text information; and, on the basis of the layout information and the label, using a large language model to recognize text information matched with the specified field from the text information to which the label has been added, the large language model being a large language model that has been trained by using the industry knowledge base. In the embodiments of the present disclosure, document recognition results more conform to industry attributes, improving document recognition accuracy.

Description

Document intelligent recognition method, device, electronic device and storage medium

[0001] This disclosure claims priority to Chinese patent application number 202410084634.3, filed with the Patent Office of China on January 19, 2024, entitled “Document Intelligent Recognition Method, Device, Electronic Device and Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of document recognition technology, and in particular to a method, device, electronic device, and storage medium for intelligent document recognition. Background Art

[0003] Foreign trade business processes often involve extensive document processing. Manually entering, verifying, and identifying document information requires significant labor costs. Currently, some companies are using AI (artificial intelligence) to optimize document processing, improving internal operational efficiency and streamlining document business processes. However, these AI technologies generally rely on general domain knowledge to identify document information, resulting in a lack of industry-specific attributes for foreign trade, which in turn reduces recognition accuracy. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, and storage medium for intelligent document recognition to improve the accuracy of document recognition.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for intelligent document recognition, comprising:

[0006] Extract text information and layout information from the document to be recognized;

[0007] Get the template containing the specified field;

[0008] Adding tags to the text information based on the industry knowledge base corresponding to the document;

[0009] Using a large language model, based on the layout information and the tags, to identify text information that matches the designated field in the tagged text information;

[0010] The large language model is a large language model trained using the industry knowledge base.

[0011] In a second aspect, an embodiment of the present disclosure provides a document intelligent recognition device, comprising:

[0012] An extraction module, used to extract text information and layout information from the document to be recognized;

[0013] The acquisition module is used to obtain the template containing the specified field;

[0014] A labeling module, configured to add labels to the text information based on the industry knowledge base corresponding to the document;

[0015] an identification module, configured to use a large language model to identify text information matching the specified field in the tagged text information based on the layout information and the tags;

[0016] The large language model is a large language model trained using the industry knowledge base.

[0017] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the above methods.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program, which, when executed in a computer, causes the computer to execute any one of the methods described above.

[0020] Compared with the prior art, the present disclosure has the following advantages:

[0021] This approach extracts text and layout information from the document to be recognized, obtains a template containing designated fields, tags the text based on the document's corresponding industry knowledge base, and uses a large language model to identify text matching the designated fields within the tagged text based on the layout and tags. The large language model, trained using the industry knowledge base, possesses industry-specific knowledge. By adding tags to the text, the model provides recognition results that are more consistent with industry attributes, improving document recognition accuracy.

[0022] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0024] FIG1 is a schematic diagram of an application scenario of the document intelligent recognition method provided by the present disclosure;

[0025] FIG2 is a flow chart of a method for intelligent document recognition according to an embodiment of the present disclosure;

[0026] FIG3 is a flow chart of a method for intelligent document recognition according to another embodiment of the present disclosure;

[0027] FIG4 is a schematic diagram of a document according to another embodiment of the present disclosure;

[0028] FIG5 is a schematic diagram of a document intelligent recognition process according to another embodiment of the present disclosure;

[0029] FIG6 is a schematic diagram of a framework for implementing intelligent document recognition according to another embodiment of the present disclosure;

[0030] FIG7 is a schematic diagram comparing various document recognition methods according to another embodiment of the present disclosure;

[0031] FIG8 is a structural block diagram of a document intelligent recognition device according to another embodiment of the present disclosure;

[0032] FIG9 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0034] To facilitate understanding of the technical solutions of the embodiments of the present disclosure, the following describes the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be combined with the technical solutions of the embodiments of the present disclosure in any way, and all of them fall within the scope of protection of the embodiments of the present disclosure.

[0035] First, the terms used in this disclosure are explained.

[0036] Document processing: refers to the editing, formatting and typesetting of documents to meet specific needs.

[0037] Template definition: refers to creating a template for a document so that documents that meet specific requirements can be quickly generated.

[0038] Industry data management: refers to the collection, organization, storage and management of data in a specific industry to support business decision-making and analysis.

[0039] Recognition processing: refers to the use of artificial intelligence technology to identify and analyze text, images, or sounds to improve work efficiency and accuracy.

[0040] Feedback evaluation: refers to the evaluation and analysis of user feedback to improve product and service quality.

[0041] Cross-border trade scenarios: refers to scenarios used in cross-border trade, such as international trade contracts, international cargo transportation, etc.

[0042] Risk control and compliance assistance to improve efficiency: refers to the use of risk control and compliance technologies to assist in improving business efficiency and ensuring business compliance.

[0043] Digital upgrade of the document chain: refers to the digitization and intelligence of the document chain to achieve automated management and tracking of documents.

[0044] R&D costs: refers to the costs of developing and maintaining products, including labor costs, technology costs, and equipment costs.

[0045] Usage cost: refers to the cost of using the product, including purchase cost, maintenance cost and training cost, etc.

[0046] LLM (Large Language Model): A natural language processing model based on deep learning. It is trained using a large corpus and typically has billions to trillions of parameters. It can handle various natural language tasks such as natural language generation, text classification, text summarization, question answering, dialogue, machine translation, or speech recognition.

[0047] LangChain: A programming framework that facilitates the use of LLM in applications. It provides an open-source Python library that encapsulates extensive LLM application development logic and tool integration. It includes standard modular components, integrates and consolidates different large language models, and simplifies the process of creating applications powered by LLM and chat models.

[0048] The document intelligent recognition method provided by the embodiments of the present disclosure can be applied to any electronic device, including but not limited to: computers, mobile terminals, tablet computers, laptop computers or servers, etc. The documents involved in the embodiments of the present disclosure can be documents in any format. The document formats include but are not limited to: word, excel, pdf, ppt or pictures, etc. The fields involved in the documents can be general domains, such as education, science, sports or transportation, etc., or vertical domains, including but not limited to foreign trade, medical care, games, banks, automobiles or aviation, etc. The documents in the embodiments of the present disclosure can be documents of a certain industry, usually including multiple types. Taking the foreign trade industry as an example, foreign trade documents include but are not limited to: customs declarations, bills of exchange, invoices, packing lists, bills of lading, insurance policies, insurance certificates, certificates of origin, verification certificates or GSP documents, etc.

[0049] This method can provide intelligent identification and information comprehension capabilities for various documents and can be applied to scenarios such as import and export declarations, freight forwarding booking, cross-border trade, or financial risk control. Specific application scenarios include, but are not limited to, customs brokers identifying declarations, buyers identifying invoices, and cross-border e-commerce platforms identifying credit insurance certificates. This method can improve the efficiency of risk control and compliance assistance, digitally upgrade the document chain, and enhance product competitiveness and user experience. It solves the difficulties and problems users face in filling out order forms, sorting order fields, and verifying document information, eliminating the need for extensive manpower and offering the advantages of low cost and high recognition accuracy.

[0050] Figure 1 is a schematic diagram of an exemplary application scenario for implementing the document intelligent recognition method of an embodiment of the present disclosure. The electronic devices in this application scenario include a server and a client. The server in the cloud can communicate with multiple clients. Among them, there can be various types of clients, such as computers, mobile phones, tablets or laptops, etc. The server in the cloud can be deployed as needed, including centralized deployment or distributed deployment. For example, multiple servers or one server can be deployed in the cloud, and the figure only takes the deployment of one server as an example. In the above application scenario, any electronic device can execute the above document intelligent recognition method to achieve intelligent recognition based on industry attributes.

[0051] An embodiment of the present disclosure provides a method for intelligent document recognition. FIG2 is a flow chart of the method for intelligent document recognition according to an embodiment of the present disclosure. The method may include the following steps.

[0052] S201: Extracting text information and layout information from the document to be recognized.

[0053] The document to be identified in the embodiment of the present disclosure can be a document with industry attributes, such as a document in a certain vertical domain, or a document without industry attributes, such as a general document not related to a specific field. Regardless of the type of document, it has certain layout information.

[0054] In the embodiments of the present disclosure, layout information refers to all formatting information on a document's pages, excluding textual content. This information characterizes the layout, style, and position of text, and may include, but is not limited to, at least one of the following: position information, wireframe information, table information, distance information, or style information. If a document contains multiple pages, the layout information may include layout information for each page in the document.

[0055] Position information refers to the coordinates of text on the page, while wireframe information refers to information such as lines or frames on the page. Tables are a common format in documents. For example, multiple product information is represented using multiple rows, with each row containing one product. Information such as the product name, price, and quantity is represented using multiple columns. Therefore, table information includes the number of rows and columns, row height, column width, or cell merging. Distance information refers to the distance between text at different locations on the page, which can indicate the distance between them. Distance information can reflect the relationship between text to a certain extent. Generally, text that is farther apart has a weaker relationship, while text that is farther apart has a stronger relationship. For example, text that is in the same row or adjacent to each other in typesetting has a strong relationship. Style information includes but is not limited to font, font size, capitalization, color, alignment, full-width or double-column layout, number of lines per page, number of words per line, line spacing, or indentation.

[0056] S202: Acquire a template containing a specified field.

[0057] In the embodiments of the present disclosure, multiple templates can be pre-generated for the user to select, or a template can be customized according to the user's needs. The templates in the embodiments of the present disclosure can be understood as user-required templates. The user can select the template they need from the template library, or customize the template they need according to their own intentions.

[0058] In the disclosed embodiments, the designated fields included in the template are used to represent the fields that the user wishes to have intelligently identified in the document, and can be understood as target fields or required fields. For example, a template corresponding to a courier order may include designated fields such as the sender's name, sender's address, sender's phone number, recipient's name, recipient's address, or recipient's phone number. A template corresponding to an invoice may include designated fields such as the invoice header, invoice number, purchasing company, purchasing address, invoice date, product name, quantity, unit price, or total price.

[0059] S203: Add tags to the text information based on the industry knowledge base corresponding to the document.

[0060] In the disclosed embodiment, one or more industry knowledge bases can be pre-built, mainly for the storage of industry knowledge. When the document to be identified has industry characteristics, the corresponding industry knowledge base can be searched according to the industry to which it belongs, thereby adding industry attribute tags to the text information.

[0061] The tags in the embodiments of this disclosure refer to semantic tags. One or more semantic tags can be added to a text. Semantic tags are used to identify the professional meaning of the text within the industry. For example, semantic tags for the text FOB may include: Free On Board, Free On Board, Transaction Method, or FOB.

[0062] S204: Using the large language model, based on the layout information and the tags, identify text information that matches the specified field in the tagged text information.

[0063] The large language model in the embodiment of the present disclosure is a large language model trained using an industry knowledge base, so it has the ability to reserve industry knowledge, and the recognition results given are more in line with industry attributes and have higher accuracy.

[0064] In the embodiment of the present disclosure, since the text extracted from the document usually contains a lot of content, it may include important core information and usually also include unimportant auxiliary information. Therefore, the role of intelligent document recognition is to find the text information that matches the specified field in the template, that is, to identify the text information that meets the user's needs and obtain the results the user wants. Among them, the text information that matches the specified field means that the text information meets the definition of the specified field. For example, the specified field is quantity, the matching text information is 60, the specified field is unit price, the matching text information is $22.00, the specified field is name, and the matching text information is FAN MOTOR≤125W.

[0065] In one embodiment, the above-mentioned adding labels to text information based on the industry knowledge base corresponding to the document may include: determining the industry knowledge base corresponding to the document according to the industry to which the document belongs, finding professional vocabulary belonging to the industry in the text information based on the industry knowledge base, and adding one or more semantic type labels to the professional vocabulary in the text information.

[0066] In one embodiment, the above-mentioned use of a large language model to identify text information matching a specified field in tagged text information based on layout information and tags may include: using the specified field as a question and combining the layout information and tags to generate prompt words, using the large language model to understand the meaning of the prompt words and generate corresponding answers, thereby obtaining text information matching the specified field.

[0067] In one embodiment, the above-mentioned use of a large language model to understand the meaning of the prompt word and generate a corresponding answer to obtain text information matching the specified field may include: using the large language model to perform a single or multiple rounds of dialogue according to a specified process arrangement, understanding the meaning of the prompt word and generating a corresponding answer to obtain text information matching the specified field.

[0068] In one embodiment, the above method may further include: using the LangChain tool to pre-set a single-round or multi-round dialogue for interacting with the large language model to obtain a process arrangement of the large language model.

[0069] In one embodiment, the above method may also include: constructing corresponding industry knowledge bases for multiple industries in advance, setting data features for each industry, generating corresponding data sets for each industry knowledge base, generating dialogue text based on sample information in the data set, and fine-tuning the large language model with the dialogue text as input and output that conforms to the above data features.

[0070] In one embodiment, the above-mentioned obtaining of a template containing a specified field may include: obtaining a template selected by a user from a plurality of pre-generated templates containing the specified field, or determining the specified field according to the user's intention and generating a template containing the specified field.

[0071] In one embodiment, the above method may further include: filtering non-compliant content of text information matching the specified field according to risk control rules, and performing formal standardization processing on the filtered text information.

[0072] In one embodiment, the above method may further include: pre-setting multiple types of documents, and for each type of document, selecting content representing document characteristics as designated fields and generating a template containing the designated fields.

[0073] The method provided in this embodiment extracts text and layout information from the document to be identified, obtains a template containing a specified field, tags the text based on the industry knowledge base corresponding to the document, and uses a large language model to identify text matching the specified field within the tagged text based on the layout and tags. The large language model is trained using the industry knowledge base and possesses industry knowledge. Based on the tagged text, the recognition results provided are more consistent with industry attributes, thereby improving document recognition accuracy.

[0074] Another embodiment of the present disclosure provides a method for intelligent document recognition. FIG3 is a flow chart of the method for intelligent document recognition according to another embodiment of the present disclosure. The method may include the following steps.

[0075] S301: Extracting text information and layout information from the document to be recognized.

[0076] The layout information in the embodiment of the present disclosure may include at least one of the following: position information, wireframe information, table information, distance information, or style information.

[0077] In one embodiment, an OCR (Optical Character Recognition) algorithm can be used to identify text information in a document to be identified, and an image recognition algorithm can be used to identify publication information in the document. Image recognition algorithms include, but are not limited to, ResNet (Residual Network) 50 algorithms, RPN (Region Proposal Network) algorithms, or HSV (Hue Saturation Value) algorithms, etc., which are not specifically limited in the present embodiment.

[0078] S302: Acquire a template selected by the user from a plurality of pre-generated templates containing designated fields, or determine the designated fields according to the user's intention and generate a template containing the designated fields.

[0079] In one embodiment, the above-mentioned pre-generated template can be implemented by the following steps:

[0080] Multiple types of documents are pre-set. For each type of document, the content representing the characteristics of the document is selected as the designated field and a template containing the designated field is generated.

[0081] In the disclosed embodiments, there may be one or more pre-generated templates, and the number of designated fields in each template may also be one or more, without specific limitation. If there are multiple document types, one or more templates may be set for each document type. If multiple templates are associated with a single document type, the designated fields contained in each template may be partially identical or completely different, without specific limitation.

[0082] For example, a customs declaration can include various types, including import declaration, export declaration, cargo invoice, land transport document, airway bill, ocean import bill, ocean export bill, export tax rebate verification form, commodity packing list, trade contract, or certificate of origin. An import declaration can correspond to two templates. Template A includes designated fields: declaration number, import / export identification, declaration date, country of departure / arrival, and port of shipment / arrival. Template B includes designated fields: declaration number, import / export identification, declaration date, country of departure / arrival, port of shipment / arrival, means of transport, freight / insurance, trade method, and transaction method.

[0083] Figure 4 is a document schematic diagram of an embodiment of the present disclosure. As shown in Figure 4, the left side is a schematic diagram of an invoice document. Among them, it includes information such as the invoice number, the purchasing company, the product name and the invoice date. The table on the right illustrates all the text information that can be extracted from the invoice. The key column in the table represents the field, and the value column represents the content corresponding to the field. For example, the content corresponding to the field invoice date is 2020-12-11. All fields in the key column in the table can be used as designated fields to set the corresponding template for the invoice, that is, some or all fields can be selected from the column to generate an invoice template. The specific number of selected fields is not limited.

[0084] S303: Determine the industry knowledge base corresponding to the document according to the industry to which the document belongs.

[0085] In the disclosed embodiment, multiple industry knowledge bases can be pre-constructed, and each industry corresponds to an industry knowledge base. According to the industry to which the document to be identified belongs, the industry knowledge base corresponding to the industry can be found in the multiple industry knowledge bases that have been constructed. In one embodiment, the specific industry to which the document to be identified belongs can be known, that is, the industry information to which the document to be identified belongs has been obtained before document recognition. In this scenario, the corresponding industry knowledge base can be directly found to execute subsequent steps. In another embodiment, the industry to which the document to be identified belongs is unknown, and sorting technology can be used to sort the documents to be identified to obtain the industry that matches the document to be identified, that is, the industry to which it belongs, and then find the industry knowledge base corresponding to the industry to execute subsequent steps. Among them, the sorting technology can judge the document according to preset rules to identify the matching industry, which will not be explained in detail here.

[0086] In addition, the large language model used in the embodiment of the present disclosure can also be pre-trained using multiple industry knowledge bases constructed, so that the trained large language model has the knowledge reserve of the corresponding industry, can answer questions with industry characteristics, and give answers with industry attributes.

[0087] S304: Finding industry-specific vocabulary in the text information based on the industry knowledge base.

[0088] The text information extracted from the document to be identified can be one or more words or one or more paragraphs. Based on a specific industry knowledge base, specialized vocabulary related to the industry to which the document to be identified belongs can be found in the text information. For example, if the document to be identified belongs to the foreign trade industry, specialized vocabulary related to the foreign trade industry can be found in the text information based on the foreign trade industry knowledge base.

[0089] S305: Add one or more semantic type tags to the professional vocabulary in the text information.

[0090] In the disclosed embodiments, semantic tags are used to identify the industry-specific meaning of a term. For example, semantic tags for the term "FOB" might include: "Free on Board," "Delivery on Board," "Transaction Method," or "Free on Board." This industry-specific tagging approach helps generate more precise prompts, which in turn helps the large language model provide more accurate, industry-specific answers.

[0091] In one implementation, BERT (Bidirectional Encoder Representation from Transformers) can be used to complete the tag addition task. Of course, other algorithms or models can also be used to implement it, and this embodiment of the present disclosure does not specifically limit this.

[0092] S306: Generate prompt words based on the specified field as a question and in combination with the layout information and tags.

[0093] In the embodiment of the present disclosure, using a specified field as a question means inputting the specified field into the large language model. The purpose of interacting with the large language model is to expect the large language model to give an answer to the question, that is, text information matching the specified field.

[0094] Because the designated field input contains relatively little information, combining the acquired layout information with the tags already added to the text can generate more precise and industry-specific prompts, making the question description more detailed and specific. Consequently, when these prompts are fed into the large language model, more accurate answers can be obtained, improving document recognition accuracy.

[0095] S307: Use the large language model to perform single or multiple rounds of dialogue according to the specified process, understand the meaning of the prompt words and generate corresponding answers to obtain text information that matches the specified field.

[0096] Among them, the large language model is a large language model trained using an industry knowledge base.

[0097] In the disclosed embodiment, the training process of the large language model can be performed during initialization, and may specifically include the following steps: constructing corresponding industry knowledge bases for multiple industries in advance, setting data features for each industry, generating corresponding data sets for each industry knowledge base, generating dialogue text based on sample information in the data set, and fine-tuning the large language model using the dialogue text as input and output that meets the above-mentioned data features.

[0098] Each industry corresponds to different data features. For example, in the customs declaration industry, data features may include features such as the declaration form number, country of destination, and commodity; in the express delivery industry, data features may include features such as the consignee and delivery address. Each industry knowledge base has its own corresponding dataset, which can be one or more, and a dataset can include one or more samples. Sample information can include, but is not limited to: source files, OCR text, source information, annotation information, or image feature files. Based on this sample information, conversational text can be generated as input for the LLM to train the LLM, ensuring that the LLM output conforms to the data features of the corresponding industry. Training based on datasets from different industries will ultimately fine-tune the LLM to form an industry-specific LLM. Furthermore, in addition to files and images, samples can also use text data and include layout information. This improves the quality of the input content, thereby increasing the accuracy of training and the precision of the model.

[0099] In one implementation, the LangChain tool can also be used in advance to set up single-round or multi-round dialogues to interact with the large language model, obtain the process arrangement of the large language model, and then ensure that the large language model executes single-round or multi-round dialogues according to the process arrangement to complete the recognition of text information matching the specified field.

[0100] In the disclosed embodiments, process orchestration refers to the ability to follow a specified process orchestration during interaction with a large language model, such as whether it is a single-round or multi-round interaction, specifically the number of rounds, the sequence of each round, and the relationship between the questions in the current round and the results of the previous round. After setting the specified process orchestration, when using the large language model, the interaction with the large language model is carried out according to the set process orchestration, thereby completing the recognition of text information matching the specified field.

[0101] S308: For text information that matches the specified field, filter out non-compliant content according to risk control rules, and perform format standardization on the filtered text information.

[0102] In the embodiment of the present disclosure, after the text information is identified, post-processing can also be performed. Among them, post-processing includes but is not limited to: risk control processing and / or standardization processing. Risk control processing refers to filtering or deleting non-compliant content in text information according to risk control rules, and the text information obtained after processing is all legal and compliant text information. Standardization processing refers to the formal processing of text information, such as whether to uniformly use Chinese or English, whether to use text or code, how many digits after the decimal point are displayed, etc., so as to make it conform to standardization.

[0103] Figure 5 is a schematic diagram of the intelligent document recognition process according to another embodiment of the present disclosure. As shown in Figure 5, the intelligent document recognition process may include: first, extracting text and layout information from various types of business documents to be recognized, including coordinate information, layout information, and text information, i.e., information based on optical character recognition (OCR). Furthermore, obtaining a template containing designated fields, accessing the industry database corresponding to the document, and adding industry tags to the text information based on the industry database to obtain information based on the industry tags. Then, combining the OCR-based information with the industry tag-based information to obtain prompt words with layout and industry attributes, these prompt words are input into a pre-trained large language model. The large language model, based on the layout information and tags, identifies text matching the designated fields within the tagged text. Finally, the output of the large language model may be subjected to business post-processing, including rule processing related to laws and regulations, recognition correction related to text form specifications, or scenario-specific processing. The results obtained from this post-processing can serve as the basis for feedback evaluation and can also be used for iterative training of the large language model, thereby further improving its recognition accuracy.

[0104] The method provided in this embodiment extracts text and layout information from the document to be recognized, obtains a template containing a specified field, tags the text based on the document's corresponding industry knowledge base, and uses a large language model to identify text matching the specified field within the tagged text based on the layout information and tags. The large language model is trained using the industry knowledge base and possesses industry knowledge. Based on the tagged text, the recognition results provided are more consistent with industry attributes, improving document recognition accuracy. Furthermore, the method supports a variety of document types, such as express delivery bills, customs declarations, letters of credit, and ocean bills of lading, meeting the needs of diverse scenarios. With advantages such as low R&D costs and high recognition accuracy, it supports cross-border trade scenarios, providing intelligent recognition and information comprehension capabilities for documents such as express delivery bills, customs declarations, letters of credit, and ocean bills of lading. It is widely applicable to scenarios such as import and export declarations, freight forwarding booking, and financial risk management. It can help customers sort through large amounts of information, addressing the difficulties of ordering fields, complex order forms, and poor user experience, thereby improving customer ordering efficiency and experience. It can reduce personnel operating costs, improve business scale development, build capability barriers based on accumulated industry knowledge in various vertical domains, and help risk control departments verify the authenticity of contract performance through documents submitted by customers, thereby improving risk control capabilities.

[0105] Figure 6 is a schematic diagram of the document intelligent recognition implementation framework according to another embodiment of the present disclosure. As shown in Figure 6, the document intelligent recognition method provided in this embodiment of the present disclosure can be implemented using the framework shown in the figure. This framework comprises five components: a document processing center, a template center, an industry data center, a recognition R&D center, and a feedback and evaluation center. The document processing center is responsible for processing documents in various formats, such as Excel, Word, PDF, or PNG, extracting information such as text, layout, barcodes, or seals. The template center is responsible for defining and storing template information, such as the name, type, description, and dictionary of designated fields for different types of documents. The industry data center is responsible for building industry knowledge bases, one for each industry. The recognition R&D center is responsible for document sample management, workflow orchestration for LLM interactions, single- and multi-turn dialogue templates, tagging, content debugging, version management, and SOP publishing. The feedback and evaluation center is responsible for document recognition capability assessment, intervention support, and feedback iteration. This framework can be applied to a variety of business scenarios, including but not limited to Scenario A involving security services, Scenario B involving foreign exchange, Scenario C involving logistics, and scenarios involving tax calculation or search.

[0106] The following uses a logistics scenario as an example, with reference to Figure 6, to illustrate the intelligent document recognition process. In this logistics scenario, the first step is for the document processing center to pre-process the image of the logistics document to be recognized, extracting information such as the publication format, barcode, or seal. Furthermore, the document content is extracted and character correction, format standardization, and cleaning are performed. The second step is for the template center to retrieve a template selected by the user from a list of known templates, or to generate a template containing specified fields based on user intent. Multiple known templates are typically pre-generated by the template center and contain one or more specified fields. These fields, determined based on user intent, may include the recipient's name, phone number, and address, as well as the sender's name, phone number, and address. The third step is for the industry data center to locate a logistics industry knowledge base from among multiple established knowledge bases based on the logistics industry to which the document belongs. Using this knowledge base, the industry-specific vocabulary is identified and semantically tagged within the text information retrieved from the document processing center. In the fourth step, the Document Intelligent Recognition Center uses the specified field in the template obtained in the second step as a question and, combined with the layout information and labels, generates prompt words that are input into the large language model. The large language model then interprets the prompt words and generates corresponding answers, obtaining text information that matches the specified field as the final result of logistics document recognition. The large language model is pre-trained based on multiple industry knowledge bases, and its processes are orchestrated using LangChain. In the fifth step, the Feedback Evaluation Center evaluates the capabilities of the aforementioned framework based on the recognition results of multiple logistics documents and performs feedback iterations to optimize the parameters of the large language model, thereby improving model accuracy. Through the cooperation and coordination of the five centers within the aforementioned framework, the large language model can acquire industry knowledge reserves. Based on the labeling of text information, the recognition results given are more consistent with industry attributes, thereby improving the accuracy of document recognition.

[0107] Figure 7 is a schematic diagram comparing various document recognition methods according to another embodiment of the present disclosure. As shown in Figure 7, the processing results corresponding to the three document recognition extraction methods are shown. The unordered text extraction method can obtain basic text information, but fails to identify the product quantity and unit price as two different products. The OCR-based layout extraction method can identify the information of two products, with the correct quantity and unit price, but fails to recognize specialized foreign trade terms. The final method, an intelligent document recognition method or industry-specific extraction method provided by this embodiment, adds multiple semantic tags to the text information, thus providing accurate recognition results for specialized terms. In the example shown in the figure, the semantic tags added to the text "NINGBO" include the province and customs code 3301; the semantic tags added to the text "FOB" include "Free on Board" and "Transaction method"; the semantic tags added to the text "THALAND" include "Country" and the binary code "TH"; and the semantic tags added to the text "WEIGUAND" include "Company" and 330196032. Therefore, the final recognition result correctly identifies the text matching the specified transaction method field as "FOB." The three tables below the figure are the processing results obtained by the three extraction methods. It can be clearly seen from the tables that compared with the first two extraction methods, the industry-specific extraction method provided by the embodiment of the present disclosure can give recognition results that are more in line with industry attributes, greatly improving the accuracy of document recognition.

[0108] Corresponding to the application scenario and method of the method provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides a document intelligent recognition device. As shown in Figure 8, a structural block diagram of the document intelligent recognition device of an embodiment of the present disclosure is shown. The device may include:

[0109] Extraction module 801, used to extract text information and layout information from the document to be recognized;

[0110] The acquisition module 802 is used to acquire a template containing a specified field;

[0111] The tag module 803 is used to add tags to the text information based on the industry knowledge base corresponding to the document;

[0112] The recognition module 804 is configured to use a large language model to identify text information that matches a specified field in the tagged text information based on the layout information and the tags;

[0113] The large language model is a large language model trained using an industry knowledge base.

[0114] In one embodiment, the label module 803 is specifically used to: determine the industry knowledge base corresponding to the document based on the industry to which the document belongs, find professional vocabulary belonging to the industry in the text information based on the industry knowledge base, and add one or more semantic type labels to the professional vocabulary in the text information.

[0115] In one embodiment, the recognition module 804 is specifically used to: use the specified field as a question and combine the layout information and labels to generate prompt words, use the large language model to understand the meaning of the prompt words and generate corresponding answers, and obtain text information matching the specified field.

[0116] In one embodiment, the recognition module 804 is specifically used to: use a large language model to perform single-round or multi-round dialogues according to a specified process arrangement, understand the meaning of the prompt words and generate corresponding answers, and obtain text information that matches the specified field.

[0117] In one embodiment, the above device may further include:

[0118] The setup module is used to pre-set single-round or multi-round dialogues with the large language model using the LangChain tool to obtain the process orchestration of the large language model.

[0119] In one embodiment, the above device may further include:

[0120] The training module is used to pre-build corresponding industry knowledge bases for multiple industries, set the data characteristics of each industry, generate corresponding data sets for each industry knowledge base, generate dialogue text based on the sample information in the data set, and fine-tune the large language model with the dialogue text as input and the output meeting the above data characteristics.

[0121] In one embodiment, the acquisition module 802 is specifically configured to: acquire a template selected by a user from a plurality of pre-generated templates containing a specified field, or determine a specified field according to a user's intention and generate a template containing the specified field.

[0122] In one embodiment, the apparatus further comprises:

[0123] The post-processing module is used to filter out non-compliant content in text information that matches the specified fields according to risk control rules, and to perform formal standardization on the filtered text information.

[0124] In one embodiment, the apparatus further comprises:

[0125] The template generation module is used to pre-set multiple types of documents, and for each type of document, select the content that represents the characteristics of the document as the specified field and generate a template containing the specified field.

[0126] In one embodiment, the layout information includes at least one of the following: position information, wireframe information, table information, distance information, or style information.

[0127] The functions of each module in each device of the embodiment of the present disclosure can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0128] The above-mentioned device provided in this embodiment extracts text information and layout information from the document to be recognized, obtains a template containing a specified field, tags the text information based on the industry knowledge base corresponding to the document, and uses a large language model to identify text information that matches the specified field in the tagged text information based on the layout information and tags. The large language model is a large language model trained using the industry knowledge base and has the ability to accumulate industry knowledge. Based on the tagging of text information, the recognition results provided are more consistent with industry attributes, thereby improving the accuracy of document recognition.

[0129] Figure 9 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in Figure 9, the electronic device includes a memory 910 and a processor 920. Memory 910 stores a computer program executable on processor 920. When processor 920 executes the computer program, it implements the methods described in the above embodiments. There can be one or more memory 910 and processor 920.

[0130] The electronic device further includes a communication interface 930 for communicating with external devices and performing data exchange transmission.

[0131] If the memory 910, processor 920, and communication interface 930 are implemented independently, the memory 910, processor 920, and communication interface 930 can be interconnected via a bus and communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0132] Optionally, in a specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

[0133] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present disclosure when the program is executed by a processor.

[0134] The embodiments of the present disclosure further provide a computer program, which, when executed in a computer, enables the computer to execute the method provided in the embodiments of the present disclosure.

[0135] An embodiment of the present disclosure further provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present disclosure.

[0136] An embodiment of the present disclosure also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0137] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0138] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct RAM bus random access memory (DR RAM).

[0139] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present disclosure are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0140] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0142] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present disclosure includes additional implementations in which the functions may be performed in a different order than shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0143] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with such instruction execution systems, apparatuses or devices.

[0144] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0145] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0146] It should be noted that the embodiments of the present disclosure may involve the use of user data. In actual applications, user-specific personal data may be used in the scenarios described herein within the scope permitted by applicable laws and regulations, subject to compliance with applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0147] The above description is merely an exemplary embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope of the present disclosure, and such modifications or substitutions should be included within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for intelligent document recognition, the method comprising: Extracting text information and layout information from the document to be recognized; Obtaining a template containing specified fields; Adding tags to the text information based on the industry knowledge base corresponding to the document; Using a large language model to identify, from the text information with tags added, the text information that matches the specified fields according to the layout information and the tags; Wherein, the large language model is a large language model trained using the industry knowledge base.

2. The method according to claim 1, wherein Adding tags to the text information based on the industry knowledge base corresponding to the document includes: Determining the industry knowledge base corresponding to the document according to the industry to which the document belongs; Finding, from the text information, the professional terms belonging to the industry according to the industry knowledge base; Adding tags of one or more semantic types to the professional terms in the text information.

3. The method according to claim 1 or 2, wherein Using a large language model to identify, from the text information with tags added, the text information that matches the specified fields according to the layout information and the tags includes: Generating a prompt word with the specified field as the question and combining the layout information and the tags; Using the large language model to understand the meaning of the prompt word and generate a corresponding answer to obtain the text information that matches the specified fields.

4. The method according to claim 3, wherein, Using the large language model to understand the meaning of the prompt word and generate a corresponding answer to obtain the text information that matches the specified fields includes: Using the large language model to perform single-round or multi-round conversations according to a specified process arrangement to understand the meaning of the prompt word and generate a corresponding answer to obtain the text information that matches the specified fields.

5. The method according to claim 4, further comprising: Previously using the LangChain tool to set single-round or multi-round conversations for interacting with the large language model to obtain the process arrangement of the large language model.

6. The method according to any one of claims 1-5, further comprising: Previously constructing respective industry knowledge bases for multiple industries and setting data characteristics of each industry; Generating corresponding data sets for each industry knowledge base; Generating dialogue texts according to the sample information in the data sets; Using the dialogue texts as inputs and the outputs conforming to the data characteristics to fine-tune and train the large language model.

7. The method according to any one of claims 1-6, wherein, Obtaining a template containing specified fields includes: Obtaining the template selected by the user from multiple pre-generated templates containing specified fields; Or determining the specified fields according to the user's intention and generating a template containing the specified fields.

8. The method according to any one of claims 1-7, further comprising: Filtering non-compliant content from the text information that matches the specified fields according to risk control rules; Performing formal normalization processing on the filtered text information.

9. The method according to any one of claims 1-8, further comprising: Previously setting multiple types of documents; For each type of document, selecting the content characterizing the document characteristics as the specified fields and generating a template containing the specified fields.

10. The method according to any one of claims 1-9, wherein, The layout information includes at least one of the following: position information, wireframe information, table information, distance information, or style information.

11. A document intelligent recognition device, the device comprising: An extraction module, configured to extract text information and layout information from a document to be recognized; An acquisition module, configured to acquire a template containing specified fields; A tagging module, configured to add tags to the text information based on the industry knowledge base corresponding to the document; A recognition module, configured to use a large language model to recognize, from the tagged text information, the text information that matches the specified fields according to the layout information and the tags; wherein, the large language model is a large language model trained using the industry knowledge base.

12. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, where the processor implements the method according to any one of claims 1-10 when executing the computer program.

13. A computer-readable storage medium, in which a computer program is stored, and the computer program implements the method according to any one of claims 1-10 when executed by a processor.

14. A computer program, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 10.

Images