Document information extraction method and device, computer equipment and readable storage medium

Abstract

The invention relates to the field of artificial intelligence, and particularly discloses a document information extraction method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a document image of a target document; performing content identification on the document image to obtain structured information corresponding to the target document; performing feature extraction based on content information and position information included in the structured message to obtain structured features of the target document; fusing the structured features and image features obtained by performing feature extraction on the document image to obtain fused features of the target document; and performing information extraction on the target document based on the document category represented by the fusion feature to obtain document information of the target document. By adopting the method, the accuracy of the document information extraction result can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a document information extraction method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] In many business scenarios, it is necessary to extract information from documents that do not have a fixed format in order to carry out subsequent business operations. These documents are usually paper-based, and staff need to find useful information from these documents and enter them into the system.

[0003] With the rapid development of artificial intelligence technology, information extraction technology based on artificial intelligence has emerged. Artificial intelligence technology is applied to the document information extraction process, and document information can be extracted by identifying the text content in the document. However, the use of traditional technology to extract information from different documents indiscriminately is likely to introduce more redundant information while extracting valid information. Therefore, the use of traditional technology has the problem of inaccurate document information extraction results. Summary of the invention

[0004] Based on this, it is necessary to provide a document information extraction method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of information extraction results in response to the above technical problems.

[0005] In a first aspect, the present application provides a document information extraction method, comprising:

[0006] Acquire a document image of a target document;

[0007] Performing content recognition on the document image to obtain structured information corresponding to the target document; the structured information includes content information contained in the target document and location information of the content information in the target document;

[0008] Extracting features based on each of the content information and each of the location information to obtain structured features of the target document;

[0009] fusing the structural features and the image features obtained by extracting features from the document image to obtain fused features of the target document;

[0010] Based on the document category represented by the fusion feature, information is extracted from the target document to obtain the document information of the target document.

[0011] In one embodiment, the extracting features based on each of the content information and each of the location information to obtain the structured features of the target document includes:

[0012] Extracting content features from each piece of content information to obtain content features of each piece of content information;

[0013] Extracting position features from each piece of position information to obtain position features of each piece of content information;

[0014] The content features and the location features are integrated to obtain the structured features of the target document.

[0015] In one embodiment, the fusing of the content features and the location features to obtain the structured features of the target document includes:

[0016] For each piece of content information, the content feature and the position feature of the content information are integrated to obtain a content position feature of the content information;

[0017] The content location features of each of the content information are integrated to obtain the structural features of the target document.

[0018] In one embodiment, the document information extraction method further includes:

[0019] Obtain a document classification model; the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels;

[0020] The fused features are input into the document classification model, and the document category of the target document is determined based on the output of the document classification model.

[0021] In one embodiment, the document information extraction method further includes:

[0022] Obtain candidate keywords for each of multiple candidate document categories;

[0023] Determine a candidate keyword matching the content information from each of the candidate keywords, and determine a candidate document category corresponding to the candidate keyword;

[0024] When the number of the candidate document categories is one, determining the candidate document category as the document category of the target document;

[0025] When there are multiple candidate document categories, a step of extracting features based on each piece of content information and each piece of position information is performed.

[0026] In one embodiment, extracting information from the target document based on the document category represented by the fusion feature to obtain the document information of the target document includes:

[0027] Acquire candidate information extraction models corresponding to a plurality of candidate document categories respectively; the training samples of the candidate information extraction models include structured information samples and document information samples corresponding to the candidate document categories;

[0028] In the case that a document category represented by the fusion feature exists in each of the candidate document categories, a candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document.

[0029] In one embodiment, the document information extraction method further includes:

[0030] When the document category represented by the fusion feature does not exist in each of the candidate document categories, obtaining an information extraction rule configured for the document category;

[0031] The structured information is extracted according to the information extraction rule to obtain the document information of the target document.

[0032] In a second aspect, the present application also provides a document information extraction device, comprising:

[0033] An acquisition module, used for acquiring a document image of a target document;

[0034] A content recognition module, configured to perform content recognition on the document image to obtain structured information corresponding to the target document; the structured information includes content information contained in the target document and location information of the content information in the target document;

[0035] A structured feature extraction module, used for extracting features based on each of the content information and each of the location information to obtain structured features of the target document;

[0036] A feature fusion module, used to fuse the structural features and the image features obtained by extracting features from the document image to obtain fused features of the target document;

[0037] The information extraction module is used to extract information from the target document based on the document category represented by the fusion feature to obtain the document information of the target document.

[0038] In one embodiment, the structured feature extraction module includes:

[0039] A content feature extraction unit, used to extract content features from each piece of content information to obtain content features of each piece of content information;

[0040] A position feature extraction unit, used to extract position features from each piece of position information to obtain position features of each piece of content information;

[0041] The fusion unit is used to fuse the content features and the location features to obtain the structured features of the target document.

[0042] In one embodiment, the fusion unit is specifically used for:

[0043] For each piece of content information, the content feature and the position feature of the content information are integrated to obtain a content position feature of the content information;

[0044] The content location features of each of the content information are integrated to obtain the structural features of the target document.

[0045] In one embodiment, the document information extraction device further includes a document classification module for:

[0046] Obtain a document classification model; the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels;

[0047] The fused features are input into the document classification model, and the document category of the target document is determined based on the output of the document classification model.

[0048] In one embodiment, the document information extraction device further includes a document classification module for:

[0049] Obtain candidate keywords for each of multiple candidate document categories;

[0050] Determine a candidate keyword matching the content information from each of the candidate keywords, and determine a candidate document category corresponding to the candidate keyword;

[0051] When the number of the candidate document categories is one, determining the candidate document category as the document category of the target document;

[0052] The structured feature extraction module, specifically, when there are multiple candidate document categories, executes a step of extracting features based on each content information and each location information.

[0053] In one embodiment, the information extraction module is specifically used to:

[0054] Acquire candidate information extraction models corresponding to a plurality of candidate document categories respectively; the training samples of the candidate information extraction models include structured information samples and document information samples corresponding to the candidate document categories;

[0055] In the case that a document category represented by the fusion feature exists in each of the candidate document categories, a candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document.

[0056] In one embodiment, the information extraction module is further used to:

[0057] When the document category represented by the fusion feature does not exist in each of the candidate document categories, obtaining an information extraction rule configured for the document category;

[0058] The structured information is extracted according to the information extraction rule to obtain the document information of the target document.

[0059] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned document information extraction method when executing the computer program.

[0060] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned document information extraction method are implemented.

[0061] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned document information extraction method when executed by a processor.

[0062] The document information extraction method, apparatus, computer equipment, computer-readable storage medium and computer program product described above acquire a document image of a target document; perform content recognition on the document image to obtain structured information corresponding to the target document; perform feature extraction based on the content information and location information included in the structured message to obtain structured features of the target document; fuse the structured features and the image features obtained by extracting features from the document image to obtain fused features of the target document; perform information extraction on the target document based on the document category represented by the fused features to obtain document information of the target document. On the one hand, based on the document category represented by the fused features, information is extracted from the target document to obtain the document information of the target document. Personalized information extraction can be performed for documents of different categories, thereby avoiding the extraction of useless redundant information to a certain extent, which is beneficial to improving the accuracy of the document information extraction results. On the other hand, the fused features of the target document are obtained by fusing structured features and image features, which is equivalent to extracting features of the target document from multiple dimensions, which can improve the ability of the fused features to express the characteristics of the target document, thereby improving the accuracy of the document category determined based on the fused features, which is beneficial to further improve the accuracy of the document information extracted based on the document category. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 An application environment diagram of a document information extraction method in an embodiment;

[0065] Figure 2 A schematic diagram of a process of extracting document information in one embodiment;

[0066] Figure 3 It is a flowchart of extracting features based on each content information and each location information to obtain the structured features of the target document in one embodiment;

[0067] Figure 4 A schematic diagram of a process for determining a document category of a target document in one embodiment;

[0068] Figure 5 It is a schematic diagram of a process of extracting information from a target document based on the document category represented by the fusion feature in one embodiment to obtain the document information of the target document;

[0069] Figure 6 A schematic diagram of a flow chart of a document information extraction method in another embodiment;

[0070] Figure 7 is a structural block diagram of a document information extraction device in one embodiment;

[0071] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0072] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0074] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0075] The document information extraction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0076] It should be noted that the document information extraction method in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or can be applied to a system including the terminal 102 and the server 104, and implemented through the interaction between the terminal 102 and the server 104. Taking the case where the server 104 is executed alone as an example, the server 104 can obtain the document image of the target document from the terminal; perform content recognition on the document image to obtain the structured information corresponding to the target document; perform feature extraction based on the content information and location information included in the structured message to obtain the structured features of the target document; fuse the structured features and the image features obtained by feature extraction of the document image to obtain the fused features of the target document; extract information from the target document based on the document category represented by the fused features to obtain the document information of the target document. Among them, the structured information of the target document includes the content information contained in the target document and the location information of the content information in the target document.

[0077] In an exemplary embodiment, Figure 2 As shown, a document information extraction method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0078] Step S202, obtaining a document image of the target document.

[0079] Among them, the target document refers to the document that needs to be extracted. The document can be a written certificate obtained or filled out when the business occurs, which records the actual situation of the transaction and the matter. It is the original data and important basis for accounting. The specific categories of target documents are not unique. For example, it can be documents such as freight documents, bills of exchange, commercial invoices, insurance documents, packing lists, certificates of origin, inspection certificates, etc. generated in the cross-border business review scenario of the bank, or documents such as customs declarations, commercial invoices, trade contracts, etc. generated in the foreign exchange remittance scenario of the bank, or documents such as medical receipts generated in the insurance life insurance claims business scenario.

[0080] Specifically, the document image of the target document can be collected based on computer vision technology. The document image is the source of document information. The document image can be an image obtained by collecting the target document through an image collection device such as a scanner or a camera. Therefore, the terminal can obtain the document image and extract the document information based on the document image. Furthermore, the specific way for the terminal to obtain the document image can be active acquisition or passive reception.

[0081] Step S204: perform content recognition on the document image to obtain structured information corresponding to the target document.

[0082] Among them, structured information refers to the information that is decomposed into multiple interrelated components after analysis, and each component has a clear hierarchical structure. The structured information of the target document includes the content information contained in the target document and the location information of the content information in the target document. The content information can specifically be text content, table content, image content, etc. The location information can be represented by vertex coordinates. For example, the vertex coordinates of the text box where the text content is located, the table vertex coordinates of the table content, and the image vertex coordinates of the image content. Furthermore, the coordinate system where the vertex coordinates are located can be a two-dimensional coordinate system with any vertex of the target document as the origin.

[0083] In practical applications, information that cannot be fully digitized is called unstructured information, such as the document image in this application, which is unstructured information. The process of determining the structured information corresponding to the target document is equivalent to converting the unstructured document image into structured information. In an optional embodiment, the structured information can specifically be html (HyperText Markup Language) text.

[0084] Specifically, the terminal can perform content recognition on the document image based on optical character recognition technology (OCR) or intelligent character recognition technology (ICR) to obtain the structured information corresponding to the target document. Among them, OCR refers to the process in which electronic devices such as scanners or digital cameras check the characters and lines printed on paper, determine their shapes by detecting dark and light patterns, and then use character recognition methods to translate the shapes into computer text and tables. By performing content recognition on the document image, it is possible to understand the image layout of the document and restore the document image structure information, which is convenient for subsequent document classification and information extraction.

[0085] In an optional embodiment, the terminal can perform posture correction on the acquired document image so that the text in the corrected document image is positive, thereby improving the reliability of subsequent text detection and text recognition. Then, for the corrected document image, the text detection model is used to detect the coordinate information of each of the multiple text blocks in the document image. Optionally, the text detection model can include a text line detection module and a text block positioning module. Among them, the text line detection module is used to detect the coordinate information of each of the multiple text lines in the document image; the text positioning module is used to divide the multiple text lines into multiple text blocks based on the coordinate information of each of the multiple text lines, and for any text block, based on the coordinate information of each of the at least one text line contained in the text block, obtain the coordinate information of the text block. After detecting the position of the text block, the terminal can perform text recognition on each text block based on the text recognition model to determine the text sequence contained in each text block. On the other hand, the terminal can also detect whether the document structure in the document image contains a table. If a table is contained, the coordinate information of each of the multiple cells in the detected table is obtained.

[0086] Among them, both the text detection model and the text recognition model can adopt existing networks. For example, the text detection model can adopt the existing PSENET (Shape Robust Text Detection with Progressive ScaleExpansion Network), and the text recognition model can adopt the existing CNN (Convolutional Recurrent Neural Network)-based text recognition network. Furthermore, the text recognition model can simultaneously recognize English text and Chinese text to improve the accuracy of text recognition. In addition, the table detection process can also be implemented through a table detection model. The table detection model can, for example, be a neural network model based on PANET (Path Aggregation Network). It should be noted that in addition to the above-mentioned network structure, the text detection model, the text recognition model and the table detection model of the embodiment of the present application can also adopt other network structures, which are not limited here.

[0087] Step S206: extract features based on the content information and the location information to obtain structured features of the target document.

[0088] Among them, feature extraction is an indispensable part of machine learning and data science. It helps the model better understand and process data by extracting meaningful features from raw data, thereby improving the accuracy of processing results. The structured features of the target document are used to characterize the characteristics of the target document in the document image structure. The specific data form of the structured feature can be a vector or a matrix. Specifically, the terminal can perform feature extraction based on each content information and each location information to obtain the structured features of the target document. The specific method of feature extraction is not unique. For example, feature extraction can be achieved based on machine learning models such as RNN (Recurrent Neural Network), CNN (Convolutional Neural Networks) and Transformer models.

[0089] Optionally, the terminal may first obtain the structural information corresponding to each content information in the target document by fusing the content information and the position information corresponding to the content information, and then extract the features of each structural information to obtain the corresponding structural features of each content information, and finally fuse the structural features to obtain the structural features of the target document. Optionally, the terminal may also extract the content features of each content information separately to obtain the content features of each content information, extract the position features of each position information separately to obtain the position features of each content information, and then fuse the content features and the position features to obtain the structural features of the target document. The specific method of fusing multiple information or multiple features may be splicing, calculation, etc., which is not limited here.

[0090] Step S208, fusing the structured features and the image features obtained by extracting features from the document image to obtain fused features of the target document.

[0091] Among them, image features are a set of attributes used to describe the content and characteristics of an image in the field of computer vision and image processing. In this application, the image features of a document image are used to characterize the image content and characteristics of the document image. The specific data form of the image features can be a vector or a matrix.

[0092] Furthermore, image features may include texture features, shape features, spatial relationship features, and statistical features, etc. Texture features are used to describe the surface structure and details of an image; shape features are used to describe the geometric form of an image object; spatial relationship features are used to describe the relative position and spatial relationship between different objects in an image, which helps to understand the overall layout and structure of the image; and statistical features are used to describe the overall distribution of an image, which can reflect the amount of information and contrast of the image.

[0093] Specifically, the terminal can extract features of the document image based on methods such as convolutional neural networks or autoencoders to obtain image features of the document image. Then, through splicing or feature operations, the structural features and image features are fused to obtain the fused features of the target document.

[0094] Step S210: extract information of the target document based on the document category represented by the fusion feature to obtain the document information of the target document.

[0095] Specifically, after obtaining the fusion feature of the target document, the terminal may determine the document category to which the target document belongs based on the fusion feature, and then extract information from the target document based on the document category to obtain the document information of the target document.

[0096] Optionally, the terminal may obtain information extraction rules configured for the document category, and then extract information from the target document according to the information extraction rules to obtain the document information of the target document. The information extraction rules may be represented by an information extraction template or by keywords to be extracted. Optionally, the terminal may also obtain information extraction models trained for each document type, and then input the structured information of the target document into the information extraction model corresponding to the target document to extract document information and obtain the document information of the target document.

[0097] In the above document information extraction method, a document image of a target document is obtained; content recognition is performed on the document image to obtain structured information corresponding to the target document; feature extraction is performed based on the content information and location information included in the structured message to obtain structured features of the target document; the structured features and the image features obtained by extracting features from the document image are fused to obtain fused features of the target document; based on the document category represented by the fused features, information extraction is performed on the target document to obtain document information of the target document. On the one hand, based on the document category represented by the fused features, information is extracted from the target document to obtain the document information of the target document. Personalized information extraction can be performed for documents of different categories, thereby avoiding the extraction of useless redundant information to a certain extent, which is beneficial to improving the accuracy of the document information extraction results. On the other hand, the fused features of the target document are obtained by fusing structured features and image features, which is equivalent to extracting features of the target document from multiple dimensions, which can improve the ability of the fused features to express the characteristics of the target document, thereby improving the accuracy of the document category determined based on the fused features, which is beneficial to further improve the accuracy of the document information extracted based on the document category.

[0098] In one embodiment, Figure 3 As shown, step S206 includes:

[0099] Step S302 : extracting content features of each piece of content information to obtain the content features of each piece of content information.

[0100] Among them, content features refer to features that can characterize the characteristics of content information. The specific data form of the content feature can be a vector or a matrix. Specifically, the terminal can convert content information into content features by using the bag of words model (Bag of Words), TF-IDF (Term Frequency-Inverse Document Frequency), n-gram model, etc. to achieve content feature extraction. The terminal can also extract content features based on deep learning technology and apply deep convolutional networks.

[0101] Step S304: extracting position features from each piece of position information to obtain position features of each piece of content information.

[0102] Among them, the location feature refers to a feature that can characterize the characteristics of the location information. Similar to the content feature, the specific data form of the location feature can be a vector or a matrix. Specifically, the terminal can use a pre-trained location feature extraction network based on a deep learning method to extract the location feature corresponding to the content information. Optionally, the terminal can use a triple loss network to learn the location embedding method, which can further explore the characteristics of the location information and obtain the location feature.

[0103] Step S306: fuse the content features and the location features to obtain the structured features of the target document.

[0104] Specifically, after obtaining the content features and location features of each content information, the terminal can fuse the content features and location features to obtain the structural features of the target document. The specific method of fusing multiple features can be splicing or feature operation. The specific algorithm of feature operation can include at least one of addition, subtraction, multiplication, etc.

[0105] Taking the case of realizing feature fusion through splicing as an example, the terminal can sequentially splice each content feature and each position feature according to the position of the content information in the target document to obtain a structured feature; the terminal can also splice each content feature to obtain a fused content feature, splice each position feature to obtain a fused position feature, and then splice the fused content feature and the fused position feature to obtain a structured feature of the target document.

[0106] In the above embodiment, content features and position features are first extracted separately and then fused to obtain structured features, which can ensure that the structured features can characterize the content characteristics of the target document and the location distribution characteristics of different contents in the document, thereby ensuring the accuracy of the document category determined based on the structured features.

[0107] In one embodiment, step S306 includes: for each content information, fusing the content feature and the position feature of the content information to obtain the content position feature of the content information; fusing the content position features of each of the content information to obtain the structural features of the target document.

[0108] Specifically, the terminal can fuse the content features and position features of each content information to obtain the content position features of the content information. The content position features can characterize the features of the content information in terms of specific content (such as text content) and the features of the position of the content information in the target document. After obtaining the content position features of each content information, the terminal can obtain the structural features of the target document by fusing the content position features.

[0109] Optionally, the terminal may obtain the structured features of the target document by splicing the location features of each content.

[0110] Optionally, the terminal may perform weighted summation on each content position feature to obtain the structured feature of the target document. The weight of each content position feature in the weighted summation process may be determined based on at least one of the amount of information of the content information corresponding to the content position feature, or the size of the region of the position information corresponding to the content position feature. The weight may be positively correlated with the amount of information and the size of the region. Thus, a larger weight may be assigned to content information with a larger amount of information or content information occupying a larger area, which is conducive to further improving the accuracy of the structured feature.

[0111] In the above embodiment, the content features and position features are first fused according to the content information to obtain the content position features, and then further fused to obtain the structured features, so that the structured features can have stronger content and position representation capabilities, which is beneficial to further ensure the accuracy of the document category determined based on the structured features.

[0112] In one embodiment, the document information extraction method further includes: obtaining a document classification model; inputting the fusion feature into the document classification model, and determining the document category of the target document based on the output of the document classification model.

[0113] Among them, the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels. The document classification model has the ability to identify the document category represented by the document category label. That is to say, in the case of this embodiment, the document classification model can be a multi-classification model. The sample fusion feature refers to the fusion feature of the document sample, which can characterize the structural and image characteristics of the document sample. The document sample carries a document category label. That is, the document sample refers to a sample marked with a document category. The document category can be, for example, an insurance document, a bill of exchange, a packing list, etc., which is not limited here.

[0114] Specifically, the terminal can directly obtain the document classification model, or train the document classification model based on multiple document samples carrying category labels. The training process of the document classification model requires learning the sample fusion features carrying document category labels. The process of determining the sample fusion features based on the document samples refers to the process of determining the fusion features based on the target document above, which will not be repeated here.

[0115] Furthermore, after acquiring the document classification model, the terminal may input the fusion features into the document classification model, and determine the document category of the target document based on the output of the document classification model. Optionally, the output of the document classification model may include the confidence of each of the multiple reference categories, and the terminal may determine the document category of the target document based on the reference category with the highest confidence. Among them, each reference category refers to a plurality of document categories represented by each document category label. Optionally, the terminal may determine the reference category with the highest confidence as the document category of the target document; the terminal may also determine the reference category corresponding to the highest confidence as the document category of the target document when the highest confidence is greater than the confidence threshold.

[0116] In the above embodiment, the document category of the target document is determined based on the document classification model obtained through machine learning, which can improve the accuracy of the document category.

[0117] In one embodiment, Figure 4 As shown, the document information extraction method also includes:

[0118] Step S402, obtaining candidate keywords for each of a plurality of candidate document categories.

[0119] It is understandable that different document categories usually differ in content composition. For example, freight documents usually include cargo specifications; insurance documents usually include insurance numbers. Specifically, corresponding candidate keywords can be pre-configured for multiple candidate document categories. Thus, the terminal can obtain candidate keywords for each of the multiple candidate document categories. Among them, the number of candidate keywords corresponding to a candidate document category can be one or more. There may be overlaps among the multiple candidate keywords corresponding to different candidate document categories.

[0120] Step S404: determining candidate keywords that match the content information from the candidate keywords, and determining candidate document categories corresponding to the candidate keywords.

[0121] Among them, the candidate keyword matches the content information, which may mean that the content information contains the candidate keyword, or that the content information and the candidate keyword are semantically similar. Specifically, the terminal can match and analyze each content information contained in the target document with each candidate keyword, and determine the matching degree between the content information and each candidate keyword. In the case that each candidate keyword contains a candidate keyword that matches any content information, the candidate document category corresponding to the candidate keyword is further determined.

[0122] Step S406: When the number of candidate document categories is one, determine the candidate document category as the document category of the target document.

[0123] Specifically, when the number of candidate document categories is one, it means that the candidate keyword is a specific keyword of the candidate document category, and the document category can be determined by keyword matching. In this case, the terminal can determine the candidate document category as the document category of the target document.

[0124] When there are multiple candidate document categories, the document category cannot be determined only by keyword matching. It is necessary to further perform feature extraction steps based on each content information and each location information, so that the document category of the target document can be further determined from the candidate document categories based on the fusion features of the target document.

[0125] In the above embodiment, combining keywords and fusion features to classify documents can improve the accuracy of classification results on the one hand, and on the other hand, first performing keyword matching and then further classifying based on fusion features can improve work efficiency in the document classification process.

[0126] In one embodiment, Figure 5 As shown, step S210 includes:

[0127] Step S502, obtaining candidate information extraction models corresponding to a plurality of candidate document categories.

[0128] The training samples of the information extraction model corresponding to the candidate document category include structured information samples and document information samples corresponding to the candidate document category. The structured information samples and the document information samples correspond one to one. The structured information samples are obtained by performing content recognition on the document images of the document samples. The document information samples are obtained by performing information extraction on the document samples.

[0129] Specifically, since the information extraction requirements of different document categories are different, candidate information extraction models applicable to a certain candidate document category can be trained in advance for multiple candidate document categories. For example, for candidate document category A, by obtaining multiple document samples belonging to candidate document category A, document information samples of structured information samples corresponding to each document sample are constructed to obtain a training set. Then, the initial model is trained based on the training set to obtain a candidate information extraction model A applicable to candidate document category A. The initial model can be a pre-trained large language model.

[0130] Step S504: when there is a document category represented by the fusion feature in each candidate document category, the candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document.

[0131] Specifically, when there is a document category represented by the fusion feature in each candidate document category, the terminal can further determine the target information extraction model corresponding to the document category from each candidate information extraction model, and use the target information extraction model to extract the structured information of the target document to obtain the document information of the target document. In a specific implementation, the extracted document information is represented by a key-value pair.

[0132] In the above embodiments, on the one hand, using an information extraction model that matches the document category to extract information from the target document can improve the accuracy of the extracted document information while improving work efficiency; on the other hand, using the information extraction model to extract structured information from the target document can further simplify the information extraction process and improve work efficiency compared to directly extracting information from unstructured documents.

[0133] In a specific embodiment, please continue to refer to Figure 5 , the document information extraction method also includes:

[0134] Step S506: when the document category represented by the fusion feature does not exist in each candidate document category, obtain the information extraction rule configured for the document category.

[0135] The information extraction rules may be represented by information extraction templates or by keywords that are expected to be extracted.

[0136] Specifically, when the document category represented by the fusion feature does not exist in each candidate document category, it means that the current information extraction model cannot complete the information extraction of the target document. In this case, the terminal can extract document information based on rules. Optionally, the terminal can prompt the staff to configure the corresponding information extraction rules for the document category, so that the information extraction rules configured for the document category can be obtained in response to the rule configuration operation of the staff. Optionally, the terminal can obtain the information extraction rules pre-configured for the document category.

[0137] Step S508: extract the structured information according to the information extraction rules to obtain the document information of the target document.

[0138] After determining the information extraction rule, the terminal can extract the structured information according to the information extraction rule to obtain the document information of the target document. For example, the information extraction rule can be configured with a keyword key, and the terminal can extract the value corresponding to the keyword key from the structured information according to each keyword key to obtain the document information of the target document.

[0139] In the above embodiment, for target documents to which the information extraction model cannot be applied, information is extracted using a rule-based method, which can ensure that information can be extracted for target documents of different categories, thereby responding to application requirements in different application scenarios, which is conducive to improving the flexibility of the document information extraction method.

[0140] In a specific embodiment, a method for extracting document information is provided. The method for extracting document information can be applied to a computer device, which can be a terminal or a server. For example, the computer device is a terminal. Figure 6 As shown, the document information extraction method may include the following steps:

[0141] Step S601, obtaining a document image of a target document and candidate keywords of a plurality of candidate document categories;

[0142] Step S602, performing content recognition on the document image to obtain structured information corresponding to the target document;

[0143] The structured information includes content information contained in the target document and location information of the content information in the target document;

[0144] Step S603, determining candidate keywords matching the content information from the candidate keywords, and determining candidate document categories corresponding to the candidate keywords;

[0145] Step S604: when the number of candidate document categories is one, determine the candidate document category as the document category of the target document;

[0146] Step S605, when there are multiple candidate document categories, extract content features from each content information to obtain content features of each content information;

[0147] Step S606, extracting position features from each piece of position information to obtain position features of each piece of content information;

[0148] Step S607, for each piece of content information, the content feature and the position feature of the content information are integrated to obtain the content position feature of the content information;

[0149] Step S608, fusing the content location features of each content information to obtain the structural features of the target document;

[0150] Step S609, fusing the structured features and the image features extracted from the document image to obtain the fused features of the target document;

[0151] Step S610, inputting the fused features into the document classification model, and determining the document category of the target document based on the output of the document classification model;

[0152] Among them, the document classification model is a machine learning model, which is trained based on the sample fusion features carrying document category labels;

[0153] Step S611, obtaining candidate information extraction models corresponding to multiple candidate document categories respectively;

[0154] The training samples of the candidate information extraction model include structured information samples and document information samples corresponding to the candidate document categories;

[0155] Step S612: when there is a document category represented by the fusion feature in each candidate document category, the candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document;

[0156] Step S613, when the document category represented by the fusion feature does not exist in each candidate document category, obtaining the information extraction rule configured for the document category;

[0157] Step S614: extract the structured information according to the information extraction rules to obtain the document information of the target document.

[0158] The above-mentioned document information extraction method, on the one hand, extracts information of the target document based on the document category represented by the fused features to obtain the document information of the target document, and can perform personalized information extraction for documents of different categories, thereby avoiding the extraction of useless redundant information to a certain extent, which is beneficial to improving the accuracy of the document information extraction results; on the other hand, the fused features of the target document are obtained by fusing structured features and image features, which is equivalent to extracting features of the target document from multiple dimensions, which can improve the ability of the fused features to express the characteristics of the target document, thereby improving the accuracy of the document category determined based on the fused features, which is beneficial to further improve the accuracy of the document information extracted based on the document category.

[0159] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0160] Based on the same inventive concept, the embodiment of the present application also provides a document information extraction device for implementing the document information extraction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more document information extraction device embodiments provided below can refer to the limitations of the document information extraction method above, and will not be repeated here.

[0161] In an exemplary embodiment, Figure 7 As shown, a document information extraction device is provided, comprising:

[0162] An acquisition module 701 is used to acquire a document image of a target document;

[0163] The content recognition module 702 is used to perform content recognition on the document image to obtain structured information corresponding to the target document; the structured information includes content information contained in the target document and location information of the content information in the target document;

[0164] The structured feature extraction module 703 is used to extract features based on each content information and each location information to obtain the structured features of the target document;

[0165] A feature fusion module 704 is used to fuse the structural features and the image features obtained by extracting the features of the document image to obtain the fusion features of the target document;

[0166] The information extraction module 705 is used to extract information from the target document based on the document category represented by the fusion feature to obtain the document information of the target document.

[0167] In one embodiment, the structured feature extraction module 703 includes: a content feature extraction unit, which is used to extract content features from each content information respectively to obtain the content features of each content information; a position feature extraction unit, which is used to extract position features from each position information respectively to obtain the position features of each content information; and a fusion unit, which is used to fuse each content feature and each position feature to obtain the structured features of the target document.

[0168] In one embodiment, the fusion unit is specifically used to: for each content information, fuse the content feature and the position feature of the content information to obtain the content position feature of the content information; fuse the content position features of each content information to obtain the structural features of the target document.

[0169] In one of the embodiments, the document information extraction device also includes a document classification module, which is used to: obtain a document classification model; the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels; input the fusion features into the document classification model, and determine the document category of the target document based on the output of the document classification model.

[0170] In one of the embodiments, the document information extraction device also includes a document classification module, which is used to: obtain candidate keywords for each of multiple candidate document categories; determine alternative keywords that match the content information from each candidate keyword, and determine the alternative document category corresponding to the alternative keyword; when the number of alternative document categories is one, determine the alternative document category as the document category of the target document; and a structured feature extraction module, specifically when the number of alternative document categories is multiple, execute the step of extracting features based on each content information and each location information.

[0171] In one of the embodiments, the information extraction module 705 is specifically used to: obtain candidate information extraction models corresponding to multiple candidate document categories; training samples of the candidate information extraction model include structured information samples and document information samples corresponding to the candidate document categories; when there is a document category represented by the fusion feature in each candidate document category, use the candidate information extraction model corresponding to the document category to extract information from the structured information to obtain the document information of the target document.

[0172] In one of the embodiments, the information extraction module 706 is also used to: when there is no document category represented by the fusion feature in each candidate document category, obtain the information extraction rules configured for the document category; extract the structured information according to the information extraction rules to obtain the document information of the target document.

[0173] Each module in the document information extraction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0174] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the above-mentioned document information extraction method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a document information extraction method is implemented.

[0175] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a document information extraction method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0176] Those skilled in the art will understand that Figure 8 or Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method embodiments are implemented.

[0179] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned method embodiments when executed by a processor.

[0180] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A document information extraction method, characterized in that: The method comprises: Acquire a document image of a target document; Performing content recognition on the document image to obtain structured information corresponding to the target document; the structured information includes content information contained in the target document and location information of the content information in the target document; Extracting features based on each of the content information and each of the location information to obtain structured features of the target document; fusing the structural features and the image features obtained by extracting features from the document image to obtain fused features of the target document; Based on the document category represented by the fusion feature, information is extracted from the target document to obtain the document information of the target document.

2. The method according to claim 1, characterized in that The extracting features based on each of the content information and each of the location information to obtain the structured features of the target document includes: Extracting content features from each piece of content information to obtain content features of each piece of content information; Extracting position features from each piece of position information to obtain position features of each piece of content information; The content features and the location features are integrated to obtain the structured features of the target document.

3. The method according to claim 2, characterized in that The fusing of the content features and the location features to obtain the structural features of the target document includes: For each piece of content information, the content feature and the position feature of the content information are integrated to obtain a content position feature of the content information; The content location features of each of the content information are integrated to obtain the structural features of the target document.

4. The method according to claim 1, characterized in that The method further comprises: Obtain a document classification model; the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels; The fused features are input into the document classification model, and the document category of the target document is determined based on the output of the document classification model.

5. The method according to claim 1, characterized in that The method further comprises: Obtain candidate keywords for each of multiple candidate document categories; Determine a candidate keyword matching the content information from each of the candidate keywords, and determine a candidate document category corresponding to the candidate keyword; When the number of the candidate document categories is one, determining the candidate document category as the document category of the target document; When there are multiple candidate document categories, a step of extracting features based on each piece of content information and each piece of position information is performed.

6. The method according to any one of claims 1 to 5, characterized in that The step of extracting information from the target document based on the document category represented by the fusion feature to obtain document information of the target document includes: Acquire candidate information extraction models corresponding to a plurality of candidate document categories respectively; the training samples of the candidate information extraction models include structured information samples and document information samples corresponding to the candidate document categories; In the case that a document category represented by the fusion feature exists in each of the candidate document categories, a candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document.

7. The method according to claim 6, characterized in that The method further comprises: When the document category represented by the fusion feature does not exist in each of the candidate document categories, obtaining an information extraction rule configured for the document category; The structured information is extracted according to the information extraction rule to obtain the document information of the target document.

8. A document information extraction device, characterized in that: The device comprises: An acquisition module, used for acquiring a document image of a target document; A content recognition module, configured to perform content recognition on the document image to obtain structured information corresponding to the target document; the structured information includes content information contained in the target document and location information of the content information in the target document; A structured feature extraction module, used for extracting features based on each of the content information and each of the location information to obtain structured features of the target document; A feature fusion module, used to fuse the structural features and the image features obtained by extracting features from the document image to obtain fused features of the target document; The information extraction module is used to extract information from the target document based on the document category represented by the fusion feature to obtain the document information of the target document.

9. The device according to claim 8, characterized in that The structured feature extraction module comprises: A content feature extraction unit, used to extract content features from each piece of content information to obtain content features of each piece of content information; A position feature extraction unit, used to extract position features from each piece of position information to obtain position features of each piece of content information; The fusion unit is used to fuse the content features and the location features to obtain the structured features of the target document.

10. The device according to claim 9, characterized in that The fusion unit is specifically used for: For each piece of content information, the content feature and the position feature of the content information are integrated to obtain a content position feature of the content information; The content location features of each of the content information are integrated to obtain the structural features of the target document.

11. The device according to claim 8, characterized in that The device also includes a document classification module, which is used to: Obtain a document classification model; the document classification model is a machine learning model, which is trained based on sample fusion features carrying document category labels; The fused features are input into the document classification model, and the document category of the target document is determined based on the output of the document classification model.

12. The device according to claim 8, characterized in that The device also includes a document classification module, which is used to: Obtain candidate keywords for each of multiple candidate document categories; Determine a candidate keyword matching the content information from each of the candidate keywords, and determine a candidate document category corresponding to the candidate keyword; When the number of the candidate document categories is one, determining the candidate document category as the document category of the target document; The structured feature extraction module, specifically, when there are multiple candidate document categories, executes a step of extracting features based on each content information and each location information.

13. The device according to any one of claims 8 to 12, characterized in that The information extraction module is specifically used for: Acquire candidate information extraction models corresponding to a plurality of candidate document categories respectively; the training samples of the candidate information extraction models include structured information samples and document information samples corresponding to the candidate document categories; In the case that a document category represented by the fusion feature exists in each of the candidate document categories, a candidate information extraction model corresponding to the document category is used to extract information from the structured information to obtain the document information of the target document.

14. The device according to claim 13, characterized in that The information extraction module is also used for: When the document category represented by the fusion feature does not exist in each of the candidate document categories, obtaining an information extraction rule configured for the document category; The structured information is extracted according to the information extraction rule to obtain the document information of the target document.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images