Automatically classify documents and identify metadata hierarchically using machine learning and fuzzy matching

Through a multi-stage hierarchical method, combined with text and image-based metadata recognition technology, the problems of electronic document metadata recognition accuracy and computing resource consumption are solved, and more efficient electronic document classification and metadata extraction are achieved.

CN109783635BActive Publication Date: 2024-11-08ACCENTURE GLOBAL SOLUTIONS LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201811219473.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-11-13
Filing Date
2018-10-19
Publication Date
2024-11-08
Estimated Expiration
2038-10-19

AI Technical Summary

Technical Problem

The prior art has insufficient accuracy in the identification and classification of metadata of electronic documents, excessive computing resources are consumed, and recognition technology based on text or image alone is difficult to provide satisfactory results.

Method used

A multi-stage hierarchical approach is adopted, including text-based document classification, image-based metadata recognition and fuzzy text matching, and electronic documents are classified into predefined categories through machine learning algorithms, improving the accuracy of metadata recognition and reducing computing needs.

Benefits of technology

Improve the accuracy of electronic document metadata identification, reduce the computing resource consumption for model development and update, and enhance the analysis accuracy for categories of interest.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN109783635B_ABST
    Figure CN109783635B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to automatically classifying documents and identifying metadata in a hierarchical manner using machine learning and fuzzy matching. A hierarchical document classification system is disclosed. The system includes: a text-based document classifier model for classifying an input electronic document into one of a set of predefined document categories. The system also includes: an image-based metadata identification model for classifying electronic documents of a specific document category into a set of metadata categories. The system also includes: a fuzzy text matcher for supplementing the classification accuracy of the image-based metadata identification model to obtain a metadata category for the input electronic document.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Patent Application No. 15 / 810,885 filed with the U.S. Patent Office on November 13, 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to hierarchical classification of documents, identification / marking of metadata, and attributes of documents. Background Art

[0004] Electronic documents may contain embedded text and images. They may be created for a variety of purposes and generated in a variety of electronic file formats. Each page of an electronic document file may also be associated with metadata and attributes that may or may not be directly embedded in the content or data header of the electronic document file. Automatic classification of electronic documents and identification / marking of metadata and attributes associated with each page of an electronic document may facilitate organization, archiving, and more efficient use of these electronic documents. Summary of the invention

[0005] Embodiments of the present disclosure relate to automatically classifying documents and identifying metadata in a hierarchical manner using machine learning and fuzzy matching. A hierarchical document classification system is disclosed. The system includes: a text-based document classifier model for classifying an input electronic document into one of a set of predefined document categories. The system also includes: an image-based metadata identification model for classifying electronic documents of a specific document category into a set of metadata categories. The system also includes: a fuzzy text matcher for supplementing the classification accuracy of the image-based metadata identification model to obtain a metadata category for the input electronic document. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 An exemplary computer system for implementing hierarchical classification and metadata recognition / identification of electronic documents is shown.

[0007] Figure 2 The logic and data flow of an exemplary implementation for hierarchical classification and metadata identification of electronic documents is shown.

[0008] Figure 3 Logic and data flow are shown for building an exemplary text-based document classifier model that can be implemented as part of a process for hierarchical classification and metadata recognition / identification of electronic documents.

[0009] Figure 4Logic and data flow for building an exemplary image-based metadata recognition / identification model is shown, which may be implemented as part of a process for hierarchical classification and metadata recognition / identification of electronic documents.

[0010] Figure 5 The image-based metadata recognition / identification modeling process implemented based on convolutional neural network technology is shown.

[0011] Figure 6 Logic and data flow for building an exemplary fuzzy text matcher that may be implemented as part of a process for hierarchical classification and metadata recognition / identification of electronic documents is shown. DETAILED DESCRIPTION

[0012] Electronic documents can contain text, images and other components arranged as pages. Electronic documents can also be stored in various electronic file formats, including but not limited to: file formats based on page description languages ​​such as PostScript, PDF and XML, and binary file formats such as Microsoft Word. For example, page description languages ​​can be used to specify various components in a document page, including their content and their position and layout relationship. In particular, the text information contained in the document page can be embedded as a plain text data component, which is represented as a string with font and position / layout information, or can be embedded as a bitmap image component. In one example, the entire electronic document page can be a bitmap image carrying embedded text information and other graphics and layout information. In particular, an electronic document obtained by optical scanning and digitizing a paper source can contain a bitmap page of an image.

[0013] Electronic documents can originate from various sources and contain different content. For example, electronic documents in the form of document files can also include multiple pages, each containing different content. Automatic document recognition, identification, classification and cataloging tasks can require identifying relevant pages from a large collection of electronic documents and extracting metadata from each relevant page. Such metadata recognition / identification can be achieved using content recognition / identification and / or classifier models developed based on machine learning techniques. Here, the term "metadata" can broadly refer to the attributes of electronic document pages. This attribute may not be included in, for example, the data header of the corresponding electronic file. For example, a collection of electronic documents may contain pages of supplier invoices and many other information. In this case, metadata may refer to the supplier name or ID associated with the invoice contained in the electronic document collection. Such metadata of electronic document pages may belong to a predefined set of metadata categories, such as a supplier name set / ID set. Metadata information can be embedded in the text, graphics or layout of electronic documents. Identifying / identifying such metadata can help better extract information of interest, tags, and organize electronic documents. Although the term "electronic document" can be generally used to refer to an electronic file that can contain multiple pages, in order to simplify the description, it will be used synonymously with the term "page" in the following disclosure. In this way, each page in a document page collection can be referred to as an electronic document, and a document collection can refer to pages parsed and extracted from electronic document files and other sources.

[0014] Electronic documents (or pages) of a particular class of interest (e.g., supplier invoices) and requiring metadata recognition / identification may be only a small portion of a large collection of electronic documents. Models based on machine learning algorithms for providing direct metadata recognition / identification in a large collection of electronic documents may require a large number of document features, a huge search space, to achieve good metadata recognition / identification accuracy. Therefore, the development and updating of these models may consume an excessive amount of computing resources. The resulting models may still be inaccurate due to noise from document pages that is not related to, for example, supplier invoices. In addition, metadata information may be embedded in text information or graphics and layout information in electronic documents, so a single model based only on text recognition / identification technology or only on image recognition / identification technology may not be sufficient to provide generally satisfactory metadata recognition / identification accuracy.

[0015] The following disclosure provides a multi-stage hierarchical approach to the metadata recognition / identification problem for improving accuracy while reducing computational requirements during model development, model updating, and feature computation of input electronic documents. In one exemplary implementation, the multi-stage metadata recognition / identification method includes multiple (e.g., three) hierarchical stages involving, for example, a text-based document classification and filtering stage, an image-based metadata recognition / identification stage, and a supplementary fuzzy text matching stage to further improve the accuracy of metadata recognition / identification. The model for text-based document classification and the model for image-based metadata recognition / identification can be based on any suitable machine learning algorithm for text classification and image classification, respectively. The term "metadata identification" and the term "metadata recognition" are used synonymously and can refer to both recognizing / identifying text metadata information embedded in an input electronic document (e.g., a supplier name) and graphic / layout metadata patterns in the input electronic document, which are features of a particular type of document (e.g., a supplier invoice) associated with particular metadata (e.g., a particular supplier).

[0016] In one implementation, a text-based document classifier model can be used to classify input electronic documents into a predefined set of document categories, including, for example, supplier invoices and other supplier document categories. Electronic documents that are classified into the predefined set of document categories but are not of interest can be filtered. In this way, in the next stage of the hierarchical method, only documents belonging to the category of interest need to be processed by the metadata identification model to extract metadata information. Because only a small set of electronic documents in the input electronic documents (e.g., supplier invoices) need to be further analyzed, the development of metadata identification models based on machine learning techniques can rely on more targeted and narrower training and testing corpora, and rely on better defined machine learning features to provide image-based metadata identification models with higher accuracy and less computational complexity. In one implementation, the metadata identification model can be based on image recognition and classification techniques, because unique metadata features can often be embedded in the layout and graphic components of the input electronic documents. For example, a specific supplier (with a unique supplier name or ID) can use an invoice format that contains layout or image features that are unique to that supplier. Although text content in an electronic document may also contain entity names and ID information, identifying metadata via text recognition may not be as accurate because the entity names and ID information input into the electronic document may generally relate to entities other than the entity that authored the electronic file and issued the invoice contained in the electronic document.

[0017] In some cases, an input electronic document may be identified by an image-based metadata identification model as belonging to a particular metadata category (e.g., a particular supplier name / ID), but with a low identification confidence score. This may occur, for example, when the input electronic document belongs to a supplier whose invoices were not included in the training data corpus when the image-based metadata identification model was established. In these cases, a complementary fuzzy text matcher based on a hierarchical approach that performs comparisons between features of the text in the input electronic document and features of metadata items in an independent metadata database can help improve the accuracy of the image-based metadata identification model. The fuzzy text matcher can confirm the metadata identification made by the image-based metadata identification model or confirm the metadata identification by proposing a different but more likely metadata category for the input electronic document. The metadata database on which the fuzzy text matcher relies may contain a more extensive set of metadata that covers metadata categories, such as supplier names and IDs, that may not have a correspondence in the training data set (e.g., supplier invoices) used to establish the image-based metadata identification model.

[0018] exist Figure 1 In FIG. 1 , a system 100 for inputting metadata identification of electronic documents is shown as being implemented by a computer 101. The computer 101 may include a communication interface 102, a system circuit 104, an input / output (I / O) interface 106, a storage device 109, and a display circuit 108, which generates a machine interface 110 locally or for remote display, for example, in a web browser running on a local or remote machine. The machine interface 110 and the I / O interface 106 may include a GUI, a touch-sensitive display, voice or facial recognition input, buttons, switches, speakers, and other user interface elements. Other examples of the I / O interface 106 include microphones, video and still image cameras, headphone and microphone input / output jacks, universal serial bus (USB) connectors, memory card slots, and other types of inputs. The I / O interface 106 may also include magnetic or optical media interfaces (e.g., CDROM or DVD drives), serial and parallel bus interfaces, and keyboard and mouse interfaces.

[0019] The communication interface 102 may include a wireless transmitter and receiver ("transceiver") 112 and any antenna 114 used by the transmit and receive circuitry of the transceiver 112. The transceiver 112 and antenna 114 may support Wi-Fi network communications, for example, under any version of IEEE 802.11 (e.g., 802.11n or 802.11ac). The communication interface 102 may also include a wired transceiver 116. The wired transceiver 116 may provide a physical layer interface for any of a wide range of communication protocols, such as any type of Ethernet, Data Over Cable Service Interface Specification (DOCSIS), Digital Subscriber Line (DSL), Synchronous Optical Network (SONET), or other protocols.

[0020] Storage 109 can be used to store various initial, intermediate or final data or models for building, updating and operating document classification, metadata identification and fuzzy text matching models. Storage 109 can also store data corpora for developing document classification and metadata identification models. Storage 109 can also be used to implement a database of metadata required for fuzzy text matching models. Storage 109 can be centralized or distributed. For example, it can be remotely hosted by a cloud computing service provider.

[0021] The system circuit 104 may include hardware, software, firmware, or other circuits in any combination. The system circuit 104 may be implemented, for example, with one or more systems on a chip (SoC), application specific integrated circuits (ASICs), microprocessors, discrete analog and digital circuits, and other circuits. The system circuit 104 is part of the implementation of any desired functionality associated with the construction, maintenance, and application of a custom graph knowledge base. As just one example, the system circuit 104 may include one or more instruction processors 118 and a memory 120. The memory 120 stores, for example, control instructions 124 and an operating system 122. In one implementation, the instruction processor 118 executes the control instructions 124 and the operating system 122 to perform any desired functionality associated with models for document classification, metadata identification, and fuzzy text matching.

[0022] Figure 2 Shown for Figure 1 1. An exemplary logic and data flow 200 for hierarchical classification and metadata identification of electronic documents implemented in the computer system 100 of FIG. 1. The logic and data flow 200 includes logic and data flow for a text-based document classification and filtering process 220, logic and data flow for an image-based metadata identification process 230, and logic and data flow for a fuzzy text matching process 240. The text-based document classification and filtering process 220, the image-based metadata identification process 230, and the fuzzy text matching process 240 are hierarchically related, as shown in FIG. Figure 2Specifically, the filtered output documents 228 of the text-based document classification and filtering process 220 are input to the image-based metadata identification process 230 and the fuzzy text matching process 240 .

[0023] for Figure 2 In the text-based document classification and filtering process 220, the input electronic document 210 is first converted into a text document (222). The input electronic document may contain document pages from various sources. For example, a multi-page electronic document file may be divided into multiple documents, each corresponding to a page. The conversion of the input electronic document into a text document may be achieved by, for example, optical character recognition (OCR). In this way, the text information contained in the input electronic document may be extracted and converted into a string. The converted text document may be classified by a text-based document classifier model 226 (224). Specifically, the text features of the converted text file may be calculated using a predefined text feature calculation algorithm (225). The calculated text features may be input into a text-based document classifier model, which classifies the converted text document into one of a predefined document category set (226). The predefined document category set may include, but is not limited to, supplier invoices, supplier plans, product catalogs, product manuals, warranties, supply contracts, etc.

[0024] In one implementation, the input electronic documents may be filtered to remove electronic documents that do not belong to the document category of interest (227). Thus, only the electronic documents of interest (228) remain and are further analyzed by the image-based metadata identification process 230 and the fuzzy text matching process 240. For example, supplier invoices may be the only document category of interest. Thus, only electronic documents classified as supplier invoices by the text-based document classification process 220 may be further analyzed to identify metadata.

[0025] for Figure 2 The image-based metadata identification process 230 may first obtain images of the filtered documents 228 (250). Each image may correspond to a document (page). In one implementation, the input documents 210 corresponding to some of the filtered electronic documents 228 may already be in image format, and these electronic documents may not need to be converted to images. Some other input electronic documents 210 may initially be in a format other than images, so after the filtering process in 220, they may be converted to images (250).

[0026] Each of the images of the input electronic documents belonging to the document category of interest may then be processed to extract a region of interest (234). The region of interest for each image or document page may be a portion or part of the image having a predefined size and location. For example, the region of interest may be the upper left quadrant of the image. Extraction of the region of interest helps to further reduce the amount of data that may need to be processed by the image-based identification process 230. The predetermination of the location and size of the region of interest may be based on historical data. For example, if a supplier invoice is a document category of interest, and the supplier invoice typically contains text or graphic information with supplier metadata embedded in the data in the upper left quadrant, the region of interest may be predefined as the upper left quadrant of the image. The image may be cropped and / or deformed and / or realigned to obtain the region of interest.

[0027] To handle the case where various suppliers may place metadata carrying information in different parts of their invoices, you can run Figure 2 Multiple parallel processes 230 are performed, each process specifies a different predefined size and location for the region of interest. The analysis results of the parallel metadata identification processes 230 can be combined to determine metadata information for a document. For example, one of the parallel processes 230 can extract the upper left quadrant of an image, while another parallel process 230 can extract the upper right quadrant of the image. As will be described below, the image-based metadata identification modeling 237 can be based on a regression algorithm (rather than a binary classifier) ​​that outputs a probability that the input region of interest is associated with each metadata category in a predefined set of metadata categories. In the event that the parallel image-based metadata identification processes 230 are being executed, the most likely metadata category for a particular input electronic document among the metadata categories identified by the parallel metadata identification processes can be determined as the metadata category for the particular input electronic document.

[0028] In one implementation, the extracted region of interest may be preprocessed (234) prior to image-based metadata identification modeling 236. The region of interest may be resized and / or deformed and / or realigned. For example, preprocessing of the region of interest may include reducing the resolution of the region of interest. Specifically, the number of imaging pixels of each region of interest may be reduced by aggregating and averaging a predetermined number of adjacent pixels together. For example, a region of interest having 200 by 200 imaging pixels may be reduced to 50 by 50 pixels. Such resolution reduction may help reduce the computational requirements of the image-based metadata identification process 230 without much information loss, because metadata information embedded in an image corresponding to an input electronic document may generally be identified at a relatively low resolution. Preprocessing of the extracted region of interest may further include deforming the region of interest. For example, such deformation may be designed to detect and straighten skew in the region of interest. For example, such skew may be introduced when the input electronic document is generated from an original paper document source using an optical scanner or camera. For example, a supplier invoice may generally include a table with parallel lines. In this way, lines that are not straight or parallel can be detected in the region of interest and these lines can be corrected when pre-processing the region of interest.For example, correction for skew can be achieved by determining a transformation matrix that can be applied to the pixelated data of the region of interest.

[0029] The pre-processed regions of interest of the filtered electronic documents may then be analyzed by image-based metadata identification modeling 236. Specifically, image features of the converted text document may be extracted or learned using a predefined algorithm for image feature extraction (237). The extracted image features may be input into an image-based metadata identification model 238, which classifies the regions of interest into one of the predefined metadata categories. The image-based metadata identification model may be based on, for example, a regression algorithm, and thus may be designed to determine the probability that a particular region of interest is associated with each of the predefined metadata categories. For example, the image-based metadata identification model may be designed to determine the probability that a region of interest of a particular image of an input electronic document from the supplier invoice category is associated with each of the predefined supplier name or ID group. In one exemplary implementation, the image feature extraction 237 and the image classifier model may be based on a neural network approach such as a convolutional neural network.

[0030] The image-based metadata identification modeling 236 may not provide accurate metadata identification for some input regions of interest. For example, the probability that the input region of interest is associated with any particular predefined metadata category, as determined by the image-based metadata identification modeling 236, may not stand out among other predefined metadata categories, indicating that the metadata identification process 236 is ambiguous about assigning a metadata category to the image corresponding to the input region of interest. In one case, images of metadata categories corresponding to some regions of interest of the input electronic documents processed by the image-based metadata identification modeling 236 may not be part of the training data corpus of the image-based metadata identification model 238. In other cases, template images of particular metadata categories used in the training data corpus may evolve and change. In still other cases, the same template image in the training data corpus may correspond to multiple metadata categories (e.g., the same invoice template image corresponds to multiple different supplier names or IDs). In these various cases, Figure 2 The image-based metadata identification modeling 236 may not be able to unambiguously identify a single correct metadata category for the input region of interest with reasonable accuracy. Figure 2 The fuzzy text matching process 240 further supplements those regions of interest with metadata identifications of corresponding input electronic documents.

[0031] In one implementation, the fuzzy text matching process 240 includes a metadata database 244 in communication with the fuzzy text matching model 242. The metadata database 244 may contain metadata corresponding to the following electronic documents that do not exist or are otherwise not included in the training data corpus for the image-based metadata identification model 238, which is developed for the image-based metadata identification process 230. For example, there may be a complete list of past, current, and future suppliers. Metadata for these suppliers, such as supplier names and IDs, may be maintained in the metadata database 244. Actual invoices may only exist for a subset of these suppliers. Invoices from other potential suppliers may not have been received before. The training and testing corpora of the above image-based metadata identification model 238 may correspondingly include only invoice documents or images of a subset of potential suppliers. When the input electronic document contains an invoice issued by one of these potential suppliers, the image-based metadata identification model 238 may not accurately determine the metadata category of these input documents.

[0032] In these cases, because the textual metadata for these potential suppliers may be available in the metadata database 244, and the fuzzy text matching process 240 may use such information to provide better metadata identification for the input electronic documents associated with these potential suppliers. In particular, the fuzzy matching process 240 may include: identifying metadata entries in the metadata database 244 that approximately match the textual information contained in the input electronic documents. The final metadata identifications for these electronic documents may be determined by considering both the probabilities provided by the image-based metadata identification process 230 and the suggestions of the fuzzy text matching process 240 (260).

[0033] The textual metadata information in metadata database 244 may be extracted from a variety of sources. These sources may be text-based. Alternatively, these sources may be image-based. For example, images of documents other than invoices associated with a supplier may be obtained and these images may be converted to textual information by, for example, an OCR process, and metadata may be further extracted from the converted textual information and maintained by metadata database 244.

[0034] Figure 3 shows the method for establishing Figure 2 300 of the logic and data flow of an exemplary document classifier model 226 used in the text-based document classification process 220 of FIG. The development of the document classifier model may include: building a corpus of text documents or a corpus of documents labeled with a predefined set of document categories 310. The development of the document classifier model 226 may also include: a text feature extraction process 320 based on a machine learning algorithm and iterative document classifier modeling 330.

[0035] The document corpus 310 labeled with a predefined set of document categories can be divided into a training text document set 312 with labels 314 and a test text document set 316 with labels 318. The predefined set of document categories can be determined based on the scope of the electronic document collection. For example, the electronic documents can be related to supplier communications and can include supplier invoices, supplier plans, product catalogs, product manuals, warranties, supply contracts, etc. Accordingly, the electronic documents can be labeled with these exemplary categories. The electronic documents in the corpus 310 can be manually labeled, or can be labeled using an automated process.

[0036] A multidimensional feature space can be defined to represent electronic documents in the document corpus 310. The multidimensional feature space can be formed by a set of text features, and these features can be further extracted from each document in the training document set 312 to represent the document in the multidimensional feature space (320). Each document in the training text document set 312 can be represented by a vector in the multidimensional feature space, which corresponds to a point in the multidimensional feature space having coordinates determined by the extracted features. The machine learning algorithm in the document classifier modeling process 330 is then responsible for determining rules and parameters that can be used to cluster or partition the multidimensional feature space so that points in the multidimensional feature space representing documents with the same document label in the training text documents fall roughly in the same partition according to the evaluation function of the machine learning algorithm. The partitioned multidimensional feature space and corresponding parameters form the basis of the text-based document classification model 226.

[0037] The dimension of the text feature space can be predefined. For example, a bag of words can be used to determine the text feature space. Other implementations for constructing a multidimensional text feature space can include: a concept feature space, a context feature space based on n-grams, a feature space extracted mechanically, and a document structure feature space. The dimensionality of the multidimensional feature space can be limited for computational proficiency. For example, a scoring function can be used to rank feature dimensions, and only a predetermined number of optimal feature dimensions are used to form a multidimensional feature space of a text-based document classifier model 226. In one implementation, the text features can be a set of words or phrases, and the ranking of these features can be determined by the frequency with which these words or phrases appear in the document corpus 310.

[0038] Document classifier modeling 330 can be based on various machine learning algorithms, including various classification algorithms and regression algorithms. For example, these classification and regression algorithms can be based on random forest regression, linear regression, logistic regression, naive Bayes classifier, Bayesian network, ordinary least squares regression, principal component regression, etc. An evaluation function with multiple modeling parameters can be designed to cluster or partition the multidimensional feature space so that points representing documents with the same document label in the training text document in the multidimensional feature space fall roughly in the same partition. The modeling parameters form the basis of the text-based document classifier model 226. When using the test text document set 316 in the document corpus 310, the established classification model can be further tested and improved. In particular, the features (320) of the documents in the test text document set 316 can be similarly extracted and input into the text-based document classifier model 226, and these features are classified according to the modeling parameters. The resulting classification can be compared with the test text document label 318. The document classifier modeling 330 may be applied iteratively to refine the modeling parameters so that the resulting document classifier model 226 achieves a predetermined accuracy in classifying the test text document set 316. The final text-based document classifier model may be used in Figure 2 In the text-based document classification process 220. Figure 2 The text feature calculation 225 follows accordingly Figure 3 The same feature extraction process 320 of .

[0039] Figure 4 shows the method for establishing Figure 2 4. Logic and data flow 400 of an exemplary image-based metadata identification model 238 used in the image-based metadata identification process 230 of FIG. 4. The image-based metadata identification model 238 may be an image classifier for classifying input images into a predetermined set of metadata categories (e.g., vendor name or ID). Development of the image-based metadata identification model may include: building a corpus 410 of images labeled with a predefined set of metadata categories (e.g., vendor name or ID). Figure 2 In one implementation, the development of the image-based metadata identification model 238 may further include: image feature extraction or learning based on a machine learning algorithm (320) and iterative image-based metadata identification modeling 430.

[0040] The image corpus 410 labeled with a predefined set of metadata categories can be divided into a training image set 412 with labels 414 and a test image set 416 with labels 418. The predefined set of metadata categories can be determined based on the scope of the images in the image corpus 410. For example, the images in the image corpus 410 can be related to invoices from various suppliers. The metadata categories can correspond to supplier names or IDs. Therefore, the images in the image corpus 410 can be labeled with these supplier names or IDs accordingly. The images in the corpus can be manually labeled, or can be labeled using an automated process.

[0041] exist Figure 4 In the implementation of , the image feature extraction 420 based on the machine learning algorithm and the image-based metadata identification modeling 430 can be separate processes. Specifically, a multidimensional image feature space can be determined or learned to represent the images in the image corpus 410. The multidimensional image feature space can be formed by the image feature set, and these image features can be learned and extracted from each of the images in the training image set 412 to represent the image in the multidimensional image feature space (420). Each of the images in the training image set can be represented by a vector in the multidimensional image feature space, which corresponds to a point in the multidimensional image feature space, which has coordinates determined by the extracted image features. Then, the machine learning algorithm in the image-based metadata identification modeling process 430 is responsible for determining model rules and parameters, which can be used as the basis of the image-based metadata identification model 238 using the metadata category labels 414 for the training images 412. When using the test image set 416 and the test image labels 418, the model rules and parameters can be further improved. The final image-based metadata identification model 238 may then be used to derive the probability that the input image is any one of a predetermined set of metadata categories.

[0042] Alternatively, the image-based metadata identification modeling can be implemented based on multi-layer deep learning technology, including but not limited to convolutional neural network (CNN). In this way, the image feature extraction 237 and the image-based metadata identification model 238 can be implemented as a multi-layer CNN as a whole, such as Figure 5 An exemplary logic and data flow is shown in FIG. 500 .

[0043] The exemplary CNN-based metadata identification modeling logic and data flow 500 may include an image feature extraction layer 510 and a classification layer 540 for iterative learning of image features, and a classification model (560 and 570) using a labeled training image 512. Specifically, the training image 512 is convolutionally filtered using N image features (each image feature in the form of an image block 514), as shown in 516, to obtain N feature maps 518. Each of the N image features is randomly initialized. Figure 5 In the example of , the training image is normalized to a predetermined size of, for example, 32 by 32 (as shown in 526s). The size of the image block for each image feature can be predetermined (e.g., 5 by 5, as shown in 514s). After the convolution filter 516, the 32 by 32 training image 512 becomes N feature maps 518 of 28 by 28 (as shown in 524). The number N (522) of feature maps corresponds to the number of image features 514. Each point (e.g., point 520) in the feature map 518 corresponds to the convolution of one of the image features 514 and a 5×5 block of the training image 512. In this example, the size of the feature map 518 is 28 by 28 because the 5 by 5 feature 514 produces 28 by 28 unique locations when moving across the entire 32 by 32 training image. The feature maps 518 can then be max-pooled to reduce their size (532) to obtain the max-pooled feature map 528. For example, max pooling can divide each feature map into a predetermined number of groups, for example, 2 by 2 adjacent points, and only keep the maximum point in the group. Figure 5 In the example of , the max-pooled feature map 528 includes N 14 by 14 feature maps (as shown in 530). The processes 516 and 532 can be repeated in multiple convolutional and max-pooling layers using higher-level image features. The resulting final max-pooled feature map can be vectorized in 534 to obtain a fully connected layer 542. In some alternatives, the fully connected layer can be implemented as multiple layers. The (multiple) fully connected layers 542 can then be associated with the class labels of the training images, for example, supplier A 544, supplier B 546, supplier C 548, or supplier D 550.

[0044] The above process represents a forward propagation 560. The initial random image features 514 may be refined to minimize any error in classifying the training images 512 into their correct labels via a back propagation process 570. The forward propagation process 560 and the back propagation process 570 are iteratively performed for a large number of training images to obtain common image features that provide the best classification consistent with the labels of the training images. Finally, the input image may be processed by the forward propagation 560 to obtain a predicted probability that the input image is associated with any of the suppliers A, B, C, or D. Thus, the CNN-based image classification process 500 may be used to implement Figure 2 Image-based metadata identification modeling 236.

[0045] Figure 6 shows that can be used as Figure 2 6. The exemplary fuzzy text matching process 240 includes extracting n-grams 632, 624, and 636 (630) from an input text document 640, accessing a metadata database 510 having a plurality of metadata data items 611, and running a fuzzy text matching algorithm 650 on the extracted n-grams and the metadata data items to obtain the best metadata data item that matches the extracted n-gram.

[0046] The n-grams of the input electronic document are basically terms of a predefined length in the input electronic document. These terms may include single words (unigrams), two words (bigrams), three words (trigrams), or more. For example, a trigram of a document may include all three consecutive word patterns that appear in the document. Using terms with multiple consecutive words to characterize the input document can help reduce false positives in the fuzzy text matching process 600, thereby supplementing and improving rather than reducing Figure 2 The accuracy of the image-based metadata identification process 240 is determined by the image-based metadata identification process 240. For example, the word "Dell" may refer to the supplier Dell company, but may also appear in the electronic document as a product name rather than a supplier name associated with the metadata category. Figure 6 The illustration of FIG. 64 shows only the extraction of singletons, bigrams, and trigrams from the input electronic document 640, but higher-level n-grams can be similarly extracted. The number of n-grams to be included in the input electronic document 640 can be determined by considering the trade-off between computational requirements and modeling accuracy gain. Figure 6 The number of levels of n-grams in the fuzzy matching process.

[0047] Metadata database 610 may contain metadata data items represented by table 611. The metadata data items may be logically arranged in rows and columns. In one example, rows 620, 622, 624, 626, and 628 may represent metadata for different vendors. Columns 612, 614, 616, and 618 may represent different types of metadata. Alternatively, data columns 611 may be referred to as metadata fields. One of the columns or fields (e.g., metadata field 612) may represent a predefined metadata category. In this way, Figure 2 The input electronic document will be labeled with one of the metadata data items of metadata field 612, such as supplier A, supplier B, supplier C, and supplier D. For example, other metadata fields may include supplier address field 614, supplier contact field 616, and supplier phone number field 618.

[0048] The n-grams 632, 634, and 636 extracted from the input document 640 and the metadata data item 611 can be input into the fuzzy text matching algorithm 650. In one implementation, fuzzy matching between the n-grams and the metadata data items can be performed for each metadata column or field. A predetermined number of best matching metadata data items can be determined for each metadata field, as shown in 660, 662, 664, and 666. For example, the 20 best matches in each metadata field can be identified.

[0049] A fuzzy match may be performed between each metadata data item and all extracted n-grams 632, 634, and 636 of the input document. The matches may be weighted, for example, giving different weights to units, bigrams, and trigrams. For example, trigrams may be given a greater weight than bigrams and units, and bigrams may be given a higher weight than units because higher-level n-grams may carry more syntactic information.

[0050] The best matches from each metadata field may be further compared to extract a predetermined number of final matches (670). Different weights may be given to different fields during this comparison process. For example, a greater weight may be given to the address 614 than to the phone number field 618. In one exemplary implementation, the final top three metadata items may be determined. The corresponding data item 670 in the metadata category field 612 may also be identified as the final best matching metadata category 680. In some implementations, a preferred weight may be given to a metadata row with multiple best matching data items. Such a preferred weight may be given because multiple matching data items within the same row indicate that the input document contains a more detailed description of the entity associated with that row of the metadata database, and therefore the document may be preferably tagged with such an entity (or metadata category).

[0051] The implementation disclosed above provides improvements in computing technology. For example, text data in electronic documents can be processed with string data types and representations. The implementation disclosed herein includes: transforming a document block with embedded text into a data matrix representing a bitmap image of the document block, thereby providing a data structure that is more efficiently processed by using hardware-accelerated matrix operations. The data matrix also contains information other than the string content of the embedded text, which can be identified via machine learning of data patterns in the matrix data structure.

[0052] The above exemplary implementations facilitate automated document processing, identification, and organization, such as invoice processing and supplier identification. Similar methods can be used, for example, for delivery tracking (e.g., by courier / logistics service providers) to classify documents as "delivery records" and extract supplier metadata from the delivery records. As another example, the above implementations can be used for insurance claims processing to classify claim documents into different types of claims, such as motor vehicle insurance claims, medical insurance claims, etc., and extract / identify, for example, claimant metadata only from medical claims (e.g., hospital, clinic, and doctor metadata information), or extract / identify manufacturer metadata information only from motor vehicle insurance claims.

[0053] The following principles can be widely applied in industrial and manufacturing environments: use imaging classification to identify metadata and use fuzzy text matching to achieve better accuracy or to supplement when the training images for the image classification model are incomplete. For example, identifying the manufacturer of an electronic component in an electronics assembly plant can rely on image recognition of a set of electronics manufacturers' logos. This image-based recognition can be further supplemented by extracting text from the images of the electronic components and fuzzy matching the extracted text with a metadata database of the electronics manufacturers, which can be more comprehensive than the training image database used for the image-based classification model. Therefore, manufacturers of electronic components with logos that are not part of the training images (e.g., the manufacturer has changed to a new logo) can still be identified with reasonable accuracy via the supplementary fuzzy text matching process.

[0054] Although the above disclosure uses text-based document classification and image-based metadata identification as examples, the principle of cascading multiple classification or regression models to simplify classification problems and reduce computational intensity can be widely applied to other scenarios where large and diverse feature sets and feature spaces are involved.

[0055] The methods, devices, processes, and logic described above may be implemented in many different ways and in many different combinations of hardware and software. For example, all or part of the implementation may be a circuit including an instruction processor, such as a central processing unit (CPU), a microcontroller, or a microprocessor; an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA); or a circuit including discrete logic or other circuit components, including analog circuit components, digital circuit components, or both; or any combination thereof. As an example, the circuit may include discrete interconnected hardware components, and / or may be combined on a single integrated circuit die, distributed among multiple integrated circuit dies, or implemented as a multi-chip module (MCM) of multiple integrated circuit dies in a common package.

[0056] The circuit may also include or have access to instructions for execution by the circuit. These instructions may be stored on a tangible storage medium other than a transient signal, such as flash memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM); or on a magnetic disk or optical disk, such as a compact disk read-only memory (CDROM), a hard disk drive (HDD), or other magnetic disk or optical disk; or in or on another machine-readable medium. A product such as a computer program product may include a storage medium and instructions stored in and on the medium, and the instructions, when executed by circuits in a device, may cause the device to implement any of the processes described above or shown in the accompanying drawings.

[0057] These implementations can be distributed as circuits among multiple system components, such as between multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures can be stored and managed separately, can be merged into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many different ways, including as data structures such as linked lists, hash tables, arrays, records, objects, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), separate programs, distributed among several memories and processors, or implemented in many different ways, such as in libraries, such as shared libraries (e.g., dynamic link libraries DLLs). For example, a DLL can store instructions that, when executed by a circuit, perform any of the processing described above or shown in the accompanying drawings.

[0058] Various implementations have been described in detail. However, many other implementations are possible.

Claims

1. A system comprising: a database (244) comprising metadata data elements organized in metadata fields (611), and a fuzzy text matching model (242), wherein the metadata fields (611) include at least a metadata category field, the metadata category field containing metadata category data elements; A memory (120) for storing a text-based document classifier model (142) established based on a machine learning algorithm, a metadata category identification model (238) trained using a training data corpus (410), and a fuzzy text matcher (146); as well as A system circuit in communication with the database (244) and the memory (120), the system circuit being configured to: Receive electronic documents; In response to determining, using the text-based document classifier model (142), that the electronic document belongs to a predetermined set of document categories: obtaining an image corresponding to the electronic document; obtaining a first association of the image with a first metadata category data element of the metadata category field using the metadata category identification model (238); using the fuzzy text matcher (146) to obtain a second association of the electronic document with a subset of the metadata data elements of the database; Based on the first association and the second association, determining a single metadata category data element of the first metadata category data element and a set of metadata category data elements corresponding to the subset of the metadata data elements of the database as a label for the electronic document; as well as The second association of the electronic document with the subset of the metadata data elements of the database is obtained using the fuzzy text matcher (146) by: extracting n-grams (632, 634, 636) of text contained in the electronic document to represent the electronic document; as well as performing a fuzzy match between the n-gram (632, 634, 636) and the metadata data elements of the database to identify the subset of the metadata data elements of the database that matches the n-gram (632, 634, 636) and to obtain the second association, wherein performing the fuzzy match comprises: performing fuzzy matching between the n-gram and the metadata data elements of the database to identify a predetermined number of best matching metadata data elements within each of the metadata fields (611) to obtain a set of metadata data elements that best match the n-gram (632, 634, 636) of the electronic document, and to obtain a corresponding association; and identifying the subset of the metadata data elements by weighted comparison of corresponding associations of the set of metadata data elements that best matches the n-tuple (632, 634, 636); wherein the database (244) contains metadata corresponding to electronic documents that are not present in or otherwise not included in the training data corpus (410) for the metadata category identification model (238); wherein the n-tuple (632, 634, 636) includes a single-tuple (632), a binary set (634), and a triplet (636); and The fuzzy matching between the n-tuple (632, 634, 636) and the data items of the database includes: setting different weights on the single-tuple (632), the binary set (634) and the triple-tuple (636).

2. The system according to claim 1: Wherein the system circuit is further configured to: obtain a region of interest from the image corresponding to the electronic document; and The system circuitry is configured to obtain the first association by applying the metadata category identification model (238) to the region of interest.

3. The system of claim 2, wherein the region of interest comprises a portion of the image, wherein the system circuitry is configured to obtain the region of interest from the image corresponding to the electronic document by: determining the position and size of the region of interest relative to the image; and The image is cropped and / or deformed and / or realigned according to the position and the size of the region of interest to obtain the region of interest from the image.

4. The system of claim 3, wherein the region of interest comprises a pixel array, and wherein pre-processing the cropped image comprises at least one of reducing pixel resolution and correcting for skew.

5. The system of claim 1, wherein the triplet (636) is given a higher weight than the binary set (634), and the binary set (634) is given a higher weight than the singleton set (632).

6. A method comprising: An electronic document is received by a system having a memory (120), a database (244), and system circuitry, the memory being used to store a text-based document classifier model (142) built based on a machine learning algorithm, a metadata category identification model (238) trained using a training data corpus (410), and a fuzzy text matcher (146), the database (244) including metadata data elements organized in metadata fields (611), and the fuzzy text matching model (242), wherein the metadata fields (611) include at least a metadata category field, the metadata category field containing metadata category data elements; In response to determining, using the text-based document classifier model (142), that the electronic document belongs to a predetermined set of document categories: obtaining an image corresponding to the electronic document; obtaining a first association of the image with a first metadata category in a set of metadata categories using the metadata category identification model (238) stored in the memory (120); using the fuzzy text matcher (146) stored in the memory (120) to obtain a second association of the electronic document with a subset of the metadata data elements of the database; Based on the first association and the second association, determining a single metadata category in the first metadata category and a set of metadata categories corresponding to the subset of the metadata data elements of the database as a label for the electronic document; The second association of the electronic document with the subset of the metadata data elements of the database is obtained using the fuzzy text matcher (146) by: extracting n-grams (632, 634, 636) of text contained in the electronic document to represent the electronic document; performing a fuzzy match between the n-gram (632, 634, 636) and the metadata data elements of the database to identify the subset of the metadata data elements of the database that match the n-gram (632, 634, 636) and to obtain the second association, wherein performing the fuzzy match comprises: performing fuzzy matching between the n-gram and the metadata data elements of the database to identify a predetermined number of best matching metadata data elements within each of the metadata fields (611) to obtain a set of metadata data elements that best match the n-gram (632, 634, 636) of the electronic document, and to obtain a corresponding association; and identifying the subset of the metadata data elements by weighted comparison of corresponding associations of the set of metadata data elements that best matches the n-tuple (632, 634, 636); wherein the database (244) contains metadata corresponding to electronic documents that are not present in or otherwise not included in the training data corpus (410) for the metadata category identification model (238); wherein the n-tuple (632, 634, 636) includes a single-tuple (632), a binary set (634), and a triplet (636); and The fuzzy matching between the n-tuple (632, 634, 636) and the data items of the database includes: setting different weights on the single-tuple (632), the binary set (634) and the triple-tuple (636).

7. The method according to claim 6, further comprising: The electronic document is converted into a corresponding text document, and wherein the electronic document is determined to belong to the predetermined document category set by: inputting the corresponding text document into the text-based document classifier model (142) to obtain a classification; as well as The classification is determined to be within the predetermined document category set.

8. The method according to claim 6 or claim 7, further configured to: obtain a region of interest from the image corresponding to the electronic document, and wherein obtaining the first association comprises: The metadata category identification model (238) is applied to the region of interest.

9. The method of claim 8, wherein the region of interest comprises a portion of the image, wherein obtaining the region of interest from the image corresponding to the electronic document comprises: determining the position and size of the region of interest relative to the image; as well as cropping and / or deforming and / or realigning the image according to the position and the size of the region of interest to obtain the region of interest from the image; and / or Wherein obtaining the region of interest from the image corresponding to the electronic document comprises: determining the position and size of the region of interest relative to the image; cropping the image according to the position and the size of the region of interest to obtain a cropped image; and The cropped image is preprocessed to obtain the region of interest.