Methods, systems, and computer program products for adaptive document understanding

Abstract

A method, system, and program are provided to create multiple page clusters in a feature space from multiple feature vectors corresponding to multiple unstructured pages. The method, system, and program products assign one of multiple machine learning models to each of the multiple page clusters based on the relationship in the feature space between the multiple page clusters and multiple training clusters corresponding to the multiple machine learning models. The method, system, and program products identify a page in the multiple page clusters that corresponds to a selected page in the multiple unstructured pages, and convert the selected unstructured page into a structured page using the selected machine learning model assigned to the identified page cluster.

Description

Background Technology

[0001] Machine learning algorithms build machine learning models based on sample data (called training data) to make predictions or decisions without being explicitly programmed. The process of training a machine learning model involves feeding the machine learning algorithm training data from which it learns, and the artifact created from the training process is the machine learning model. The training data includes correct answers called targets or target attributes, and the machine learning algorithm finds patterns in the training data that map input data attributes to target attributes, and outputs a machine learning model that captures those patterns.

[0002] Structured data refers to data residing in fixed fields within files or records and therefore easily analyzed. Unstructured data (or unstructured information) is information that does not have a predefined data model or is not organized in a predefined way. Unstructured information is typically text-intensive but can include data such as dates and numbers. Furthermore, unstructured data is often irregular and ambiguous, making it difficult for traditional programs to interpret such irregularities and ambiguities.

[0003] Intelligent Document Understanding (SDU) methods transform unstructured documents into structured data using machine learning. In SDU, users input annotations onto training documents extracted from the input document, and these documents are used as teaching images to train the model. However, a challenge currently identified in SDU systems is that page formats often vary between pages and various types of documents. Some pages may be in a two-column format, others may include graphics and images, while still others may be traditional paragraph-based alphabetical formats.

[0004] Therefore, using a single machine learning model to cover different page formats is difficult and leads to reduced conversion accuracy. Furthermore, existing SDU training methods require a minimum amount of training data extraction to effectively train machine learning models (e.g., random sampling). This allows for the selection of documents biased towards specific formats, which also reduces the conversion accuracy of the machine learning model. Summary of the Invention

[0005] According to one embodiment of this disclosure, a method, system, and program are provided, wherein a plurality of clusters are created in a feature space from multiple feature vectors corresponding to a plurality of unstructured pages. The method, system, and program products assign one of the multiple machine learning models to each of the multiple clusters based on the relationship in the feature space between the multiple page clusters and multiple training clusters corresponding to multiple machine learning models. The method, system, and program products identify a page in the multiple page clusters that corresponds to a selected one of the multiple unstructured pages, and transform the selected unstructured page into a structured page using the selected one of the multiple machine learning models assigned to the identified page cluster. In this embodiment, the method, system, and program products improve the accuracy of data transformation by adaptively selecting the best-fit machine learning model from the multiple machine learning models to transform unstructured data into structured data.

[0006] According to another embodiment of this disclosure, a method, system, and program product are provided to divide an unstructured document into multiple unstructured pages. The method, system, and program product select one of the multiple unstructured pages and define a set of character regions and a corresponding set of positions within the selected unstructured page. The method, system, and program product calculate a set of character region feature vectors corresponding to the set of character regions based on the corresponding position group of the character regions and the content group within their corresponding character regions. In this embodiment, the method, system, and program product calculate multiple fine-tuned feature vectors for each unstructured page based on content type and positioning within the page.

[0007] According to another embodiment of this disclosure, a method, system, and program product are provided to calculate one of a plurality of feature vectors of a selected unstructured page based on a character region feature vector set, and to map the selected feature vector to a feature space. In this embodiment, the method, system, and program product combine a plurality of feature vectors of a specific unstructured page into a finely tuned single page feature vector that best describes the content type and content location in the unstructured page.

[0008] According to another embodiment of this disclosure, a method, system, and program product are provided in which hierarchical clustering is performed on selected feature vectors. The method, system, and program product identify one of a plurality of page cluster centers corresponding to multiple page clusters in the feature space that are closest to the selected feature vector. The method, system, and program product add the selected feature vector to the identified one of the plurality of page clusters corresponding to the identified page cluster center. In this embodiment, the method, system, and program product group page feature vectors into clusters based on comparable features ultimately assigned to a specific one of multiple machine learning models.

[0009] According to another embodiment of this disclosure, a method, system, and program product are provided for calculating multiple page cluster centers based on multiple page clusters. The method, system, and program product calculate multiple training cluster centers based on multiple training clusters. The method, system, and program product select one of the multiple page cluster centers. The method, system, and program product identify one of a plurality of training cluster centers in the feature space that is closest to the selected page cluster center. The method, system, and program product assign one of the plurality of machine learning models corresponding to the identified training center cluster to the page cluster corresponding to the selected page cluster center. In this embodiment, the method, system, and program product adaptively assign each machine learning model to a specific page cluster center so that each page cluster uses its own machine learning model to transform unstructured data into structured data.

[0010] According to another embodiment of this disclosure, a method, system, and program product are provided to identify different pages corresponding to different pages in a plurality of unstructured pages within a plurality of page clusters. The method, system, and program product use different machine learning models from a plurality of machine learning models assigned to different page clusters to transform the different unstructured pages into different structured pages. In this embodiment, when different unstructured pages have characteristics different from previously transformed unstructured pages, the method, system, and program product use different machine learning models to transform the different unstructured pages into structured pages.

[0011] According to another embodiment of this disclosure, a method is provided in which a method, system, and program product trains a selected machine learning model using a portion of a plurality of unstructured documents corresponding to an identified cluster of pages. The method, system, and program product use the trained machine learning model to perform the transformation and add the trained machine learning model to the plurality of machine learning models. In this embodiment, the method, system, and program product further trains each machine learning model on a portion of the unstructured pages to better refine the transformation capabilities of each machine learning model.

[0012] According to another embodiment of this disclosure, a method is provided in which a plurality of unstructured pages include a plurality of unstructured page types, and each of the plurality of unstructured page types is assigned one of a plurality of machine learning models to perform a transformation. In this embodiment, the method, system, and program product assign a specific machine learning model to transform each specific unstructured page into a structured page based on the type of the unstructured page.

[0013] The foregoing is an overview and therefore necessarily includes simplifications, generalizations, and omissions of details; thus, those skilled in the art will recognize that this overview is merely illustrative and not intended to be limiting in any way. Other aspects, inventive features, and advantages of this disclosure, as defined only by the claims, will become clear in the following non-limiting detailed description. Attached Figure Description

[0014] This disclosure can be better understood by referring to the accompanying drawings, and its various objects, features and advantages will become apparent to those skilled in the art, wherein:

[0015] Figure 1 This is a block diagram of a data processing system that can implement the methods described herein;

[0016] Figure 2 Provided Figure 1 The information processing system environment shown is an extension to illustrate that the methods described herein can be performed on a wide variety of information processing systems operating in a networked environment;

[0017] Figure 3 It is a diagram depicting an adaptive document understanding system that trains a machine learning model and uses the trained machine learning model to extract structure.

[0018] Figure 4 This is a diagram depicting an adaptive document understanding system 300 that uses unstructured documents to train model 335;

[0019] Figure 5 This is a diagram describing an adaptive document understanding system 300, which determines a suitable machine learning model for each unstructured page and uses its chosen machine learning model to transform each unstructured page into a structured page.

[0020] Figure 6 This is an exemplary flowchart showing the steps taken to train model 335 using document 400;

[0021] Figure 7 This is an exemplary flowchart illustrating the steps taken in hierarchical clustering;

[0022] Figure 8 This is an exemplary flowchart illustrating the steps taken to apply model 335 to a specific model inference for unstructured page 420;

[0023] Figure 9 This is a diagram illustrating how the adaptive document understanding system 300 segments a page into character regions, creates character region feature vectors for the character regions, and creates page feature vectors from the character region feature vectors.

[0024] Figure 10 This is a diagram depicting an adaptive document understanding system 300 that maps document pages to a feature space during the model training phase; and

[0025] Figure 11 This is a diagram illustrating an adaptive document understanding system 300 that maps page feature vectors to a feature space and selects machine learning methods to apply to the page based on distance measurements from the cluster center. Detailed Implementation

[0026] The terminology used herein is for the purpose of describing particular embodiments and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0027] All the means or steps of the multifunctional element in the following claims are intended to include any structure, material, action, and equivalent for performing functions in combination with other claimed elements, such as those specifically claimed. The description of this disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the form of the disclosure. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Embodiments have been chosen and described in order to best explain the principles and practical application of this disclosure, and to enable others skilled in the art to understand this disclosure with respect to various embodiments having various modifications suitable for the specific intended use.

[0028] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0029] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DvD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0030] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0031] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (iSA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0032] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0033] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0034] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0035] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0036] Figure 1 An information processing system 100 is shown, which is a simplified example of a computer system capable of performing the computational operations described herein. The information processing system 100 includes one or more processors 110 coupled to a processor interface bus 112. The processor interface bus 112 connects the processors 110 to a northbridge 115, also referred to as a memory controller hub (MCH). The northbridge 115 is connected to system memory 120 and provides the processors 110 with means of accessing system memory. A graphics controller 125 is also connected to the northbridge 115. In one embodiment, a peripheral component interconnect (PCI) fast bus 118 connects the northbridge 115 to the graphics controller 125. The graphics controller 125 is connected to a display device 130, such as a computer monitor.

[0037] Northbridge 115 and Southbridge 135 are connected to each other via bus 119. In some embodiments, the bus is a Direct Media Interface (DMI) bus that transmits data at high speed in each direction between Northbridge 115 and Southbridge 135. In some embodiments, a PCi bus connects the Northbridge and Southbridge. Southbridge 135, also known as the Input / Output (I / O) Controller Hub (ICH), is a chip that typically implements the ability to operate at a slower speed than that provided by the Northbridge. Southbridge 135 typically provides different buses for connecting different components. These buses include, for example, PCI and PCI Express buses, ISA buses, system management buses (SMBus or SMB), and / or low pin count (LPC) buses. LPC buses typically connect low-bandwidth devices such as boot ROM 196 and “traditional” I / O devices (using “super I / O” chips). “Traditional” I / O devices (198) may include, for example, serial and parallel ports, keyboard, mouse, and / or floppy disk controllers. Other components that are often included in the Southbridge 135 include a direct memory access (DMA) controller, a programmable interrupt controller (PIC), and a storage device controller that uses bus 184 to connect the Southbridge 135 to a non-volatile storage device 185 (such as a hard disk drive).

[0038] The ExpressCard 155 is a slot for connecting hot-swappable devices to an information processing system. The ExpressCard 155 supports both PCI Express (High-Speed ​​Bus) and Universal Serial Bus (USB) connectivity because it uses both USB and PCI Express buses to connect to the Southbridge 135. The Southbridge 135 includes a USB controller 140, which provides USB connectivity to devices connected to USB. These devices include a webcam (camera) 150, an infrared (IR) receiver 148, a keyboard and touchpad 144, and a Bluetooth device 146 providing a wireless personal area network (PAN). The USB controller 140 also provides USB connectivity with various other USB-connected devices 142, such as mice, removable non-volatile storage devices 145, modems, network interface cards (NICs), Integrated Services Digital Network (ISDN) connectors, fax machines, printers, USB hubs, and many other types of USB-connected devices. Although the removable non-volatile storage device 145 is shown as a USB-connected device, the removable non-volatile storage device 145 can be connected using different interfaces (such as FireWire interfaces).

[0039] Wireless Local Area Network (LAN) device 175 is connected to Southbridge 135 via PCI or PCI Express bus 172. LAN device 175 typically implements one of the air modulation technologies of the Institute of Electrical and Electronics Engineers (IEEE 802.11) standard, all using the same protocol to conduct wireless communication between information processing system 100 and another computer system or device. Optical storage device 190 is connected to Southbridge 135 via Serial Analog Telephone Adapter (ATA) (SATA) bus 188. Serial ATA adapters and devices communicate via a high-speed serial link. The Serial ATA bus also connects Southbridge 135 to other forms of storage devices, such as hard drives. Audio circuitry 160 (e.g., a sound card) is connected to Southbridge 135 via bus 158. Audio circuitry 160 also provides functions associated with audio hardware, such as audio line inputs and optical digital audio input ports 162, optical digital outputs and headphone folders 164, internal speakers 166, and internal microphones 168. Ethernet controller 170 is connected to Southbridge 135 using a bus (e.g., PCI or PCI Express bus). Ethernet controller 170 connects information processing system 100 to computer networks, such as local area networks (LANs), the Internet, and other public and private computer networks.

[0040] Although Figure 1 An information processing system is shown, but information processing systems can take many forms. For example, an information processing system can take the form of a desktop computer, server, portable, laptop, notebook, or other form factor computer or data processing system. Furthermore, an information processing system can take other forms, such as a personal digital assistant (PDA), gaming device, automatic teller machine (ATM), portable telephone device, communication device, or other device including a processor and memory.

[0041] Figure 2 Provided Figure 1 The illustrated extension of the information processing system environment demonstrates that the methods described herein can be executed on a wide variety of information processing systems operating in a networked environment. The types of information processing systems range from small handheld devices such as handheld computers / mobile phones 210 to mainframe systems such as mainframe computers 270. Examples of handheld computers 210 include personal digital assistants (PDAs), personal entertainment devices such as Mobile Image Experts Group Layer 3 audio (MP3) players, portable televisions, and CD-ROM players. Other examples of information processing systems include pen or tablet computers 220, laptop or notebook computers 230, workstations 240, personal computer systems 250, and servers 260. Figure 2Other types of information processing systems shown separately are represented by information processing system 280. As shown, different information processing systems can be networked together using computer network 200. Types of computer networks that can be used to interconnect different information processing systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, the public switched telephone network (PSTN), other wireless networks, and any other network topologies that can be used to interconnect information processing systems. Many information processing systems include non-volatile data storage, such as hard disk drives and / or non-volatile memory. Figure 2 The embodiments of the information processing system shown include separate non-volatile data storage devices (more specifically, server 260 utilizes non-volatile data storage device 265, mainframe computer 270 utilizes non-volatile data storage device 275, and information processing system 280 utilizes non-volatile data storage device 285). The non-volatile data storage devices can be components external to different information processing systems, or components that can be components internal to one of the information processing systems. Furthermore, various technologies can be used to share the removable non-volatile storage device 145 between two or more information processing systems, such as connecting the removable non-volatile storage device 145 to a USB port or other connector of the information processing system.

[0042] As mentioned above, unstructured data appears in many different formats, but existing intelligent document understanding (SDU) methods employ various training data extraction methods for training. Consequently, documents can be selected, with a bias towards suboptimal formats, leading to reduced conversion accuracy. Figures 3 to 11 A method is described that can be executed on an information processing system to enhance the accuracy of transforming unstructured data into structured data by adaptively training multiple machine learning models and selecting the best-fitting model. The method automatically converts documents into feature data (feature vectors) based on character regions; clusters the feature vectors in the feature space based on format; and trains multiple machine learning models specific to their respective formats based on the clustering.

[0043] Then, once the machine learning model is trained, the method adaptively selects the best-fit machine learning model based on the feature vectors of the pages mapped to the feature space to transform specific pages of unstructured data into structured data. In one embodiment, clustering is performed using similarity between images or similarity between words in a sentence. In another embodiment, a more important segmentation model is created from the clusters by using feature values ​​for format extraction. In this embodiment, feature values ​​are shared with subsequent format extraction, thus reducing time, saving resources, and providing high affinity to the application.

[0044] Figure 3This is a diagram depicting an adaptive document understanding system that trains multiple machine learning models and uses these trained models to intelligently transform unstructured data into structured data. The Adaptive Document Understanding (ADU) system 300 comprises two phases: a model training phase 310 and a model inference phase 320. As discussed herein, the model training phase 310 independently trains models 335, which include model A 340, model B 350, model C 360, and model n 370. Then, the model inference phase 320 uses models 335 to transform unstructured data in a document storage 330 into structured data.

[0045] Document storage 330 includes various unstructured document types, such as two-column formatted articles, financial articles with graphics and images, and text based on traditional paragraphs. The model training phase 310 divides the documents from document storage 330 into pages, and then extracts characters from each page (text, images, etc.). The model training phase 310 converts each page into feature data (vectors) in a format indicating information based on the size and position of the characters on the extracted page, and stores the feature information in feature space 380. Then, the model training phase 310 groups the pages into clusters based on the feature data via hierarchical clustering, and determines the centroid of each page cluster, referred to herein as the page cluster center or page cluster center vector (see [link to documentation] for further details). Figure 4 , Figure 6 , Figure 7 , Figure 10 And the corresponding text).

[0046] Next, the model training phase 310 compares the page cluster centers with the training cluster centers in the feature space. The training cluster centers correspond to the training clusters used to initially train the machine learning model 335. Then, the model training phase 310 assigns one of the machine learning models 335 as a base model for each of the page cluster centers (see [link to model training phase]). Figure 10 And the corresponding text to obtain further details).

[0047] Next, model training phase 310 randomly extracts unstructured pages from each page cluster and inputs annotations into the extracted unstructured pages. Then, model training phase 310 uses the annotated unstructured pages to train models 340, 350, 360, and 370 for their respective page clusters. For example, if model A 340 is assigned to a page cluster from... Figure 10 If there is a page cluster A1000, then model A340 is trained on unstructured pages with annotations randomly extracted from page cluster A1000.

[0048] In one embodiment, model 335 has a preference for different content types such as images and text. In this embodiment, the adaptive document understanding system 300 selects which model 335 best fits a particular cluster based on the cluster's position in feature space 380 (e.g., clusters with high image feature values ​​are assigned to image-based machine learning models).

[0049] Once model 335 has completed training, the model inference phase 320 divides the documents from document storage 330 into pages, or uses the pages as is if they have already been divided by the model training phase 310. Then, the model inference phase 320 selects one of the pages, extracts characters from the selected page, and converts the selected page into feature data (vectors) in a format indicative of the selected page based on the size and position of the extracted characters (see [link to relevant documentation] for more details). Figure 9 (and corresponding text).

[0050] Then, the model inference phase 320 maps the feature vectors of the selected pages to the feature space 380 and identifies the nearest page cluster center in the model memory 390 (see [link to model inference] for further details). Figure 11 (and corresponding text). Then, the model inference stage 320 selects a model 335 from the model memory 390 that corresponds to the nearest page cluster center. Next, the model inference stage 320 uses the selected model 335 to convert the selected unstructured page into structured data.

[0051] Then, in the model inference phase 320, different unstructured pages are selected from the document storage 330, a best-fit model is chosen for each page, and the selected best-fit model is used to transform each unstructured page into a different structured page. The model inference phase repeats this process and selects a best-fit model to transform each unstructured page in the document storage 330 into a structured page (see [link to model inference]). Figure 5 (And the corresponding text to obtain further details).

[0052] Figure 4 This is a diagram depicting an adaptive document understanding system 300 that uses unstructured documents to select and train model 335. The adaptive document understanding system 300 (particularly in the model training phase 310) divides the reconstructed document 400 into a set of unstructured pages 420 (d...). i,p ), where i is the index of the document and p is the index of the page.

[0053] In one embodiment, the adaptive document understanding system 300 then extracts character regions from each of the unstructured pages 420 using conventional OCR techniques or existing techniques implemented in the SDU. Optical character recognition (OCR) is the conversion of images of typed, handwritten, or printed text into machine-encoded text, whether from scanned documents, photographs of documents, scene photographs, or caption text overlaid on images.

[0054] For each page in the unstructured page 420, the adaptive document understanding system 300 converts the character region into feature data F={f i,p Each feature data indicates the format (e.g., the number of character regions, the location of the regions, the size of the regions, and the size of the characters) (see [reference]). Figure 9 (And the corresponding text to obtain further details).

[0055] The adaptive document understanding system 300 maps each feature vector into a feature space 380. The adaptive document understanding system 300 then segments the unstructured page 420 into clusters based on their feature mappings. For example, image-dense pages are mapped to specific regions of feature space 380, and text-dense pages are mapped to different regions of feature space 380. The feature vectors of the adaptive document understanding system 300 also take into account the positions of images and text on the page during the mapping process (see [link to documentation]). Figure 9 , 10 (And the corresponding text to obtain further details). Figure 4 The results of feature space clustering are shown as page type A 430, page type B 440, and page type C 450.

[0056] In one embodiment, the adaptive document understanding system 300 creates an optimal number of page clusters (430, 440, 450) for training by dividing unstructured pages 420 into documents with a distance (difference) between them not less than a threshold Lth based on feature data F via hierarchical clustering (see [link]). Figure 7 (And the corresponding text to obtain further details).

[0057] The adaptive document understanding system 300 compares the page cluster centers with the training cluster centers corresponding to the training clusters used to initially train the machine learning model 335. The adaptive document understanding system 300 then assigns one of the machine learning models 335 as the base model for each of the page cluster centers (see [link to system description]). Figure 10 (And the corresponding text to obtain further details).

[0058] The adaptive document understanding system 300 randomly extracts training page samples 435, 445, and 455 from the page cluster, annotates each of the extracted training page samples, and uses the annotated documents to train models A 340, B 350, and C 360, respectively.

[0059] Figure 5 This is an exemplary diagram illustrating an adaptive document understanding system 300, which selects an appropriate machine learning model for each unstructured page and uses the selected machine learning model to transform each unstructured page into a structured page. The adaptive document understanding system 300 (specifically, model inference stage 320) divides the unstructured document 400 into unstructured pages 420. The adaptive document understanding system 300 extracts character regions from each of the unstructured pages 420 using conventional OCR techniques or existing techniques implemented in the SDU, and then converts the character regions into feature data F={f...} as described above. i,p Each feature data indicates the format of things like the number of character regions, the location of the regions, the size of the regions, and the size of the characters (see [link to documentation] for more details). Figure 9 (and the corresponding text).

[0060] Then, the adaptive document understanding system 300 selects one of the unstructured pages 420 and maps its feature vector to the feature space 380. The adaptive document understanding system 300 then identifies the page cluster center (gct) that is closest to the mapped feature vector (see [link to relevant documentation] for further details). Figure 11 (and corresponding text). Then, the adaptive document understanding system 300 selects a model 335 corresponding to the nearest identified cluster center, i.e., a machine learning model trained on unstructured pages in the corresponding cluster. Furthermore, the adaptive document understanding system 300 uses one of the selected models 335 to transform the selected unstructured page 420 into a structured page 540. The adaptive document understanding system 300 repeats this process, individually selecting one of the models 335 for each of the unstructured pages 420 to transform the unstructured data into structured data.

[0061] Figure 6 This is an exemplary flowchart showing the steps taken to train model 335 using unstructured documents 400. Figure 6 Processing begins at 600, followed by step 610, where the unstructured document 400 from document storage 330 is divided into unstructured pages 420. In step 620, the process extracts character regions from the pages and converts these regions into feature vectors f(i,p) (see [link to documentation] for more details). Figure 9 (and corresponding text). Then, the process maps the feature vectors to feature space 380 (see...). Figure 10(And the corresponding text to obtain further details).

[0062] At the predefined process box 630, this process uses hierarchical clustering to group the vector set F={f i,p} Divide into clusters that are at least Lth apart (difference) from each other (see) Figure 7 (See the corresponding text for further details). See also Figure 10 The process divides the vectors into page clusters A1000, B1040, and C1080.

[0063] In step 635, the process calculates the center-to-center distance between the page cluster center corresponding to the existing machine learning model and the training cluster center (see [link to relevant documentation] for further details). Figure 10 (and corresponding text). The process determines whether the center-to-center distance between the page cluster center and the training cluster center is equal to or less than a threshold Lth, such as half of the minimum center-to-center distance between clusters (determination 640). In other words, in one implementation, the process determines whether to assign a specific machine learning model to a specific page cluster center. In another implementation, the process may identify a portion of the page cluster centers that are adjacent to the training cluster centers and assign the specific machine learning model to some page cluster centers while not assigning the specific machine learning model to other page cluster centers.

[0064] If the center-to-center distance between the center of the page cluster and the center of the training cluster is not equal to or lower than the threshold Lth, then the decision is to go to the "No" branch in step 640. Therefore, in step 650, the process randomly initializes the model 335, which does not use individual data to train the model, because each page cluster is sufficiently different from each training cluster in the feature space 380.

[0065] On the other hand, if the center-to-center distance between the page cluster center and the training cluster center is equal to or less than the threshold Lth, then branch 640 is determined to be the "yes" branch. In step 660, the process assigns the most recent existing machine learning model to the page cluster center as the base model for transfer learning and fine-tuning (discussed below). For example, a user may have a machine learning model that has already been trained on a paper from "Conference A", and if the paper formats are similar, the process uses the machine learning model as the base model to train on a paper from "Conference B" to create a new model.

[0066] In step 670, the process randomly extracts training pages from each page cluster and labels the extracted pages. In step 675, the process selects a first set of annotated pages and the corresponding machine learning model (e.g., type A sample 430). In step 675, the process trains the selected model using the corresponding set of annotated training pages. See also Figure 4 The process uses 435 annotated samples of type A to train model A 340.

[0067] In step 685, the process stores the trained model in model 335 and the corresponding page cluster center of the page cluster in model storage 390. The process then ends at 695.

[0068] Figure 7 This is an exemplary flowchart illustrating the steps taken to aggregate unstructured page feature vectors in the feature space. Figure 7 Processing begins at step 700, followed by step 710, where the process selects the first "t", where "t" is the identifier of the cluster. Each document belongs to a separate cluster.

[0069] In step 720, the process sets the feature data f(i,p) of each page into a cluster c(t) and sets the cluster center g(ct) = f(i,p). In step 730, the process calculates the distance l(ct,c't) between each pair of g(ct) and g(c't) (e.g., Euclidean distance calculation). In step 740, the process combines pairs whose distance l(ct,c't) is less than a threshold Lth into clusters in ascending order of distance to create a new cluster set c(t'). In this step, neighboring clusters are merged and a new cluster is created, where t' represents a newly generated identifier.

[0070] The process determines whether to create a new cluster (decision 750). If a new cluster is to be created, then decision 750 branches to the "yes" branch. In step 760, for each cluster c(t'), the process calculates a new set of page cluster centers g(c') based on f(i,p) belonging to page cluster c(t'). Then, the process loops back to steps 730 and 740 to calculate the new distance between the cluster centers and combine the clusters if necessary. This loop continues until the clusters are sufficiently separated from each other and there are no more new clusters to create, at which point decision 750 branches to the "no" branch, exiting the loop. Figure 7 Subsequent processing returns to the calling routine at 795 (see...) Figure 6 (And the corresponding text to obtain further details).

[0071] Figure 8 This is an exemplary flowchart illustrating the steps taken to apply model 335 to a specific model inference for unstructured page 420. Figure 8 The process begins at step 800, followed by step 810, where the unstructured document 400 is divided into unstructured pages 420. In step 820, the process extracts character regions from the pages and converts these regions into feature vectors f(i,p). As previously described, in one embodiment, steps 810 and 820 may be omitted if steps 610 and 620 were performed most recently.

[0072] In step 830, the process selects a first unstructured page, and in step 835, the process maps the feature vector of the selected unstructured page into feature space 380. In step 840, the process computes f(i,p) of the selected page in feature space 380 and the distance between all page cluster centers from the training phase (see [link to training phase] for more details). Figure 11 (and corresponding text).

[0073] In step 850, the process identifies the page cluster center closest to f(i,p) and selects the model to be assigned to the identified page cluster center. (See also...) Figure 11 Cluster center A1010 is closest to feature vector 1110; therefore, model A340 is selected. In step 860, the process uses the selected model to convert the selected unstructured page into a structured page 540. In step 870, the process stores the structured document in document storage 330.

[0074] The process determines if there are more unstructured pages to be converted into structured pages (Decision 880). If there are more pages to be converted, then Decision 880 branches to the "Yes" branch, which loops back to select and process the next unstructured page. This loop continues until every page has been processed, at which point Decision 880 branches to the "No" branch, exiting the loop. Figure 8 The processing ended at point 895.

[0075] Figure 9 This is a diagram illustrating how the adaptive document understanding system 300 segments a page into character regions, creates feature vectors for the character regions, and creates page feature vectors from the character region feature vectors.

[0076] The adaptive document understanding system 300 analyzes the unstructured page 900 and divides it into three character regions: x 910, y 920, and z 930. Then, the adaptive document understanding system 300 uses techniques such as OCR (data x / position x 940, data y / position y 945, data z / position z 950) discussed above to capture the data and location information corresponding to each region.

[0077] Then, the adaptive document understanding system 300 uses a feature vector generator 960 to generate respective region vectors x 960, y 970, and z 980 based on their corresponding data / location data. Next, the feature vector generator 960 combines the region vectors x 960, y 970, and z 980 into a single feature vector 990 to correspond to the unstructured page. The adaptive document understanding system 300 then creates a page feature vector 990 from the character region feature vectors. In one embodiment, the adaptive document understanding system 300 i) applies a sum or average between character regions with similar font sizes; ii) uses only the largest N character regions; and / or iii) sorts the vectors of character regions according to surface regions or coordinates to identify different unstructured page types.

[0078] Then, the adaptive document understanding system 300 maps the page feature vector 990 to the feature space 380 to determine which model 330 to select for converting the unstructured page 900 into a structured page (see [link]). Figure 11 (And the corresponding text to obtain further details).

[0079] Figure 10 This is a diagram depicting an adaptive document understanding system 300 that maps unstructured pages to a feature space and clusters the mappings during the model training phase. Figure 10 Different unstructured page mappings (points) are shown. The adaptive document understanding system 300 uses, for example,... Figure 7 The hierarchical clustering steps shown cluster the maps into clusters A100, B1020, and C1040. As discussed in this paper, the adaptive document understanding system 300 uses cluster analysis to group the map set such that maps in the same group are more similar to each other than maps in other groups. The similarity between maps is often determined using distance measures across different dimensions of the dataset.

[0080] Figure 10 Different training clusters are also shown, including feature data (e.g., pages) used to initially train model 335. Training cluster A 1020 corresponds to the feature data used to train model A 340. Training cluster B 1070 corresponds to the feature data used to train model B 350. And, training cluster C 1090 corresponds to the feature data used to train model C 360.

[0081] The Adaptive Document Understanding System 300 calculates the distance between the page cluster center and the training center cluster in order to assign the appropriate machine learning model to each page cluster center. Figure 10The diagram shows that distance d1 is the shortest distance between page cluster center A 1010 and training cluster center A 1030. Therefore, model A 340 is assigned to page cluster center A 1010. Distance d2 is the shortest distance between page cluster center B 1050 and training cluster center B 1070. Therefore, model B 350 is assigned to page cluster center B 1050. And, distance d3 is the shortest distance between page cluster center C 1085 and training cluster center C 1095. Therefore, model C 360 is assigned to page cluster center C 1085.

[0082] Once the adaptive document understanding system 300 completes the model training phase, it uses the feature space locations of page cluster centers A 1010, B 1050, and C 1085 to determine which machine learning model to select for converting a specific unstructured page into a structured page (see [link to system description]). Figure 11 (And the corresponding text to obtain further details).

[0083] Figure 11 This is a diagram depicting an adaptive document understanding system 300 that maps page feature vectors to a feature space and selects a machine learning model to transform a specific unstructured page into a structured page.

[0084] Adaptive document understanding system 300 (e.g., model inference stage 320) uses Figure 9 The method shown generates a page feature vector 990. Then, the adaptive document understanding system 300 calculates the distances from mapping 1110 to page cluster centers A 1010 (distance A), B 1050 (distance B), and C 1085 (distance C). The adaptive document understanding system 300 then selects an appropriate machine learning model based on the distance measurements. Figure 11 It is shown that distance A is a shorter distance, therefore, the adaptive document understanding system 300 selects model A 340 to apply to the unstructured page 900 to transform the page from an unstructured page to a structured page.

[0085] While specific embodiments of this disclosure have been shown and described, it will be apparent to those skilled in the art that changes and modifications may be made based on the teachings herein without departing from this disclosure and its broader aspects. Therefore, the appended claims are intended to cover within their scope all such changes and modifications within the true spirit and scope of this disclosure. Furthermore, it should be understood that this disclosure is limited only by the appended claims. Those skilled in the art will understand that if a specific number of the introduced claim elements are intended, such an intention will be explicitly stated in the claims, and in the absence of such a statement, there is no such limitation. As a non-limiting example, to aid understanding, the following appended claims contain the use of the introductory phrases “at least one” and “one or more” to introduce claim elements. However, the use of such phrases should not be construed as implying that introducing a claim element by the indefinite article “a (a)” or “an” limits any particular claim containing such an introduced claim element to a disclosure containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and the indefinite articles “a (a)” or “an”; the same applies to the use of definite articles in claims.

Claims

1. A computer-implemented method, comprising: Multiple page clusters are created in the feature space based on multiple feature vectors corresponding to multiple unstructured pages; Based on the relationship between the multiple page clusters in the feature space and the multiple training clusters corresponding to the multiple machine learning models, one machine learning model from the multiple machine learning models is assigned to each of the multiple page clusters. Assigning one machine learning model to each page cluster includes: Calculate multiple page cluster centers based on the aforementioned multiple page clusters; Multiple training cluster centers are calculated based on multiple training clusters corresponding to multiple machine learning models; Select one of the multiple page cluster centers; Identify one of the plurality of training cluster centers in the feature space that is closest to the selected page cluster center; and One of the multiple machine learning models corresponding to the identified training center cluster is assigned to the page cluster corresponding to the selected page cluster center. Identify the page cluster among the plurality of page clusters that corresponds to a selected unstructured page among the plurality of unstructured pages; and The selected unstructured pages are transformed into structured pages using one of the multiple machine learning models assigned to the identified page clusters.

2. The method according to claim 1, further comprising: Divide multiple unstructured documents into multiple unstructured pages; Select one of the plurality of unstructured pages; Define a set of character regions and a corresponding set of positions in the selected unstructured page; as well as Based on the set of positions corresponding to the set of character regions and the set of contents within the character regions corresponding to the positions in the set of positions, calculate the set of character region feature vectors corresponding to the set of character regions.

3. The method according to claim 2, further comprising: Based on the character region feature vector set, calculate one feature vector selected from the plurality of feature vectors for the selected unstructured page; as well as The selected feature vectors are mapped to the feature space.

4. The method of claim 3, further comprising: Perform hierarchical clustering on the selected feature vectors, wherein the hierarchical clustering further includes: The identified page cluster center that is closest to the selected feature vector in the feature space among the multiple page cluster centers corresponding to the multiple page clusters; and The selected feature vector is added to one of the identified page clusters among the plurality of page clusters corresponding to the center of the identified page cluster.

5. The method according to claim 1, further comprising: Identify different page clusters among the multiple page clusters corresponding to different pages among the multiple unstructured pages; as well as Using different machine learning models from the plurality of machine learning models assigned to the different page clusters, the different unstructured pages are transformed into different structured pages.

6. The method of claim 1, further comprising: The selected machine learning model is trained using a portion of the plurality of unstructured documents corresponding to the identified page clusters; The transformation is performed using a trained machine learning model; as well as The trained machine learning model is added to the plurality of machine learning models.

7. The method according to claim 1, wherein, The plurality of unstructured pages includes a plurality of unstructured page types, and each of the plurality of unstructured page types is assigned to one of the plurality of machine learning models to perform the transformation.

8. An information processing system, comprising: One or more processors; A memory coupled to at least one of the processors; A computer program instruction set, which is stored in the memory and executed by at least one of the processors to perform the following actions: Multiple page clusters are created in the feature space based on multiple feature vectors corresponding to multiple unstructured pages; Based on the relationship between the multiple page clusters in the feature space and the multiple training clusters corresponding to the multiple machine learning models, one machine learning model from the multiple machine learning models is assigned to each of the multiple page clusters. Assigning one machine learning model to each page cluster includes: Calculate multiple page cluster centers based on the aforementioned multiple page clusters; Multiple training cluster centers are calculated based on multiple training clusters corresponding to multiple machine learning models; Select one of the multiple page cluster centers; Identify one of the plurality of training cluster centers in the feature space that is closest to the selected page cluster center; and One of the multiple machine learning models corresponding to the identified training center cluster is assigned to the page cluster corresponding to the selected page cluster center. Identify the page cluster among the plurality of page clusters that corresponds to a selected unstructured page among the plurality of unstructured pages; and The selected unstructured pages are transformed into structured pages using one of the multiple machine learning models assigned to the identified page clusters.

9. The information processing system according to claim 8, wherein the processor performs further actions, including: Divide multiple unstructured documents into multiple unstructured pages; Select one of the plurality of unstructured pages; Define a set of character regions and a corresponding set of positions in the selected unstructured page; as well as Based on the set of positions corresponding to the character regions and the set of contents within the character regions corresponding to the positions in the set of positions, calculate the set of character region feature vectors corresponding to the set of character regions.

10. The information processing system according to claim 9, wherein the processor performs further actions, including: Based on the character region feature vector set, calculate one feature vector selected from the plurality of feature vectors for the selected unstructured page; as well as The selected feature vectors are mapped to the feature space.

11. The information processing system of claim 10, wherein the processor performs further actions, including: Perform hierarchical clustering on the selected feature vectors, wherein the hierarchical clustering further includes: Identify the page cluster center in the feature space that is closest to the selected feature vector among the multiple page cluster centers corresponding to the multiple page clusters; and The selected feature vector is added to one of the identified page clusters among the plurality of page clusters corresponding to the center of the identified page cluster.

12. The information processing system according to claim 8, wherein the processor performs further actions, including: Identify different page clusters among the multiple page clusters corresponding to different pages among the multiple unstructured pages; as well as Using different machine learning models from the plurality of machine learning models assigned to the different page clusters, the different unstructured pages are transformed into different structured pages.

13. The information processing system according to claim 8, wherein the processor performs further actions, including: The selected machine learning model is trained using a portion of the plurality of unstructured documents corresponding to the identified page clusters; The transformation is performed using a trained machine learning model; as well as The trained machine learning model is added to the plurality of machine learning models.

14. The information processing system according to claim 8, wherein, The plurality of unstructured pages includes a plurality of unstructured page types, and each of the plurality of unstructured page types is assigned to one of the plurality of machine learning models to perform the transformation.

15. A computer program product stored in a computer-readable storage medium, comprising computer program code, which, when executed by an information processing system, causes the information processing system to perform actions, the actions including: Multiple page clusters are created in the feature space based on multiple feature vectors corresponding to multiple unstructured pages; Based on the relationship between the multiple page clusters in the feature space and the multiple training clusters corresponding to the multiple machine learning models, one machine learning model from the multiple machine learning models is assigned to each of the multiple page clusters. Assigning one machine learning model to each page cluster includes: Calculate multiple page cluster centers based on the aforementioned multiple page clusters; Multiple training cluster centers are calculated based on multiple training clusters corresponding to multiple machine learning models; Select one of the multiple page cluster centers; Identify one of the plurality of training cluster centers in the feature space that is closest to the selected page cluster center; and One of the multiple machine learning models corresponding to the identified training center cluster is assigned to the page cluster corresponding to the selected page cluster center. Identify the page cluster among the plurality of page clusters that corresponds to a selected unstructured page among the plurality of unstructured pages; and The selected unstructured pages are transformed into structured pages using one of the multiple machine learning models assigned to the identified page clusters.

16. The computer program product of claim 15, wherein the information processing system performs further actions, including: Divide multiple unstructured documents into multiple unstructured pages; Select one of the plurality of unstructured pages; Define a set of character regions and a corresponding set of positions in the selected unstructured page; as well as Based on the set of positions corresponding to the character regions and the set of contents within the character regions corresponding to the positions in the set of positions, calculate the set of character region feature vectors corresponding to the set of character regions.

17. The computer program product of claim 16, wherein the information processing system performs further actions, including: Based on the character region feature vector set, calculate one feature vector selected from the plurality of feature vectors for the selected unstructured page; as well as The selected feature vectors are mapped to the feature space.

18. The computer program product of claim 17, wherein, The information processing system performs further actions, including: Perform hierarchical clustering on the selected feature vectors, wherein the hierarchical clustering further includes: Identify the page cluster center in the feature space that is closest to the selected feature vector among the multiple page cluster centers corresponding to the multiple page clusters; and The selected feature vector is added to one of the identified page clusters among the plurality of page clusters corresponding to the center of the identified page cluster.

19. The computer program product of claim 15, wherein the information processing system performs further actions, including: Identify different page clusters among the multiple page clusters corresponding to different pages among the multiple unstructured pages; as well as Using different machine learning models from the plurality of machine learning models assigned to the different page clusters, the different unstructured pages are transformed into different structured pages.

20. The computer program product of claim 15, wherein the information processing system performs further actions, including: The selected machine learning model is trained using a portion of the plurality of unstructured documents corresponding to the identified page clusters; The transformation is performed using a trained machine learning model; as well as The trained machine learning model is added to the plurality of machine learning models.

21. The computer program product of claim 15, wherein, The plurality of unstructured pages includes a plurality of unstructured page types, and each of the plurality of unstructured page types is assigned to one of the plurality of machine learning models to perform the transformation.

22. A computer-implemented method, comprising: Divide each of the multiple unstructured documents into multiple unstructured pages; Calculate multiple feature vectors for the multiple unstructured pages; Each of the plurality of feature vectors is mapped to a feature space, wherein the feature space further includes a plurality of training cluster centers corresponding to a plurality of training data sets used to train a plurality of machine learning models; Multiple page clusters are created in the feature space based on the multiple feature vectors; Calculate multiple page cluster centers based on the aforementioned multiple page clusters; Determine the training cluster center from among the plurality of training cluster centers that is closest to the selected page cluster center among the plurality of page cluster centers in the feature space; Identify one machine learning model from the plurality of machine learning models that corresponds to the selected training cluster center; Assign the identified machine learning models to the selected page cluster centers; Select one of the multiple feature vectors corresponding to a selected unstructured page among the multiple unstructured pages; Map the selected feature vectors in the feature space; In response to determining that the center of the selected page cluster is closest to the selected feature vector in the feature space, the selected unstructured page is transformed into a structured page using the identified machine learning model.

Images