A Document Processing Framework for Robotic Process Automation

The DPF for RPA addresses vendor lock-in and complexity by allowing flexible integration of multiple vendor products on a unified platform, enabling users to define taxonomies and process documents using a combination of algorithms, thereby enhancing document processing efficiency and scalability.

JP2022549974A5Inactive Publication Date: 2024-11-08UIPATH INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2020552804
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-09-30
Filing Date
2020-08-28
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing document processing tools are limited to single-platform solutions, require separate workflows for multiple vendors, and lack flexibility for integrating custom algorithms, leading to vendor lock-in and increased complexity for users.

Method used

A document processing framework (DPF) for robotic process automation (RPA) that allows plug-and-play use of different vendor products on a unified platform, enabling users to define taxonomies, classify and extract data using a combination of algorithms, and support third-party components for document processing tasks.

Benefits of technology

Facilitates flexible and unified document processing across multiple vendors, reducing vendor lock-in and enabling users to combine, prioritize, and filter processing techniques based on specific needs, while supporting scalability and extensibility for custom solutions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

This document processing framework (DPF) for robotic process automation (RPA) is provided. The DPF allows plug-and-play use of different vendor products on the same platform, and users can configure basic schemas for document processing and document understanding workflows. The DPF may allow users to define taxonomies, digitize files, classify files into one or more document types, verify classifications, extract data, verify extracted data, train classifiers, and / or train extractors. Public packages may be provided for software developers to use to manage the DPF and build their own classifier and extractor components.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Nonprovisional Patent Application No. 16 / 587,386, filed September 30, 2019. The subject matter of this previously filed application is hereby incorporated by reference in its entirety.

[0002] The present invention relates generally to robotic process automation (RPA), and more specifically to a document processing framework (DPF) for RPA. [Background technology]

[0003] Most of the electronic document processing tools on the market process documents on a single platform. Also, most vendors in the document processing space offer black-box solutions that do not allow for interchangeable components with third-party products. This limitation means that customers who want to use multiple vendors must set up two or more parallel integrations, train employees to use two or more different tools, and provide separate user interfaces to perform these tasks. This problem exists in any solution that aims to integrate multiple technologies for document processing and is not necessarily specific to RPA. Certainly, separate workflows are required for each vendor.

[0004] Additionally, users who want to use their own custom document processing algorithms need to build the entire infrastructure themselves to be able to integrate and use them in the end-to-end process, including but not limited to handling digitization using multiple OCR vendors, handling taxonomy mapping between different solutions, integrating and configuring document processing algorithms from different vendors, consolidating results from multiple vendors under a unified data format, etc. Given the above, an improved document processing framework solution could be beneficial. Summary of the Invention

[0005] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, appreciated, or solved by current computer-based document processing technology. For example, some embodiments of the present invention relate to DPFs for RPA.

[0006] In an embodiment, a computer program is embodied on a non-transitory computer readable medium. The program causes at least one processor to execute one or more classifications in an RPA workflow. device and storing results of the automatic classification in a classification data structure. The program is also configured to cause the at least one processor to automatically classify files into one or more document types using the classification data structure. device and storing the automatically extracted data in an extraction data structure. The program is further configured to cause the at least one processor to output the automatically extracted data.

[0007] In another embodiment, the computer-implemented method includes receiving, by a computing system, a list of defined document types and associated fields for each of the defined document types from a taxonomy manager, and storing, by the computing system, the list of document types and associated fields in a master taxonomy data structure. The computer-implemented method also includes, by the computing system, generating, by the computing system, a list of the document types and associated fields for each of the defined document types from a taxonomy manager, and storing, by the computing system, the list of the document types and associated fields in a master taxonomy data structure. device and storing, by the computing system, results of the automatic classification in a classification data structure. The computer-implemented method further includes outputting, by the computing system, results of the automatic classification.

[0008] In yet another embodiment, a system includes a memory storing computer program instructions and at least one processor configured to execute the computer program instructions. The instructions are configured for the at least one processor to receive, from a taxonomy manager, a list of defined document types and fields associated with each of the defined document types. The instructions are also configured for the at least one processor to perform a digitization activity in an RPA workflow and output a text version of the file and a Document Object Model (DOM). The instructions are further configured for the at least one processor to digitize the file to one or more taxonomies in the RPA workflow. device to automatically classify documents into one or more document types and then extract one or more documents within an RPA workflow device The instructions are further configured for the at least one processor to automatically extract data from the classified documents using the classification information. Additionally, the instructions are further configured for the at least one processor to output the automatically extracted data. [Brief description of the drawings]

[0009] So that the advantages of certain embodiments of the invention may be readily understood, a more particular description of the invention briefly described above will now be depicted with reference to specific embodiments which are illustrated in the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting its scope, but the invention will be depicted and explained with further specificity and detail through the use of the following accompanying drawings, in which:

[0010]

Figure 1

[0011]

Figure 2

[0012]

Figure 3

[0013]

Figure 4

[0014]

Figure 5

[0015]

Figure 6A

[0016]

Figure 6B

[0017]

Figure 6C

[0018]

Figure 6D

[0019]

Figure 7A

[0020]

Figure 7B

[0021]

Figure 8A

[0022]

Figure 8B

[0023]

Figure 9A

[0024]

Figure 9B

[0025]

Figure 9C

[0026]

Figure 9D

[0027]

Figure 9E

[0028]

Figure 10

[0029]

Figure 11

[0030]

Figure 12

[0031] Detailed Description of the Embodiments Some embodiments relate to a document processing framework (DPF) for robotic process automation (RPA). In some embodiments, the DPF allows for plug-and-play use of different vendor products on the same platform, where the user may set up a basic schema for document processing and document understanding workflows. Thus, the DPF may allow the user to process documents throughout the flow using algorithms within the application or third-party algorithms for tasks such as, but not limited to, digitization, document classification, and data extraction. This may provide a unified experience and avoid vendor lock-in. With this plug-and-play approach, the DPF of some embodiments supports the combination, substitution, prioritization, and filtering of document processing technologies based on the needs of each use case, thereby providing flexibility in selecting the right solution for each specific problem to be solved.

[0032] In some embodiments, the DPF allows a user to classify by: (1) defining a taxonomy (i.e., a collection of document types); (2) processing and digitizing files to obtain machine-readable text; (3) classifying files into one or more document types by assigning a document type and a page range to each classification result using a combination of available algorithms for document classification; (4) verifying the verification via a tool that allows a human to review / correct / manually perform the file decomposition and classification; (5) extracting data by assigning values ​​identified within the document or from page ranges to requested fields using a combination of available algorithms for data extraction; (6) verifying the extraction via a tool that allows a human to review / correct / manually perform the data extraction; and (7) verifying that the classification algorithm is configured to receive human feedback and trigger a feedback loop if necessary. device and / or (8) ensuring that the extraction algorithm is configured to receive human feedback and trigger a feedback loop as needed. device Make sure to train them.

[0033] The DPF of some embodiments also allows software developers (e.g., third-party company developers or individual contributors) to manage and categorize the DPF. device and Extract device The present invention includes a public package that can be used to build your own document processing components, including but not limited to components. In some embodiments, the public package includes but is not limited to: (1) data structures for taxonomies, documents, and extraction results along with functionality to help manage them; (2) custom taxonomies; device and Extract device(3) abstract classes and interfaces used to build components, and / or (4) data extraction and classification input / output data structures and contracts used in reporting the output of the classification or data extraction. This public package may ensure that the entire DPF is fully scalable, open, and can be used by third parties to develop use-case or solution-specific components in a uniform manner that is fully compatible with the DPF.

[0034] FIG. 1 is an architecture diagram illustrating an RPA system 100, according to an embodiment of the present invention. The RPA system 100 includes a designer 110 that enables developers to design and implement workflows. The designer 110 provides solutions for application integration, as well as automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 110 can facilitate the development of automation projects, which are graphical representations of business processes. In simple terms, the designer 110 facilitates the development and deployment of workflows and robots.

[0035] Automation projects enable rule-based process automation by giving developers control over the execution order and relationships between a custom set of steps developed in a workflow, defined herein as "activities." One commercial example of an embodiment of the Designer 110 is UiPath Studio™. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0036] Workflow types include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suited to linear processes, allowing flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suited to more complex business logic, allowing for the integration of decisions and the connection of activities in more diverse ways via multiple branching logic operators. FSMs may be particularly suited to large workflows. FSMs may use a finite number of states during their execution that are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suited to determining the behavior of a workflow when an execution error is encountered or to debug the process.

[0037] Once a workflow is developed in Designer 110, the execution of the business process is orchestrated by Conductor 120, which coordinates one or more Robots 130 that execute the workflow developed in Designer 110. One commercial example of an embodiment of Conductor 120 is UiPath Orchestrator™. Conductor 120 facilitates the management of the creation, monitoring, and deployment of resources in an environment. Conductor 120 can act as an integration point with third party solutions and applications.

[0038] The conductor 120 may manage all the robots 130, connecting and running them from a centralized point. Types of robots 130 that may be managed include, but are not limited to, attended robots 132, unattended robots 134, development robots (similar to unattended robots 134 but used for development and testing purposes), and non-production robots (similar to attended robots 132 but used for development and testing purposes). Attended robots 132 are triggered by user events and operate side-by-side with humans on the same computing system. Attended robots 132 may be used with the conductor 120 for a centralized process deployment and logging medium. Attended robots 132 may assist human users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be started from the conductor 120 on this type of robot and / or they cannot be run under a locked screen. In certain embodiments, attended robots 132 may only be launched from the robot tray or from a command prompt. The attended robot 132, in some embodiments, preferably operates under human supervision.

[0039] Unattended robots 134 may operate unattended in a virtual environment and automate many processes. Unattended robots 134 may be responsible for providing remote execution, monitoring, scheduling, and work queue support. Debugging for all robot types may be performed in the designer 110 in some embodiments. Both attended and unattended robots may automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those produced by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0040] The conductor 120 may have various capabilities including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between the robots 130 and the conductor 120 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to the robots 130 assigned for execution. Configuration may include maintenance and delivery of robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., an SQL database) and / or another storage mechanism (e.g., ElasticSearch®, which provides the ability to store large data sets and perform queries quickly. The conductor 120 may provide interconnectivity by acting as a centralized point of communication for third-party solutions and / or applications.

[0041] The robot 130 is an execution agent that executes workflows built in the designer 110. One commercial example of some embodiments of the robot(s) 130 is UiPath Robots™. In some embodiments, the robot 130 installs the Microsoft Windows Service Control Manager (SCM) management service by default. As a result, such a robot 130 can open an interactive Windows session under the local system account and may have Windows service rights.

[0042] In some embodiments, the robot 130 may be installed in user mode, meaning that for such a robot 130, it has the same rights as the user to which the given robot 130 is installed. This feature may also be available for high density (HD) robots, ensuring maximum utilization of each machine. In some embodiments, either type of robot 130 may be configured in an HD environment.

[0043] The robot 130 in some embodiments is divided into multiple components, each specialized for a particular automation task. The robot components in some embodiments include, but are not limited to, an SCM managed robot service, a user mode robot service, an executor, an agent, and a command line. The SCM managed robot service manages and monitors the Windows session and acts as a proxy between the conductor 120 and the execution host (i.e., the computing system on which the robot 130 runs). These services trust and manage the credentials of the robot 130. The console application is launched by the SCM under the local system.

[0044] The user-mode Robot Service in some embodiments manages and monitors Windows sessions and acts as a proxy between the conductor 120 and the execution host. The user-mode Robot Service may trust and manage credentials for the Robot 130. If the SCM Management Robot Service is not installed, a Windows application may be launched automatically.

[0045] An Executor may run a given job under a Windows session (i.e., execute a workflow). An Executor may be aware of per-monitor dots-per-inch (DPI) settings. An Agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent may be a client of a Service. An Agent may ask to start or stop a job or change settings. A Command Line is a client of a Service. A Command Line is a console application that can request to start a job and wait for its output.

[0046] As explained above, the separation of the components of the robot 130 helps developers, support users, and computing systems to more easily execute, identify, and track what each component is doing. In this way, special behaviors can be configured for each component, such as setting different firewall rules for executors and services. The executor may always be aware of per-monitor DPI settings in some embodiments. As a result, workflows may run at any DPI regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 110 may be made independent of the browser zoom level. For applications that are not DPI aware or are marked as not DPI aware intentionally, in some embodiments, DPI may be disabled.

[0047] FIG. 2 is an architecture diagram illustrating a deployed RPA system 200 according to an embodiment of the present invention. In some embodiments, the RPA system 200 may be or be part of the RPA system 100 of FIG. 1. It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 210 includes an executor 212, an agent 214, and a designer 216. However, in some embodiments, the designer 216 may not be running on the computing system 210. The executor 212 executes the process. As shown in FIG. 2, multiple business projects may be running simultaneously. The agent 214 (e.g., a Windows service) is a single connection point for all the executors 212 in this embodiment. All messages in this embodiment are logged to the conductor 230, which further processes them via the database server 240, the indexer server 250, or both. As described above with respect to FIG. 1, the executor 212 may be a robotic component.

[0048] In some embodiments, a robot represents an association between a machine name and a username. A robot may manage multiple executors simultaneously. In a computing system that supports multiple interactive sessions running simultaneously (such as Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to as the HD Robot above.

[0049] The agent 214 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading necessary versions of packages to be executed. Communication between the agent 214 and the conductor 230 is, in some embodiments, always initiated by the agent 214. In a notification scenario, the agent 214 may open a WebSocket channel that is later used by the conductor 230 to send commands (e.g., start, stop, etc.) to the robot.

[0050] The server side includes a presentation layer (web application 232, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoints 234, notifications and monitoring 236), a service layer (API implementation / business logic 238), and a persistence layer (database server 240, indexer server 250). Conductor 230 includes web application 232, OData REST API endpoints 234, notifications and monitoring 236, and API implementation / business logic 238. In some embodiments, most actions that a user performs in the conductor 230 interface (e.g., via browser 220) are performed by calling various APIs. Such operations may include, but are not limited to, launching jobs on a robot, adding / removing data in a queue, scheduling jobs to run unattended, etc., without departing from the scope of the invention. Web application 232 is the visual layer of the server platform. In this embodiment, web application 232 uses HyperText Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. A user interacts with web pages from web application 232, in this embodiment via browser 220, to perform various operations to control conductor 230. For example, a user may create robot groups, assign packages to robots, analyze per-robot and / or per-process logs, and start and stop robots.

[0051] In addition to the web application 232, the conductor 230 also includes a services layer that exposes an OData REST API endpoint 234. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both the web application 232 and the agent 214, which in this embodiment is a supervisor of one or more robots on a client computer.

[0052] The REST API of the present embodiment covers configuration, logging, monitoring, and queuing functionality. The configuration endpoint may be used in some embodiments to define and configure users, permissions, robots, assets, releases, and environments of an application. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by the robot, and other environment specific information. The deployment REST endpoint may be used by the robot to query the version of the package that should be executed when a start job command is used in the conductor 230. The queuing REST endpoint may be responsible for managing queues and queue items, such as adding data to the queue, getting transactions from the queue, and setting the status of transactions.

[0053] The monitoring REST endpoints may monitor the web application 232 and the agents 214. The notification and monitoring API 236 may be a REST endpoint used to register the agents 214, deliver configuration settings to the agents 214, and send and receive notifications between the server and the agents 214. The notification and monitoring API 236 may use WebSocket communications in some embodiments.

[0054] The persistence layer, in this embodiment, includes a pair of servers - a database server 240 (e.g., a SQL server) and an indexer server 250. The database server 240 in this embodiment stores configurations for robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed, in some embodiments, via a web application 232. The database server 240 may also manage queues and queue items. In some embodiments, the database server 240 may store messages logged by robots (in addition to or instead of the indexer server 250).

[0055] Optionally in some embodiments, indexer server 250 stores and indexes information logged by the robots. In certain embodiments, indexer server 250 may be disabled via a configuration setting. In some embodiments, indexer server 250 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via logging REST endpoint(s) to indexer server 250, where they are indexed for future use.

[0056] FIG. 3 is an architecture diagram illustrating the relationship 300 between a designer 310, activities 320, 330, and a driver 340, according to an embodiment of the present invention. As can be seen, a developer uses the designer 310 to develop a workflow to be executed by the robot. The workflow may include user-defined activities 320 and UI automation activities 330. In some embodiments, non-textual visual components in an image may be identified, referred to herein as computer vision (CV). Some CV activities related to such components include, but are not limited to, click, type, get text, hover, detect presence or absence of element, update scope, highlight, and the like. In some embodiments, click identifies an element and clicks on it, for example, using CV, optical character recognition (OCR), fuzzy text matching, and multi-anchors. Type may identify an element using the above and type within the element. Get text may locate a particular text and scan it using OCR. Hover may identify an element and hover over it. Detect presence or absence of element may check to detect the presence or absence of an element on the screen using the techniques described above. In some embodiments, there may be hundreds or even thousands of activities that may be implemented in the designer 310. However, any number and / or types of activities may be utilized without departing from the scope of the present invention.

[0057] UI automation activities 330 are a subset of specialized low-level activities that are written in low-level code (e.g., CV activities) and facilitate interactions with the screen. UI automation activities 330 facilitate these interactions via drivers 340 that enable the robot to interact with desired software. For example, drivers 340 may include OS drivers 342, browser drivers 344, VM ​​drivers 346, enterprise application drivers 348, etc.

[0058] Drivers 340 may interact with the OS at a low level, looking for hooks, monitoring keys, etc. They may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity plays the same role in these different applications via drivers 340.

[0059] FIG. 4 is an architecture diagram illustrating an RPA system 400, according to an embodiment of the present invention. In some embodiments, the RPA system 400 may be or may include the RPA systems 100 and / or 200 of FIG. 1 and / or FIG. 2. The RPA system 400 includes multiple client computing systems 410 that execute robots. The computing systems 410 may communicate with a conductor computing system 420 via web applications executed thereon. The conductor computing system 420 may in turn communicate with a database server 430 and an optional indexer server 440.

[0060] 1 and 3, it should be noted that although web applications are used in these embodiments, any suitable client / server software may be used without departing from the scope of the present invention. For example, the conductor may run a server-side application on a client computing system that communicates with a non-web-based client software application.

[0061] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to provide a DPF for an RPA, according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. The computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor(s) 510 coupled to the bus 505 for processing information. The processor(s) 510 may be any type of general or application-specific processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one of the processor(s) 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, neuromorphic circuitry may not require typical components of a von Neumann computing architecture.

[0062] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage such as magnetic or optical disks, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media may be any available media accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. Also, the media may be removable, non-removable, or both.

[0063] Additionally, the computing system 500 includes a communications device 520, such as a transceiver, to provide access to a communications network via wireless and / or wired connections. In some embodiments, the communications device 520 may be configured to support any of the following wireless technologies: Frequency Division Multiple Access (FDMA), Single Carrier FDMA (SC-FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), Orthogonal Frequency Division Multiple Access (OFDM), Orthogonal Frequency Division Multiple Access (OFDMA), Global System for Mobile (GSM) communications, General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), cdma2000, Wideband CDMA (W-CDMA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), High Speed ​​Packet Access (HSPA), Long Term Evolution (LTE), LTE Advanced (LTE-A), LTE-Advanced (LTE-B), LTE-Advanced-B, LTE-Advanced-C ... Advanced), 802.11x, Wi-Fi, Zigbee, Ultra-Wide Band (UWB), 802.16x, 802.15, Home Node-B (HnB), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Near-Field Communications (NFC), 5th Generation (5G), New Radio (NR), any combination thereof, and / or any other currently existing or future implemented communications standards and / or protocols without departing from the scope of the present invention.In some embodiments, the communications device 520 may include one or more antennas that are a singular antenna, an array of antennas, phased antennas, switched antennas, beamforming antennas, beamsteering antennas, combinations thereof, and / or any other antenna configuration without departing from the scope of the invention.

[0064] The processor(s) 510 are further coupled via bus 505 to a display 525, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, a Retina® display, an in-plane switching (IPS) display, or any other suitable display for displaying information to a user. The display 525 may be configured as a touch (haptic) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-touch display, or the like, using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signaling, acoustic pulse recognition, frustrated total internal reflection, or the like. Any suitable display device and haptic I / O may be used without departing from the scope of the invention.

[0065] A keyboard 530 and cursor control device 535, such as a computer mouse, touchpad, and the like, are further coupled to the bus 505 to allow a user to interface with the computing system. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely through the display 525 and / or touchpad (not shown). The type and combination of input devices may be used as a matter of design choice. In certain embodiments, no physical input devices and / or displays are present. For example, a user may interact with computing system 500 remotely via another computing system in communication with computing system 500, or computing system 500 may operate autonomously.

[0066] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules further include a DPF module 545 configured to perform all or a portion of the processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0067] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the functions described above as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. Indeed, the methods, systems, and apparatus disclosed herein may be implemented in localized and distributed forms consistent with computing technologies, including cloud computing systems.

[0068] It should be noted that some of the system features described herein are presented as modules to better emphasize implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.

[0069] The modules may be implemented at least partially in software for execution by various types of processors. For example, the identified units of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, the identified modules that are executable need not be physically located together, but may include separate instructions stored in different locations that, when logically combined, comprise the modules and accomplish the purpose stated for the modules. Furthermore, the modules may be stored, for example, on a hard disk drive, a flash device, RAM, tape, and / or any other such non-transitory computer-readable medium used to store data without departing from the scope of the present invention.

[0070] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within a module, and may be embodied and organized in any suitable form within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network.

[0071] Taxonomy definition and management

[0072] Prior to designing a workflow for document processing, the DPF of some embodiments allows a user to define a list of document types to be subject to classification and data extraction, along with associated fields for each of the document types (i.e., information to be searched for in the document and associated types). This may be accomplished in some embodiments using a taxonomy manager. The defined taxonomy information may be stored as a master taxonomy in a public proprietary data structure defined as a class in a public package so that it can be programmatically used by both software developers and RPA developers to manipulate taxonomy objects. In some embodiments, this class may be, for example, DocumentTaxonomy in UiPath Studio™.

[0073] 6A-6D are screenshots illustrating a taxonomy manager 600, according to an embodiment of the present invention. Taxonomy manager 600 includes a document type definition interface 610 (see also FIG. 6B), a document type details interface 620 (see also FIG. 6C), and an edit fields interface 630 (see also FIG. 6D). Collectively, interfaces 610, 620, 630 allow a user to define a list of document types, provide details of the document types, and edit fields of the document types.

[0074] The document type definition interface 610 allows the user to add new document types or edit or delete existing document types. The interface also provides the ability to select groups and categories and to search for groups and categories. Groups and categories are used in this embodiment to better organize the collection of document types in the taxonomy. Based on these settings in the interface 610, group filtering and category filtering may be available (see also FIG. 6B). Each document type may be assigned to groups and categories via the interface 620 (see also FIG. 6C). While retaining the same objective of capturing document processing metadata, the taxonomy organization (e.g., by groups, categories, document types, fields, etc.) may change over time. Furthermore, the data structures, user interfaces, and contracts that public agencies enter into may also evolve in design, packaging, content, functionality, etc.

[0075] The document type details interface 620 allows the user to provide a name for the document type and to select a group and category for the document type. Optionally, a document type code may be specified. Fields associated with the document type are displayed and the user may add, edit, or delete fields.

[0076] The edit field interface 630 allows a user to edit the properties of a given field. The available properties may vary based on several factors, including, but not limited to, the field type, category, group, etc.

[0077] Digitizing your files

[0078] In some embodiments, digitization of a file may be performed using a Digitize activity 710 in a workflow of an RPA tool 700 (e.g., UiPast Studio™), which outputs a text version of the processed file along with a Document Object Model (DOM). See Figures 7A and 7B. The DOM may be captured in a proprietary data structure of the RPA tool, which may be defined as a class in a public package. This class may be used programmatically by software developers and RPA developers to manipulate the DOM object. The DOM captures information about what the file contains on each page, including typed sections (e.g., paragraphs, tables, columns, vertical text, headers, headings, footers, etc.), typed word groups (sentences, table cells, etc.), and word-level information (e.g., absolute position on each page, OCR confidence, content, etc.), all augmented with rotation, skew, relative width and height information, etc.

[0079] The file digitization component may be built with plug-and-play capabilities related to the OCR engine used, so that it can use any suitable OCR engine implementation that respects the defined contracts enforced by the document processing framework. The digitization component may also be enhanced to implement a voting system that allows the use of multiple OCR engines and reports the best combined result of all engines used. For example, the result on which the most OCR engines agreed upon may be selected. In certain embodiments, such results may be weighted based on the accuracy of a given OCR system. However, it should be noted that in some embodiments, the voting / OCR result integration algorithm may employ any type of metric or confidence calculation, whether based on human feedback or not, to determine and report the best text output available without departing from the scope of the present invention.

[0080] Classify files into one or more document types

[0081] This feature allows the user to select one or more desired classifications for classifying documents. device Examples of classification algorithms include, but are not limited to, layout-based classification, sentiment-based classification, feature-based classification, natural language processing (NLP)-based classification, machine learning (ML)-based classification, deep learning-based classification, image-based classification, keyword-based classification, and color-based classification. The user may, in some embodiments, select any number of classifications. device Implementations can be added, removed, modified, replaced, or categorized as plugins. device In certain embodiments, multiple different classifications of the same type may be used. device (e.g. multiple image-based classification device ) may be used. In some embodiments, the user may device The acceptance criteria may be defined based on a minimum confidence threshold of .

[0082] In some embodiments, a user may create a master taxonomy (e.g., designed and defined using the Taxonomy Manager in some embodiments) and a given taxonomy for the purpose of bringing all data to a common denominator in a unified manner. device This function can perform taxonomy mapping from any internal taxonomy for a given taxonomy. device is reporting an internal set of "classes" (e.g. document types) resulting from a classification that should be mapped (translated) into a master taxonomy. device may report the class "INV" if the incoming document is an invoice, but the master taxonomy may contain a document type called "Expense Invoice". device It is possible to specify that the "INV" reported by corresponds to the "Expense Invoice" defined in the master taxonomy.

[0083] In certain embodiments, the user may select a category for each classification used. device In some embodiments, the user may also determine which classifications to prioritize results from. device The component is allowed to resolve a particular document type and which classification device This allows you to determine what components are restricted from resolving. This functionality allows you to define what each classification used for a particular use case is. device The classification of documents can be fine-tuned to take into account known peculiarities and limitations of implementations (i.e., components) of .

[0084] The document classification function uses classification based on digitized files and taxonomies. device It may also be possible to guarantee the same type of output regardless of the number of documents. The output of the document classification function may be captured in proprietary data structures defined as classes in a public package. The classes may be used programmatically by software developers and RPA developers to manipulate the classification result objects. In some embodiments, these classes may be ClassifierResult and ClassificationResult defined in the Public Data Contract package of UiPath Studio™. In certain embodiments, different classifiers may be combined to automatically determine which result is better. device Implementing a more complex result mediation system between components, and different classifications for different page ranges device It may also include a system for combining results from the components. This functionality can be used when splitting a file into multiple logical documents, for example when a single file contains a package of documents where each document is scanned one after the other into the same PDF file.

[0085] 8A is a screenshot illustrating a classification document coverage workflow 800 according to an embodiment of the present invention. deviceActivity 802 and fixed-form classification device Activity 804. The user may select a classification device Clicking on link 806 will take you to the classification within workflow 800. device The classification document coverage component may, in some embodiments, comprise any desired number of classifications. device It can be constructed using:

[0086] Configuration classification device Clicking on link 806 will take you to the classification in Figure 8B. device A configuration interface 810 opens, which provides a list of document types 820 that can be processed in the first column of the configuration table, and the next columns each list one classification used in the scope in the order in which they are configured in the scope component. device In this example, the keyword-based classification device Configuration pane 830 and fixed format classification device A configuration pane 840 allows the user to set the minimum confidence level 832, 842 for each classification. device The classification results are device The user may also filter the classifications based on a minimum confidence threshold set individually for each device Composition (here, keyword-based classification) device Configuration pane 830 and fixed format classification device Classify into each document type via check boxes in the configuration pane 840) device Users can choose to apply either, both, or none of the following categories: device Optionally, you may enter an internal taxonomy unique ID in the text field next to the checkbox for document types that use the internal taxonomy.

[0087] In some embodiments, the classification document scope activity is device Not just by category, but by classification deviceand document type level. device The configuration, prioritization algorithms, user interface, data structures, and contracts entered into by public agencies may undergo changes and adjustments while maintaining the same objective of facilitating document classification in an open and extensible framework.

[0088] Checking automatic classification

[0089] In some embodiments, the classification review module provides a user interface for reviewing, correcting, and / or manually processing files for classification. A user may be able to review the automatic classification by viewing the page ranges and automatically associated document types of the processed files, perform corrections at both the page range and document type level, remove or add new sections with associated document types, and / or process the documents completely manually if desired.

[0090] The user interface of some embodiments is simple to use and provides a visual approach to understanding and interpreting individual pages within a file. The user interface in some embodiments can be used independently of the classification components used in the previous automated classification phase, providing a single, unified, state-of-the-art user interface for document classification review and processing. This approach eliminates the need for a person to learn and use different screens depending on the classification technology employed for document classification, thereby unifying the overall experience and reducing the learning curve and overall time required for document processing.

[0091] Automatic data extraction from documents

[0092] In some embodiments, the automated data extraction function includes an in-workflow extraction deviceSuch an embodiment allows standard input, standard output, and extraction to be used interchangeably. device There may be an intermediary between them and a fallback mechanism. device Examples of template-based extraction device ,Layout-based extraction device ,Keyword-based extraction device ,Regular Expression Based Extraction device ,Context-based extraction device ,Label / anchor-based extraction device ,Pattern-Based Extraction device ,Natural language processing based extraction device ,Machine learning based extraction device ,Metadata-based extraction device These may include, but are not limited to:

[0093] In some embodiments, the user can plug in as many extractions as desired. device In a particular embodiment, the user may add an extraction device In some embodiments, the user can delete, modify, replace, and reorder each extraction used. device In some embodiments, the user may define acceptance criteria based on a minimum confidence threshold for the field / extraction. device An acceptance criterion may be defined based on a minimum confidence threshold for the combination of

[0094] In some embodiments, a user may create a master taxonomy (e.g., designed and defined using the Taxonomy Manager in some embodiments) and one or more extracts for the purpose of bringing all data to a common denominator in a unified manner. device It can perform taxonomy mapping from an internal taxonomy. This function is implemented by device However, as a result of the data extraction, the “capabilities” (i.e., extractions) are to be mapped (translated) to the master taxonomy. device This can be used to report on an internal set of fields that can be extracted by the user. deviceFor example, the extraction device may report the value of a field called "INV-NO" when processing an invoice, but the master taxonomy may contain a field called "Invoice Number". device that the "INV-NO" reported from corresponds to the "Invoice Number" defined in the Master Taxonomy.

[0095] The user can also select which extraction device A component is allowed to resolve a particular field and which extraction device These functions determine what components are restricted to resolving. device The document extraction can be fine-tuned to take into account known peculiarities and limitations of the implementation (i.e., component) of device may be implemented as an activity in an RPA workflow in some embodiments.

[0096] 9A is a screenshot illustrating a workflow 900 for data extraction coverage according to an embodiment of the present invention. device Activity 902 and fixed format extraction device Activity 904 includes a configuration extractor. device Clicking on link 906 opens the extraction device may be configured.

[0097] Configuration extraction device Clicking on link 906 will take you to the extract of Figure 9B. device A configuration interface 910 opens. Document types and fields from the master taxonomy can be viewed in a document types and fields pane 920 (see also FIG. 9C), where fields (e.g., "Invoice Number") are grouped under their respective document types (e.g., "Invoice"). device Each extraction devicepane (e.g., in this embodiment, FlexiCapture™ Extraction device pane 930 (see also FIG. 9D)) and fixed format extraction device The minimum confidence percentage can also be applied on a field-by-field basis by selecting the corresponding check box in pane 940 (see also FIG. 9E). device Specific panes (e.g. FlexiCapture™ Extraction device Pain 930 and fixed format extraction device The address may be specified via text fields 932, 942 in a pane 940.

[0098] The document extraction function of this embodiment is an extraction method used based on a digitized file and a taxonomy. device For example, FlexiCapture™ is a third-party tool that has its own internal taxonomy. device To bring them to a common denominator, an alias can be added in this embodiment. For example, as shown below, the FlexiCapture™ extract device The Invoice Number field in pane 930 is a fixed format extraction device The output of the document extraction function may be captured in a proprietary data structure defined as a class in a public package that can be used programmatically by both software developers and RPA developers to manipulate the extraction result objects.

[0099] In some embodiments, these classes may be, for example, ExtractorResult and ExtractionResult defined in the UiPath Studio™ Open Data Contracts package. In certain embodiments, different extractions may be combined to automatically determine which result is better. deviceImplementing a more complex result mediation system between components, and different extractions for the same fields device Extracting from a component device The system may include a system for combining or chaining processes. This functionality may be used to identify minutiae data from a larger identification. For example, in some embodiments, the first extraction device may extract the entire sentence, while a chained second extraction device is the first extract device One may target the identification of specific information from the sentences identified by

[0100] In essence, the DPF of some embodiments receives classification results with file and page boundaries, attempts to identify data for each relevant field, uses extraction algorithms contained in the workflow, collects the results, and then performs validation, ordering, and extraction. device Based on the linkage, we report the best result for each field. In the example in Figure 9A-E, two extracts device -FlexiCapture(TM) extraction device 902 and fixed format extraction device 904- is used. The document extraction module of this embodiment uses fixed-format extraction due to the order selected by the RPA developer between these two plugins in the workflow 900. device FlexiCapture™ Extraction device The document extraction module prioritizes the results returned by 902. The document extraction module prioritizes the results returned by FlexiCapture™ extraction if the user sets this value in the Minimum confidence field 932. device 902) is set to 0%, it may not accept any reported extractions that are below 70% for the results reported by 902, whereas it may not accept fixed-form extractions if the minimum confidence field 942 is set to 0%. device The document extraction module may accept any classification reported by 904. deviceIt may look for acceptable extractions performed by 902 and, if acceptable, report the results. device If 902 does not report an acceptable extraction, the document extraction module performs a fixed-form extraction device A search for acceptable extractions may be performed by 904 .

[0101] In some embodiments, the extraction results may be further enhanced with business-specific information or validation, specific formatting, and / or other post-processing. In certain embodiments, the output of the automated data extraction component may be checked / corrected / enhanced using any other RPA workflow activities available in any desired combination. These activities may be used to manipulate the output of the automated data extraction, adding, removing, or modifying certain values ​​included in the results. This may be used, for example, for complex use cases where the extracted data needs to be enriched with information from databases or other sources that are external to the single file processed at the time.

[0102] Verifying Automatic Data Extraction

[0103] In some embodiments, the data extraction review module provides a user interface for reviewing, correcting, and / or manually processing the data points (i.e., extracted information) reported by the data extraction module. A user may review the results from the automated data extraction in the following ways: (1) viewing the values, evidence, the location in the document where the extracted values ​​were found, the reliability of the extracted values, etc. from the processed portion of the document (e.g., a specific page range for classification); (2) performing corrections by editing OCR errors of correctly reported values, correcting the location of a reported value, replacing a reported value entirely, adding values ​​missed by the automated process, and / or removing values ​​incorrectly identified by the automated process; (3) interacting directly with a graphical representation of the original file being processed by selecting ranges, regions, word(s), etc. and performing operations directly on the original file view; (4) viewing and performing operations on a text version of the document; and / or (5) processing the document completely manually, if desired.

[0104] The user interface of some embodiments is easy to use and provides a visual approach to understand and make decisions about individual fields and pieces of data identified and extracted in the file. The user interface in certain embodiments can be made to be usable independently of the data extraction component used in the previous automatic classification and data extraction phase, providing a single unified state-of-the-art user interface for document data extraction review and processing. This approach eliminates the need for a person to learn and use different screens depending on the data validation technology employed for data validation, thereby unifying the overall experience and reducing the learning curve and overall time for document processing. The data validation component of some embodiments allows for the definition and use of custom validation rules that can be checked before allowing a person to submit a processed document via the document validation function. In this way, the module can ensure full extensibility and be open to custom implementations depending on the business use cases to be covered.

[0105] classification device and Extract device Training

[0106] In some embodiments, the classification device The training module is a classification that can learn from human feedback. device Facilitates the completion of the feedback loop for classification. device The training module accumulates information for each classification that allows the algorithm to learn from human-reviewed data and human feedback. device (e.g., by using the taxonomy mapping feature) in an understandable format and with the correct permissions enabled (e.g., training taxonomy device Extracts that were launched (by using the field-level launch feature) device This approach may ensure that the classification used is consistent with the deviceIt can provide a unified experience to close the feedback loop independent of the classification(s) used for the actual file classification. device Component independence, which allows RPA developers to, for example, pre-determine the classification process before it is launched for the actual file classification. device , allowing for "on-the-fly" training and eliminating the need for large amounts of up-front annotation files specific to each classification component.

[0107] In some embodiments, the extraction device The training module is a classification that can learn from human feedback. device Facilitates the completion of the feedback loop for extraction device The training module extracts information that enables the algorithm to learn from human-reviewed data and human feedback. device (e.g., by using taxonomy mapping functionality) in a format that is understandable and with the correct permissions enabled (e.g., training extraction device Extracts that were launched (by using the field-level launch feature) device This approach ensures that all extractions are reached in a uniform way. device It may provide a unified experience to close the feedback loop, independent of which component(s) are used, and may also provide a unified experience to close the feedback loop that is used for the actual file processing for data extraction. device No need to depend on components. This allows RPA developers to use, for example, the Extract device Extract before launching device , which allows training to be done "on the fly" and eliminates the need for large amounts of up-front annotation files specific to each extraction component.

[0108] FIG. 10 illustrates a classification system according to an embodiment of the present invention. device and Extract device1 is a screenshot showing a training workflow 1000 for training classifiers. device Scope Activity 1010 is a configuration classification device Configurable keyword-based classification by clicking link 1014 device Including 1012. device The functionality of the training configuration interface is similar to that of the classification system of FIG. 8B, except that in some embodiments, there is no need to set a minimum confidence threshold. device This may be similar to the configuration interface 810. device A training configuration interface 1100 is shown in FIG. 11. However, in certain embodiments, device The training configuration interface is based on document types and classifications. device It may further include training set parameters, which may be unique. device The function of the training configuration interface is the extraction of Figure 9B. device It may be similar to the configuration interface 910 and, in certain embodiments, may include additional training configuration parameters.

[0109] 12 is a flow chart illustrating a process 1200 for implementing a DPF for RPA, according to an embodiment of the invention. The process is to define and store a general taxonomy at 1210. In some embodiments, this may include executing a taxonomy manager that provides an interface to facilitate definition of a list of document types to be classified and subject to data extraction, receiving the defined list of document types and associated fields for each of the defined document types along with associated fields for each of the document types, and storing the list of document types and associated fields in a master taxonomy data structure.

[0110] The RPA workflow then executes a digitization activity, which outputs 1220 a text version of the file and a Document Object Model (DOM) stored in a DOM data structure. In some embodiments, the DOM includes information related to typed sections, typed word groups, and word level information within the file, augmented with rotation, skew, and relative width and height information. In certain embodiments, the digitization activity uses multiple OCR engines, and the digitization activity includes implementing a voting system for the multiple OCR engines and outputting a best combination result from the multiple OCR engines.

[0111] Organize files into one or more categories for RPA workflows device , and outputting 1230 the automatic classification information. In some embodiments, this information may be stored in a classification data structure. In some embodiments, the one or more classifications may be used to classify the document into one or more document types. device is configured to perform layout-based classification, sentiment-based classification, feature-based classification, natural language processing (NLP)-based classification, machine learning (ML)-based classification, deep learning-based classification, image-based classification, keyword-based classification, color-based classification, any combination thereof, or any other document classification methodology. In certain embodiments, the classification is performed using one or more classifications. device In some embodiments, the classification includes using an acceptance criterion based on a minimum confidence threshold for each of the one or more classifications. device For each classification device In certain embodiments, the classification includes mapping the master taxonomy and the internal taxonomy of the RPA workflow. device Based on the order of each classification device One or more taxonomies for use in classification based on document type to prioritize results from device Classification of device Select one or more categoriesdevice assigning a minimum confidence field to at least one of the first, second, third, fourth, or any combination thereof.

[0112] A classification verification module is executed that provides an interface for reviewing, correcting, and / or manually processing the file for classification, and the automatic classification information is output at 1240. device Classification that facilitates the completion of the feedback loop device A training module is executed at 1250, and one or more extractions are performed in the RPA workflow. device The automated extraction process extracts data from the classified documents using the extracted data structure and outputs the automatically extracted data at 1260. This extracted data may be stored in an extraction data structure in some embodiments. In some embodiments, the extraction process may be performed using an extraction data structure in an RPA workflow. device Extract each one based on the order of device Prioritize results from one or more extracts for use in extraction based on the type of document device Extraction of device Select one or more extracts device assigning a minimum confidence field to at least one of the first, second, third, fourth, or any combination thereof.

[0113] A data extraction validation module is executed that provides an interface for correcting and / or manually processing data points from the data extraction, and the validated extracted data is output at 1270. device Completing the feedback loop of extraction device A training module is executed at 1280. Then, processed data including classification results and extracted information is written out at 1290. It should be noted that once the taxonomy is defined, based on the needs of the use case, in some embodiments, one or more of steps 1220-1290 may be optional.

[0114] After process 1200 is complete, the verified information can be made available for use in other systems. For example, the information may be inserted into a database or Excel file, a notification of a particular value may be sent, such as by email, etc. As a non-limiting example, process 1200 of FIG. 12 may start with a PDF file and end with knowledge that the file is an invoice, the total amount on the invoice is $42, the vendor is "XYZ Bookstore," and the purchase date is July 23, 2019. These values ​​can now be programmatically inserted into, for example, an accounting system.

[0115] The process steps performed in FIG. 12 may be performed by a computer program encoding instructions to a processor(s) to perform at least a portion of the process described in FIG. 12 according to an embodiment of the present invention. The computer program may be embodied in a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard disk drive, a flash device, a RAM, a tape, and / or any other such medium or combination of media used to store data. The computer program may include coded instructions for controlling a processor(s) of a computing system (e.g., processor(s) 510 of computing system 500 of FIG. 5) to implement all or a portion of the process steps described in FIG. 12, which may also be stored in a computer readable medium.

[0116] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules in operable communication with each other and designed to send information or instructions to a display. The computer program may be configured to run on a general purpose computer, an ASIC, or any other suitable device.

[0117] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying figures, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0118] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, reference throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearance of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification does not necessarily refer to the same group of all embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0119] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or in any embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussions of features and advantages throughout this specification, as well as similar language, may, but do not necessarily, refer to the same embodiment.

[0120] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. A person skilled in the relevant art will recognize that the invention may be practiced without certain features or advantages of one or more specific embodiments. In other embodiments, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the invention.

[0121] Those of ordinary skill in the art will readily appreciate that the invention as described above can be practiced using steps in a different order and / or with hardware elements in different configurations than those disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will become apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. A computer program embodied on a non-transitory computer readable medium, the computer program comprising: In a robotic process automation (RPA) workflow, automatically classifying files into one or more document types using one or more classifiers; storing the results of said automatic classification in a classification data structure; automatically extracting data from the classified files using one or more extractors within the RPA workflow; storing the automatically extracted data in an extraction data structure; and outputting the automatically extracted data.

2. 2. The computer program product of claim 1, wherein the one or more classifiers are configured to perform layout-based classification, emotion-based classification, feature-based classification, natural language processing (NLP)-based classification, machine learning (ML)-based classification, deep learning-based classification, image-based classification, keyword-based classification, color-based classification, or any combination thereof.

3. The computer program product of claim 1 , wherein the automatically classifying comprises using an acceptance criterion based on a minimum confidence threshold for each of the one or more classifiers.

4. The computer program product of claim 1 , wherein the automatically classifying comprises mapping a master taxonomy and an internal taxonomy of each of the one or more classifiers.

5. The computer program further comprises:

2. The computer program product of claim 1 , configured to prioritize results from each classifier based on an order of classifiers in the RPA workflow, select the one or more classifiers for use in the automatic classification based on the document type, assign a minimum confidence threshold to at least one of the one or more classifiers, or any combination thereof.

6. The computer program further comprises: The computer program product of claim 1 , configured to execute a classification verification module that provides an interface for reviewing, correcting, and / or manually processing the files for automatic classification.

7. 2. The computer program product of claim 1 , wherein the one or more extractors are configured to perform template-based extraction, layout-based extraction, keyword-based extraction, regular expression-based extraction, context-based extraction, label / anchor-based extraction, pattern-based extraction, natural language processing-based extraction, machine learning extraction, deep learning extraction, metadata-based extraction, or any combination thereof.

8. The computer program product of claim 1 , wherein automatically extracting the data comprises using an acceptance criterion based on a minimum confidence threshold for each of the one or more extractors.

9. The computer program product of claim 1 , wherein automatically extracting the data comprises mapping a master taxonomy and an internal taxonomy of each of the one or more extractors.

10. The computer program further comprises:

2. The computer program product of claim 1 , configured to prioritize results from each extractor based on an order of extractors in the RPA workflow, select an extractor of the one or more extractors for use in automatically extracting the data based on the document type, assign a minimum confidence threshold to at least one of the one or more extractors, or any combination thereof.

11. The computer program further comprises: The computer program product of claim 1 , configured to execute a data extraction validation module that provides an interface for correcting and / or manually processing data points from the automatically extracting the data.

12. The computer program further comprises: The computer program product of claim 1 , configured to execute an extractor training module that facilitates completion of a feedback loop of the one or more extractors.

13. The computer program product further comprises: The computer program product of claim 1 , configured to execute a classifier training module that facilitates completion of a feedback loop for the one or more classifiers.

14. The computer program further comprises: executing a taxonomy manager that provides an interface that facilitates definition of a list of document types that are subject to said automatic classification and said automatic data extraction, together with associated fields for each said document type; receiving, via the taxonomy manager, a list of the defined document types and fields associated with each of the defined document types; The computer program product of claim 1 , configured to store the list of defined document types and the associated fields in a master taxonomy data structure.

15. The computer program further comprises: Execute digitalization activities with Robotic Process Automation (RPA) workflows, 2. The computer program product of claim 1 configured to output a text version of a file stored in a Document Object Model (DOM) data structure and a DOM.

16. 16. The computer program product of claim 15, wherein the DOM includes information related to typed sections, typed word groups, and word level information within the file augmented with rotation, skew, relative width and height information, or any combination thereof.

17. the digitization activity employs multiple optical character recognition (OCR) engines; and implementing a voting system for said plurality of OCR engines; The computer program product of claim 15 , further comprising outputting an optimal combination result from the plurality of OCR engines.

18. The computer program further comprises: The computer program product of claim 1 , configured to export the automatically classified results and the automatically extracted data for use in other systems.

19. Executing a taxonomy manager that provides an interface that facilitates the definition of a list of document types to be automatically classified and from which data is to be automatically extracted, together with associated fields for each said document type; receiving, by a computing system, from the taxonomy manager, a list of the defined document types and fields associated with each of the defined document types; storing, by said computing system, the list of defined document types and the associated fields in a master taxonomy data structure; automatically classifying, by the computing system, files into one or more document types using one or more classifiers in a robotic process automation (RPA) workflow; storing, by the computing system, the results of the automatic classification in a classification data structure; A computer-implemented method including outputting, by the computing system, results of the automatic classification.

20. executing, by the computing system, a digitization activity within the RPA workflow; outputting, by the computing system, a text version of the file stored in a Document Object Model (DOM) data structure and the DOM; 20. The computer-implemented method of claim 19, wherein the DOM includes information related to typed sections, typed word groups, and word level information within the file augmented with rotation, skew, relative width and height information, or any combination thereof.

21. 20. The computer-implemented method of claim 19, further comprising executing, by the computing system, a classifier training module that facilitates completing a feedback loop for the one or more classifiers.

22. 20. The computer-implemented method of claim 19, further comprising executing, by the computing system, a classification verification module that provides an interface for reviewing, correcting, and / or manually processing the files for automatic classification.

23. automatically extracting data from the classified documents using one or more extractors in the RPA workflow; 20. The computer-implemented method of claim 19, further comprising storing, by the computing system, the automatically extracted data in an extraction data structure.

24. 24. The computer-implemented method of claim 23, further comprising executing, by the computing system, an extractor training module that facilitates completion of a feedback loop of the one or more extractors.

25. 24. The computer-implemented method of claim 23, further comprising executing, by the computing system, a data extraction validation module that provides an interface for correcting and / or manually processing data points from the automatically extracting data.

26. a memory for storing computer program instructions; and at least one processor configured to execute the computer program instructions, the computer program instructions including: Executing a taxonomy manager that provides an interface that facilitates the definition of a list of document types to be automatically classified and from which data is to be automatically extracted, together with associated fields for each said document type; receiving from the taxonomy manager a list of the defined document types and fields associated with each of the defined document types; Executing digitization activities in a Robotic Process Automation (RPA) workflow, outputting a text version of the file and a Document Object Model (DOM), automatically classifying the files into one or more document types using one or more classifiers in the RPA workflow; automatically extracting data from the classified documents using one or more extractors within the RPA workflow; and outputting the automatically extracted data.

27. The automatically classifying further includes prioritizing results from each classifier based on an order of classifiers in the RPA workflow, selecting a classifier of the one or more classifiers for use in the automatically classifying based on the document type, assigning a minimum confidence threshold to at least one of the one or more classifiers, or any combination thereof; 27. The system of claim 26, wherein automatically extracting the data further comprises prioritizing results from each extractor based on an order of extractors in the RPA workflow, selecting an extractor of the one or more extractors for use in automatically extracting the data based on the document type, assigning a minimum confidence threshold to at least one of the one or more extractors, or any combination thereof.

28. The computer program instructions further include causing the at least one processor to:

27. The system of claim 26, configured to execute an extractor training module that facilitates completion of a feedback loop of the one or more extractors.

29. The computer program instructions further include causing the at least one processor to:

27. The system of claim 26, configured to execute a classification verification module that provides an interface for reviewing, correcting, and / or manually processing the files for automatic classification.

30. The computer program instructions further include causing the at least one processor to:

27. The system of claim 26, configured to execute an extractor training module that facilitates completion of a feedback loop of the one or more extractors.

31. The computer program instructions further include causing the at least one processor to:

27. The system of claim 26, configured to execute a data extraction validation module that provides an interface for correcting and / or manually processing data points from automatically extracting the data.