Using generative artificial intelligence to supplement automated information extraction

Abstract

To provide systems, methods and media for using generative AI to supplement automated information extraction.SOLUTION: In a hyper automation system 100, computer vision (CV) and / or optical character recognition (OCR) models and a generative artificial intelligence (AI) model 132 are used together to extract information (e.g., names, dates, invoice numbers, etc.) from a source. Acceptance threshold(s) are used to receive predictions for extracted data elements from the models, and the prediction from the generative AI model is preferred or a human is tasked with reviewing the element. If no models meet their respective acceptance thresholds, these element(s) are marked for subsequent human review, or a human can mediate to correct these elements. The models may then be retrained using these labeled elements.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates generally to artificial intelligence (AI), and more specifically to the use of generative AI to supplement automated information extraction. [Background technology]

[0002] Computer vision (CV) models and / or optical character recognition (OCR) models may be used to identify graphical elements and / or text present in a source (e.g., a document, a web page, a form, a scanned invoice, etc.). Such models typically identify candidate graphical elements and / or text, as well as provide a confidence score for the candidates. However, these techniques may not always accurately identify graphical elements in a source. Thus, improved and / or alternative approaches may be beneficial. Summary of the Invention

[0003] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, appreciated, or solved by current AI techniques. For example, some embodiments of the present invention relate to using generative AI to supplement automated information extraction.

[0004] In an embodiment, one or more non-transitory computer-readable media store a computer program for using generative AI to supplement automated information extraction. The computer program is configured for at least one processor to provide source information as input to the CV model, the OCR model, or both, and the generative AI model, process the source information, and extract data elements therein. The computer program is also configured for at least one processor to receive output from the generative AI model, the CV model, and / or the OCR model. The output includes a plurality of extracted data element predictions for the extracted data elements and respective confidence scores. The computer program is further configured for at least one processor to analyze the plurality of extracted data element predictions and the respective confidence scores from the generative AI model, and the CV model and / or the OCR model to determine whether the model matches the plurality of extracted data element predictions. Further, the computer program is configured for the at least one processor to output the extracted element prediction and / or the extracted data element in response to the generative AI model and the CV and / or OCR model matching an extracted data element prediction of the plurality of extracted data element predictions and a confidence score of the respective extracted data element prediction that exceeds a respective tolerance threshold of the model.

[0005] In another embodiment, one or more computing systems include a memory storing computer program instructions for using generative AI to supplement automated information extraction, and at least one processor configured to execute the computer program instructions. The computer program instructions are configured for the at least one processor to receive output from the generative AI model and the CV model and / or the OCR model. The output includes a plurality of extracted data element predictions and respective confidence scores for the data elements extracted from the source information. The computer program instructions are also configured for the at least one processor to analyze the plurality of extracted data element predictions and the respective confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the model matches the plurality of extracted data element predictions. The computer program instructions are further configured for the at least one processor to request human assistance to identify the respective correct data elements in response to the generative AI model and the CV model and / or the OCR model not matching an extracted data element prediction of the plurality of extracted data element predictions or the generative AI model and the CV and / or the OCR model not meeting or exceeding their respective tolerance thresholds for the extracted data element predictions.

[0006] In yet another embodiment, a computer-implemented method of using generative AI to supplement automated information extraction includes providing source information as input to a CV model, an OCR model, or both, and the generative AI model to process the source information and extract data elements therein by a robotic process automation (RPA) robot executed by at least one processor of a computing system. The computer-implemented method also includes receiving, by the RPA robot, output from the generative AI model and the CV model and / or the OCR model. The output includes a plurality of extracted data element predictions and respective confidence scores for the extracted data elements. The computer-implemented method further includes analyzing, by the RPA robot, the plurality of extracted data element predictions and the respective confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the models match the plurality of extracted data element predictions. The computer-implemented method further includes, in response to the generative AI model and the CV and / or OCR model matching a confidence score of the extracted data element prediction of the plurality of extracted data element predictions that exceeds a respective tolerance threshold of the extracted data element prediction and the model, outputting, by the RPA robot, the extracted element prediction and / or the extracted data element. The computer-implemented method also includes, in response to the generative AI model and the CV and / or OCR model not matching an extracted data element prediction of the plurality of extracted data element predictions or the generative AI model and the CV and / or OCR model not meeting or exceeding their respective tolerance thresholds of the extracted data element prediction, requesting human assistance by the RPA robot to identify the correct respective data element. [Brief description of the drawings]

[0007] So that the advantages of certain embodiments of the invention may be readily understood, a more particular description of the invention briefly described above will now be depicted with reference to specific embodiments which are illustrated in the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not to be considered as limiting its scope, but the invention will be depicted and explained with further specificity and detail through the use of the following accompanying drawings, in which:

[0008] [Figure 1] FIG. 1 is an architectural diagram illustrating a hyper-automation system, in accordance with an embodiment of the present invention.

[0009] [Diagram 2] FIG. 1 is an architectural diagram illustrating an RPA system according to an embodiment of the present invention.

[0010] [Diagram 3] FIG. 1 is an architectural diagram illustrating a deployed RPA system according to an embodiment of the present invention.

[0011] [Figure 4] FIG. 2 is an architecture diagram illustrating the relationships between designers, activities, and drivers according to an embodiment of the present invention.

[0012] [Diagram 5] FIG. 1 is an architectural diagram illustrating a computing system configured to use generative AI to supplement automated information extraction, according to an embodiment of the invention.

[0013] [Figure 6A] 1 illustrates an example of a neural network trained to perform automated information extraction, according to an embodiment of the present invention.

[0014] [Figure 6B] 1 illustrates an example of a neuron according to an embodiment of the present invention.

[0015] [Figure 7] 1 is a flowchart illustrating a process for training an AI / ML model(s) in accordance with an embodiment of the present invention.

[0016] [Figure 8] 1 is a flowchart illustrating a process of using generative AI to supplement automated information extraction, according to an embodiment of the present invention.

[0017] [Figure 9] 1 is a flowchart illustrating another process of using generative AI to supplement automated information extraction, according to an embodiment of the present invention.

[0018] Unless otherwise noted, like reference characters denote corresponding features consistently throughout the attached drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Detailed Description of the Embodiments Some embodiments relate to using generative AI to supplement automated information extraction. CV and / or OCR models and generative AI models are used together to extract information (e.g., names, dates, invoice numbers, etc.) from sources. In some embodiments, the OCR model may be part of the CV model.

[0020] If the model's confidence score is above a certain common tolerance threshold, or may be above an individual tolerance threshold for each model type (e.g., 95% for generative AI, 99% for OCR, etc.), and both the CV or OCR model and the generative AI model agree with the prediction, no further validation takes place and the element extracted from the source may be accepted for further processing (e.g., understanding the document, performing semantic matching, etc.). If the predictions for an element of the CV and / or OCR model(s) and the generative AI model differ and the generative AI model is above the common threshold or its respective tolerance threshold, the element predicted and extracted by the generative AI model may be used. In some embodiments, the element extracted by the generative AI model may be used to retrain the CV and / or OCR model, and the generative AI model was above the common tolerance threshold or its respective tolerance threshold, but the CV and / or OCR model(s) was not. Alternatively, the disputed element may be referred to a human for review. If elements in the source are not identified by any of the models above the threshold, these element(s) are marked for subsequent human review or a human can relay these element(s) to be corrected. These labeled element(s) may then be used to retrain the model.

[0021] FIG. 1 is an architecture diagram illustrating a hyperautomation system 100, according to an embodiment of the present invention. As used herein, "hyperautomation" refers to an automation system that brings together process automation components, integration tools, and technologies that amplify the ability to automate work. For example, in some embodiments, robotic process automation (RPA) is used at the core of the hyperautomation system, and in certain embodiments, the automation capabilities may be extended by AI / machine learning (ML), process mining, analytics, and / or other advanced tools. As the hyperautomation system learns processes, trains AI / ML models, and employs analytics, for example, more knowledge work may be automated, and computing systems within an organization, for example, both those used by individuals and those operating autonomously, may all be engaged as participants in the hyperautomation process. The hyperautomation system of some embodiments enables users and organizations to discover, understand, and extend automation efficiently and effectively.

[0022] The hyperautomation system 100 includes user computing systems such as a desktop computer 102, a tablet 104, and a smartphone 106. However, any desired user computing system may be used without departing from the scope of the present invention, including, but not limited to, a smart watch, a laptop computer, a server, an Internet of Things (IoT) device, and the like. Also, although three user computing systems are shown in FIG. 1, any suitable number of user computing systems may be used without departing from the scope of the present invention. For example, in some embodiments, tens, hundreds, thousands, or millions of user computing systems may be used. The user computing systems may be actively used by a user or may run automatically without much or any user input.

[0023] Each user computing system 102, 104, 106 has a respective automation process(es) 110, 112, 114 running thereon. In some embodiments, the automation processes are stored remotely (e.g., on a server 130 or on a database 140) (accessed via the network 120) and loaded by the RPA robot to implement the automation. The automation may exist as a script (e.g., Extensible Markup Language (XML), Extensible Application Markup Language (XAML), etc.) or may be compiled into machine-readable code (e.g., as a digital link library).

[0024] The automation process(es) 110, 112, 114 may include, but are not limited to, an RPA robot, a part of an operating system, a downloadable application(s) for the respective computing system, any other suitable software and / or hardware, or any combination thereof, without departing from the scope of the present invention. In some embodiments, one or more of the process(es) 110, 112, 114 may be a listener. The listener may be an RPA robot, a part of an operating system, a downloadable application for the respective computing system, or any other software and / or hardware, without departing from the scope of the present invention. Indeed, in some embodiments, the logic of the listener(s) is implemented partially or completely via physical hardware.

[0025] The listeners monitor and record data related to user interactions with their respective computing systems and / or the operation of unattended computing systems and transmit the data over a network (e.g., a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, any combination thereof, etc.) to the core hyperautomation system 120. The data may include, but is not limited to, which button was clicked, where the mouse was moved, text entered into a field, one window was minimized and another opened, the application associated with the window, etc. In certain embodiments, the data from the listeners may be transmitted periodically as part of a heartbeat message. In some embodiments, the data may be transmitted to the core hyperautomation system 120 when a predetermined amount of data has been collected, after a predetermined period of time has passed, or both. One or more servers, such as server 130, receive the data from the listeners and store it in a database, such as database 140.

[0026] An automation process may execute logic developed in a workflow during design time. In the case of RPA, a workflow may include a set of steps, defined herein as "activities," that are executed in sequence or some other logic flow. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0027] Long-running workflows for RPA in some embodiments are master projects that support service orchestration, human intervention, and long-running transactions in unattended environments. See, for example, U.S. Patent No. 10,860,905, which is incorporated by reference in its entirety. Human intervention comes into play when a particular process requires human input for exception handling, approval, or validation before proceeding to the next step of the activity. In this case, the process execution is paused, freeing up the RPA robot until the human task is completed.

[0028] Long-running workflows may support workflow fragmentation through persistence activities and may be combined with calling process and non-user interaction activities to orchestrate human tasks with RPA robot tasks. In some embodiments, multiple or multiple computing systems may participate in the execution of the logic of a long-running workflow. Long-running workflows may execute in sessions to facilitate rapid execution. In some embodiments, long-running workflows may orchestrate background processes that execute application programming interface (API) calls and may include activities that execute in the long-running workflow session. These activities may be invoked in some embodiments by a calling process activity. Processes with user interaction activities that execute in a user session may be invoked by starting a job from a conductor activity (conductors are described in more detail later in this specification). Users may interact through tasks that require completing a form in the conductor in some embodiments. An activity may be included that causes the RPA robot to wait for a form task to complete and then resume the long-running workflow.

[0029] One or more automation process(es) 110, 112, 114 are in communication with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 may execute a conductor application on one or more servers, such as server 130. While one server 130 is shown for illustrative purposes, multiple or numerous servers in close proximity to one another or in a distributed architecture may be employed without departing from the scope of the present invention. For example, one or more servers may be provided for conductor functions, AI / ML model serving, certification, governance, and / or any other suitable functions without departing from the scope of the present invention. In some embodiments, the core hyper-automation system 120 may incorporate or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, and / or the like. In certain embodiments, the core hyper-automation system 120 may host multiple software-based servers on one or more computing systems, such as server 130. In some embodiments, one or more servers of the core hyper-automation system 120, such as server 130, may be implemented via one or more virtual machines (VMs).

[0030] In some embodiments, one or more automation process(es) 110, 112, 114 may invoke one or more AI / ML models 132 deployed on or accessible by the core hyper-automation system 120 and trained to accomplish various tasks. For example, the AI / ML models 132 may include models trained to look for different application versions, perform CV, perform OCR, generate user interface (UI) descriptors, and provide suggestions for the next activity or sequence of activities in an RPA workflow. The AI / ML models may be trained using labeled data including elements of data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created RPA workflows, screenshots of different application screens in different versions with corresponding UI elements, libraries of UI objects, etc. The AI / ML models 132 may be trained to achieve a desired confidence threshold without overfitting to a given set of training data.

[0031] The AI / ML model 132 may be trained for any suitable purpose without departing from the scope of the present invention, as discussed in more detail later herein. Two or more AI / ML models 132 may be chained (e.g., serially, in parallel, or a combination thereof) in some embodiments such that they collectively provide a collaborative output(s). The AI / ML model 132 may perform or assist in CV, OCR, document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automated RPA workflow generation, sequence extraction, clustering detection, speech-to-text translation, any combination thereof, and the like. However, any desired number and / or type(s) of AI / ML models may be used without departing from the scope of the present invention. Using multiple AI / ML models may, for example, allow a system to develop a holistic picture of what is going on on a given computing system. For example, one AI / ML model may perform OCR, another may detect buttons, another may compare sequences, and the like. The patterns may be determined by the AI / ML models individually or collectively by multiple AI / ML models. In particular embodiments, one or more AI / ML models are deployed locally on at least one of the computing systems 102, 104, 106.

[0032] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) that runs on data, and the AI / ML model itself may be, for example, a deep learning neural network (DLNN) of artificial "neurons" trained on training data. In some embodiments, the AI / ML model 132 may have multiple layers that perform various functions, such as statistical modeling (e.g., Hidden Markov Models (HMMs)), and may utilize deep learning techniques (e.g., long short-term memory (LSTM) deep learning, encoding of prior hidden states, etc.) to perform desired functions.

[0033] The hyper-automation system 100, in some embodiments, may provide four main groups of functions: (1) discovery, (2) build automation, (3) management, and (4) engagement. Automation (e.g., running on a user computing system, server, etc.) may be performed by software robots, such as RPA robots, in some embodiments. For example, attended robots, unattended robots, and / or test robots may be used. Attended robots collaborate with users to assist them with tasks (e.g., via UiPath Assistant™). Unattended robots operate independently of the user and may potentially run in the background without the user's knowledge. Test robots are unattended robots that run test cases against an application or RPA workflow. Test robots, in some embodiments, may run in parallel on multiple computing systems.

[0034] A discovery function may discover and provide automated recommendations for various business process automation opportunities. Such functions may be implemented by one or more servers, such as server 130. The discovery function may include providing an automation hub, process mining, task mining, and / or task capture, in some embodiments. An automation hub (e.g., UiPath Automation Hub™) may provide a mechanism to manage automation rollouts with visibility and control. Automation ideas may be crowdsourced from employees, for example, via a submission form. Calculations of feasibility and return on investment (ROI) for automating these ideas may be provided, documentation for future automations may be collected, and collaboration may be provided to expedite automation discovery-to-build.

[0035] Process mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) refers to the process of collecting and analyzing data from applications (such as enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify what end-to-end processes exist in an organization, how they can be effectively automated, and the impact that automation will have. This data may be obtained, for example, by listeners from user computing systems 102, 104, 106 and processed by a server, such as server 130. In some embodiments, one or more AI / ML models 132 may be employed for this purpose. This information may be exported to an automation hub to speed implementation and avoid manual information transfer. The goal of process mining may be to increase business value by automating processes in an organization. Some examples of process mining goals include, but are not limited to, increased profits, increased customer satisfaction, regulatory and / or contractual compliance, increased employee efficiency, etc.

[0036] Task mining (e.g., via UiPath Automation Cloud™ and / or UiPath AI Center™) identifies and aggregates workflows (e.g., employee workflows) and then applies AI to uncover patterns and variations in routine tasks and score such tasks for ease of automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 132 may be employed to uncover repetitive task patterns in the data. Repetitive tasks ripe for automation may then be identified. This information may be initially provided by the listener and, in some embodiments, analyzed on a server of the core hyper-automation system 120, such as server 130. Findings from task mining (e.g., XAML process data) may be exported to process documentation or designer applications such as UiPath Studio™ to more quickly create and deploy automations. Task mining in some embodiments may include taking screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements with which the user was interacting, timestamps for the interactions, etc.), collecting statistical data (e.g., performance time, number of actions, text input, etc.), editing and annotating the screenshots, specifying the types of actions to be recorded, etc.

[0037] Task capture (through UiPath Automation Cloud™ and / or UiPath AI Center™) automatically documents attended processes as users work on them or provides a framework for unattended processes. Such documentation may include process definition documents (PDDs), skeleton workflows, capturing actions for each part of the process, recording user actions and automatically generating comprehensive workflow diagrams with details about each step, tasks that are desirable to automate in formats such as Microsoft Word™ documents, XAML files, etc. Buildable workflows may, in some embodiments, be exported directly to designer applications such as UiPath Studio™. Task capture may simplify the requirements gathering process for both subject matter experts describing the process and Center of Excellence (CoE) members providing production-grade automation.

[0038] Building automations may be accomplished through designer applications (UiPath Studio™, UiPath StudioX™, UiPath Web™, etc.). For example, an RPA developer at RPA development facility 150 may use RPA designer application 154 at computing system 152 to build and test automations for various applications and environments, such as web, mobile, SAP®, and virtual desktop. API integration may be provided for various applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and workflow recorders may facilitate automation with minimal coding. Document understanding capabilities may be provided through drag-and-drop AI skills for data extraction and interpretation that invoke one or more AI / ML models 132. Such automations may handle virtually any document type and format, including tables, checkboxes, signatures, and handwriting. When data is validated or exceptions are handled, this information may be used to retrain the respective AI / ML models, improving their accuracy over time.

[0039] RPA designer application 152 may be designed to invoke one or more of trained AI / ML models 132 on server 130 and / or generative AI models 172 in a cloud environment over network 120 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.) to assist in the RPA automation development process. In some embodiments, one or more of the AI / ML models may be packaged with RPA designer application 152 or may otherwise be stored locally on computing system 150.

[0040] In some embodiments, the RPA designer application 152 and the one or more AI / ML models 132 may be configured to use an object repository stored in the database 140. The object repository may include a library of UI objects that may be used to develop RPA workflows via the RPA designer application 152. The object repository may be used to add UI descriptors to activities in the workflows of the RPA designer application 152 for UI automation. In some embodiments, one or more of the AI / ML models 132 may generate new UI descriptors and add them to the object repository in the database 140. Once an automation is completed in the designer application 152, the automation may be published on the server 130 and pushed out to the computing systems 102, 104, 106, etc.

[0041] The integration services allow developers to seamlessly combine, for example, UI automation and API automation. Automations that require APIs or span both API and non-API applications and systems may be built. A repository (e.g., UiPath Object Repository™) or marketplace (e.g., UiPath Marketplace™) for pre-built RPA and AI templates and solutions may be provided to enable developers to more quickly automate a wide variety of processes. Thus, when building automations, the hyper-automation system 100 may provide a user interface, development environment, API integration, pre-built and / or custom-built AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. The hyper-automation system 100, in some embodiments, may enable development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, which may provide automation for the hyper-automation system 100.

[0042] In some embodiments, components of the hyper-automation system 100, such as the designer application(s) and / or external rules engine, provide support for managing and enforcing governance policies to control various functions provided by the hyper-automation system 100. Governance is the ability for an organization to put policies in place to prevent users from developing automations (such as RPA robots) that can perform actions that may harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the US Health Insurance Portability and Accountability Act (HIPAA), third-party application terms of use, etc. Because developers may otherwise create automations that violate privacy laws, terms of use, etc. during the execution of their automations, some embodiments implement access control and governance restrictions at the robot and / or robot design application level. This may provide an additional level of security and compliance to the automation process development pipeline in some embodiments by preventing developers from taking dependencies on unapproved software libraries that may pose security risks or operate in a manner that violates policies, regulations, privacy laws, and / or privacy policies. See, for example, U.S. Patent Application Publication No. 2022 / 0011732, which is incorporated by reference in its entirety into this specification.

[0043] The management functions may provide management, deployment, and optimization of automation across an organization. The management functions may include orchestration, test management, AI capabilities, and / or insights, in some embodiments. The management functions of the hyper-automation system 100 may also act as an integration point with third party solutions and applications for automation applications and / or RPA robots. The management functions of the hyper-automation system 100 may include, but are not limited to, facilitating provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots, among others.

[0044] Conductor applications such as UiPath Orchestrator™ (which may be provided in some embodiments as part of UiPath Automation Cloud™ or may be provided on-premise, on a VM, on a private or public cloud, on a Linux™ VM, or as a cloud-native single container suite via UiPath Automation Suite™) provide orchestration capabilities to deploy, monitor, optimize, scale, and ensure the security of RPA robot deployments. Test suites (e.g., UiPath Test Suite™) may provide test management to monitor the quality of deployed automation. Test suites may facilitate test planning and execution, requirements fulfillment, and defect traceability. Test suites may include comprehensive test reports.

[0045] The analytics software (e.g., UiPath Insights™) can track, measure, and manage the performance of deployed automation. The analytics software can align automation actions with specific key performance indicators (KPIs) and strategic outcomes of the organization. The analytics software can present results in a dashboard format for better understanding by human users.

[0046] Data services (e.g., UiPath Data Service™) can bring data to a single, scalable, secure location, for example, stored in database 140, with a drag-and-drop storage interface. Some embodiments may provide low-code or no-code data modeling and storage to automation, while ensuring seamless access, enterprise-grade security, and scalability of data. AI capabilities may be provided by an AI center (e.g., UiPath AI Center™), which facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make such capabilities accessible to non-data scientists. Deployed automation (e.g., RPA robots) may invoke AI / ML models from the AI ​​center, such as AI / ML models 132. Performance of the AI / ML models may be monitored and trained and improved using human-verified data, such as provided by data review center 160. Human reviewers may provide labeled data to the core hyper-automation system 120 via review applications 152 on computing systems 154. For example, the human reviewer may verify that predictions by the AI / ML model 132 and / or the generative AI model 172 are accurate or may otherwise provide corrections. This dynamic input may then be saved as training data for retraining the AI / ML model 132 and / or the generative AI model 172 and may be stored in a database, such as database 140. The AI ​​center may then schedule and complete training jobs to train new versions of the AI / ML models using the training data. Both positive and negative examples may be stored and used for retraining the AI / ML model 132 and / or the generative AI model 172.

[0047] Engagement capabilities engage humans and automation as one team for seamless collaboration around a desired process. Low-code applications can be built (e.g., via UiPath Apps™), even those lacking APIs in some embodiments, to connect browser tabs and legacy software. Applications can be created quickly using a web browser, for example, through a rich library of drag-and-drop controls. Applications can be connected to one automation or multiple automations.

[0048] Action Center (e.g., UiPath Action Center™) provides an easy and efficient mechanism to hand off processes from automation to humans and vice versa. Humans can provide approvals or escalations, handle exceptions, etc. Automation can then execute the automated functions of a given workflow.

[0049] A local assistant may be provided as a launch pad for users to launch automations (e.g., UiPath Assistant™). This functionality may be provided, for example, in a tray provided by the operating system, and may allow users to interact with RPA robots and RPA robot-powered applications on their computing systems. The interface may list automations approved for a given user and allow users to execute them. These may include ready-to-use automations from the automation marketplace, the automation hub's internal automation store, etc. When automations are running, they may run as local instances in parallel with other processes on the computing system so that the user can use the computing system while the automation performs its actions. In certain embodiments, the assistant is integrated with a task capture functionality so that users can document their soon-to-be-automated processes from the assistant's launch pad.

[0050] Chatbots (e.g., UiPath Chatbots™), social messaging applications, and / or voice commands may enable users to execute automations. This may simplify access to information, tools, and resources needed to interact with customers or conduct other activities. Human-to-human conversations may be automated as easily as other processes. Triggered RPA robots thus activated may be able to perform actions such as checking order status, posting data to a CRM, etc., using plain language commands.

[0051] End-to-end measurement and governance of automation programs at any scale may be provided by the hyper-automation system 100 in some embodiments. As a result of the above, analytics (e.g., via UiPath Insights™) may be employed to understand the performance of automations. Data modeling and analytics using any combination of available business metrics and operational insights may be used for various automation processes. Custom-designed and pre-built dashboards may visualize data across desired metrics, discover new analytical insights, track performance indicators, discover ROI for automations, perform telemetry monitoring on users' computing systems, detect errors and anomalies, and debug automations. An automation management console (e.g., UiPath Automation Ops™) may be provided to manage automations throughout the automation lifecycle. Organizations may govern how automations are built, what users can do with them, and which automations users have access to.

[0052] The hyper-automation system 100, in some embodiments, provides an iterative platform: processes can be discovered, automations can be built, tested, and deployed, performance can be measured, automation usage can be easily provided to users, feedback can be obtained, AI / ML models can be trained and retrained, and the process itself can be repeated, facilitating a more robust and effective set of automations.

[0053] In some embodiments, generative AI models are used. Generative AI may generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models may be used, including but not limited to large-scale language models (LLMs), generative adversarial networks (GANs), variational autoencoders (VAEs), transformers, and the like. These models may be part of the AI / ML models 132 hosted on the server 130. For example, generative AI models may be trained on large corpora of text information to perform semantic understanding, understand the nature of what is present on the screen from the text, and the like. In certain embodiments, generative AI models 172 provided by existing cloud ML service providers, such as OpenAI®, Google®, Amazon®, Microsoft®, IBM®, Nvidia®, Facebook®, and the like, may be employed and trained to provide such capabilities. In generative AI embodiments where the generative AI model(s) 172 are hosted remotely, the server 130 may be configured to integrate with third-party APIs, such that the server 130 can send requests containing required input information to the generative AI model(s) 172 and receive the reply (e.g., semantic matching of fields between application versions, classification of application types on screens, etc.). Such embodiments can provide a more advanced and sophisticated user experience as well as access to state-of-the-art natural language processing (NLP) and other ML capabilities offered by these companies.

[0054] One aspect of generative AI models in some embodiments is the use of transfer learning. In transfer learning, a pre-trained generative AI model, such as an LLM, is fine-tuned for a specific task or domain. This allows the LLM to leverage knowledge already learned during its initial training and adapt it to a specific application. In the case of an LLM, the pre-training phase trains the LLM on a large text corpus, typically consisting of billions of words. During this phase, the LLM learns relationships between words and phrases, which allows the LLM to generate consistent, human-like responses to text-based inputs. The output of this pre-training phase is an LLM with a sophisticated understanding of the patterns underlying natural language.

[0055] The fine-tuning phase adapts the pre-trained LLM to a particular task or domain by training the LLM on a smaller dataset specific to the task. For example, in some embodiments, the LLM may be trained to analyze a particular type or types of data sources to improve its accuracy on that content. Such information may be provided as part of the training data, and the LLM may focus on these areas and learn to more accurately identify data elements therein. Fine-tuning may allow the LLM to learn the subtle nuances of a task or domain, such as the specific vocabulary and syntax used in that domain, without requiring as much data as would be necessary to train the LLM from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned LLM may achieve state-of-the-art performance on a particular task with a relatively small amount of training data.

[0056] FIG. 2 is an architecture diagram illustrating an RPA system 200, according to an embodiment of the present invention. In some embodiments, the RPA system 200 is part of the hyper-automation system 100 of FIG. 1. The RPA system 200 includes a designer 210 that enables developers to design and implement workflows. The designer 210 provides solutions for application integration, as well as automating third-party applications, management information technology (IT) tasks, and business IT processes. The designer 210 may facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 210 facilitates the development and deployment of workflows and robots. In some embodiments, the designer 210 may be an application running on a user's desktop, an application running remotely in a VM, a web application, or the like.

[0057] An automation project enables rule-based process automation by giving developers control over the order of execution and relationships between a custom set of steps developed in a workflow, defined herein as "activities" as described above. One commercial example of an embodiment of the designer 210 is UiPath Studio™. Each activity may include an action such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0058] Some types of workflows may include, but are not limited to, sequences, flowcharts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, allowing flow from one activity to another without cluttering the workflow. Flowcharts may be particularly suitable for more complex business logic, allowing for decision integration and connection of activities in more diverse ways via multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a finite number of states triggered by conditions (i.e., transitions) or activities during their execution. Global exception handlers may be particularly suitable for determining the behavior of a workflow when an execution error is encountered, or for debugging the process.

[0059] Once a workflow is developed in Designer 210, the execution of the business process is orchestrated by Conductor 220, which coordinates one or more Robots 230 that execute the workflow developed in Designer 210. One commercial example of an embodiment of Conductor 220 is UiPath Orchestrator™. Conductor 220 facilitates management of the creation, monitoring, and deployment of resources in an environment. Conductor 220 may act as an integration point with third party solutions and applications. Accordingly, in some embodiments, Conductor 220 may be part of the core hyper-automation system 120 of FIG. 1.

[0060] The conductor 220 may manage all the robots 230, connecting and executing them from a central point. Types of robots 230 that may be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234, but used for development and testing purposes), and non-production robots (similar to attended robots 232, but used for development and testing purposes). Attended robots 232 are triggered by user events and operate side-by-side with humans on the same computing system. Attended robots 232 may be used with the conductor 220 for a centralized process deployment and logging medium. Attended robots 232 may assist human users in accomplishing various tasks and may be triggered by user events. In some embodiments, processes cannot be initiated from the conductor 220 on this type of robot and / or they cannot be run under a locked screen. In certain embodiments, the attended robot 232 may only be launched from the robot tray or from a command prompt. The attended robot 232 preferably operates under human supervision in some embodiments.

[0061] Unattended robots 234 can operate unattended in a virtual environment and automate many processes. Unattended robots 234 can be responsible for providing remote execution, monitoring, scheduling, and work queue support. Debugging for all robot types can be performed in designer 210 in some embodiments. Both attended and unattended robots can automate a variety of systems and applications, including, but not limited to, mainframes, web applications, VMs, enterprise applications (e.g., those produced by SAP®, SalesForce®, Oracle®, etc.), and computing system applications (e.g., desktop and laptop applications, mobile device applications, wearable computer applications, etc.).

[0062] Conductor 220 may have various capabilities including, but not limited to, provisioning, deployment, configuration, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining connections between robots 230 and conductor 220 (e.g., web applications). Deployment may include ensuring correct delivery of package versions to robots 230 assigned for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot specific data and maintaining user permissions. Logging may include storing and indexing logs in a database (e.g., a Structured Query Language (SQL) database or a "not only" SQL (NoSQL) database) and / or another storage mechanism (e.g., ElasticSearch, which provides the ability to store large data sets and perform queries quickly). Conductor 220 may provide interconnectivity by operating as a centralized point of communication for third-party solutions and / or applications.

[0063] Robot 230 is an execution agent that implements the workflow built in designer 210. One commercial example of some embodiments of robot(s) 230 is UiPath Robots™. In some embodiments, robot 230 installs Microsoft Windows Service Control Manager (SCM) management services by default. As a result, such robot 230 can open interactive Windows sessions under the local system account and may have Windows service rights.

[0064] In some embodiments, a robot 230 may be installed in user mode, meaning that for such a robot 230, it has the same rights as the user in which the given robot 230 is installed. This feature may also be available for high density (HD) robots, ensuring maximum utilization of each machine. In some embodiments, either type of robot 230 may be configured in an HD environment.

[0065] The robot 230 in some embodiments is divided into multiple components, each specialized for a particular automation task. The robot components in some embodiments include, but are not limited to, an SCM managed robot service, a user mode robot service, an executor, an agent, and a command line. The SCM managed robot service manages and monitors the Windows session and acts as a proxy between the conductor 220 and the execution host (i.e., the computing system on which the robot 230 executes). These services are entrusted with and manage the credentials of the robot 230. The console application is launched by the SCM under the local system.

[0066] The user-mode robot service in some embodiments manages and monitors the Windows session and acts as a proxy between the conductor 220 and the execution host. The user-mode robot service may be delegated and manage the credentials of the robot 230. If the SCM management robot service is not installed, a Windows application may be launched automatically.

[0067] An Executor may execute a given job under a Windows session (i.e., execute a workflow). An Executor may be aware of per-monitor dots-per-inch (DPI) settings. An Agent may be a Windows Presentation Foundation (WPF) application that displays available jobs in a system tray window. An Agent may be a client of a service. An Agent may ask to start or stop a job or change settings. A Command Line is a client of a service. A Command Line is a console application that can request the start of a job and wait for its output.

[0068] As explained above, the separation of the robot 230 components helps developers, support users, and computing systems to more easily execute, identify, and track what each component is doing. In this way, special behaviors can be configured for each component, such as setting different firewall rules for executors and services. The executor may always be aware of per-monitor DPI settings in some embodiments. As a result, workflows may execute at any DPI regardless of the configuration of the computing system on which the workflow was created. Also, in some embodiments, projects from the designer 210 may be made independent of the browser zoom level. For applications that are not DPI aware or are marked as not DPI aware intentionally, in some embodiments, DPI may be disabled.

[0069] The RPA system 200 in this embodiment is part of a hyper-automation system. A developer can use the designer 210 to build and test RPA robots that utilize AI / ML models deployed to the core hyper-automation system 240 (e.g., as part of its AI center). Such RPA robots can send inputs for execution of the AI / ML model(s) and receive outputs therefrom via the core hyper-automation system 240.

[0070] One or more of the robots 230 may be listeners, as described above. These listeners may provide information to the core hyper-automation system 240 about what users are doing when they use their computing systems. This information may then be used by the core hyper-automation system for process mining, task mining, task capture, and the like.

[0071] An assistant / chatbot 250 may be provided on a user computing system to allow the user to launch an RPA local robot. The assistant may be located, for example, in the system tray. The chatbot may have a user interface so that the user can see the chatbot's text. Alternatively, the chatbot may not have a user interface, run in the background, and listen to the user's speech using the computing system's microphone.

[0072] In some embodiments, data labeling may be performed by a user of the computing system on which the robot is running, or on another computing system to which the robot provides information. For example, if the robot invokes an AI / ML model to perform CV on an image for a VM user, but the AI / ML model does not correctly identify a button on the screen, the user may draw a rectangle around the misidentified or unidentified component and potentially provide text with the correct identification. This information may be provided to the core hyperautomation system 240 and then later used to train a new version of the AI / ML model.

[0073] Figure 3 is an architecture diagram illustrating a deployed RPA system 300, according to an embodiment of the present invention. In some embodiments, the RPA system 300 may be part of the RPA system 200 of Figure 2 and / or the hyper-automation system 100 of Figure 1. The deployed RPA system 300 may be a cloud-based system, an on-premise system, a desktop-based system, etc., providing enterprise-level, user-level, or device-level automation solutions for automation of different computing processes.

[0074] It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robotic application 310 includes an executor 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not be running on the same computing system as the executor 312 and the agent 314. The executor 312 is running a process. As shown in FIG. 3, multiple business projects may be running simultaneously. The agent 314 (e.g., a Windows service) is a single connection point for all the executors 312 in this embodiment. All messages in this embodiment are logged into the conductor 340, which further processes them via the database server 350, the AI / ML server 360, the indexer server 370, or any combination thereof. As described above with respect to FIG. 2, the executor 312 may be a robotic component.

[0075] In some embodiments, a robot represents an association between a machine name and a username. A robot may manage multiple executors simultaneously. In computing systems that support multiple interactive sessions running simultaneously (such as Windows Server 2012), multiple robots may run simultaneously, each running in a separate Windows session using a unique username. This is referred to as the HD Robot above.

[0076] The agent 314 is also responsible for transmitting the robot's status (e.g., periodically sending "heartbeat" messages to indicate that the robot is still functioning) and downloading required versions of packages to be fulfilled. Communication between the agent 314 and the conductor 340 is, in some embodiments, always initiated by the agent 314. In a notification scenario, the agent 314 may open a WebSocket channel that is later used by the conductor 340 to send commands (e.g., start, stop, etc.) to the robot.

[0077] Listener 330 monitors and records data relating to user interactions with the operation of the attended and / or unattended computing systems on which listener 330 resides. Listener 330 may be an RPA robot, part of an operating system, a downloadable application for the respective computing system, or any other software and / or hardware without departing from the scope of the invention. Indeed, in some embodiments, the logic of the listener is implemented partially or completely via physical hardware.

[0078] The server side includes a presentation layer (web application 342, Open Data Protocol (OData) Representational State Transfer (REST) ​​Application Programming Interface (API) endpoints 344, notifications and monitoring 346), a service layer (API implementation / business logic 348), and a persistence layer (database server 350, AI / ML server 360, indexer server 370). Conductor 340 includes web application 342, OData REST API endpoints 344, notifications and monitoring 346, and API implementation / business logic 348. In some embodiments, most actions that a user performs in the Conductor 340 interface (e.g., via browser 320) are performed by calling various APIs. Such actions may include, but are not limited to, launching jobs on a robot, adding / removing data in a queue, scheduling jobs to run unattended, etc., without departing from the scope of the invention. Web application 342 is the visual layer of the server platform. In this embodiment, web application 342 uses HyperText Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format may be used without departing from the scope of the present invention. A user interacts with web pages from web application 342, in this embodiment via browser 320, to perform various operations to control conductor 340. For example, a user may create robot groups, assign packages to robots, analyze per-robot and / or per-process logs, start and stop robots, etc.

[0079] In addition to the web application 342, the conductor 340 also includes a services layer that exposes an OData REST API endpoint 344. However, other endpoints may be included without departing from the scope of the invention. The REST API is consumed by both the web application 342 and an agent 314, which in this embodiment is a supervisor of one or more robots on a client computer.

[0080] The REST API of the present embodiment covers configuration, logging, monitoring, and queuing functionality. The configuration endpoint may be used in some embodiments to define and configure users, permissions, robots, assets, releases, and environments of an application. The logging REST endpoint may be used to log various information, such as errors, explicit messages sent by the robot, and other environment specific information. The deployment REST endpoint may be used by the robot to query the version of the package that should be executed when a start job command is used in the conductor 340. The queuing REST endpoint may be responsible for managing queues and queue items, such as adding data to the queue, getting transactions from the queue, and setting the status of transactions.

[0081] Monitoring REST endpoints may monitor the web application 342 and the agents 314. Notification and monitoring API 346 may be a REST endpoint used to register agents 314, deliver configuration settings to agents 314, and send and receive notifications from the server and agents 314. Notification and monitoring API 346 may use WebSocket communications in some embodiments.

[0082] The service layer APIs, in some embodiments, may be accessed through configuration of appropriate API access paths, for example, based on whether the conductor 340 and the overall hyperautomation system have an on-premise or cloud-based deployment type. The API for the conductor 340 may provide custom methods to query statistics about various entities registered with the conductor 340. Each logical resource may be an OData entity, in some embodiments. In such entities, components such as robots, processes, queues, etc. may have properties, relationships, and behaviors. The conductor 340 APIs, in some embodiments, may be consumed by web applications 342 and / or agents 314 in two ways: by getting API access information from the conductor 340 or by registering an external application to use the OAuth flow.

[0083] The persistence layer includes three servers in this embodiment: a database server 350 (e.g., a SQL server), an AI / ML server 360 (e.g., a server providing AI / ML model provision services such as an AI center function), and an indexer server 370. The database server 350 in this embodiment stores configurations of robots, robot groups, associated processes, users, roles, schedules, etc. This information is managed via a web application 342 in some embodiments. The database server 350 may manage queues and queue items. In some embodiments, the database server 350 may store messages logged by the robots (in addition to or instead of the indexer server 370). The database server 350 may also store process mining, task mining, and / or task capture related data received, for example, from a listener 330 installed on the client side. Although no arrow is shown between the listener 330 and the database 350, it should be understood that the listener 330 can communicate with the database 350 and vice versa in some embodiments. This data may be stored in the form of PDDs, images, XAML files, etc. Listener 330 may be configured to intercept user actions, processes, tasks, and performance metrics on the respective computing system on which listener 330 resides. For example, listener 330 may record user actions (e.g., clicks, characters typed, locations, applications, active elements, times, etc.) on its respective computing system and then convert these into a suitable format to be provided to and stored in database server 350.

[0084] AI / ML Server 360 facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make such capabilities accessible to non-data scientists. Deployed automation (e.g., RPA robots) may invoke AI / ML models from AI / ML Server 360. Performance of AI / ML models may be trained and improved using monitored and human-validated data. AI / ML Server 360 may schedule and execute training jobs to train new versions of AI / ML models.

[0085] The AI / ML server 360 may store data related to AI / ML models and ML packages for configuring various ML skills for users at development time. An ML skill, as used herein, is a pre-built and trained ML model for a process that may be used, for example, by automation. The AI / ML server 360 may also store data related to document understanding techniques and frameworks, algorithms, and software packages for various AI / ML capabilities, including, but not limited to, intent analysis, NLP, speech analysis, different types of AI / ML models, etc.

[0086] Optionally in some embodiments, the indexer server 370 stores and indexes information logged by the robots. In certain embodiments, the indexer server 370 may be disabled via a configuration setting. In some embodiments, the indexer server 370 uses ElasticSearch®, a full-text search engine from an open source project. Messages logged by the robots (e.g., using activities such as log messages or line writes) may be sent via logging REST endpoint(s) to the indexer server 370, where they are indexed for future use.

[0087] FIG. 4 is an architecture diagram illustrating relationships 400 between a designer 410, activities 420, 430, 440, 450, drivers 460, APIs 470, and AI / ML models 480, according to an embodiment of the present invention. Thus, a developer uses the designer 410 to develop a workflow to be performed by a robot. Various types of activities may be displayed to the developer in some embodiments. The designer 410 may be local to the user's computing system or remote thereto (e.g., accessed via a local web browser that interacts with a VM or a remote web server). The workflow may include user-defined activities 420, API-driven activities 430, AI / ML activities 440, and / or UI automation activities 450. The user-defined activities 420 and API-driven activities 440 interact with the application via their APIs. The user-defined activities 420 and / or the AI / ML activities 440 may, in some embodiments, invoke one or more AI / ML models 480, which may be located locally and / or remotely to the computing system on which the robot is operating.

[0088] In some embodiments, non-text visual components in an image may be identified, referred to herein as CV. However, it is noted that in some embodiments, CV incorporates OCR. CV may be performed at least in part by AI / ML model(s) 480. Some CV activities related to such components may include, but are not limited to, extracting text from segmented label data using OCR, fuzzy text matching, cropping segmented label data using ML, comparing extracted text in label data to ground truth data, etc. In some embodiments, the number of activities that may be implemented in user-defined activities 420 may be hundreds or thousands. However, any number and / or type of activities may be used without departing from the scope of the present invention.

[0089] UI automation activities 450 are a subset of specialized low-level activities that are written in low-level code and facilitate interactions with the screen. UI automation activities 450 facilitate these interactions via drivers 460 that enable the robot to interact with desired software. For example, drivers 460 may include operating system (OS) drivers 462, browser drivers 464, VM ​​drivers 466, enterprise application drivers 468, etc. In some embodiments, one or more AI / ML models 480 may be used by UI automation activities 450 to perform interactions with the computing system. In certain embodiments, AI / ML models 480 may augment drivers 460 or replace them entirely. Indeed, in certain embodiments, drivers 460 are not included.

[0090] Driver 460 may interact with the OS at a low level, such as looking for hooks, monitoring keys, etc., via OS driver 462. Driver 460 may facilitate integration with Chrome®, IE®, Citrix®, SAP®, etc. For example, a "click" activity plays the same role in these different applications via driver 460.

[0091] FIG. 5 is an architecture diagram illustrating a computing system 500 configured to use generative AI to supplement automated information extraction, according to an embodiment of the present invention. In some embodiments, the computing system 500 may be one or more computing systems depicted and / or described herein. In certain embodiments, the computing system 500 may be part of a hyper-automation system such as those shown in FIGS. 1 and 2. The computing system 500 includes a bus 505 or other communication mechanism for communicating information and a processor(s) 510 coupled to the bus 505 for processing information. The processor(s) 510 may be any type of general or application-specific processor, including a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. The processor(s) 510 may also have multiple processing cores, and at least some of the cores may be configured to perform specific functions. In some embodiments, multiple parallel processing may be used. In certain embodiments, at least one processor(s) 510 may be a neuromorphic circuit that includes processing elements that mimic biological neurons. In some embodiments, a neuromorphic circuit may not require typical components of a von Neumann computing architecture.

[0092] The computing system 500 further includes a memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache, a static storage device such as a magnetic or optical disk, or other types of non-transitory computer-readable media, or any combination thereof. The non-transitory computer-readable media may be any available media accessible by the processor(s) 510 and may include volatile media, non-volatile media, or both. Also, the media may be removable, non-removable, or both. The computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 may include one or more antennas that are a single antenna, an array of antennas, phased antennas, switched antennas, beamforming antennas, beam-steering antennas, combinations thereof, and / or any other antenna configuration without departing from the scope of the present invention.

[0093] The processor(s) 510 are further coupled to a display 525 via a bus 505. Any suitable display device and haptic I / O may be used without departing from the scope of the invention. A keyboard 530 and a cursor control device 535, such as a computer mouse, touchpad, and the like, are further coupled to the bus 505 to allow a user to interface with the computing system 500. However, in certain embodiments, a physical keyboard and mouse may not be present and the user may interact with the device solely via the display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In certain embodiments, no physical input devices and / or displays are present. For example, a user may interact with the computing system 500 remotely via another computing system in communication with it, or the computing system 500 may operate autonomously.

[0094] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules further include an automated information extraction model 545 configured to perform all or a portion of the AI / ML processes described herein or derivatives thereof. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0095] Those skilled in the art will appreciate that the "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or combination of devices without departing from the scope of the present invention. Presenting the above-described functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but rather to provide an example of many embodiments of the present invention. Indeed, the methods, systems, and apparatuses disclosed herein may be implemented in localized and distributed forms consistent with computing techniques, including cloud computing systems. The computing system may be part of or otherwise accessible by a local area network (LAN), a mobile communications network, a satellite communications network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, and the like. Any local or distributed architecture may be used without departing from the scope of the present invention.

[0096] It should be noted that some of the system features described herein are presented as modules to better emphasize implementation independence. For example, a module may be implemented as a hardware circuit comprising custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, graphics processing units, and the like.

[0097] Modules may also be implemented at least in part in software for execution by various types of processors. For example, an identified unit of executable code may include one or more physical or logical blocks of computer instructions, which may be organized, for example, as an object, procedure, or function. Nevertheless, executable identified modules need not be physically located together, but may include separate instructions stored in different locations, which when logically combined, comprise the modules, to accomplish the purpose stated for the modules. Furthermore, modules may be stored on non-transitory computer readable media, such as, for example, hard disk drives, flash devices, RAM, tape, and / or any other non-transitory computer readable media used to store data without departing from the scope of the present invention.

[0098] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within a module, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network.

[0099] Various types of AI / ML models may be trained and deployed without departing from the scope of the present invention. For example, FIG. 6A illustrates an example of a neural network 600 (e.g., a CV model, an OCR model, a generative AI model, etc.) trained to perform automated information extraction according to an embodiment of the present invention. The neural network 600 includes a number of hidden layers. Although both DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, SLNNs may have only one or two layers in some cases, and typically fewer than DLNNs. Typically, the architecture of a neural network includes an input layer, multiple hidden layers, and an output layer, as in the case of the neural network 600.

[0100] DLNNs often have many layers (10, 50, 200, etc.), with subsequent layers typically reusing features from previous layers to compute more complex and general functions. SLNNs, on the other hand, tend to have only a few layers and train relatively quickly, since expert features are pre-created from raw data samples; however, feature extraction is laborious. DLNNs, on the other hand, usually do not require expert features, but they take longer to train and tend to have more layers.

[0101] In both approaches, the layers are trained simultaneously on the training set and are typically checked for overfitting on a separate cross-validation set. Both techniques produce excellent results, and there is considerable enthusiasm for both approaches. The optimal size, shape, and number of individual layers depend on the problem being addressed by each neural network.

[0102] Returning to Figure 6A, text, screenshots, graphical elements, and contextual information (e.g., information about the application the source is related to, the type of source, etc.) are provided as inputs to the J neurons in hidden layer 1. In this example, all of these inputs are fed to each neuron, however, a variety of architectures are possible that may be used individually or in combination, including, but not limited to, feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graphics networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid state machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks without departing from the scope of the invention.

[0103] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on for all hidden layers until the last hidden layer provides its output as an input for the output layer. Although multiple suggestions are shown herein as outputs, in some embodiments, only a single output suggestion is provided. In certain embodiments, the suggestions are ranked based on confidence scores.

[0104] Note that the numbers of neurons I, J, K, and L are not necessarily equal. Thus, any desired number of layers may be used in a given layer of neural network 600 without departing from the scope of the present invention. Indeed, in certain embodiments, the types of neurons in a given layer may not all be the same.

[0105] The neural network 600 is trained to assign confidence score(s) to appropriate outputs. To reduce inaccurate predictions, in some embodiments, only those results with confidence scores equal to or greater than a confidence threshold may be provided. For example, if the confidence threshold is 80%, outputs with confidence scores above this amount may be used and the rest may be ignored.

[0106] Note that neural networks are probabilistic constructs that typically have a confidence score(s). This can be a score that the AI / ML model learned based on how often similar inputs were correctly identified during training. Some common types of confidence scores include a decimal number between 0 and 1 (which can also be interpreted as a confidence percentage), a numeric value between negative infinity and positive infinity, a set of formulas (e.g., "low", "medium", and "high"), etc. To obtain more accurate confidence scores, various post-processing calibration techniques such as temperature scaling, batch normalization, weight decay, and negative log-likelihood (NLL) can also be used.

[0107] The "neurons" of a neural network are algorithmically implemented as mathematical functions, usually based on the functionality of biological neurons. Neurons receive weighted inputs and have a sum and activation function that governs whether they pass on an output to the next layer. This activation function may be a nonlinear, thresholded activity function where nothing happens if the value is below a threshold, and responds linearly when the function exceeds the threshold (i.e., rectified linear unit (ReLU) nonlinearity). Sum and ReLU functions are used in deep learning because real neurons may have roughly similar activity functions. Through linear transformations, information may be subtracted, added, etc. In essence, neurons act as gating functions that pass on outputs to the next layer governed by their underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.

[0108] JPEG2025026286000002.jpg84144

[0109] JPEG2025026286000003.jpg47144

[0110] JPEG2025026286000004.jpg30132

[0111] In this case, neuron 610 is a single-layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that the range of values ​​of the activation function weights and / or output value(s) may vary in some embodiments without departing from the scope of the present invention.

[0112] A goal, or "reward function," is often employed. The reward function guides the search of the state space, exploring intermediate transitions and steps with both short-term and long-term rewards to attempt to achieve the goal (e.g., finding the most accurate answer to a user query based on relevant metrics). During training, various labeled data are fed through the neural network 600. Successful identification strengthens the weights of the inputs to the neurons, while failure to identify weakens those weights. A cost function such as mean squared error (MSE) or gradient descent may be used to ensure that slightly incorrect predictions are punished much less than those that are significantly incorrect. If the performance of the AI / ML model does not improve after a certain number of training iterations, the data scientist may change the reward function, correct the incorrect predictions, etc.

[0113] Backpropagation is a technique for optimizing synaptic weights in feedforward neural networks. Backpropagation can be used to "pop up" the hidden layers of a neural network to see how much loss each node is carrying, and then update the weights to minimize the loss, giving lower weights to nodes with higher error rates and vice versa. In other words, backpropagation allows data scientists to iteratively adjust the weights to minimize the difference between the actual output and the desired output.

[0114] Backpropagation algorithms are mathematically based on optimization theory. In supervised learning, training data with known outputs is passed through a neural network and an error is calculated from the known target output using a cost function, which gives the backpropagation error. An error is calculated at the output and this error is converted into a modification of the network weights that minimizes the error.

[0115] JPEG2025026286000005.jpg72144

[0116] JPEG2025026286000006.jpg23144

[0117] JPEG2025026286000007.jpg124144

[0118] JPEG2025026286000008.jpg107116

[0119] JPEG2025026286000009.jpg84144

[0120] The AI / ML model may be trained over multiple epochs until it reaches a good level of accuracy (e.g., 97% or higher using an F2 or F4 threshold for detection, approximately 2,000 epochs). This accuracy level may be determined in some embodiments using an F1 score, an F2 score, an F4 score, or any other suitable technique that does not depart from the scope of the present invention. Once trained on training data, the AI / ML model may be tested on a set of evaluation data that the AI / ML model has not encountered before. This helps to ensure that the AI / ML model does not "overfit" such that it performs well on the training data but not on other data.

[0121] In some embodiments, it may not be known what level of accuracy an AI / ML model is capable of achieving. Thus, if the accuracy of the AI / ML model begins to degrade when analyzing the evaluation data (i.e., the model performs well on the training data but begins to perform poorly on the evaluation data), the AI / ML model can undergo further epochs of training on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only when the accuracy reaches a certain level or if the accuracy of the trained AI / ML model is superior to the existing deployed AI / ML model. In certain embodiments, a collection of trained AI / ML models can be used to accomplish a task. For example, one model can be trained to perform CV, another model can be trained to perform OCR, yet another model can be a generative AI model that identifies data elements in a source, etc.

[0122] In some embodiments, a transformer network such as SentenceTransformers™, a state-of-the-art Python™ framework for sentence, text, and image embedding, can be used. Such a transformer network learns associations of words and phrases with both high and low scores. This trains an AI / ML model to determine what is close to the input and what is not, respectively. Rather than using only word / phrase pairs, the transformer network may also use field lengths and field types.

[0123] In some embodiments, NLP techniques such as word2vec, BERT, GPT-3, ChatGPT, and other LLMs can be used to facilitate semantic understanding and provide more accurate and human-like answers, as described above. Other techniques such as clustering algorithms can be used to find similarities between groups of elements. Clustering algorithms can include, but are not limited to, density-based algorithms, distribution-based algorithms, centroid-based algorithms, and hierarchical-based algorithms. K-means clustering algorithms, DBSCAN clustering algorithms, Gaussian Mixture Models (GMM) algorithms, Balanced Iterative Reduction and Clustering with Hierarchy (BIRCH) algorithms, etc. Such techniques can also be useful for classification.

[0124] FIG. 7 is a flow chart illustrating a process 700 for training an AI / ML model(s), according to an embodiment of the present invention. In some embodiments, the AI / ML model(s) may be generative AI models, as described above. The neural network architecture of an AI / ML model typically includes multiple layers of neurons, including an input layer, an output layer, and a hidden layer. See, for example, FIGS. 6A and 6B. The input layer receives text-based input(s) and the output layer generates text-based response(s). Intermediate hidden layers process the input data and generate intermediate representations of the inputs that are used to generate the outputs. These hidden layers may include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transformer neurons.

[0125] The training process begins at 710 with providing text, screenshots, graphical elements, and labeled or unlabeled contextual information. The AI / ML model is then trained for multiple epochs at 720, and the results are reviewed at 730. Although various types of AI / ML models may be used, LLMs and other generative AI models are typically trained using a process called "supervised learning," described above. Supervised learning involves providing the model with a large dataset, which it uses to learn the relationships between inputs and outputs. During the training process, the model adjusts the weights and biases of neurons in the neural network to minimize the difference between the predicted outputs in the training dataset and the actual outputs.

[0126] One aspect of the model in some embodiments is the use of transfer learning. For example, transfer learning may utilize a pre-trained model, such as ChatGPT, that is fine-tuned for a particular task or domain in step 720. This allows the model to leverage already learned knowledge from the pre-training phase and adapt it to a particular application through the training phase in step 720.

[0127] The pre-training phase involves training the model based on an initial set of training data, which may be more general. In this phase, the model learns relationships in the data. The fine-tuning phase (e.g., in some embodiments, performed in addition to or instead of the initial training phase during step 720, when the pre-trained model is used as the initial basis for the final model) adapts the pre-trained model to a specific task or domain by training the model on a smaller data set specific to the task. For example, in some embodiments, the model may focus on a specific type(s) of data source. This may help the model identify data elements therein more accurately than a pre-trained generative AI model alone. Fine-tuning may allow the model to learn the nuances of the source, such as specific vocabulary and syntax, specific graphical characteristics, specific data formats, etc., without requiring as much data as would be required to train the model from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model may achieve state-of-the-art performance on a specific task, even with relatively little additional training data.

[0128] If the AI / ML model does not meet the desired confidence threshold at 740, the training data is supplemented and / or the reward function is modified to help the AI / ML model better achieve its objective at 750, and the process returns to step 720. If the AI / ML model meets the confidence threshold at 740, the AI / ML model is tested against evaluation data at 760 to ensure that the AI / ML model generalizes well and that the AI / ML model does not over-fit with respect to the training data. The evaluation data includes information that the AI / ML model has not previously processed. If the confidence threshold is met at 770 for the evaluation data, the AI / ML model is deployed at 780. If not, the process returns to step 750 and the AI / ML model is further trained.

[0129] 8 is a flow chart illustrating a process for using generative AI to supplement automated information extraction, according to an embodiment of the present invention. The process begins at 810 with providing source information as input to a CV and / or OCR model and a generative AI model to process the source information and extract data elements therein. For example, HTML, XAML, text, images, etc. may be obtained from the source and provided to these models as needed. This information may vary depending on the type of model. For example, CV and OCR models may be provided with images as input, while generative AI models may be provided with XAML data, OCR model output, etc.

[0130] The CV and / or OCR models and generative AI models process the respective source information provided to each model at 820 to extract data elements therein, and the output from these models is analyzed by the RPA robot or another software application at 830 to determine whether the models match their predictions for the extracted data elements. The model output may include predictions for the extracted data elements, their respective confidence scores, the extracted data elements themselves, etc. If the CV and / or OCR models match their predictions at 840 and the CV and / or OCR models and generative AI models have identified all data elements in the source at least with their respective tolerance thresholds (whether common or unique to that model), the common extracted element predictions and / or the respective data elements are provided as output at 850.

[0131] However, if the model does not match the prediction at 840, or regardless of model match, the CV and / or OCR model and / or generative AI model are below their respective tolerance thresholds, human assistance is requested at 860 (i.e., a human is brought into the loop). Labeled element data from the human reviewer(s) is then incorporated into each model's training data to meet the tolerance threshold for that element. For example, if the prediction is successful even if below the tolerance threshold, the labeled element may be provided as a positive example for retraining. If the prediction was incorrect, correcting the labeled data may help to successfully identify the element after a future retrain. This labeled data is then used to retrain the AI / ML model(s) at 870.

[0132] 9 is a flow chart illustrating a process 900 for using generative AI to supplement automated information extraction, according to an embodiment of the invention. The process begins at 910 with providing source information as input to a CV and / or OCR model and a generative AI model to process the source information and extract data elements therein. For example, HTML, XAML, text, images, etc. may be obtained from the source and provided to these models as needed. This information may vary depending on the type of model. For example, CV and OCR models may be provided with images as input, while generative AI models may be provided with XAML data, OCR model output, etc.

[0133] The CV and / or OCR models and generative AI models process the respective source information provided to each model at 920 to extract data elements therein, and the output from these models is analyzed by the RPA robot or another software application at 930 to determine whether the models match their predictions for the extracted data elements. The model output may include predictions for the extracted data elements, their respective confidence scores, the extracted data elements themselves, etc. If the CV and / or OCR models match the predictions at 940 and the CV and / or OCR models and generative AI models have identified all data elements in the source at least with their respective tolerance thresholds (either common or unique to that model), the common extracted element predictions and / or the respective data elements are provided as output at 950.

[0134] However, if the model predictions do not match at 960 and the element(s) are identified by the generative AI above a common or generative AI tolerance threshold, the extracted data element prediction(s) at 970 and / or the respective data element(s) extracted by the generative AI model are used and the failed element(s) labeled in the generative AI prediction are provided to retrain the CV or OCR model. If the CV and / or OCR models and the generative AI model at 960 are below their respective tolerance thresholds, human assistance is requested at 980 (e.g., if the process can await correction by a human user) and / or the element(s) are marked for subsequent retraining, regardless of model match. The labeled element data is incorporated into the training data of each model to meet the tolerance threshold for that element. For example, if the prediction was successful despite being below the tolerance threshold, the labeled element may be provided as a positive example for retraining. If the prediction was incorrect, correcting the labeled data may help successfully identify the element after future retraining. The AI / ML model(s) are then retrained at 990 using this labeled data.

[0135] The process steps performed in Figures 8 and 9 may be performed by a computer program encoding instructions to a processor(s) to perform at least a portion of the process(es) described in Figures 8 and 9 according to an embodiment of the present invention. The computer program may be stored on a non-transitory computer-readable medium. The computer-readable medium may be, but is not limited to, a hard disk drive, a flash device, a RAM, a tape, and / or any other such medium or combination of media used to store data. The computer program may include coded instructions for controlling a processor(s) of a computing system (e.g., processor(s) 510 of computing system 500 of Figure 5) to implement all or a portion of the process steps described in Figures 8 and 9, which may also be stored on a computer-readable medium.

[0136] The computer program may be implemented in hardware, software, or a hybrid implementation. The computer program may be composed of modules in operable communication with each other and designed to send information or instructions to a display. The computer program may be configured to run on a general purpose computer, an ASIC, or any other suitable device.

[0137] It will be readily understood that the components of the various embodiments of the present invention, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the detailed description of the embodiments of the present invention, as represented in the accompanying figures, is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention.

[0138] The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, reference throughout this specification to "certain embodiments," "some embodiments," or similar language means that the particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearance of "certain embodiments," "some embodiments," "other embodiments," or similar language throughout this specification does not necessarily refer to the same group of all embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0139] It should be noted that references to features, advantages, or similar language throughout this specification do not imply that all of the features and advantages that may be realized in the present invention should be in any single embodiment of the present invention, or that any embodiment of the present invention. Rather, language referring to features and advantages is understood to mean that the particular feature, advantage, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, discussions of features and advantages throughout this specification, as well as similar language, may, but do not necessarily, refer to the same embodiment.

[0140] Furthermore, the described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the invention may be practiced without the specific features or advantages of one or more particular embodiments. In other instances, additional features and advantages may be recognized in certain embodiments, although they may not be present in all embodiments of the invention.

[0141] Those of ordinary skill in the art will readily appreciate that the invention as described above can be practiced using steps in a different order and / or with hardware elements in different configurations than those disclosed. Thus, while the invention has been described based on these preferred embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative configurations will become apparent while remaining within the spirit and scope of the invention. Accordingly, reference should be made to the appended claims to determine the scope of the invention.

Claims

1. 1. One or more non-transitory computer readable media storing a computer program for using generative artificial intelligence (AI) to supplement automated information extraction, the computer program comprising: providing source information as input to a computer vision (CV) model, an optical character recognition (OCR) model, or both, and a generative artificial intelligence (AI) model, and processing said source information to extract data elements therein; receiving output from the generative AI model and the CV model and / or the OCR model, the output including a plurality of extracted data element predictions and respective confidence scores for the extracted data elements; analyzing the plurality of extracted data element predictions and the respective confidence scores from the generative AI model, and the CV model and / or the OCR model to determine whether the models match the plurality of extracted data element predictions; one or more non-transitory computer-readable media configured to: output the extracted element prediction and / or the extracted data element in response to the generative AI model and the CV and / or OCR model matching an extracted data element prediction of the plurality of extracted data element predictions and a confidence score of the respective extracted data element prediction that exceeds a respective tolerance threshold of the model.

2. The computer program further comprises:

2. The one or more non-transitory computer-readable media of claim 1, configured to request human assistance to identify a correct respective data element in response to the generative AI model and the CV and / or OCR model not meeting or exceeding their respective tolerance thresholds of an extracted data element prediction or the generative AI model and the CV and / or OCR model not matching an extracted data element prediction of the plurality of extracted data element predictions.

3. The computer program further comprises:

3. The one or more non-transitory computer-readable media of claim 2, wherein the generative AI model and the CV and / or OCR model are configured to mark a respective data element for retraining the generative AI model and the CV and / or OCR model in response to an extracted data element prediction not meeting or exceeding its respective tolerance threshold.

4. 10. The one or more non-transitory computer-readable media of claim 1, wherein the source information comprises HyperText Markup Language (HTML) code, Extensible Application Markup Language (XAML) code, text, an image, or any combination thereof.

5. 10. The one or more non-transitory computer-readable media of claim 1 , wherein the source information provided to the generative AI model is different from the source information provided to the CV model and / or the OCR model.

6. 10. The one or more non-transitory computer readable media of claim 1, wherein the computer program is a Robotic Process Automation (RPA) robot.

7. 10. The one or more non-transitory computer-readable media of claim 1, wherein the generative AI model is a large-scale language model (LLM), a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer.

8. a memory storing computer program instructions for using generative artificial intelligence (AI) to supplement the automated information extraction; and at least one processor configured to execute the computer program instructions, the computer program instructions comprising: receiving output from a generative artificial intelligence (AI) model and a computer vision (CV) model and / or an optical character recognition (OCR) model, said output including a plurality of extracted data element predictions and respective confidence scores for data elements extracted from the source information; analyzing the plurality of extracted data element predictions and the respective confidence scores from the generative AI model, and the CV model and / or the OCR model to determine whether the models match the plurality of extracted data element predictions; one or more computing systems configured to request human assistance to identify a correct respective data element in response to the generative AI model and the CV model and / or OCR model not matching an extracted data element prediction of the plurality of extracted data element predictions or the generative AI model and the CV and / or OCR model not meeting or exceeding their respective tolerance thresholds of the extracted data element prediction.

9. The computer program instructions further include causing the at least one processor to:

10. The one or more computing systems of claim 8, wherein the generative AI model and the CV and / or OCR model are configured to output the extracted element prediction and / or the extracted data element in response to matching an extracted data element prediction of the plurality of extracted data element predictions and a confidence score of the respective extracted data element prediction that exceeds a respective tolerance threshold of the model.

10. The computer program instructions further include causing the at least one processor to:

10. The one or more computing systems of claim 8, wherein the generative AI model and the CV and / or OCR model are configured to mark a respective data element for retraining the generative AI model and the CV and / or OCR model in response to an extracted data element prediction not meeting or exceeding its respective tolerance threshold.

11. 10. The one or more computing systems of claim 8, wherein the source information comprises HyperText Markup Language (HTML) code, Extensible Application Markup Language (XAML) code, text, an image, or any combination thereof.

12. 10. The one or more computing systems of claim 8, wherein the source information provided to the generative AI model is different from the source information provided to the CV model and / or the OCR model.

13. 10. The one or more computing systems of claim 8, wherein the computer program instructions are or include a robotic process automation (RPA) robot.

14. 10. The one or more computing systems of claim 8, wherein the generative AI model is a large-scale language model (LLM), a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer.

15. The computer program instructions further include causing the at least one processor to:

10. The one or more computing systems of claim 8, configured to provide the source information as input to the generative AI model and the CV and / or OCR model, and to process the source information to extract the data elements therein.

16. 1. A computer-implemented method for using generative artificial intelligence (AI) to supplement automated information extraction, comprising: providing source information as input to a computer vision (CV) model, an optical character recognition (OCR) model, or both, and a generative artificial intelligence (AI) model by a robotic process automation (RPA) robot executed by at least one processor of a computing system, and processing the source information to extract data elements therein; receiving, by the RPA robot, output from the generative AI model and the CV model and / or the OCR model, the output including a plurality of extracted data element predictions and respective confidence scores for the extracted data elements; analyzing, by the RPA robot, the plurality of extracted data element predictions and the respective confidence scores from the generative AI model, and the CV model and / or the OCR model to determine whether the models match the plurality of extracted data element predictions; outputting, by the RPA robot, the extracted element prediction and / or the extracted data element in response to the generative AI model and the CV and / or OCR model matching an extracted data element prediction of the plurality of extracted data element predictions and a confidence score of the respective extracted data element prediction that exceeds a respective tolerance threshold of the model; responsive to the generative AI model and the CV model and / or OCR model not matching an extracted data element prediction of the plurality of extracted data element predictions or the generative AI model and the CV and / or OCR model not meeting or exceeding their respective tolerance thresholds of the extracted data element predictions, requesting human assistance by the RPA robot to identify a correct respective data element.

17. 17. The computer-implemented method of claim 16, further comprising, in response to the generative AI model and the CV and / or OCR model not meeting or exceeding their respective tolerance thresholds of extracted data element predictions, marking, by the RPA robot, the respective data element for retraining the generative AI model and the CV and / or OCR model.

18. 20. The computer-implemented method of claim 16, wherein the source information comprises HyperText Markup Language (HTML) code, Extensible Application Markup Language (XAML) code, text, an image, or any combination thereof.

19. 17. The computer-implemented method of claim 16, wherein the source information provided to the generative AI model is different from the source information provided to the CV model and / or the OCR model.

20. 17. The computer-implemented method of claim 16, wherein the generative AI model is a large-scale language model (LLM), a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer.

Images