Supplementing automatic information extraction using generative artificial intelligence

Abstract

Supplemental automatic information extraction using generative AIs is disclosed. A computer vision (CV) and / or optical character recognition (OCR) model and a generative artificial intelligence (AI) model are used together to extract information (e.g., name, date, invoice number, etc.) from a source. The acceptance threshold (s) may be used to accept predictions for extracted data elements from the model, and predictions from the generative AI model may be preferred, or human may be responsible for reviewing the elements. If no model satisfies its respective acceptance threshold (whether generic or specific to the model), the element (s) may be marked for subsequent manual review, or may be manually recycled to correct the element (s). The marked elements can then be used to retrain the model.

Description

Technical Field

[0001] The present invention relates generally to artificial intelligence (AI), and more particularly to using generative AI to supplement automatic information extraction. Background Art

[0002] Computer vision (CV) models and / or optical character recognition (OCR) models can be used to identify graphical elements and / or text present in a source (e.g., a document, a web page, a form, a scanned invoice, etc.). Such models will typically identify candidate graphical elements and / or text and provide confidence scores for the candidates. However, these techniques may not always accurately identify graphical elements in a source. Therefore, improved and / or alternative approaches may be beneficial. Summary of the invention

[0003] Certain embodiments of the present invention may provide solutions to problems and needs in the field that have not yet been fully identified, understood, or solved by current AI technology.For example, some embodiments of the present invention involve using generative AI to supplement automatic information extraction.

[0004] In one embodiment, one or more non-transitory computer-readable media store a computer program for supplementing automatic information extraction using generative AI. The computer program is configured to cause at least one processor to provide source information as input to a CV model, an OCR model, or both, and to provide it to a generative AI model to process the source information and extract data elements therein. The computer program is also configured to cause at least one processor to receive output from a generative AI model and a CV model and / or an OCR model. The output includes multiple extracted data element predictions and corresponding confidence scores for the extracted data elements. The computer program is also configured to cause at least one processor to analyze multiple extracted data element predictions and corresponding confidence scores from a generative AI model and a CV model and / or an OCR model to determine whether the model is consistent with respect to multiple extracted data element predictions. In addition, the computer program is configured to cause at least one processor to output extracted element predictions and / or extracted data elements in response to a generative AI model and a CV model and / or an OCR model reaching a consensus on the extracted data element predictions in multiple extracted data element predictions and the confidence score for the corresponding extracted data element prediction exceeds the corresponding acceptance threshold for the model.

[0005] In another embodiment, one or more computing systems include a memory storing computer program instructions for supplementing automatic information extraction with generative AI, and at least one processor configured to execute the computer program instructions. The computer program instructions are configured to cause at least one processor to receive outputs from a generative AI model and a CV model and / or an OCR model. The output includes multiple extracted data element predictions and corresponding confidence scores for data elements extracted from source information. The computer program instructions are also configured to cause at least one processor to analyze multiple extracted data element predictions and corresponding confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the model is consistent with multiple extracted data element predictions. The computer program instructions are also configured to cause at least one processor to respond to the generative AI model and the CV model and / or the OCR model not reaching a consensus on the extracted data element predictions in the multiple extracted data element predictions or the generative AI model and the CV model and / or the OCR model not meeting or exceeding their corresponding acceptance thresholds for extracting data element predictions and requesting manual assistance to identify the correct corresponding data element.

[0006] In another embodiment, a computer-implemented method for supplementing automatic information extraction with generative AI includes a robotic process automation (RPA) robot executed by at least one processor of a computing system providing source information as input to a CV model, an OCR model, or both, and providing it to a generative AI model to process the source information and extract data elements therein. The computer-implemented method also includes receiving outputs from a generative AI model and a CV model and / or an OCR model by the RPA robot. The output includes multiple extracted data element predictions and corresponding confidence scores for the extracted data elements. The computer-implemented method also includes analyzing multiple extracted data element predictions and corresponding confidence scores from the generative AI model and the CV model and / or the OCR model by the RPA robot to determine whether the model agrees on multiple extracted data element predictions. In addition, the computer-implemented method includes: in response to the generative AI model and the CV model and / or the OCR model agreeing on the extracted data element predictions in the multiple extracted data element predictions and the confidence score for the corresponding extracted data element prediction exceeds the corresponding acceptance threshold for the model, the RPA robot outputs the extracted element prediction and / or the extracted data element. In addition, the computer-implemented method includes: in response to the generative AI model and the CV model and / or the OCR model failing to reach a consensus on an extracted data element prediction among multiple extracted data element predictions, or the generative AI model and the CV model and / or the OCR model failing to meet or exceed their corresponding acceptance thresholds for the extracted data element predictions, the RPA robot requests human assistance to identify the correct corresponding data element. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to readily understand the advantages of certain embodiments of the present invention, the invention briefly described above will be described in more detail with reference to specific embodiments shown in the accompanying drawings. Although it should be understood that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0008] Figure 1 is an architectural diagram showing a hyper-automation system according to an embodiment of the present invention;

[0009] Figure 2 is an architectural diagram showing an RPA system according to an embodiment of the present invention;

[0010] Figure 3 is an architectural diagram showing an RPA system deployed according to an embodiment of the present invention;

[0011] Figure 4 is an architectural diagram illustrating the relationship between a designer, an activity, and a driver according to an embodiment of the present invention;

[0012] Figure 5 is an architectural diagram illustrating a computing system configured to supplement automatic information extraction using generative AI according to an embodiment of the present invention;

[0013] Fig. 6A shows an example of a neural network that has been trained to perform automatic information extraction according to an embodiment of the present invention;

[0014] Figure 6B shows an example of a neuron according to an embodiment of the present invention;

[0015] Figure 7 is a flow chart illustrating a process for training (multiple) AI / ML models according to an embodiment of the present invention;

[0016] Figure 8 is a flow chart illustrating a process for using generative AI to supplement automatic information extraction according to an embodiment of the present invention; and

[0017] Fig. 9 is a flow chart illustrating another process for using generative AI to supplement automatic information extraction according to an embodiment of the present invention.

[0018] Unless otherwise stated, similar reference numerals denote corresponding features consistently throughout the drawings. DETAILED DESCRIPTION

[0019] Some embodiments involve using generative AI to supplement automatic information extraction. The CV model and / or the OCR model and the generative AI model are used together to extract information (e.g., name, date, invoice number, etc.) from a source. In some embodiments, the OCR model can be part of the CV model.

[0020] If the confidence score of the model exceeds a specific common acceptance threshold, or a potential independent acceptance threshold for each model type (e.g., 95% for generative AI and 99% for OCR, etc.), and the CV or OCR model and the generative AI model agree on the prediction, no further verification is performed, and the elements extracted from the source will be accepted for further processing (e.g., for document understanding, performing semantic matching, etc.). If (multiple) CV models and / or OCR models and the generative AI model have different predictions for the elements, and the generative AI model is above the common threshold or its corresponding acceptance threshold, the elements predicted and extracted by the generative AI model will be used. In some embodiments, the CV model and / or OCR model can be retrained using elements extracted by the generative AI model, where the generative AI model is above the common acceptance threshold or its acceptance threshold, but the (multiple) CV model and / or OCR model does not. Alternatively, controversial elements can be submitted to manual review. In the case where elements in the source are not identified by any model as being above any threshold, these (multiple) elements can be marked for subsequent manual review, or manual work can be used in a loop to correct these (multiple) elements. The model can then be retrained using these(multiple) labeled elements.

[0021] Figure 1 is an architectural diagram showing a hyper-automation system 100 according to an embodiment of the present invention. "Hyper-automation" as used herein refers to an automation system that combines components of process automation, integrated tools, and technologies that enhance work automation capabilities. For example, in some embodiments, robotic process automation (RPA) can be used at the core of a hyper-automation system, and in certain embodiments, automation capabilities can be extended using AI / machine learning (ML), process mining, analytics, and / or other advanced tools. As the hyper-automation system learns processes, trains AI / ML models, and employs analytics, for example, more and more knowledge work can be automated, and computing systems in an organization (e.g., computing systems used by individuals and computing systems that operate autonomously) can all participate in the hyper-automation process. The hyper-automation system of some embodiments allows users and organizations to efficiently and effectively discover, understand, and extend automation.

[0022] The hyper-automated system 100 includes user computing systems such as a desktop computer 102, a tablet computer 104, and a smartphone 106. However, any desired user computing system may be used without departing from the scope of the present invention, including but not limited to smart watches, laptop computers, servers, Internet of Things (IoT) devices, etc. Figure 1 , but any suitable number of user computing systems may be used without departing from the scope of the invention. For example, in some embodiments, dozens, hundreds, thousands, or millions of user computing systems may be used. The user computing systems may be actively used by the user, or may run automatically without much or any user input.

[0023] Each user computing system 102, 104, 106 has (a plurality of) corresponding automation processes 110, 112, 114 running thereon. In some embodiments, the automation processes are stored remotely (e.g., on a server 130 or in a database 140 and accessed via a network 120) and loaded by an RPA robot to implement automation. The automation may exist as a script (e.g., Extensible Markup Language (XML), Extensible Application Markup Language (XAML), etc.), or may be compiled into machine-readable code (e.g., as a digital link library).

[0024] Without departing from the scope of the present invention, the (multiple) automated processes 110, 112, 114 may include, but are not limited to, an RPA robot, a portion of an operating system, a (multiple) downloadable application of a corresponding computing system, any other suitable software and / or hardware, or any combination of these. In some embodiments, without departing from the scope of the present invention, one or more of the (multiple) processes 110, 112, 114 may be a listener. The listener may be an RPA robot, a portion of an operating system, a downloadable application of a corresponding computing system, or any other software and / or hardware. In fact, in some embodiments, the logic of the (multiple) listeners is partially or completely implemented via physical hardware.

[0025] The listener monitors and records data related to the user's interaction with the corresponding computing system and / or the operation of the unattended computing system, and sends the data to the core hyper-automation system 120 via a network (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.). The data may include, but is not limited to, which buttons are clicked, where the mouse is moved, the text entered in the field, one window is minimized and another window is open, the application associated with the window, etc. In some embodiments, the data from the listener may be sent periodically as part of a heartbeat message. In some embodiments, the data may be sent to the core hyper-automation system 120 once a predetermined amount of data has been collected, after a predetermined period of time has passed, or both. One or more servers, such as server 130, receive the data from the listener and store it in a database, such as database 140.

[0026] The automated process can execute the logic developed in the workflow at design time. In the case of RPA, a workflow can include a set of steps (defined as "activities" in this article) that are executed in a sequence or some other logical flow. Each activity can include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0027] In some embodiments, RPA's long-running workflows are the main project that supports service orchestration, human intervention, and long-running transactions in an unattended environment. For example, see U.S. Patent No. 10,860,905, which is incorporated herein by reference in its entirety. Human intervention comes into play when certain processes require human input to handle exceptions, approvals, or verifications before entering the next step of the activity. In this case, process execution is paused, freeing up the RPA robot until the human task is completed.

[0028] Long-running workflows can support workflow fragmentation via persistent activities, and can be combined with calling processes and non-user interactive activities to orchestrate manual tasks with RPA robot tasks. In some embodiments, multiple or many computing systems may participate in the logic of executing long-running workflows. Long-running workflows can run in sessions for fast execution. In some embodiments, long-running workflows can orchestrate background processes that may include activities that execute application programming interface (API) calls and run in long-running workflow sessions. In some embodiments, these activities can be called by calling process activities. A process with user interactive activities running in a user session can be called by starting a job from a conductor activity (the conductor will be described in more detail later in this article). In some embodiments, a user can interact with tasks that require completing a form in a conductor. It may include activities that cause the RPA robot to wait for the form task to complete and then resume the long-running workflow.

[0029] One or more of the (multiple) automated processes 110, 112, 114 communicate with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 may run a guide application on one or more servers (such as server 130). Although one server 130 is shown for illustrative purposes, multiple or many servers in close proximity or distributed architectures may be employed without departing from the scope of the present invention. For example, without departing from the scope of the present invention, one or more servers may be provided for guide functions, AI / ML model services, authentication, governance, and / or any other suitable functions. In some embodiments, the core hyper-automation system 120 may be incorporated into or as part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, etc. In some embodiments, the core hyper-automation system 120 may host multiple software-based servers (such as server 130) on one or more computing systems. In some embodiments, one or more servers (such as server 130) of the core hyper-automation system 120 may be implemented via one or more virtual machines (VMs).

[0030] In some embodiments, one or more of the automated processes 110, 112, 114 may call one or more AI / ML models 132 deployed on or accessible by the core hyperautomation system 120 and trained to accomplish various tasks. For example, the AI / ML models 132 may include models trained to find various application versions, perform CV, perform OCR, generate user interface (UI) descriptors, provide suggestions for the next activity or sequence of activities in an RPA workflow, etc. The AI / ML models may be trained using labeled data including, but not limited to, elements from data sources (e.g., web pages, forms, scanned documents, application interfaces, screens, etc.), previously created RPA workflows, screenshots of various application screens of various versions and their corresponding UI elements, libraries of UI objects, etc. The AI / ML models 132 may be trained to achieve a desired confidence threshold while not overfitting a given training data set.

[0031] Without departing from the scope of the present invention, the AI / ML model 132 may be trained for any suitable purpose, which will be discussed in more detail below. In some embodiments, two or more AI / ML models 132 may be linked (e.g., in series, in parallel, or a combination thereof) so that they jointly provide (multiple) collaborative outputs. The AI / ML model 132 may perform or assist in CV, OCR, document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automatic RPA workflow generation, sequence extraction, cluster detection, audio to text conversion, any combination thereof, etc. However, without departing from the scope of the present invention, any desired number and / or (multiple) types of AI / ML models may be used. For example, using multiple AI / ML models may allow the system to develop a global picture of what is happening on a given computing system. For example, one AI / ML model may perform OCR, another AI / ML model may detect buttons, another AI / ML model may compare sequences, and so on. The pattern may be determined by one AI / ML model alone or by multiple AI / ML models together. In certain embodiments, one or more AI / ML models are deployed locally on at least one of computing systems 102 , 104 , 106 .

[0032] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) that operates on data, and for example, the AI / ML model itself may be a deep learning neural network (DLNN) of trained artificial "neurons" trained on training data. In some embodiments, the AI / ML model 132 may have multiple layers that perform various functions, such as statistical modeling (e.g., hidden Markov models (HMMs)), and utilize deep learning techniques (e.g., long short-term memory (LSTM) deep learning, encoding of previous hidden states, etc.) to perform the desired function.

[0033] In some embodiments, the hyperautomation system 100 can provide four main groups of functions: (1) discovery; (2) building automation; (3) management; and (4) engagement. In some embodiments, the automation (e.g., running on a user computing system, server, etc.) can be run by a software robot such as an RPA robot. For example, attended robots, unattended robots, and / or testing robots can be used. Attended robots work with users to assist them in completing tasks (e.g., via UiPath Assistant TM ). Unattended robots work independently of users and can run in the background without the user's knowledge. Test robots are unattended robots that are used to run test cases against applications or RPA workflows. In some embodiments, test robots can run in parallel on multiple computing systems.

[0034] The discovery functionality may discover and provide automatic recommendations for different opportunities for business process automation. Such functionality may be implemented by one or more servers, such as server 130. In some embodiments, the discovery functionality may include providing an automation hub, process mining, task mining, and / or task capture. An automation hub (e.g., UiPath Automation Hub) may include: TM ) can provide a mechanism for managing automation rollouts with visibility and control. For example, automation ideas can be crowdsourced from employees via a submission form. Feasibility and return on investment (ROI) calculations for automating these ideas can be provided, documentation for future automation can be collected, and collaboration can be provided to get faster builds from automation discoveries.

[0035] Process mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM) refers to the process of collecting and analyzing data from applications (e.g., enterprise resource planning (ERP) applications, customer relationship management (CRM) applications, email applications, call center applications, etc.) to identify which end-to-end processes exist in the organization and how to effectively automate these processes and indicate the impact of automation. This data can be collected by a listener from user computing systems 102, 104, 106 and processed by a server (such as server 130). In some embodiments, one or more AI / ML models 132 can be used for this purpose. This information can be exported to an automation hub to speed up implementation and avoid manual information transfer. The goal of process mining can be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increasing profits, improving customer satisfaction, complying with regulations and / or contracts, improving employee efficiency, and the like.

[0036] Task mining (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM ) identifies and aggregates workflows (e.g., employee workflows), and then applies AI to reveal patterns and variations in daily tasks, scoring these tasks for automation and potential savings (e.g., time and / or cost savings). One or more AI / ML models 132 can be employed to reveal repetitive task patterns in the data. Repetitive tasks that are ripe for automation can then be identified. In some embodiments, this information can initially be provided by a listener and analyzed on a server (such as server 130) of the core hyperautomation system 120. The results of task mining (e.g., XAML process data) can be exported to a process document or designer application (such as UiPath Studio TM ) to create and deploy automation faster. In some embodiments, task mining can include taking screenshots of user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements that the user is interacting with, timestamps of interactions, etc.), collecting statistics (e.g., execution time, number of actions, text entries, etc.), editing and annotating screenshots, specifying action types to be recorded, and the like.

[0037] Task capture (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM) automatically records the process of participation as the user works, or provides a framework for unattended processes. Such documents can include the desired tasks to be automated in the form of a process definition document (PDD), a framework workflow, capturing actions for each part of the process, recording user actions, and automatically generating a comprehensive workflow diagram including details of each step, Microsoft Documents, XAML files, etc. In some embodiments, build-ready workflows can be exported directly to a designer application such as UiPath Studio TM Task capture can streamline the requirements gathering process for subject matter experts who interpret the process and Center of Excellence (CoE) members who provide production-grade automation.

[0038] Building automation can be applied via a designer (e.g., UiPath Studio TM 、UiPath StudioX TM or UiPath Web TM ) implementation. For example, RPA developers of RPA development facility 150 can use RPA designer application 154 of computing system 152 to build and test automation for various applications and environments, such as web, mobile, and virtual desktops. API integration can be provided for a variety of applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and a workflow recorder make automation easier with minimal coding. Document understanding capabilities can be provided through drag-and-drop AI skills for data extraction and interpretation that invoke one or more AI / ML models 132. This automation can handle nearly any document type and format, including forms, checkboxes, signatures, and handwriting. When data is validated or exceptions are handled, this information can be used to retrain the corresponding AI / ML model to improve its accuracy over time.

[0039] The RPA designer application 152 can be designed to call one or more trained AI / ML models 132 on the server 130 and / or a generative AI model 172 in a cloud environment via a network 120 (e.g., a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.) to assist the RPA automation development process. In some embodiments, one or more of the AI / ML models can be packaged with the RPA designer application 152 or otherwise stored locally on the computing system 150.

[0040] In some embodiments, the RPA designer application 152 and one or more of the AI / ML models 132 may be configured to use an object library stored in the database 140. The object library may include a UI object library that can be used to develop RPA workflows via the RPA designer application 152. The object library can be used to add UI descriptors to activities in the workflow of the RPA designer application 152 for UI automation. In some embodiments, one or more of the AI / ML models 132 can generate new UI descriptors and add them to the object library in the database 140. Once the automation is completed in the designer application 152, they can be published on the server 130, pushed out to the computing systems 102, 104, 106, and so on.

[0041] For example, integration services can allow developers to seamlessly combine UI automation with API automation. Automations can be built that require APIs or traverse both API and non-API applications and systems. Repositories for pre-built RPA and AI templates and solutions can be provided (e.g., UiPath Object Repository TM ) or a marketplace (e.g., UiPathMarketplace TM ) to allow developers to automate various processes faster. Therefore, when building automation, the hyper-automation system 100 can provide a user interface, development environment, API integration, pre-built and / or customized AI / ML models, development templates, integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyper-automation system 100 enables the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, which can provide automation for the hyper-automation system 100.

[0042] In some embodiments, components of the hyperautomation system 100 (such as (multiple) designer applications and / or external rule engines) provide support for managing and enforcing governance policies that are used to control various functions provided by the hyperautomation system 100. Governance is the ability of an organization to establish policies to prevent users from developing automations (e.g., RPA robots) that are capable of taking actions that could harm the organization, such as violating the EU General Data Protection Regulation (GDPR), the U.S. Health Insurance Portability and Accountability Act (HIPAA), third-party application terms of service, etc. Because developers may create automations that violate privacy laws, terms of service, etc. when executing their automations, some embodiments implement access control and governance restrictions at the robot and / or robot design application level. In some embodiments, this can provide an additional level of security and compliance for the automation process development pipeline by preventing developers from relying on unapproved software libraries that may introduce security risks or work in a manner that violates policies, regulations, privacy laws, and / or privacy policies. For example, see U.S. Patent Application Publication No. 2022 / 0011732, the entire contents of which are incorporated herein by reference.

[0043] The management functions may provide management, deployment, and optimization of automation throughout the organization. In some embodiments, the management functions may include orchestration, test management, AI capabilities, and / or insights. The management functions of the hyper-automation system 100 may also serve as an integration point with third-party solutions and applications for automation applications and / or RPA robots. The management capabilities of the hyper-automation system 100 may include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots.

[0044] Orchestrator applications, such as UiPath Orchestrator TM (In some embodiments, this can be used as a UiPath Automation Cloud TM as part of a , either locally, in a virtual machine, in a private or public cloud, on Linux TM In a VM or via UiPath Automation Suite TM Delivered as a cloud-native single-container suite), it provides orchestration capabilities for deploying, monitoring, optimizing, scaling, and securing RPA robot deployments. Test suites (e.g., UiPathTest Suite TM ) can provide test management to monitor the quality of deployed automation. Test suites can facilitate test planning and execution, requirements fulfillment, and defect traceability. Test suites can include comprehensive test reporting.

[0045] Analytics software (e.g., UiPath InsightsTM ) can track, measure, and manage the performance of deployed automation. Analytics software can align automated actions with specific key performance indicators (KPIs) and strategic outcomes for the organization. Analytics software can present results in a dashboard format for better understanding by human users.

[0046] For example, data services (e.g., UiPath Data Service TM ) can be stored in database 140 and brought to a single, scalable, secure place through a drag-and-drop storage interface. Some embodiments can provide low-code or no-code data modeling and storage for automation while ensuring seamless access to data, enterprise-grade security, and scalability. AI capabilities can be powered by an AI center (e.g., UiPath AI Center TM ) is provided, which facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make these capabilities accessible even to non-data scientists. Deployed automation (e.g., RPA robots) can call AI / ML models, such as AI / ML model 132, from the AI ​​center. The performance of the AI / ML model can be monitored, trained, and improved using human validation data, such as provided by the data review center 160. A human reviewer can provide labeled data to the core hyper-automation system 120 via the review application 152 on the computing system 154. For example, a human reviewer can verify that the predictions made by the AI / ML model 132 and / or the generative AI model 172 are accurate or otherwise provide corrections. This dynamic input can then be saved as training data for retraining the AI / ML model 132 and / or the generative AI model 172, and can be stored in a database such as the database 140. The AI ​​center can then schedule and execute training jobs to train a new version of the AI / ML model using the training data. Both positive and negative examples may be stored and used for retraining of the AI / ML model 132 and / or the generative AI model 172 .

[0047] Engagement capabilities bring humans and automation together as a team to seamlessly collaborate on desired processes. You can build low-code applications (e.g., via UiPath Apps TM ) to connect browser tabs and traditional software, even in the absence of an API in some embodiments. For example, applications can be quickly created using a web browser through a rich library of drag-and-drop controls. An application can be connected to a single automation or multiple automations.

[0048] Action Center (for example, UiPath Action Center TM) provides a simple and efficient mechanism to hand over automated processes to humans and vice versa. Humans can provide approvals or escalations, make exceptions, etc. Automation can then perform the automated functions of a given workflow.

[0049] A local assistant can be provided as a launchpad for users to start automation (e.g., UiPath Assistant TM ). This functionality may be provided in a tray provided, for example, by an operating system, and may allow a user to interact with RPA robots and RPA robot-driven applications on their computing system. The interface may list automations approved for a given user and allow the user to run them. These may include off-the-shelf automations from an automation marketplace, an internal automation shop in an automation center, and the like. When automations run, they may run as local instances in parallel with other processes on the computing system, so that the user can use the computing system while the automation performs its actions. In some embodiments, the assistant is integrated with a task capture function, so that a user can record their processes that are about to be automated from the assistant launchpad.

[0050] Chatbots (e.g., UiPath Chatbots TM ), social messaging apps, and / or voice commands can enable users to run automations. This can simplify access to the information, tools, and resources that users need to interact with customers or perform other activities. Conversations between people can be easily automated, as can other processes. Triggered RPA robots launched in this way can perform actions such as checking order status, posting data in a CRM, etc., possibly using plain language commands.

[0051] In some embodiments, the hyperautomation system 100 can provide end-to-end measurement and governance of automation programs of any size. In accordance with the above, analytics can be used to understand the performance of automation (e.g., via UiPathInsights TM ). Data modeling and analysis using any combination of available business metrics and operational insights can be used for a variety of automation processes. Custom designed and pre-built dashboards allow visualization of data across desired metrics, discovery of new analytical insights, tracking of performance metrics, discovery of ROI for automation, performance monitoring on user computing systems, detection of errors and anomalies, and debugging of automations. Automation management consoles (e.g., UiPath Automation Ops) can be provided TM ) to manage automations throughout the automation lifecycle. An organization can govern how automations are built, what users can do with them, and which automations users can access.

[0052] In some embodiments, the hyperautomation system 100 provides an iterative platform. Processes can be discovered, automation can be built, tested, and deployed, performance can be measured, use of automation can be easily provided to users, feedback can be captured, AI / ML models can be trained and retrained, and processes can repeat themselves. This helps achieve a more robust and effective automation suite.

[0053] In some embodiments, generative AI models are used. Generative AI can generate various types of content, such as text, images, audio, and synthetic data. Various types of generative AI models can be used, including but not limited to large language models (LLMs), generative adversarial networks (GANs), variational autoencoders (VAEs), transformers, etc. These models can be part of the AI / ML model 132 hosted on the server 130. For example, a generative AI model can be trained on a large corpus of textual information to perform semantic understanding, understand the nature of the content presented on the screen from the text, and so on. In some embodiments, the generative AI model is provided by an existing cloud ML service provider such as etc.) can be used and trained to provide such functionality. In a generative AI embodiment where (multiple) generative AI models 172 are remotely hosted, server 130 can be configured to integrate with a third-party API that allows server 130 to send requests including necessary input information to (multiple) generative AI models 172 and receive responses in return (e.g., semantic matching of fields between application versions, classification of application types on a screen, etc.). Such embodiments can provide a more advanced and sophisticated user experience, as well as provide access to existing natural language processing (NLP) and other ML capabilities provided by these companies.

[0054] In some embodiments, one aspect of generative AI models is the use of transfer learning. In transfer learning, a pre-trained generative AI model (such as an LLM) is fine-tuned on a specific task or domain. This enables the LLM to leverage knowledge already learned during its initial training and adapt it to a specific application. In the case of an LLM, the pre-training phase involves training the LLM on a large text corpus, typically consisting of billions of words. In this phase, the LLM learns the relationships between words and phrases, which enables the LLM to generate coherent, human-like responses to text-based inputs. The output of this pre-training phase is an LLM that has a high level of understanding of the basic patterns of natural language.

[0055] In the fine-tuning phase, the pre-trained LLM is adapted to a specific task or domain by training the LLM on a smaller dataset specific to the task. For example, in some embodiments, the LLM can be trained to analyze a certain type or multiple types of data sources to improve its accuracy about their contents. Such information can be provided as part of the training data, and the LLM can learn to focus on these areas and more accurately identify the data elements therein. Fine-tuning allows the LLM to learn the nuances of the task or domain, such as the specific vocabulary and grammar used in that domain, without requiring as much data as would be required if the LLM were trained from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned LLM can achieve the performance of the state of the art on a specific task with a relatively small amount of training data.

[0056] Figure 2 2 is an architecture diagram showing an RPA system 200 according to an embodiment of the present invention. Figure 1 The RPA system 200 includes a designer 210 that allows developers to design and implement workflows. The designer 210 can provide solutions for application integration and automation of third-party applications, management of information technology (IT) tasks, and business IT processes. The designer 210 can facilitate the development of automation projects, which are graphical representations of business processes. Simply put, the designer 210 facilitates the development and deployment of workflows and robots. In some embodiments, the designer 210 can be an application running on a user's desktop, an application running remotely in a VM, a web application, etc.

[0057] Automation projects enable automation of rule-based processes by giving developers control over the order of execution and the relationships between sets of custom steps (defined herein as “activities” above) developed in a workflow. A commercial example of an embodiment of the designer 210 is UiPath Studio TM Each activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0058] Some types of workflows may include, but are not limited to, sequences, flow charts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly useful for linear processes, thereby enabling flow from one activity to another without confusing the workflow. Flow charts may be particularly useful for more complex business logic, enabling decision integration and activity connections in a more diverse manner through multiple branching logic operators. FSMs may be particularly useful for large workflows. FSMs may use a limited number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly useful for determining workflow behavior and debugging processes when encountering execution errors.

[0059] Once a workflow is developed in the designer 210, the execution of the business process is orchestrated by the director 220, which orchestrates one or more robots 230 that execute the workflow developed in the designer 210. One commercial example of an embodiment of the director 220 is the UiPath Orchestrator TM Director 220 facilitates the creation, monitoring, and deployment of resources in the management environment. Director 220 may serve as an integration point with third-party solutions and applications. As described above, in some embodiments, Director 220 may be Figure 1 Part of the core hyper-automation system 120.

[0060] The guide 220 can manage a fleet of robots 230, connecting and executing robots 230 from a centralized point. The types of robots 230 that can be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234, but for development and testing purposes), and non-production robots (similar to attended robots 232, but for development and testing purposes). The attended robots 232 are triggered by user events and operate with people on the same computing system. The attended robots 232 can be used together with the guide 220 for centralized process deployment and logging media. The attended robots 232 can help manual users complete various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the guide 220 on this type of robot, and / or they cannot be run under a lock screen. In some embodiments, the attended robots 232 can only be started from the robot tray or from a command prompt. In some embodiments, the attended robots 232 should be run under manual supervision.

[0061] Unattended robots 234 run unattended in a virtual environment and can automate many processes. Unattended robots 234 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging of all robot types can be run in the designer 210. Both attended and unattended robots can automate various systems and applications, including but not limited to mainframes, web applications, VMs, enterprise applications (e.g., etc.) and computing system applications (e.g., desktop and notebook computer applications, mobile device applications, wearable computer applications, etc.).

[0062] The director 220 can have various capabilities, including, but not limited to, provisioning, deploying, configuring, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning can include creating and maintaining connections between the robots 230 and the director 220 (e.g., web applications). Deployment can include ensuring that branch versions are properly delivered to assigned robots 230 for execution. Configuration can include maintenance and delivery of robot environment and process configurations. Queuing can include providing management of queues and queue items. Monitoring can include tracking robot identification data and maintaining user permissions. Logging can include storing logs to a database (e.g., a structured query language (SQL) database or a "non-unique" SQL (NoSQL) database) and / or other storage mechanisms (e.g., a SQL database that provides the ability to store and quickly query large data sets). ) and indexing. Director 220 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0063] Robot 230 is an execution agent that implements the workflow constructed in designer 210. One commercial example of some embodiments of robot(s) 230 is UiPath Robots TM In some embodiments, the robot 230 installs Microsoft Services managed by the Service Control Manager (SCM). Therefore, such a robot 230 can open interactive session, and has Permissions for services.

[0064] In some embodiments, robots 230 can be installed in user mode. For such robots 230, this means that they have the same rights as the user who installed the given robot 230. This feature can also be used for high density (HD) robots to ensure that the maximum potential of each machine is fully utilized. In some embodiments, any type of robot 230 can be configured in an HD environment.

[0065] In some embodiments, the robot 230 is split into several components, each dedicated to a specific automation task. In some embodiments, the robot components include but are not limited to SCM-managed robot services, user mode robot services, executors, agents, and command lines. SCM-managed robot service management and monitoring The SCM starts a console application under the local system.

[0066] In some embodiments, user-mode robot services manage and monitor session and acts as a proxy between the boot 220 and the execution host. The user mode robot service can be trusted and manage the credentials of the robot 230. If the SCM managed robot service is not installed, then Applications can be started automatically.

[0067] The actuator can be The executor can be aware of the dots per inch (DPI) setting of each monitor. The agent can be a window that displays available jobs in the system tray. A Windows Presentation Foundation (WPF) application. An agent can be a client of a service. An agent can request to start or stop a job and change settings. A command line is a client of a service. A command line is a console application that can request to start a job and wait for its output.

[0068] As described above, splitting the components of the robot 230 helps developers, support users, and computing systems to more easily run, identify, and track what each component is executing. Special behaviors can be configured for each component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting of each monitor. Therefore, the workflow can be executed at any DPI, regardless of the configuration of the computing system on which the workflow is created. In some embodiments, the project from the designer 210 can also be independent of the browser zoom level. In some embodiments, DPI can be disabled for applications that are DPI unaware or deliberately marked as unaware.

[0069] The RPA system 200 in this embodiment is part of a hyper-automation system. Developers can use the designer 210 to build and test RPA robots that utilize AI / ML models deployed in the core hyper-automation system 240 (e.g., as part of its AI center). Such RPA robots can send inputs for executing (multiple) AI / ML models and receive outputs therefrom via the core hyper-automation system 240.

[0070] As described above, one or more robots 230 may be listeners. These listeners may provide information to the core hyper-automation system 240 about what users are doing when using their computing systems. This information may then be used by the core hyper-automation system for process mining, task mining, task capture, and the like.

[0071] An assistant / chatbot 250 may be provided on the user computing system to allow the user to launch an RPA local robot. For example, the assistant may be located in the system tray. The chatbot may have a user interface so that the user can see the text in the chatbot. Alternatively, the chatbot may not have a user interface but run in the background to listen to the user's voice using the computing system's microphone.

[0072] In some embodiments, data labeling can be performed by a user of the computing system on which the robot is executing, or on another computing system to which the robot provides information. For example, if the robot calls an AI / ML model that performs CV on an image of a VM user, but the AI / ML model does not correctly identify a button on the screen, the user can draw a rectangle around the incorrectly identified or unidentified component and potentially provide text with the correct identification. This information can be provided to the core hyperautomation system 240 and then later used to train a new version of the AI / ML model.

[0073] Figure 3 is an architectural diagram showing an RPA system 300 deployed according to an embodiment of the present invention. In some embodiments, the RPA system 300 may be Figure 2 RPA system 200 and / or Figure 1 The deployed RPA system 300 may be a cloud-based system, a local system, a desktop-based system that provides enterprise-level, user-level, or device-level automation solutions for different computing process automation, etc.

[0074] It should be noted that the client side, the server side, or both may include any desired number of computing systems without departing from the scope of the present invention. On the client side, the robot application 310 includes an executor 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not be running on the same computing system as the executor 312 and the agent 314. The executor 312 is running a process. Multiple business projects can be running simultaneously, such as Figure 3 Agent 314 (e.g., The Executor 312 service is the single point of contact for all executors 312 in this embodiment. All messages in this embodiment are logged to the Director 340, which further processes the messages via the database server 350, AI / ML server 360, indexer server 370, or any combination thereof. Figure 2 As discussed, the actuator 312 may be a robotic component.

[0075] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. On a computing system that supports running multiple interactive sessions simultaneously (e.g., Server 2012), multiple robots can run simultaneously, each with a unique username in a separate This is the HD robot mentioned above.

[0076] The agent 314 is also responsible for sending the status of the robot (e.g., periodically sending a "heartbeat" message indicating that the robot is still running) and downloading the desired version of the package to be executed. In some embodiments, communication between the agent 314 and the director 340 is always initiated by the agent 314. In the notification scenario, the agent 314 can open a WebSocket channel that is later used by the director 340 to send commands to the robot (e.g., start, stop, etc.).

[0077] The listener 330 monitors and records data related to the user's interaction with the attended computing system and / or the operation of the unattended computing system in which the listener 330 resides. Without departing from the scope of the present invention, the listener 330 may be an RPA robot, part of an operating system, a downloadable application for a corresponding computing system, or any other software and / or hardware. In fact, in some embodiments, the logic of the listener is partially or completely implemented via physical hardware.

[0078] On the server side, it includes a presentation layer (web application 342, open data protocol (OData) representational state transfer (REST) ​​application programming interface (API) endpoint 344, and notification and monitoring 346), a service layer (API implementation / business logic 348), and a persistence layer (database server 350, AI / ML server 360, and indexer server 370). The director 340 includes a web application 342, an OData REST API endpoint 344, notification and monitoring 346, and an API implementation / business logic 348. In some embodiments, most actions performed by a user in the interface of the director 340 (e.g., via a browser 320) are performed by calling various APIs. Without departing from the scope of the present invention, such actions may include, but are not limited to, starting a job on a robot, adding / deleting data in a queue, scheduling a job to run unattended, etc. The web application 342 is the visual layer of the server platform. In this embodiment, the web application 342 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, without departing from the scope of the present invention, any desired markup language, scripting language, or any other format may be used. In this embodiment, the user interacts with a web page from web application 342 via browser 320 to perform various actions to control director 340. For example, the user can create robot groups, assign groups to robots, analyze logs per robot and / or per process, start and stop robots, and so on.

[0079] In addition to the web application 342, the director 340 also includes a service layer that exposes an OData REST API endpoint 344. However, other endpoints may be included without departing from the scope of the present invention. The REST API is consumed by both the web application 342 and the agent 314. In this embodiment, the agent 314 is a supervisor of one or more robots on a client computer.

[0080] The REST API in this embodiment includes configuration, logging, monitoring and queuing functions. In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint can be used to log different information, such as errors, explicit messages sent by the robot, and other environment-specific information. If the start job command is used in the director 340, the robot can use the deployment REST endpoint to query the grouped version that should be executed. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, and so on.

[0081] The monitoring REST endpoint can monitor the web application 342 and the agent 314. The notification and monitoring API 346 can be a REST endpoint for registering the agent 314, delivering configuration settings to the agent 314, and for sending / receiving notifications from the server and the agent 314. In some embodiments, the notification and monitoring API 346 can also use WebSocket communication.

[0082] In some embodiments, the API in the service layer can be accessed by configuring an appropriate API access path, for example, based on whether the director 340 and the entire hyper-automation system have an on-premises deployment type or a cloud-based deployment type. The API for the director 340 may provide customized methods for querying statistical information about various entities registered in the director 340. In some embodiments, each logical resource may be an OData entity. In such an entity, components such as robots, processes, queues, etc. may have attributes, relationships, and operations. In some embodiments, the web application 342 and / or the agent 314 may consume the API of the director 340 in two ways: by obtaining API access information from the director 340, or by registering an external application to use the OAuth flow.

[0083] In this embodiment, the persistence layer includes three servers - a database server 350 (e.g., a SQL server), an AI / ML server 360 (e.g., a server providing AI / ML model services, such as an AI center function), and an indexer server 370. The database server 350 in this embodiment stores configurations of robots, robot groups, related processes, users, roles, schedules, etc. In some embodiments, this information is managed by a web application 342. The database server 350 can manage queues and queue items. In some embodiments, the database server 350 can store messages logged by the robot log (in addition to or instead of the indexer server 370). The database server 350 can also store process mining, task mining, and / or task capture related data, such as received from a listener 330 installed on the client side. Although no arrow is shown between the listener 330 and the database 350, it should be understood that in some embodiments, the listener 330 can communicate with the database 350, and vice versa. The data can be stored in the form of a PDD, an image, a XAML file, etc. Listener 330 may be configured to intercept user actions, processes, tasks, and performance metrics on the corresponding computing system where listener 330 is located. For example, listener 330 may record user actions (e.g., clicks, typed characters, locations, applications, active elements, time, etc.) on its corresponding computing system and then convert these actions into a suitable format to be provided to and stored in database server 350.

[0084] The AI / ML server 360 facilitates the incorporation of AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options may make these capabilities accessible even to non-data scientists. Deployed automation devices (e.g., RPA robots) can call AI / ML models from the AI / ML server 360. The performance of AI / ML models can be monitored and trained and improved using manually verified data. The AI / ML server 360 can schedule and execute training jobs to train new versions of AI / ML models.

[0085] The AI / ML server 360 may store data related to AI / ML models and ML groups for configuring various ML skills for users at the time of development. ML skills as used herein are pre-built and trained ML models for processes, for example, which may be used by automation. The AI / ML server 360 may also store data related to document understanding techniques and frameworks, algorithms, and software packages for various AI / ML capabilities, including but not limited to intent analysis, NLP, speech analysis, different types of AI / ML models, and the like.

[0086] Indexer server 370 (which is optional in some embodiments) stores and indexes information logged by the robot logs. In some embodiments, indexer server 370 can be disabled through configuration settings. In some embodiments, indexer server 370 uses (which is an open source project full-text search engine.) Messages logged by the robot (e.g., using activities such as logging messages or writing lines) can be sent to the indexer server 370 through the logging REST endpoint(s), where they are indexed for future use.

[0087] Figure 44 is an architectural diagram showing the relationship 400 between the designer 410, activities 420, 430, 440, 450, driver 460, API 470 and AI / ML model 480 according to an embodiment of the present invention. According to the above content, the developer uses the designer 410 to develop a workflow performed by the robot. In some embodiments, various types of activities can be displayed to the developer. The designer 410 can be local or remote to the user's computing system (for example, accessed via a VM or a local web browser interacting with a remote web server). The workflow can include user-defined activities 420, API-driven activities 430, AI / ML activities 440 and / or UI automation activities 450. User-defined activities 420 and API-driven activities 440 interact with the application via their APIs. In some embodiments, user-defined activities 420 and / or AI / ML activities 440 can call one or more AI / ML models 480, which can be located locally and / or remotely from the computing system on which the robot operates.

[0088] Some embodiments are capable of identifying non-textual visual components in images, referred to herein as CV. However, it should be noted that in some embodiments, CV is combined with OCR. CV can be performed at least in part by (multiple) AI / ML models 480. Some CV activities associated with such components may include, but are not limited to, extracting text from segmented label data using OCR, fuzzy text matching, using ML to crop segmented label data, comparing text extracted from label data with basic fact data, and the like. In some embodiments, hundreds or even thousands of activities can be implemented in user-defined activities 420. However, without departing from the scope of the present invention, any number and / or type of activities may be used.

[0089] UI automation activities 450 are a subset of special lower-level activities written in lower-level code and facilitating interaction with the screen. UI automation activities 450 facilitate these interactions via drivers 460, which allow the robot to interact with the desired software. For example, drivers 460 may include operating system (OS) drivers 462, browser drivers 464, VM ​​drivers 466, enterprise application drivers 468, etc. In some embodiments, one or more of the AI / ML models 480 may be used by UI automation activities 450 to perform interactions with the computing system. In some embodiments, AI / ML models 480 may enhance drivers 460 or replace them completely. In fact, in some embodiments, drivers 460 are not included.

[0090] Driver 460 can interact with the OS at a low level via OS driver 462 to find hooks, monitor keys, etc. Driver 460 can facilitate communication with For example, the "click" activity performs the same role in these different applications via the driver 460.

[0091] Figure 5 is an architectural diagram illustrating a computing system 500 configured to use generative AI to supplement automatic information extraction according to an embodiment of the present invention. In some embodiments, computing system 500 may be one or more of the computing systems depicted and / or described herein. In some embodiments, computing system 500 may be a computing system such as Figure 1 and Figure 2 Part of the hyper-automation system shown. Computing system 500 includes a bus 505 or other communication mechanism for transmitting information, and (multiple) processors 510 coupled to bus 505 for processing information. (Multiple) processors 510 can be any type of general or special processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. (Multiple) processors 510 can also have multiple processing cores, and at least some of these cores can be configured to perform specific functions. In some embodiments, multi-parallel processing can be used. In some embodiments, at least one of (multiple) processors 510 can be a neuromorphic circuit including a processing element that simulates a biological neuron. In some embodiments, the neuromorphic circuit may not require typical components of the von Neumann computing architecture.

[0092] The computing system 500 also includes a memory 515 for storing information and instructions to be executed by the (multiple) processors 510. The memory 515 may include random access memory (RAM), read-only memory (ROM), flash memory, cache, static memory such as a magnetic disk or optical disk, or any other type of non-transitory computer-readable medium or any combination of combinations thereof. The non-transitory computer-readable medium may be any available medium that can be accessed by the (multiple) processors 510, and may include volatile media, non-volatile media, or both. The medium may also be removable, non-removable, or both. The computing system 500 includes a communication device 520, such as a transceiver, to provide access to a communication network via a wireless and / or wired connection. In some embodiments, without departing from the scope of the present invention, the communication device 520 may include one or more antennas that are singular, arrayed, phased, switched, beamformed, beam-steered, combinations thereof, and / or any other antenna configuration.

[0093] Processor(s) 510 are also coupled to display 525 via bus 505. Any suitable display device and tactile I / O may be used without departing from the scope of the present invention. Keyboard 530 and cursor control device 535 (such as a computer mouse, touchpad, etc.) are further coupled to bus 505 to enable a user to interface with computing system 500. However, in some embodiments, there may be no physical keyboard and mouse, and the user may interact with the device only through display 525 and / or touchpad (not shown). Any type and combination of input devices may be used as a matter of design choice. In some embodiments, there are no physical input devices and / or displays. For example, a user may interact with computing system 500 remotely via another computing system communicating therewith, or computing system 500 may operate autonomously.

[0094] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. These modules include an operating system 540 for the computing system 500. The modules also include an automatic information extraction model 545 that is configured to perform all or part of the AI / ML processes described herein or their derivatives. The computing system 500 may include one or more additional functional modules 550 that include additional functionality.

[0095] Those skilled in the art will appreciate that, without departing from the scope of the present invention, a "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device, or a combination of devices. Presenting the above functions as being performed by a "system" is not intended to limit the scope of the present invention in any way, but is intended to provide an example of many embodiments of the present invention. In fact, the methods, systems, and devices disclosed herein can be implemented in a localized and distributed form consistent with computing technology, including cloud computing systems. The computing system can be part of or otherwise accessible by a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, etc. Without departing from the scope of the present invention, any localized or distributed architecture can be used.

[0096] It should be noted that some of the system features described in this specification have been presented as modules in order to more particularly emphasize their implementation independence. For example, a module can be implemented as a hardware circuit that includes a custom very large scale integrated circuit (VLSI) circuit or gate array, an off-the-shelf semi-initiator such as a logic chip, a transistor, or other discrete components. A module can also be implemented in a programmable hardware device such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, etc.

[0097] Module can also be implemented at least in part in software for various types of processor execution.Identified executable code unit can for example include one or more physical or logical blocks of computer instructions, and this computer instruction can for example be organized as object, process or function.However, the executable program of identified module does not need to be physically located together, but can include different instructions stored in different locations, and these instructions include this module and realize the above-mentioned purpose of this module when being logically linked together.In addition, without departing from the scope of the present invention, module can be stored on computer readable medium, and this computer readable medium can be for example hard disk driver, flash memory device, RAM, magnetic tape and / or any other such non-transient computer readable medium for storing data.

[0098] In fact, a module of executable code can be a single instruction or multiple instructions, and can even be distributed across several different code segments, between different programs, and between several memory devices. Similarly, operational data can be identified and shown within a module, and can be embodied in any suitable form and organized in any suitable type of data structure. The operational data can be collected as a single data set, or can be distributed across different locations, including on different storage devices, and can exist at least in part only as electronic signals on a system or network.

[0099] Various types of AI / ML models may be trained and deployed without departing from the scope of the present invention. For example, Fig. 6A An example of a neural network 600 that has been trained to perform automatic information extraction according to an embodiment of the present invention (e.g., a CV model, an OCR model, a generative AI model, etc.) is shown. The neural network 600 includes multiple hidden layers. Both DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, although in some cases, SLNNs may have only one or two layers, and are typically less than DLNNs. Typically, a neural network architecture includes an input layer, multiple intermediate layers, and an output layer, as in the case of neural network 600.

[0100] DLNNs typically have many layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse features from previous layers to compute more complex general functions. On the other hand, SLNNs tend to have only a few layers and are relatively fast to train because expert features are created in advance from raw data samples. However, feature extraction is laborious. On the other hand, DLNNs typically do not require expert features, but tend to take longer to train and have more layers.

[0101] For both approaches, the layers are trained simultaneously on a training set, usually with a check for overfitting on an isolated cross-validation set. Both techniques produce excellent results, and there is considerable enthusiasm for both approaches. The optimal size, shape, and number of individual layers depends on the problem being solved by the neural network.

[0102] Back to Fig. 6A , text, screenshots, graphic elements, contextual information (e.g., information about the application to which the source belongs, the type of source, etc.) are provided as input layers and fed as inputs to the J neurons of the hidden layer 1. Although all of these inputs are fed to each neuron in this example, various architectures may be used alone or in combination without departing from the scope of the invention, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graph networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional layer networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks.

[0103] Hidden layer 2 receives input from hidden layer 1, hidden layer 3 receives input from hidden layer 2, and so on, until the last hidden layer provides its output as input to the output layer. Although multiple suggestions are shown here as outputs, in some embodiments, only a single output suggestion is provided. In some embodiments, the suggestions are ranked based on confidence scores.

[0104] It should be noted that the number of neurons I, J, K, and L is not necessarily equal. Thus, without departing from the scope of the present invention, any desired number of layers may be used for a given layer of the neural network 600. In fact, in some embodiments, the types of neurons in a given layer may not all be the same.

[0105] The neural network 600 is trained to assign confidence scores to the appropriate outputs. To reduce inaccurate predictions, in some embodiments, only those results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, then outputs with confidence scores exceeding that amount may be used, and the rest may be ignored.

[0106] It should be noted that neural networks are probabilistic structures that typically have (multiple) confidence scores. This can be a score learned by the AI / ML model based on the frequency of correctly identifying similar inputs during training. Some common types of confidence scores include decimal numbers between 0 and 1 (which can also be interpreted as confidence percentages), numbers between negative ∞ and positive ∞, a set of expressions (e.g., "low", "medium", and "high"), etc. In order to obtain more accurate confidence scores, various post-processing calibration techniques can also be used, such as temperature scaling, batch normalization, weight decay, negative log-likelihood (NLL), etc.

[0107] A "neuron" in a neural network is algorithmically implemented as a mathematical function that is typically based on the functionality of a biological neuron. Neurons receive weighted inputs and have a summation and activation function that governs whether they pass the output to the next layer. The activation function can be a nonlinear threshold activity function, where nothing happens if the value is below the threshold, but then the function responds linearly above the threshold (i.e., rectified linear unit (ReLU) nonlinearity). The summation function and the ReLU function are used for deep learning because real neurons can have approximately similar activity functions. Via linear transformations, information can be subtracted, added, etc. In essence, the neuron acts as a gating function that passes the output to the next layer, which is governed by its underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.

[0108] exist Figure 6B An example of a neuron 610 is shown in FIG. Input x1, x2, ..., x from the previous layer n are assigned corresponding weights w1, w2, ..., w n . Therefore, the collective input from the previous neuron 1 is w1x1. These weighted inputs are used in the summation function of the neuron modified by the bias, such as:

[0109]

[0110] This sum is compared to the activation function f(x) to determine whether the neuron "fires". For example, f(x) can be given by:

[0111]

[0112] Therefore, the output y of neuron 610 can be given by:

[0113]

[0114] In this case, neuron 610 is a single layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of the present invention. It should also be noted that in some embodiments, the range of values ​​of the weights and / or output values ​​of the activation function may be different without departing from the scope of the present invention.

[0115] An objective or "reward function" is typically used. The reward function explores intermediate transitions and steps with short-term and long-term rewards to guide the search of the state space and attempt to achieve the goal (e.g., finding the most accurate answer to a user query based on a relevant metric). During training, various labeled data are fed through the neural network 600. Successful identifications strengthen the weight of the neuron's input, while unsuccessful identifications weaken that weight. A cost function (such as mean squared error (MSE) or gradient descent) can be used to penalize slightly wrong predictions much less than very wrong predictions. If the performance of the AI / ML model does not improve after a certain number of training iterations, the data scientist can modify the reward function, provide corrections for incorrect predictions, and so on.

[0116] Backpropagation is a technique used to optimize synaptic weights in feed-forward neural networks. Backpropagation can be used to "uncover" the hidden layers of a neural network to see how much loss each node is responsible for, and then update the weights so that the loss is minimized by giving lower weights to nodes with higher error rates, and vice versa. In other words, backpropagation allows data scientists to repeatedly adjust the weights to minimize the difference between the actual output and the desired output.

[0117] The back-propagation algorithm is built on the mathematical foundations of optimization theory. In supervised learning, training data with known outputs is passed through a neural network and the error is calculated using a cost function with known target outputs, which provides the error for back-propagation. The error is calculated at the output and transformed into a correction to the network weights to minimize the error.

[0118] In the case of supervised learning, an example of backpropagation is provided below. A series of N nonlinear activation functions f between each layer i=1, ..., N of the network i To process the column vector input x, the output of a given layer is first multiplied by the synaptic matrix W i , and add the bias vector b i The network output o is given by

[0119] o=f N (WN f N-1 (W N-1 f N-2 (...f1(W1x+b1)...)+b N-1 )+b N ) (4)

[0121] In some embodiments, o is compared to the target output t to obtain the error that is desired to be minimized.

[0122] Optimization in the form of a gradient descent procedure can be used to modify the synaptic weights W of each layer by i to minimize the error. The gradient descent process requires computing an output o given an input x corresponding to a known target output t, and produces an error ot. This global error is then propagated backward to give a local error for the weight update, whose computation is similar but not identical to that used for the forward propagation. In particular, the backpropagation step typically requires a formula of the form p j (n j ) = f′ j (n j ) activity function, where n j is the network activity at layer j (i.e., n j =W j o j-1 +b j ), where o j =f j (n j ), and the prime symbol ' denotes the derivative of the activity function f.

[0123] The weight update can be calculated using the following formula:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129] in represents the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, o j represents f j (W j o j-1 +b j), where o0 = x. Here, the learning rate η is chosen based on machine learning considerations. Below, η is related to the neural Hebbian learning mechanism used in the neural implementation. Note that the synapses W and b can be combined into a large synaptic matrix, where the input vector is assumed to have 1s appended, and an extra column representing b synapses is included in W.

[0130] The AI / ML model may be trained over multiple epochs until it reaches a good level of accuracy (e.g., 97% or better using an F2 or F4 threshold for detection and approximately 2000 epochs). Without departing from the scope of the invention, in some embodiments, this level of accuracy may be determined using an F1 score, an F2 score, an F4 score, or any other suitable technique. Once trained on the training data, the AI / ML model may be tested on a set of evaluation data that the AI / ML model has not previously encountered. This helps ensure that the AI / ML model does not "overfit," causing it to perform well on the training data but poorly on other data.

[0131] In some embodiments, it may not be known what level of accuracy the AI / ML model can achieve. Therefore, if the accuracy of the AI / ML model begins to decline when analyzing the evaluation data (i.e., the model performs well on the training data but performs poorly on the evaluation data), the AI / ML model may undergo more training epochs on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only when the accuracy reaches a certain level or the accuracy of the trained AI / ML model is better than the existing deployed AI / ML model. In certain embodiments, a collection of trained AI / ML models can be used to complete the task. For example, one model may be trained to perform CV, another model may be trained to perform OCR, another model may be a generative AI model that identifies data elements in a source, and so on.

[0132] Some embodiments may use transformer networks such as SentenceTransformers TM , a Python library for state-of-the-art sentence, text, and image embedding TM Framework. Such a transformer network learns associations of words and phrases with both high and low scores. This trains the AI / ML model to determine which are close to the input and which are not, respectively. The transformer network can also use field lengths and field types, rather than just pairs of words / phrases.

[0133] In some embodiments, NLP techniques (such as word2vec, BERT, GPT-3, ChatGPT, other LLMs, etc.) can be used to promote semantic understanding and provide more accurate and more humane answers. Other techniques (such as clustering algorithms) can be used to find similarities between groups of elements. Clustering algorithms may include, but are not limited to, density-based algorithms, distribution-based algorithms, centroid-based algorithms, hierarchy-based algorithms, K-means clustering algorithms, DBSCAN clustering algorithms, Gaussian mixture models (GMM) algorithms, balanced iterative reduction and clustering (BIRCH) algorithms using hierarchical structures, etc. Such techniques can also help with classification.

[0134] Figure 7 700 is a flow chart illustrating a process 700 for training (multiple) AI / ML models according to an embodiment of the present invention. In some embodiments, as described above, (multiple) AI / ML models may be generative AI models. The neural network architecture of the AI / ML model typically includes multiple layers of neurons, including input, output, and hidden layers. For example, see Fig. 6A and Figure 6B The input layer receives (multiple) text-based inputs, and the output layer generates (multiple) text-based responses. Intermediate hidden layers process the input data and generate intermediate representations of the inputs for generating outputs. These hidden layers may include various types of neurons, such as convolutional neurons, recurrent neurons, and / or transformation neurons.

[0135] At 710, the training process begins by providing text, screenshots, graphical elements, and contextual information, whether labeled or unlabeled. The AI / ML model is then trained over multiple epochs at 720, and the results are reviewed at 730. While various types of AI / ML models can be used, LLM and other generative AI models are typically trained using a process known as "supervised learning," which is also discussed above. Supervised learning involves providing the model with a large data set, which the model uses to learn the relationship between inputs and outputs. During the training process, the model adjusts the weights and biases of the neurons in the neural network to minimize the difference between the predicted outputs and the actual outputs in the training data set.

[0136] In some embodiments, one aspect of the model is the use of transfer learning. For example, transfer learning can utilize a pre-trained model, such as ChatGPT, that is fine-tuned for a specific task or domain in step 720. This allows the model to leverage knowledge already learned from the pre-training phase and adapt it to a specific application via the training phase of step 720.

[0137] The pre-training phase involves training the model on an initial training data set that can be more general. In this phase, the model learns the relationships in the data. In the fine-tuning phase (for example, in some embodiments, if the pre-trained model is used as the initial basis for the final model, it is performed in addition to or in place of the initial training phase during step 720), the pre-trained model is adapted to a specific task or domain by training the model on a smaller data set specific to the task. For example, in some embodiments, the model can focus on a certain (certain) type of data source. This can help the model identify the data elements therein more accurately than a generative AI model pre-trained alone. Fine-tuning allows the model to learn the nuances of the source, such as specific vocabulary and grammar, certain graphic features, certain data formats, etc., without requiring as much data as would be required if the model was trained from scratch. By leveraging the knowledge learned in the pre-training phase, the fine-tuned model can achieve the performance of the prior art on a specific task with relatively little additional training data.

[0138] If the AI / ML model fails to meet the desired confidence threshold at 740, the training data is supplemented and / or the reward function is modified at 750 to help the AI / ML model better achieve its goal, and the process returns to step 720. If the AI / ML model meets the confidence threshold at 740, the AI / ML model is tested on the evaluation data at 760 to ensure that the AI / ML model generalizes well and that the AI / ML model is not overfit with respect to the training data. The evaluation data includes information that the AI / ML model has not processed before. If the confidence threshold of the evaluation data is met at 770, the AI / ML model is deployed at 780. If not, the process returns to step 750 and the AI / ML model is further trained.

[0139] Figure 8 is a flowchart illustrating a process for supplementing automatic information extraction using generative AI according to an embodiment of the present invention. The process begins at 810 by providing source information as input to a CV model and / or an OCR model and a generative AI model to process the source information and extract data elements therein. For example, HTML, XAML, text, images, etc. can be obtained from the source and appropriately provided to these models. Depending on the type of model, this information may vary. For example, images can be provided as input to CV and OCR models, while XAML data, OCR model output, etc. can be provided to a generative AI model.

[0140] At 820, the CV model and / or the OCR model and the generative AI model process the corresponding source information provided to each model to extract data elements therein, and at 830, the outputs of these models are analyzed by the RPA robot or another software application to determine whether the models are consistent with the extracted data element predictions. The model outputs may include the extracted data element predictions, the corresponding confidence scores, the extracted data elements themselves, etc. If the CV model and / or the OCR model are consistent in the predictions, and at 840, the CV model and / or the OCR model and the generative AI model identify all data elements in the source with at least their corresponding acceptance thresholds (whether common or specific to the model), then at 850, a common extracted element prediction and / or corresponding data element is provided as an output.

[0141] However, if at 840 the models do not agree on the prediction, or the CV model and / or the OCR model and / or the generative AI model are below their respective acceptance thresholds, regardless of whether the models agree, then at 860, human assistance (i.e., a human in the loop) is requested. The labeled element data from the (multiple) human reviewers is then merged into the training data for each model to meet the acceptance threshold for that element. For example, if below the acceptance threshold but successfully predicted, the labeled element can be provided as a positive example for retraining. If the prediction was wrong, the correction in the labeled data can help successfully identify the element in the future after retraining. The (multiple) AI / ML models are then retrained using this labeled data at 870.

[0142] Fig. 9 is a flowchart illustrating a process 900 for supplementing automatic information extraction using generative AI according to an embodiment of the present invention. The process begins at 910 by providing source information as input to a CV model and / or an OCR model and a generative AI model to process the source information and extract data elements therein. For example, HTML, XAML, text, images, etc. may be obtained from the source and appropriately provided to these models. Depending on the type of model, this information may vary. For example, images may be provided as input to CV and OCR models, while XAML data, OCR model output, etc. may be provided to a generative AI model.

[0143] At 920, the CV model and / or the OCR model and the generative AI model process the corresponding source information provided to each model to extract data elements therein, and at 930, the outputs of these models are analyzed by the RPA robot or another software application to determine whether the models are consistent with the extracted data element predictions. The model outputs may include the extracted data element predictions, the corresponding confidence scores, the extracted data elements themselves, etc. If the CV model and / or the OCR model are consistent in the predictions, and at 940, the CV model and / or the OCR model and the generative AI model identify all data elements in the source with at least their corresponding acceptance thresholds (whether common or specific to the model), then at 950, a common extracted element prediction and / or corresponding data element is provided as an output.

[0144] However, if the models do not agree on the prediction, and the (multiple) elements identified by the generative AI at 960 are above the common or generative AI acceptance threshold, then at 970, the (multiple) extracted data elements are predicted and / or the (multiple) corresponding data elements extracted by the generative AI are used, and the (multiple) failed elements marked with the generative AI predictions are provided for retraining the CV or OCR model. If the CV model and / or OCR model and the generative AI model are below their respective acceptance thresholds at 960, regardless of model consistency, at 980, human assistance is requested (e.g., if the process can wait for a human user to provide corrections) and / or the (multiple) elements are marked for subsequent retraining. The marked element data is then merged into the training data of each model to meet the acceptance threshold for the element. For example, if it is below the acceptance threshold but successfully predicted, the marked element can be provided as a positive example for retraining. If the prediction is wrong, the correction in the marked data can help to successfully identify the element in the future after retraining. The (multiple) AI / ML model is then retrained using the marked data at 990.

[0145] According to an embodiment of the present invention, Figure 8 and Fig. 9 The process steps performed in the embodiment may be performed by a computer program encoding instructions for execution by a processor(s) Figure 8 and Fig. 9 The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be, but is not limited to, a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include a computer program for controlling a processor (e.g., Figure 5 The processor(s) 510 of the computing system 500 are used to implement Figure 8 and Fig. 9The coded instructions for all or part of the processing steps described in the present invention may also be stored on a computer-readable medium.

[0146] The computer program may be implemented in hardware, software or a hybrid implementation. The computer program may consist of modules that are in operational communication with each other and are designed to transfer information or instructions for display. The computer program may be configured to operate on a general purpose computer, an ASIC or any other suitable device.

[0147] It is easy to understand that, as generally described and illustrated in the accompanying drawings herein, the components of the various embodiments of the present invention can be arranged and designed according to various different configurations. Therefore, as shown in the accompanying drawings, the detailed description of the embodiments of the present invention is not intended to limit the scope of the claimed invention, but only represents the selected embodiments of the present invention.

[0148] The features, structures or characteristics of the present invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, throughout the specification, references to "certain embodiments," "some embodiments," or similar language indicate that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of the present invention. Therefore, the appearance of the phrases "in certain embodiments," "in some embodiments," "in other embodiments," or similar language throughout this specification does not necessarily all refer to the same set of embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0149] It should be noted that in this specification, references to features, advantages, or similar language do not imply that all features and advantages that may be achieved by the present invention should be present or exist in any single embodiment of the present invention. Instead, language referring to features and advantages is understood to mean that a particular feature, advantage, or characteristic described in conjunction with an embodiment is included in at least one embodiment of the present invention. Therefore, discussions of features and advantages and similar language throughout this specification may, but do not necessarily, refer to the same embodiment.

[0150] In addition, the features, advantages and characteristics of the present invention may be combined in any suitable manner in one or more embodiments. Those skilled in the relevant art will recognize that the present invention may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments of the present invention may be recognized in certain embodiments.

[0151] Those skilled in the art will readily appreciate that the present invention as described above may be practiced with steps in different orders and / or with hardware elements that are different from the disclosed configurations. Therefore, although the present invention has been described based on these preferred embodiments, certain modifications, variations, and alternative configurations will be clear to those skilled in the art while remaining within the spirit and scope of the present invention. Therefore, in order to determine the scope and limits of the present invention, reference should be made to the appended claims.

Claims

1. One or more non-transitory computer-readable media storing a computer program for supplementing automatic information extraction with generative artificial intelligence (AI), the computer program being configured to cause at least one processor to: Providing the source information as input to a computer vision (CV) model, an optical character recognition (OCR) model, or both, and to a generative artificial intelligence (AI) model to process the source information and extract data elements therein; receiving outputs from the generative AI model and the CV model and / or the OCR model, the outputs comprising a plurality of extracted data element predictions and corresponding confidence scores for the extracted data elements; analyzing the plurality of extracted data element predictions and the corresponding confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the models agree on the plurality of extracted data element predictions; as well as In response to the generative AI model and the CV model and / or the OCR model reaching a consensus on an extracted data element prediction among the multiple extracted data element predictions and the confidence score for the corresponding extracted data element prediction exceeds the corresponding acceptance threshold for the model, outputting the extracted element prediction and / or the extracted data element.

2. The one or more non-transitory computer readable media of claim 1, wherein the computer program is further configured to cause the at least one processor to: In response to the generative AI model and the CV model and / or the OCR model not meeting or exceeding their respective acceptance thresholds for the extracted data element predictions, or the generative AI model and the CV model and / or the OCR model not reaching a consensus on the extracted data element predictions among the multiple extracted data element predictions, requesting human assistance to identify the correct corresponding data element.

3. The one or more non-transitory computer readable media of claim 2, wherein the computer program is further configured to cause the at least one processor to: In response to the generative AI model and the CV model and / or the OCR model not meeting or exceeding their respective acceptance thresholds for predictions of extracted data elements, marking the corresponding data elements for retraining the generative AI model and the CV model and / or the OCR model. 4 . The one or more non-transitory computer-readable media of claim 1 , wherein the source information comprises Hypertext Markup Language HTML code, Extensible Application Markup Language XAML code, text, an image, or any combination thereof.

5. The one or more non-transitory computer-readable media of claim 1, wherein the source information provided to the generative AI model is different from the source information provided to the CV model and / or the OCR model.

6. The one or more non-transitory computer-readable media of claim 1, wherein the computer program is a Robotic Process Automation (RPA) robot.

7. One or more non-transitory computer-readable media according to claim 1, wherein the generative AI model is a large language model (LLM), a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer.

8. One or more computing systems, including: A memory storing computer program instructions for supplementing automatic information extraction with generative artificial intelligence (AI); as well as at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to: Receiving outputs from a generative artificial intelligence (AI) model and a computer vision (CV) model and / or an optical character recognition (OCR) model, the outputs comprising a plurality of extracted data element predictions and corresponding confidence scores for data elements extracted from source information; analyzing the plurality of extracted data element predictions and the corresponding confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the models agree on the plurality of extracted data element predictions; as well as In response to the generative AI model and the CV model and / or the OCR model failing to agree on an extracted data element prediction among the multiple extracted data element predictions, or the generative AI model and the CV model and / or the OCR model failing to meet or exceed their respective acceptance thresholds for the extracted data element predictions, requesting human assistance to identify the correct corresponding data element.

9. The one or more computing systems of claim 8, wherein the computer program instructions are further configured to cause the at least one processor to: In response to the generative AI model and the CV model and / or the OCR model reaching a consensus on an extracted data element prediction among the multiple extracted data element predictions and the confidence score for the corresponding extracted data element prediction exceeds the corresponding acceptance threshold for the model, outputting the extracted element prediction and / or the extracted data element.

10. The one or more computing systems of claim 8, wherein the computer program instructions are further configured to cause the at least one processor to: In response to the generative AI model and the CV model and / or the OCR model not meeting or exceeding their respective acceptance thresholds for predictions of extracted data elements, marking the corresponding data elements for retraining the generative AI model and the CV model and / or the OCR model.

11. The one or more computing systems of claim 8, wherein the source information comprises Hypertext Markup Language (HTML) code, Extensible Application Markup Language (XAML) code, text, images, or any combination thereof.

12. The one or more computing systems of claim 8, wherein the source information provided to the generative AI model is different than the source information provided to the CV model and / or the OCR model.

13. The one or more computing systems of claim 8, wherein the computer program instructions are or include a Robotic Process Automation (RPA) robot.

14. One or more computing systems according to claim 8, wherein the generative AI model is a large language model (LLM), a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer.

15. The one or more computing systems of claim 8, wherein the computer program instructions are further configured to cause the at least one processor to: The source information is provided as input to the generative AI model and the CV model and / or the OCR model to process the source information and extract the data elements therein.

16. A computer-implemented method for using generative artificial intelligence (AI) to supplement automatic information extraction, comprising: A robotic process automation (RPA) robot executed by at least one processor of the computing system provides source information as input to a computer vision (CV) model, an optical character recognition (OCR) model, or both, and to a generative artificial intelligence (AI) model to process the source information and extract data elements therein; receiving, by the RPA robot, outputs from the generative AI model and the CV model and / or the OCR model, the outputs comprising a plurality of extracted data element predictions and corresponding confidence scores for the extracted data elements; analyzing, by the RPA robot, the plurality of extracted data element predictions and the corresponding confidence scores from the generative AI model and the CV model and / or the OCR model to determine whether the models agree on the plurality of extracted data element predictions; In response to the generative AI model and the CV model and / or the OCR model reaching an agreement on an extracted data element prediction among the plurality of extracted data element predictions and the confidence score for the corresponding extracted data element prediction exceeds a corresponding acceptance threshold for the model, outputting, by the RPA robot, the extracted element prediction and / or the extracted data element; as well as In response to the generative AI model and the CV model and / or the OCR model failing to agree on an extracted data element prediction among the multiple extracted data element predictions, or the generative AI model and the CV model and / or the OCR model failing to meet or exceed their respective acceptance thresholds for the extracted data element predictions, the RPA robot requests human assistance to identify the correct corresponding data element.

17. The computer-implemented method of claim 16, further comprising: In response to the generative AI model and the CV model and / or the OCR model not meeting or exceeding their corresponding acceptance thresholds for extracted data element predictions, the RPA robot marks the corresponding data elements for retraining the generative AI model and the CV model and / or the OCR model.

18. The computer-implemented method of claim 16, wherein the source information comprises Hypertext Markup Language HTML code, Extensible Application Markup Language XAML code, text, images, or any combination thereof.

19. The computer-implemented method of claim 16, wherein the source information provided to the generative AI model is different from the source information provided to the CV model and / or the OCR model.

20. The computer-implemented method of claim 16, wherein the generative AI model is a Large Language Model (LLM), a Generative Adversarial Network (GAN), a Variational Autoencoder (VAE), or a Transformer.

Images