Automatic data conversion during copy and paste operations

Abstract

The invention relates to automatic data conversion during copy and paste operations. A method is provided. The method is performed by a translation engine implemented as a computer program in a computing environment. The method includes selecting a conversion option to convert source data of a source field and provide an input field that prompts an input. The source data is in a first format. The method includes converting the source data into converted data based on the conversion option and the input and providing the converted data to the target field. The converted data is in a second format.

Description

Technical Field

[0001] The present disclosure relates generally to data conversion, and more particularly to a conversion engine that provides automatic data conversion during copy and paste operations. Background Art

[0002] Typically, traditional clipboard software does not provide data conversion when copying and pasting data from a source to a target. That is, traditional clipboard software copies data from one place to another. This copying is usually performed in batches without regard to formatting, fonts, or organization. In the best case, traditional clipboard software can align the location of the source with the location of the target, but this alignment does not take into account how the location of the target requires the data.

[0003] For example, dates, currencies, and temperatures from a European document cannot simply be copied and pasted into a U.S. document using traditional clipboard software because the formats of dates, currencies, and temperatures are different between European and U.S. documents. In addition, traditional clipboard software cannot or will not handle drop-down menus and other interface elements of a European document when performing copy and paste operations.

[0004] A solution is needed. Summary of the invention

[0005] According to one or more embodiments, a method is provided. The method is performed by a conversion engine implemented as a computer program in a computing environment. The method includes selecting a conversion option to convert source data of a source field and providing an input field that prompts for input. The source data is in a first format. The method includes converting the source data into converted data based on the conversion option and the input and providing the converted data to a target field. The converted data is in a second format.

[0006] According to one or more embodiments, the method may be implemented as a computer program product, a device, an apparatus and / or a system. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to facilitate understanding of the advantages of certain embodiments of the present invention, a more detailed description will be given with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments and should not be considered to limit the scope thereof, and one or more embodiments of the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0008] Figure 1 An architectural diagram of an automation system according to one or more embodiments is shown.

[0009] Figure 2 An architectural diagram of an RPA system according to one or more embodiments is shown.

[0010] Figure 3 An architectural diagram of a deployed RPA system according to one or more embodiments is shown.

[0011] Figure 4 An architectural diagram illustrating the relationship between designers, activities, and drivers according to one or more embodiments.

[0012] Figure 5 An architectural diagram of a computing system is shown in accordance with one or more embodiments.

[0013] Figure 6 An example of a neural network trained to recognize graphical elements in an image is shown in accordance with one or more embodiments.

[0014] Figure 7 An example of a neuron is shown in accordance with one or more embodiments.

[0015] Figure 8 A flowchart of a process for training (multiple) AI / ML models according to one or more embodiments is shown.

[0016] Fig. 9 A flow diagram of a process according to one or more embodiments is shown.

[0017] Fig.10 Mapping examples are shown in accordance with one or more embodiments.

[0018] Fig.11 An interface is shown in accordance with one or more embodiments.

[0019] Fig.12 An interface is shown in accordance with one or more embodiments.

[0020] Fig.13 An interface is shown in accordance with one or more embodiments.

[0021] Fig.14 An interface is shown in accordance with one or more embodiments.

[0022] Fig.15 Example code is shown in accordance with one or more embodiments.

[0023] Fig.16 An example of a conversion is shown in accordance with one or more embodiments.

[0024] Unless otherwise indicated, similar reference characters denote corresponding features consistently throughout the drawings. DETAILED DESCRIPTION

[0025] The present disclosure relates generally to data conversion and, more particularly, to a conversion engine that provides automatic data conversion during copy and paste operations. Implementing the conversion engine may be executable by a computing system (as described herein).

[0026] According to one or more embodiments, the conversion engine converts data from one format to another while performing copy and paste automation from a source to a target screen / document / table / page. For example, the conversion may include, but is not limited to, date formats, address formats, currency types, and temperature types. According to one or more embodiments, the conversion engine may be interactive, such as by requesting user input to select source data and / or fields and providing natural language text related to data conversion. According to one or more embodiments, the conversion engine may determine the scope of data conversion, such as whether advanced conversion or general conversion is required. The conversion engine may perform general conversion using a built-in conversion model. The conversion engine may perform advanced conversion using natural language processing, such as an artificial intelligence and machine learning (AI / ML) model including a generative pre-trained converter three (GPT-3). The operation of the conversion may be automated.

[0027] For example, the conversion engine maps at least one source field (e.g., one or more source fields) to at least one target field (e.g., one or more target fields), and selects a conversion option from one or more options to convert source data of the one or more source fields. Note that the conversion option can be automatically selected or selected by user input. One or more source fields can be located in a first, initial or previous document, form or page. One or more target fields can be located in a separate, subsequent or different document, form or page. The source data can be in a first format. The conversion engine also provides an input field for prompting input. Note that the input field can be provided in a user interface to be displayed to the user on a screen. The input can include a natural language query. Then, the conversion engine converts the source data into converted data according to the conversion option and the input. For example, the conversion engine converts the source data including providing the source data, the conversion option and the input to a model, and obtaining the converted data from the model. The converted data can be in a second format that is different from the first format of the source data. The conversion engine provides the converted data to at least one target field (e.g., one or more target fields). One or more advantages, technical effects, and / or benefits of the transformation engine include enabling data transformation when copying and pasting data, which is not possible in traditional clipboard software. By way of example, one or more advantages, technical effects, and / or benefits of the transformation engine that enables data transformation include providing data extraction (e.g., zip code from address field, summary and / or keywords of text), data classification (e.g., messages or text can be identified as errors, queries, feature requests, etc.), and data standardization (e.g., standard dates, phone numbers, addresses, currencies, etc.).

[0028] Figure 1 is an architectural diagram illustrating a hyper-automation system 100 according to one or more embodiments. "Hyper-automation" as used herein refers to an automation system that integrates process automation components, integration tools, and technologies that enhance work automation capabilities. For example, in some embodiments, RPA may be used at the core of a hyper-automation system, and in some embodiments, automation capabilities may be extended through artificial intelligence and / or machines (AI / ML), process mining, analytics, and / or other advanced tools. For example, as the hyper-automation system learns processes, trains AI / ML models, and employs analytics, more and more knowledge work can be automated, and computing systems in an organization (e.g., computing systems used by individuals and computing systems that operate autonomously) can all participate in the hyper-automation process. The hyper-automation system of some embodiments allows users and organizations to efficiently and effectively discover, understand, and extend automation.

[0029] The hyper-automated system 100 includes user computing systems, such as a desktop computer 102, a tablet computer 104, and a smartphone 106. However, any desired computing system may be used without departing from the scope of one or more embodiments herein, including but not limited to smart watches, laptops, servers, Internet of Things (IoT) devices, etc. In addition, although Figure 1 Three user computing systems are shown, but any suitable number of computing systems may be used without departing from the scope of one or more embodiments herein. For example, in some embodiments, tens, hundreds, thousands, or millions of computing systems may be used. The user computing systems may be actively used by the user or run automatically without much or any user input.

[0030] Each computing system 102, 104, 106 has a corresponding (multiple) automated process 110, 112, 114 running thereon. The (multiple) automated process 102, 104, 106 may include, but is not limited to, an RPA robot, a portion of an operating system, a (multiple) downloadable application for the corresponding computing system, any other suitable software and / or hardware, or any combination of these, without departing from the scope of one or more embodiments herein. In some embodiments, one or more of the (multiple) processes 110, 112, 114 may be a listener. The listener may be an RPA robot, a portion of an operating system, a downloadable application for the corresponding computing system, or any other software and / or hardware, without departing from the scope of one or more embodiments herein. In fact, in some embodiments, the logical portion or all of the (multiple) listeners are implemented by physical hardware.

[0031] The listener monitors and records data related to the user's interaction with the corresponding computing system and / or the operation of the unattended computing system, and sends the data to the core hyper-automation system 120 via a network (e.g., a local area network LAN, a mobile communication network, a satellite communication network, the Internet, any combination thereof, etc.). The data may include, but is not limited to, which buttons were clicked, where the mouse was moved, which text was placed in a field, one window was minimized and another window was opened, the application associated with the window, etc. In some embodiments, the data from the listener may be sent periodically as part of a heartbeat message. In some embodiments, the data may be sent to the core hyper-automation system 120 once a predetermined amount of data is collected, or after a predetermined time period has passed, or both. One or more servers (e.g., server 130) receive the data from the listener and store it in a database (e.g., database 140).

[0032] The automation process can execute the logic developed in the workflow during design time. In the case of RPA, a workflow can include a set of steps (defined herein as "activities") that are executed in sequence or some other logical flow. Each activity can include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows can be nested or embedded.

[0033] In some embodiments, long-running workflows for RPA are master projects that support service orchestration, human intervention, and long-running transactions in an unattended environment. See U.S. Patent No. 10,860,905, all of which is incorporated herein by reference. Human intervention comes into play when certain processes require human input to handle exceptions, approvals, or verifications before proceeding to the next step of the activity. In this case, process execution is paused, freeing up the RPA robot until the human task is completed.

[0034] Long-running workflows can support workflow fragmentation through persistent activities and can be combined with calling processes and non-user interactive activities to orchestrate human tasks with RPA robot tasks. In some embodiments, multiple or many computing systems can participate in the execution of the logic of long-running workflows. Long-running workflows can run in sessions to facilitate fast execution. In some embodiments, long-running workflows can orchestrate background processes that may include activities that execute application programming interface (API) calls and run in long-running workflow sessions. In some embodiments, these activities can be called by calling process activities. A process with user interactive activities running in a user session can be called by starting a job from a commander activity (the commander will be described in more detail later in this article). In some embodiments, a user can interact through tasks that require completing a form in the commander. Activities that cause the RPA robot to wait for the form task to complete and then resume the long-running workflow can be included.

[0035] One or more of the (multiple) automated processes 110, 112, 114 communicate with the core hyper-automation system 120. In some embodiments, the core hyper-automation system 120 may run a commander application on one or more servers (e.g., server 130). Although one server 130 is shown for illustrative purposes, multiple or many servers close to each other or in a distributed architecture may be employed without departing from the scope of one or more embodiments herein. For example, one or more servers may be provided for commander functions, AI / ML model services, authentication, governance, and / or any other suitable functions without departing from the scope of one or more embodiments herein. In some embodiments, the core hyper-automation system 120 may include or be part of a public cloud architecture, a private cloud architecture, a hybrid cloud architecture, and the like. In a certain embodiment, the core hyper-automation system 120 may host multiple software-based servers on one or more computing systems (e.g., server 130). In some embodiments, one or more servers (e.g., server 130) of the core hyper-automation system 120 may be implemented by one or more virtual machines (VMs).

[0036] In some embodiments, one or more automated processes 110, 112, 114 may call one or more AI / ML models 132 deployed on or accessible to the core hyper-automation system 120. The AI / ML model 132 may be trained for any suitable purpose without departing from the scope of one or more embodiments herein, as will be discussed in more detail herein. In some embodiments, two or more AI / ML models 132 may be linked together (e.g., in series, in parallel, or a combination of both) so that they jointly provide (multiple) collaborative outputs. The AI / ML model 132 may perform or assist in computer vision (CV), optical character recognition (OCR), document processing and / or understanding, semantic learning and / or analysis, analytical prediction, process discovery, task mining, testing, automatic RPA workflow generation, sequence extraction, cluster detection, audio to text translation, any combination thereof, etc. However, any desired number and / or type of AI / ML models may be used without departing from the scope of one or more embodiments herein. For example, using multiple AI / ML models may allow the system to develop a global picture of what is happening on a given computing system. For example, one AI / ML model can perform OCR, another can detect buttons, another can compare sequences, etc. The pattern can be determined by an AI / ML model alone or by multiple AI / ML models together. In some embodiments, one or more AI / ML models are locally deployed on at least one computing system 102, 104, 106.

[0037] In some embodiments, multiple AI / ML models 132 may be used. Each AI / ML model 132 is an algorithm (or model) that operates on data, and the AI / ML model itself may be, for example, a deep learning neural network (DLNN) of trained artificial "neurons" trained on training data. In some embodiments, the AI / ML model 132 may have multiple layers that perform various functions (e.g., statistical modeling, such as hidden Markov models HMM), and utilize deep learning techniques (e.g., long short-term memory LSTM deep learning, encoding of previous hidden states, etc.) to perform the desired function.

[0038] In some embodiments, the hyperautomation system 100 can provide four main groups of functions: (1) discovery; (2) building automation; (3) management; and (4) engagement. In some embodiments, the automation (e.g., running on a user computing system, server, etc.) can be run by a software robot (e.g., an RPA robot). For example, attended robots, unattended robots, and / or testing robots can be used. Attended robots work with users to assist them in completing tasks (e.g., through UiPath Assistant). TM). Unattended robots work independently of users and can run in the background, possibly without the user's knowledge. Test robots are unattended robots that run test cases against applications or RPA workflows. In some embodiments, test robots can run in parallel on multiple computing systems.

[0039] The discovery functionality may discover and provide automated suggestions for different opportunities for business process automation. Such functionality may be implemented by one or more servers (e.g., server 130). In some embodiments, the discovery functionality may include providing an automation hub, process mining, task mining, and / or task capture. An automation hub (e.g., UiPath Automation Hub) may include: TM ) can provide mechanisms for managing automation promotion with visibility and control. For example, automation ideas can be crowdsourced from employees through a submission form. Feasibility and return on investment (ROI) calculations for automating these ideas can be provided, records for future automation can be collected, and collaboration can be provided to get from automation discovery to build faster.

[0040] Process mining (e.g., through UiPath Automation Cloud TM and / or UiPath AI Center TM ) refers to the process of collecting and analyzing data from applications (e.g., enterprise resource planning ERP applications, customer relationship management CRM applications, email applications, call center applications, etc.) to determine what end-to-end processes exist in the organization and how to effectively automate them, as well as indicate what impact the automation will have. For example, a listener can collect this data from user computing systems 102, 104, 106, and the data is processed by a server (e.g., server 130). In some embodiments, one or more AI / ML models 132 can be used for this purpose. This information can be exported to an automation center to speed up implementation and avoid manual information transfer. The goal of process mining can be to increase business value by automating processes within an organization. Some examples of process mining goals include, but are not limited to, increasing profits, improving customer satisfaction, regulatory and / or contractual compliance, improving employee efficiency, etc.

[0041] Task mining (e.g., through UiPath Automation Cloud TM and / or UiPath AI Center TM) identifies and aggregates workflows (e.g., employee workflows) and then applies AI to uncover patterns and variations in daily tasks, scoring such tasks to simplify automation and achieve potential savings (e.g., saving time and / or cost). One or more AI / ML models 132 can be employed to uncover recurring task patterns in the data. Recurring tasks suitable for automation can then be identified. In some embodiments, this information can initially be provided by a listener and analyzed on a server (e.g., server 130) of the core hyperautomation system 120. The results of task mining (e.g., Extensible Application Markup Language XAML process data) can be exported to a process document or designer application (e.g., UiPath Studio TM ) to create and deploy automation faster. According to one or more embodiments, the operation of the transformation engine can also be performed in UiPath Studio TM is implemented as an activity.

[0042] In some embodiments, task mining can include capturing screenshots with user actions (e.g., mouse click locations, keyboard input, application windows and graphical elements with which the user interacts, timestamps of interactions, etc.), collecting statistics (e.g., execution time, number of operations, text entries, etc.), editing and annotating screenshots, specifying the types of actions to be recorded, etc.

[0043] Task capture (e.g., via UiPath Automation Cloud TM and / or UiPath AI Center TM ) automatically documents a human-attended process as the user works or provides a framework for an unattended process. Such documentation can include automation of required tasks in the form of a process definition document (PDD), a skeleton workflow that captures the operation of each part of the process, records user actions and automatically generates a comprehensive workflow diagram with detailed information for each step, Documents, XAML files, and other files. In some embodiments, the built-in workflow can be exported directly to a designer application, such as UiPath Studio TM Task capture can streamline the requirements gathering process for subject matter experts who interpret the process and Center of Excellence (CoE) members who provide production-grade automation.

[0044] Build automation can be applied via a designer (e.g., UiPath Studio TM 、UiPath StudioX TM or UiPath Web TM). For example, RPA developers at PA development facility 150 can use RPA designer application 154 of computing system 152 to build and test automation for various applications and environments, such as the World Wide Web (Web), mobile applications, and virtualized desktops. API integrations can be provided for a variety of applications, technologies, and platforms. Predefined activities, drag-and-drop modeling, and a workflow recorder make automation easier with minimal coding. Document understanding capabilities can be provided through drag-and-drop AI skills for data extraction and interpretation, which call one or more AI / ML models 132. This automation can handle nearly any document type and format, including forms, checkboxes, signatures, and handwriting. When validating data or handling exceptions, this information can be used to retrain the corresponding AI / ML model, thereby improving its accuracy over time.

[0045] For example, integration services can allow developers to seamlessly combine user interface (UI) automation with API automation. Automations can be built that require APIs or traverse API and non-API applications and systems. Repositories for pre-built RPA and AI templates and solutions can be provided (e.g., UiPath ObjectRepository TM ) or a marketplace (e.g., UiPathMarketplace TM ) to allow developers to automate various processes faster. Therefore, when building automation, the hyper-automation system 100 can provide a user interface, a development environment, API integration, pre-built and / or customized AI / ML models, development templates, an integrated development environment (IDE), and advanced AI capabilities. In some embodiments, the hyper-automation system 100 supports the development, deployment, management, configuration, monitoring, debugging, and maintenance of RPA robots, which can provide automation for the hyper-automation system 100.

[0046] In some embodiments, components of the hyperautomation system 100 (e.g., designer application(s) and / or external rule engines) provide support for managing and enforcing governance policies for controlling the various functions provided by the hyperautomation system 100. Governance is the establishment of policies by an organization to prevent users from developing automations (e.g., RPA robots) that can take actions that could harm the organization (e.g., violate the EU General Data Protection Regulation GDPR, the US Health Insurance Portability and Accountability Act HIPAA, third-party application terms of service, etc.). Because developers may otherwise create automations that violate privacy laws, terms of service, etc. when executing automations, some embodiments implement access control and governance restrictions at the robot and / or robot design application level. In some embodiments, this can provide an additional level of security and compliance to the automation process development pipeline by preventing developers from relying on unapproved software libraries that may introduce security risks or work in a manner that violates policies, regulations, privacy laws, and / or privacy policies. See U.S. Non-Provisional Patent Application No. 16 / 924,499, all of which is incorporated herein by reference.

[0047] The management functions may provide management, deployment, and optimization of automation across an organization. In some embodiments, the management functions may include orchestration, test management, AI functions, and / or insights. The management functions of the hyper-automation system 100 may also serve as an integration point with third-party solutions and applications for automated applications and / or RPA robots. The management capabilities of the hyper-automation system 100 may include, but are not limited to, facilitating the provisioning, deployment, configuration, queuing, monitoring, logging, and interconnection of RPA robots.

[0048] Orchestrator applications, such as UiPath Orchestrator TM (In some embodiments, this can be used as a UiPath Automation Cloud TM or on-premises, in a VM, in a private or public cloud, on Linux TM Available in VM or via UiPath Automation Suite TM Available as a cloud-native single-container suite), providing orchestration capabilities to deploy, monitor, optimize, scale, and secure RPA robot deployments. Test suites (e.g., UiPath Test Suite TM ) can provide test management to monitor the quality of deployed automation. Test suites can facilitate test planning and execution, requirements fulfillment, and defect traceability. Test suites can include comprehensive test reporting.

[0049] Analytics software (e.g., UiPath Insights TM) can track, measure, and manage the performance of deployed automation. Analytics software can align automation actions with specific key performance indicators (KPIs) and strategic outcomes for the organization. Analytics software can present results in a dashboard format for better understanding by human users.

[0050] Data Services (for example, UiPath Data Service TM ) can be stored in, for example, a database 140, and brought to a single, scalable, secure location through a drag-and-drop storage interface. Some embodiments provide low-code or no-code data modeling and storage for automation while ensuring seamless access to data, enterprise-grade security, and scalability. AI capabilities can be managed by an AI center (e.g., UiPath AI Center TM ) is provided, which helps to incorporate AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make such capabilities accessible even to people who are not data scientists. Deployed automation (e.g., RPA robots) can call AI / ML models, such as AI / ML model 132, from the AI ​​center. The performance of the AI / ML model can be monitored and trained and improved using manually verified data (e.g., data provided by the data review center 160). A human reviewer can provide labeled data to the core hyper-automation system 120 via the review application 152 on the computing system 154. For example, a human reviewer can verify whether the prediction of the AI / ML model 132 is accurate, and provide corrections if it is not. This dynamic input can then be saved as training data for retraining the AI / ML model 132, and can be stored in, for example, a database (e.g., database 140). The AI ​​center can then schedule and execute training jobs to train a new version of the AI / ML model using the training data. Both positive and negative examples can be stored and used to retrain the AI / ML model 132.

[0051] The engagement feature enables humans and automation to engage as a team to collaborate seamlessly on the desired process. Low-code applications can be built (e.g., via UiPath Apps TM ) to connect browser tabs and legacy software, even if the API is missing in some embodiments. For example, applications can be quickly created using a web browser through a rich library of drag-and-drop controls. Applications can be connected to a single automation or multiple automations.

[0052] Action Center (for example, UiPath Action Center TM ) provides a straightforward and efficient mechanism to hand over a process from automation to humans and vice versa. Humans can provide approvals or escalations, make exceptions, etc. Automation can then perform the automated functions of a given workflow.

[0053] A local assistant can be provided as a launchpad for users to start automation (e.g., UiPath Assistant TM ). For example, this functionality may be provided in a tray provided by the operating system and may allow users to interact with RPA robots and applications driven by RPA robots on their computing system. The interface may list automations approved for a given user and allow the user to run them. These may include off-the-shelf automations from the Automation Marketplace, internal automation storage in the Automation Center, and the like. When automations run, they may run as local instances in parallel with other processes on the computing system so that the user can use the computing system while the automation performs its actions. In some embodiments, the assistant is integrated with a task capture function so that users can record the processes they are about to automate from the Assistant Launchpad.

[0054] Chatbots (e.g., UiPath Chatbots TM ), social messaging apps, and / or voice commands can enable users to run automations. This can simplify access to the information, tools, and resources users need to interact with customers or perform other activities. Conversations between people can be easily automated, as can other processes. A triggered RPA robot launched in this way can perform actions, such as checking order status, posting data in a CRM, etc., perhaps using plain language commands.

[0055] In some embodiments, the hyperautomation system 100 can provide end-to-end measurement and management of automation programs of any size. In accordance with the above, analytics can be used to understand the performance of automation (e.g., via UiPathInsights TM ). Data modeling and analysis using any combination of available business metrics and operational insights can be used for various automation processes. Custom designed and pre-built dashboards allow visualization of data across desired metrics, discovery of new analytical insights, tracking of performance indicators, discovery of ROI for automation, performance monitoring on user computing systems, detection of errors and anomalies, and debugging of automations. An automation management console (e.g., UiPath Automation Ops) can be provided TM ) to manage automations throughout the automation lifecycle. Organizations can govern how automations are built, what users can do with automations, and which automations users can access.

[0056] In some embodiments, the hyperautomation system 100 provides an iterative platform. Processes can be discovered, automation can be built, tested, and deployed, performance can be measured, automation usage can be easily provided to users, feedback can be obtained, AI / ML models can be trained and retrained, and the process can be repeated. This helps form a more powerful and effective automation suite.

[0057] Figure 2 An architecture diagram of an RPA system 200 according to one or more embodiments is shown. In some embodiments, the RPA system 200 is Figure 1 The RPA system 200 includes a designer 210 that allows developers to design and implement workflows. The designer 210 can provide solutions for application integration, as well as automate third-party applications, manage information technology (IT) tasks, and business IT processes. The designer 210 can facilitate the development of automation projects, which are graphical representations of business processes. In short, the designer 210 facilitates the development and deployment of workflows and robots (as shown by arrow 211). In some embodiments, the designer 210 can be an application running on a user's desktop, an application running remotely in a VM, a World Wide Web application, etc.

[0058] Automation projects enable automation of rule-based processes by giving developers control over the order of execution and the relationships between custom sets of steps developed in a workflow (defined herein as "activities" as described above). A commercial example of an embodiment of the designer 210 is UiPath Studio TM Each activity may include an action, such as clicking a button, reading a file, writing to a log panel, etc. In some embodiments, workflows may be nested or embedded.

[0059] Certain types of workflows may include, but are not limited to, sequences, flow charts, finite state machines (FSMs), and / or global exception handlers. Sequences may be particularly suitable for linear processes, enabling the flow from one activity to another without cluttering the workflow. Flow charts may be particularly suitable for more complex business logic, enabling the integration of decisions and connecting activities in a more diverse manner through multiple branching logic operators. FSMs may be particularly suitable for large workflows. FSMs may use a limited number of states in their execution, which are triggered by conditions (i.e., transitions) or activities. Global exception handlers may be particularly suitable for determining workflow behavior when execution errors are encountered and for debugging processes.

[0060] Once a workflow is developed in the designer 210, the execution of the business process is orchestrated by the director 220, which orchestrates one or more robots 230 that execute the workflow developed in the designer 210. One commercial example of an embodiment of the director 220 is the UiPath Orchestrator TM The commander 220 helps manage the creation, monitoring, and deployment of resources in the environment. The commander 220 can serve as an integration point with third-party solutions and applications. In accordance with the above, in some embodiments, the commander 220 can be Figure 1 Part of the core hyper-automation system 120.

[0061] The commander 220 can manage a queue of robots 230, connect and execute (as shown by arrow 231) robots 230 from a centralized point. The types of robots 230 that can be managed include, but are not limited to, attended robots 232, unattended robots 234, development robots (similar to unattended robots 234, but for development and testing purposes), and non-production robots (similar to attended robots 232, but for development and testing purposes). Attended robots 232 are triggered by user events and operate with humans on the same computing system. Attended robots 232 can be used with commander 220 for centralized process deployment and logging media. Attended robots 232 can help human users complete various tasks and can be triggered by user events. In some embodiments, processes cannot be started from the commander 220 on this type of robot and / or they cannot run under a locked screen. In some embodiments, attended robots 232 can only be started from the robot tray or command prompt. In some embodiments, attended robots 232 should run under human supervision.

[0062] Unattended robots 234 run unattended in a virtual environment and can automate many processes. Unattended robots 234 can be responsible for remote execution, monitoring, scheduling, and providing support for work queues. In some embodiments, debugging of all robot types can be run in designer 210. Both attended and unattended robots can automate (as shown in dashed box 290) various systems and applications, including but not limited to mainframes, World Wide Web applications, VMs, enterprise applications (e.g., hosted by applications produced by the computer industry) and computing system applications (e.g., desktop and laptop computer applications, mobile device applications, wearable computer applications, etc.).

[0063] The commander 220 may have various capabilities (as indicated by arrow 232), including, but not limited to, provisioning, deploying, configuring, queuing, monitoring, logging, and / or providing interconnectivity. Provisioning may include creating and maintaining a connection between the robot 230 and the commander 220 (e.g., a World Wide Web application). Deployment may include ensuring that a software package version is properly delivered to a designated robot 230 for execution. Configuration may include maintaining and delivering robot environment and process configurations. Queuing may include providing management of queues and queue items. Monitoring may include tracking robot identification data and maintaining user permissions. Logging may include storing and indexing logs into a database (e.g., a structured query language SQL or NoSQL database) and / or another storage mechanism (e.g., It provides the ability to store and quickly query large data sets.) Director 220 can provide interconnectivity by acting as a centralized communication point for third-party solutions and / or applications.

[0064] Robot 230 is an execution agent that executes the workflow built in designer 210. One commercial example of some embodiments of robot(s) 230 is UiPath Robots TM In some embodiments, the robot 230 is installed by default Services managed by the Service Control Manager (SCM). Therefore, such a robot 230 can open interactive session, and has Permissions for services.

[0065] In some embodiments, robots 230 can be installed in user mode. For such robots 230, this means that they have the same permissions as the user who installed the given robot 230. This feature can also be used for high density (HD) robots, which ensures that the maximum potential of each machine is fully utilized. In some embodiments, any type of robot 230 can be configured in an HD environment.

[0066] In some embodiments, the robot 230 is divided into several components, each dedicated to a specific automation task. In some embodiments, the robot components include but are not limited to SCM-managed robot services, user-mode robot services, executors, agents, and command lines. SCM-managed robot service management and monitoring The console application is started by the SCM under the local system.

[0067] In some embodiments, user-mode robot services manage and monitor The SCM manages the robot service and acts as a proxy between the commander 220 and the execution host. The user mode robot service can be trusted and manage the credentials of the robot 230. If the SCM manages the robot service is not installed, it can be automatically started. application.

[0068] The actuator can be The executor can be aware of the dots per inch (DPI) setting of each monitor. The agent can be the one that displays available jobs in a system tray window. Presentation Foundation (WPF) applications. Agents can be clients of services. Agents can request to start or stop jobs and change settings. Command lines are clients of services. Command lines are console applications that can request to start jobs and wait for their output.

[0069] As described above, separating the components of the robot 230 helps developers, support users, and computing systems to more easily run, identify, and track what each component is executing. Special behaviors can be configured for each component in this way, such as setting different firewall rules for executors and services. In some embodiments, the executor can always know the DPI setting of each monitor. Therefore, workflows can be executed at any DPI, regardless of the configuration of the computing system where they were created. In some embodiments, projects from the designer 210 can also be independent of the browser zoom level. For applications that are unaware of the DPI or are deliberately marked as unaware, DPI can be disabled in some embodiments.

[0070] The RPA system 200 in this embodiment is part of a hyper-automation system. Developers can use the designer 210 to build and test RPA robots that utilize AI / ML models deployed in the core hyper-automation system 240 (e.g., as part of its AI center). Such RPA robots can send inputs for executing (multiple) AI / ML models and receive outputs from them via the core hyper-automation system 240.

[0071] As described above, one or more robots 230 may be listeners. These listeners may provide information to the core hyper-automation system 240 about what users are doing when using their computing systems. The core hyper-automation system may then use this information for process mining, task mining, task capture, and the like.

[0072] An assistant / chatbot 250 may be provided on the user's computing system to allow the user to launch the RPA local robot. For example, the assistant may be located in the system tray. The chatbot may have a user interface so the user can see the text in the chatbot. Alternatively, the chatbot may lack a user interface and run in the background, using the computing system's microphone to listen to the user's voice.

[0073] In some embodiments, data labeling can be performed by a user of the computing system on which the robot is executing, or by a user of another computing system to which the robot provides information. For example, if the robot calls an AI / ML model that performs image CV for a VM user, but the AI / ML model fails to correctly identify a button on the screen, the user can draw a rectangle around the misidentified or unidentified component and perhaps provide text with the correct identification. This information can be provided to the core hyperautomation system 240 and then later used to train a new version of the AI / ML model.

[0074] Figure 3 An architectural diagram of an RPA system 300 deployed according to one or more embodiments is shown. In some embodiments, the RPA system 300 may be Figure 2 RPA system 200 and / or Figure 1 The deployed RPA system 300 may be a cloud-based system, a local system, or a desktop-based system, which provides enterprise-level, user-level, or device-level automation solutions for automation of different computing processes, etc.

[0075] It should be noted that the client side 301, the server side 302, or both may include any desired number of computing systems without departing from the scope of one or more embodiments herein. On the client side 301, the robot application 310 includes an executor 312, an agent 314, and a designer 316. However, in some embodiments, the designer 316 may not be running on the same computing system as the executor 312 and the agent 314. The executor 312 is running a process. Multiple business projects can be running simultaneously, such as Figure 3 In this embodiment, the agent 314 (for example, The service is the single point of contact for all executors 312. All messages in this embodiment are logged to the commander 340, which further processes the messages via the database server 355, the AI / ML server 360, the indexer server 370, or any combination thereof. Figure 2 As discussed, the actuator 312 may be a robotic component.

[0076] In some embodiments, a robot represents an association between a machine name and a user name. A robot can manage multiple executors simultaneously. On a computing system that supports running multiple interactive sessions simultaneously (e.g., Server2012), multiple robots can run simultaneously, each in a separate A unique username is used in the session. This is referred to above as an HD robot.

[0077] Agent 314 is also responsible for sending the status of the robot (e.g., sending "heartbeat" messages periodically to indicate that the robot is still running) and downloading the required version of the software package to be executed. In some embodiments, communication between agent 314 and commander 340 is always initiated by agent 314. In a notification scenario, agent 314 can open a WebSocket channel that commander 330 later uses to send commands to the robot (e.g., start, stop, etc.).

[0078] The listener 330 monitors and records data related to the interaction of the user with the attended computing system and / or the operation of the unattended computing system in which the listener 330 is located. The listener 330 can be an RPA robot, part of an operating system, a downloadable application for a corresponding computing system, or any other software and / or hardware without departing from the scope of one or more embodiments herein. In fact, in some embodiments, part or all of the logic of the listener is implemented via physical hardware.

[0079] On the server side 302, a presentation layer 333, a service layer 334, and a persistence layer 336, as well as a commander 340 are included. The presentation layer 333 may include a World Wide Web application 342, an open data protocol (OData) representational state transfer (REST) ​​application programming interface (API) endpoint 344, and notification and monitoring 346. The service layer 334 may include an API implementation / business logic 348. The persistence layer 336 may include a database server 355, an AI / ML server 360, and an indexer server 370. For example, the commander 340 includes a World Wide Web application 342, an OData REST API endpoint 344, notification and monitoring 346, and an API implementation / business logic 348. In some embodiments, most actions performed by a user in the interface of the commander 340 (e.g., via the browser 320) are performed by calling various APIs. Such actions may include, but are not limited to, starting a job on a robot, adding / deleting data in a queue, scheduling a job to run unattended, etc., without departing from the scope of one or more embodiments of this document. The web application 342 can be the visual layer of the server platform. In this embodiment, the web application 342 uses Hypertext Markup Language (HTML) and JavaScript (JS). However, any desired markup language, scripting language, or any other format can be used without departing from the scope of one or more embodiments of the present invention. In this embodiment, the user interacts with the web page from the web application 342 via the browser 320 to perform various actions to control the commander 340. For example, the user can create a robot group, assign software packages to robots, analyze logs for each robot and / or each process, start and stop robots, etc.

[0080] In addition to the web application 342, the director 340 also includes a service layer 334 that exposes an OData REST API endpoint 344. However, other endpoints may be included without departing from the scope of one or more embodiments herein. The REST API is consumed by the web application 342 and the agent 314. In this embodiment, the agent 314 is a supervisor of one or more robots on a client computer.

[0081] The REST API in this embodiment includes configuration, logging, monitoring, and queuing functionality (represented by at least arrow 349). In some embodiments, the configuration endpoint can be used to define and configure application users, permissions, robots, assets, releases, and environments. The logging REST endpoint can be used to log different information, such as errors, explicit messages sent by the robot, and other environment-specific information. If the start job command is used in the commander 340, the robot can use the deployment REST endpoint to query the version of the software package that should be executed. The queuing REST endpoint can be responsible for queue and queue item management, such as adding data to the queue, getting transactions from the queue, setting the status of transactions, etc.

[0082] The monitoring REST endpoint can monitor the web application 342 and the agent 314. The notification and monitoring API 346 can be a REST endpoint for registering the agent 314, passing configuration settings to the agent 314, and for sending / receiving notifications from the server and the agent 314. In some embodiments, the notification and monitoring API 346 can also use WebSocket communication. Figure 3 As shown, one or more activities / actions described herein are represented by arrows 350 and 351 .

[0083] In some embodiments, the API in the service layer 334 can be accessed by configuring the appropriate API access path, for example, based on whether the commander 340 and the overall hyper-automation system have a native deployment type or a cloud-based deployment type. The commander 340 API can provide custom methods for querying statistics of various entities registered in the commander 340. In some embodiments, each logical resource can be an OData entity. In such an entity, components (e.g., robots, processes, queues, etc.) can have properties, relationships, and operations. In some embodiments, the commander 340 API can be occupied by the World Wide Web application 342 and / or the agent 314 in two ways: by obtaining API access information from the commander 340, or by registering an external application to use the OAuth flow.

[0084] The persistence layer 336 includes three servers in this embodiment - a database server 355 (e.g., a SQL server), an AI / ML server 360 (e.g., a server that provides AI / ML model services, such as AI center functions), and an indexer server 370. The database server 355 in this embodiment stores configurations of robots, robot groups, related processes, users, roles, arrangements, etc. In some embodiments, this information is managed by the World Wide Web application 342. The database server 355 can manage queues and queue items. In some embodiments, the database server 355 can store messages that are journaled by the robot (in addition to or in place of the indexer server 370). For example, the database server 355 can also store process mining, task mining, and / or task capture related data received from the listener 330 installed on the client side 301. Although no arrow is shown between the listener 330 and the database 355, it should be understood that in some embodiments, the listener 330 is able to communicate with the database 355, and vice versa. The data can be stored in the form of a PDD, an image, a XAML file, etc. Listener 330 may be configured to intercept user actions, processes, tasks, and performance indicators on the corresponding computing system where listener 330 is located. For example, listener 330 may record user actions (e.g., clicks, typed characters, locations, applications, active elements, time, etc.) on its corresponding computing system and then convert them into a suitable format to be provided to and stored in database server 355.

[0085] The AI / ML server 360 facilitates incorporating AI / ML models into automation. Pre-built AI / ML models, model templates, and various deployment options can make this capability accessible even to people who are not data scientists. Deployed automation systems (e.g., RPA robots) can call AI / ML models from the AI / ML server 360. The performance of AI / ML models can be monitored and trained and improved using human-validated data. The AI / ML server 360 can schedule and execute training jobs to train new versions of AI / ML models.

[0086] The AI / ML server 360 can store data related to AI / ML models and ML packages for configuring various ML skills for users at the time of development. As used herein, ML skills are ML models that are pre-built and trained for a process, for example, that can be used by automation. The AI / ML server 460 can also store data related to document understanding techniques and frameworks, algorithms, and packages for various AI / ML functions, including but not limited to intent analysis, natural language processing (NLP), speech analysis, different types of AI / ML models, and the like.

[0087] Indexer server 370 is optional in some embodiments and stores and indexes the information logged by the robot. In some embodiments, indexer server 370 can be disabled through configuration settings. In some embodiments, indexer server 370 uses This is an open source full-text search engine. Messages recorded by the robot journal (e.g., using activities such as log messages or write lines) can be sent to the indexer server 370 through the (multiple) logging REST endpoints, where they are indexed for future use.

[0088] Figure 4 An architectural diagram showing the relationship between a designer 410, activities 420, 430, 440, 450, a driver 460, an API 470, and an AI / ML model 480 according to one or more embodiments. As described herein, a developer uses the designer 410 to develop a workflow executed by a robot. In some embodiments, various types of activities may be displayed to the developer. The designer 410 may be local to the user computing system or remote (e.g., accessed via a VM or a local web browser interacting with a remote web server). A workflow may include user-defined activities 420, API-driven activities 430, AI / ML activities 440, and / or UI automation activities 450. For example (as shown by the dashed lines), user-defined activities 420 and API-driven activities 440 interact with an application via its API. In turn, in some embodiments, user-defined activities 420 and / or AI / ML activities 440 may call one or more AI / ML models 480, which may be local and / or remote to the computing system on which the robot is running.

[0089] Some embodiments are capable of identifying non-text visual components in images, referred to herein as CV. CV may be performed at least in part by (multiple) AI / ML models 480. Some CV activities related to such components may include, but are not limited to, extracting text from segmented label data using OCR, fuzzy text matching, cropping segmented label data using ML, comparing extracted text from label data with real data, etc. In some embodiments, there may be hundreds or even thousands of activities that can be implemented in user-defined activities 420. However, any number and / or type of activities may be used without departing from the scope of one or more embodiments herein.

[0090] UI automation activities 450 are a subset of special, lower-level activities that are written in lower-level code and facilitate interaction with the screen. UI automation activities 450 facilitate these interactions via drivers 460 that allow the robot to interact with the desired software. For example, drivers 460 may include operating system (OS) drivers 462, browser drivers 464, VM ​​drivers 466, enterprise application drivers 468, etc. In some embodiments, UI automation activities 450 may use one or more AI / ML models 480 to interact with the computing system. In some embodiments, AI / ML models 480 may enhance drivers 460 or replace them completely. In fact, in some embodiments, drivers 460 are not included.

[0091] Driver 460 can interact with the OS at a low level via OS driver 462, looking for hooks, monitoring keys, etc. Driver 460 can facilitate communication with For example, the "click" activity performs the same role in these different applications via the driver 460.

[0092] Figure 5 An architectural diagram of a computing system 500 configured to provide a transformation engine for RPA according to one or more embodiments is shown. In some embodiments, the computing system 500 may be one or more of the computing systems shown and / or described herein. In some embodiments, the computing system 500 may be part of a hyper-automated system, such as Figure 1 and 2 The hyper-automation system shown in . The computing system 500 includes a bus 505 or other communication mechanism for communicating information, and (multiple) processors 510 coupled to the bus 505 for processing information. (Multiple) processors 510 can be any type of general or special-purpose processor, including a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), multiple instances thereof, and / or any combination thereof. (Multiple) processors 510 can also have multiple processing cores, and at least some of the cores can be configured to perform specific functions. In some embodiments, multi-parallel processing can be used. In some embodiments, at least one of the (multiple) processors 510 can be a neuromorphic circuit including a processing element that simulates a biological neuron. In some embodiments, the neuromorphic circuit may not require typical components of the von Neumann computing architecture.

[0093] The computing system 500 also includes a memory 515 for storing information and instructions executed by the processor(s) 510. The memory 515 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), flash memory, cache, static storage (e.g., a magnetic disk or optical disk), or any other type of non-transitory computer-readable medium or a combination thereof. Non-transitory computer-readable media may be any available media that can be accessed by the processor(s) 510, and may include volatile media, non-volatile media, or both. The media may also be removable, non-removable, or both.

[0094] In addition, the computing system 500 includes a communication device 520 (e.g., a transceiver) to provide access to a communication network via wireless and / or wired connections. In some embodiments, the communication device 520 can be configured to use frequency division multiple access (FDMA), single carrier FDMA (SC-FDMA), time division multiple access (TDMA), code division multiple access (CDMA), orthogonal frequency division multiplexing (OFDM), orthogonal frequency division multiple access (OFDMA), global system for mobile communications (GSM) communications, general packet radio service (GPRS), universal mobile telecommunications system (UMTS), cdma2000, wideband CDMA (W-CDMA), high speed downlink packet access (HSDPA), high speed uplink packet access (HS The communication device 520 may include one or more antennas that are single, array, panel, phased, switched, beamformed, beamsteering, combinations thereof, and / or any other antenna configurations without departing from the scope of one or more embodiments herein.

[0095] The processor(s) 510 are also coupled to a display 525 via the bus 505, such as a plasma display, a liquid crystal display (LCD), a light emitting diode (LED) display, a field emission display (FED), an organic light emitting diode (OLED) display, a flexible OLED display, a flexible substrate display, a projection display, a 4K display, a high definition display, Display, in-plane switching (IPS) display or any other display suitable for displaying information to a user. Display 525 can be configured as a touch (tactile) display, a three-dimensional (3D) touch display, a multi-input touch display, a multi-point touch display, etc., using resistive, capacitive, surface acoustic wave (SAW) capacitive, infrared, optical imaging, dispersive signal technology, acoustic pulse recognition, frustrated total internal reflection, etc. Any suitable display device and tactile I / O can be used without departing from the scope of one or more embodiments herein.

[0096] A keyboard 530 and a cursor control device 535 (e.g., a computer mouse, touchpad, etc.) are also coupled to bus 505 to enable a user to interact with computing system 500. However, in some embodiments, a physical keyboard and mouse may not be present, and a user may interact with the device only through display 525 and / or a touchpad (not shown). Any type and combination of input devices may be used depending on design choices. In some embodiments, there is no physical input device and / or display. For example, a user may interact with computing system 500 remotely via another computing system that communicates with computing system 500, or computing system 500 may operate autonomously.

[0097] The memory 515 stores software modules that provide functionality when executed by the processor(s) 510. The modules include an operating system 540 for the computing system 500. The modules also include a module 545 (e.g., a module implementing a transformation engine) that is configured to perform all or part of the processes described herein, or derivatives thereof.

[0098] According to one or more embodiments, module 545 can map a single first field to a single target field, module 545 can map a single first field to one or more target fields, module 545 can map multiple first fields to a single target field, and module 545 can map multiple first fields to multiple target fields. Module 545 can provide an input field for prompting input. Module 545 can select a conversion option from one or more options to convert the source data of any source field. The source data can be in a first format. Module 545 can convert the source data to converted data based on the conversion option and input. The converted data can be in a second format, which is different from the first format. Module 545 can provide the converted data to one or more target fields (e.g., mapped fields).

[0099] In addition, module 545 also has AI / ML functions to provide source data to a model with transformation options and inputs (e.g., an AI / ML model including GPT-3 or any other large language model) and obtain transformed data from the model. Computing system 500 may include one or more additional function modules 550 that include additional functions.

[0100] Those skilled in the art will appreciate that a "system" may be embodied as a server, an embedded computing system, a personal computer, a console, a personal digital assistant (PDA), a mobile phone, a tablet computing device, a quantum computing system, or any other suitable computing device or device combination, without departing from the scope of one or more embodiments herein. The above functions are represented as being performed by a "system" and are not intended to limit the scope of the embodiments herein in any way, but are intended to provide an example in many embodiments. In fact, the methods, systems, and devices disclosed herein may be implemented in a localized and distributed form consistent with computing technology (including cloud computing systems). The computing system may be a part of a local area network (LAN), a mobile communication network, a satellite communication network, the Internet, a public or private cloud, a hybrid cloud, a server farm, any combination thereof, or may be otherwise accessed by them. Any localized or distributed architecture may be used without departing from the scope of one or more embodiments herein.

[0101] It should be noted that certain system features described in this specification have been presented as modules in order to more specifically emphasize their implementation independence. For example, a module can be implemented as a hardware circuit, including a custom very large scale integrated circuit (VLSI) circuit or gate array, an off-the-shelf semiconductor (e.g., a logic chip, a transistor, or other discrete component). A module can also be implemented in a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, a graphics processing unit, or other device.

[0102] Modules can also be implemented at least in part in software for various types of processors to implement. Identified executable code units can include one or more physical or logical blocks of computer instructions, which can be, for example, organized as objects, processes, or functions. Nevertheless, the executable files of identified modules do not have to be physically located together, but can include different instructions stored in different locations, which, when logically combined together, constitute modules and realize the described purpose of modules. In addition, modules can be stored on computer-readable media, which can be, for example, hard disk drives, flash memory devices, RAM, magnetic tapes, and / or any other such non-transient computer-readable media for storing data, without departing from the scope of one or more embodiments herein.

[0103] In practice, a module of executable code may be a single instruction or multiple instructions, and may even be distributed across several different code segments, between different programs, and across multiple memory devices. Similarly, operational data may be identified and described within a module herein, and may be embodied in any suitable form and organized in any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed across different locations, including different storage devices, and may exist, at least in part, only as electronic signals on a system or network.

[0104] Various types of AI / ML models may be trained and deployed without departing from the scope of one or more embodiments herein. For example, Figure 6 An example of a neural network 600 that has been trained to recognize graphical elements in an image is shown in accordance with one or more embodiments. Here, the neural network 600 receives as input the pixels of a screenshot image of a 1920×1080 screen (represented by column 610) for input “neurons” 1 through I of an input layer (represented by column 620). In this case, I is 2,073,600, which is the total number of pixels in the screenshot image.

[0105] Neural network 600 also includes multiple hidden layers (represented by columns 630 and 640). Both DLNNs and shallow learning neural networks (SLNNs) typically have multiple layers, although SLNNs may have only one or two layers in some cases and typically have fewer layers than DLNNs. Typically, a neural network architecture includes an input layer, multiple intermediate layers (e.g., hidden layers), and an output layer (represented by column 650), which is the case with neural network 600.

[0106] DLNNs typically have many layers (e.g., 10, 50, 200, etc.), and subsequent layers typically reuse features from previous layers to compute more complex, more general functions. On the other hand, SLNNs tend to have only a few layers and are relatively fast to train because expert features are created in advance from raw data samples. However, feature extraction is laborious. On the other hand, DLNNs typically do not require expert features, but tend to take longer to train and have more layers.

[0107] For both approaches, the layers are trained simultaneously on a training set, usually checking for overfitting on an isolated cross-validation set. Both techniques can give excellent results, and there is a lot of interest in both approaches. The optimal size, shape, and number of individual layers depends on the problem being solved by the respective neural network.

[0108] Back to Figure 6, the pixels provided as the input layer are fed to the J neurons of the hidden layer 1 as input. Although all pixels are fed to each neuron in this example, various architectures may be used alone or in combination, including but not limited to feedforward networks, radial basis networks, deep feedforward networks, deep convolutional inverse graph networks, convolutional neural networks, recurrent neural networks, artificial neural networks, long / short-term memory networks, gated recurrent unit networks, generative adversarial networks, liquid machines, autoencoders, variational autoencoders, denoising autoencoders, sparse autoencoders, extreme learning machines, echo state networks, Markov chains, Hopfield networks, Boltzmann machines, restricted Boltzmann machines, deep residual networks, Kohonen networks, deep belief networks, deep convolutional networks, support vector machines, neural Turing machines, or any other suitable type or combination of neural networks without departing from the scope of one or more embodiments herein.

[0109] Hidden layer 2 (630) receives input from hidden layer 1 (620), hidden layer 3 receives input from hidden layer 2 (630), and so on for all hidden layers until the final hidden layer (represented by ellipse 655) uses its output as input for the output layer. It should be noted that the number of neurons I, J, K, and L is not necessarily equal, and thus, any desired number of layers may be used for a given layer of neural network 600 without departing from the scope of one or more embodiments herein. In fact, in some embodiments, the types of neurons in a given layer may not all be the same.

[0110] The neural network 600 is trained to assign confidence scores to graphical elements that are believed to have been found in the image. In some embodiments, in order to reduce matches with unacceptably low likelihoods, only results with confidence scores that meet or exceed a confidence threshold may be provided. For example, if the confidence threshold is 80%, the outputs with confidence scores exceeding that value may be used, while the remaining outputs are ignored. In this case, the output layer indicates that two text fields (represented by outputs 661 and 662), a text label (represented by output 663), and a submit button (represented by output 665) were found. The neural network 600 may provide the location, size, image, and / or confidence scores for these elements without departing from the scope of one or more embodiments herein, which may then be used by an RPA robot or another process that uses the output for a given purpose.

[0111] It should be noted that neural networks are probabilistic constructs and typically have a confidence score. This can be a score that the AI / ML model learns during training based on how often it correctly identifies similar inputs. For example, text fields often have a rectangular shape and a white background. A neural network can learn to identify graphical elements with these characteristics with high confidence. Some common types of confidence scores include a decimal number between 0 and 1 (which can be interpreted as a confidence percentage), a number between negative ∞ and positive ∞, or a set of expressions such as "low", "medium", and "high". Various post-processing calibration techniques can also be employed to try to obtain more accurate confidence scores, such as temperature scaling, batch normalization, weight decay, negative log-likelihood (NLL), etc.

[0112] A "neuron" in a neural network is a mathematical function, usually based on the function of a biological neuron. A neuron receives weighted inputs and has a summation and activation function that controls whether the neuron passes the output to the next layer. The activation function can be a nonlinear threshold activity function, where nothing happens if the value is below the threshold, but if the value is above the threshold the function responds linearly (i.e., a rectified linear unit ReLU nonlinearity). The summation function and the ReLU function are used for deep learning because real neurons can have roughly similar activity functions. Information can be subtracted, added, etc. via linear transformations. In essence, the neuron acts as a gating function, passing the output to the next layer according to the control of its underlying mathematical function. In some embodiments, different functions may be used for at least some neurons.

[0113] Figure 7 An example of a neuron 700 is shown in FIG. 7A . The input x1, x2, ..., x1 from the previous layer n are assigned corresponding weights w1, w2, ..., w n . Therefore, the collective input from the previous neuron 1 is w1x1. These weighted inputs are used in the neuron summation function modified by the bias, for example:

[0114]

[0115] This sum is compared to the activation function f(x) (represented by block 710) to determine whether the neuron "fires." For example, f(x) may be given by the following formula:

[0116]

[0117] Therefore, the output y of neuron 700 can be given by the following formula:

[0118]

[0119] In this case, neuron 700 is a single layer perceptron. However, any suitable neuron type or combination of neuron types may be used without departing from the scope of one or more embodiments herein. It should also be noted that in some embodiments, the value ranges of the weights and / or the output values ​​of the activation function(s) may be different without departing from the scope of one or more embodiments herein.

[0120] Typically a goal or "reward function" is employed, such as in this case successfully identifying a graphic element in an image. The reward function explores intermediate transitions and steps with short-term and long-term rewards to guide the search of the state space and attempt to achieve the goal (e.g., successfully identifying a graphic element, successfully identifying the next sequence of activities for an RPA workflow, etc.).

[0121] During training, various labeled data (in this case, images) are fed through the neural network 600. Successful identifications strengthen the weight of the neuron input, while failed identifications weaken the weight of the neuron input. A cost function (such as mean squared error MSE or gradient descent) can be used to penalize slightly wrong predictions, with the penalty for slightly wrong predictions being much smaller than the penalty for very wrong predictions. If the performance of the AI / ML model does not improve after a certain number of training iterations, the data scientist can modify the reward function to provide indications of the locations of unidentified graphic elements, provide corrections for incorrectly identified graphic elements, etc.

[0122] Backpropagation is a technique for optimizing the synaptic weights in a feed-forward neural network. Backpropagation can be used to "uncover" the hidden layers of a neural network to see how much loss each node contributes to, and then update the weights to minimize the loss by giving the smallest weights to nodes with higher error rates (and vice versa). In other words, backpropagation allows data scientists to repeatedly adjust weights to minimize the difference between actual and desired outputs.

[0123] The back-propagation algorithm is mathematically based on optimization theory. In supervised learning, training data with known outputs is passed through a neural network and the error is calculated using a cost function based on the known target output, which gives the error for back-propagation. The error is calculated at the output and converted into a correction to the network weights to minimize the error.

[0124] An example of backpropagation in the context of supervised learning is provided below. A column vector input x passes through a series of N nonlinear activation functions f between each layer i=1,...,N of the network. i The process is performed where the output of a given layer is first multiplied by the synaptic matrix W i , and add the bias vector b i The network output o is given by:

[0125] o=fN (W N f N-1 (W N-1 f N-2 (...f1(W1x+b1)...)+b N-1 )+b N ) (4)

[0126] In some embodiments, o is compared with the target output t to obtain an error It is desirable to minimize the error.

[0127] An optimization in the form of a gradient descent procedure can be used to modify the synaptic weights W for each layer by i to minimize the error. The gradient descent process requires computing the output o given an input x corresponding to a known target output t, and produces an error ot. This global error is then propagated backwards to produce local errors for weight updates, which are calculated similarly to, but not identically, the forward pass. Specifically, the backpropagation step typically requires an activity function p of the form j (n j ) = f j ′(n j ), where n j is the network activity at layer j (i.e., n j =W j o j-1 +b j ), where o j =f j (n j ), the prime symbol ′ represents the derivative of the activity function f.

[0128] The weight update can be calculated by the following formula:

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] in represents the Hadamard product (i.e., the element-wise product of two vectors), T represents the matrix transpose, and o j represents f j (W j o j-1 +b j), where o0 = x. Here, the learning rate η is chosen based on machine learning considerations. Below, η is related to the neural Hebbian learning mechanism used in the neural implementation. Note that the synapses W and b can be combined into a large synaptic matrix, where the input vector is assumed to have a vector attached, and additional columns representing b synapses are included in W.

[0135] The AI / ML model may be trained for multiple epochs until a good level of accuracy is achieved (e.g., 97% accuracy or higher for approximately 2000 epochs using an F2 or F4 threshold for detection). In some embodiments, the level of accuracy may be determined using an F1 score, an F2 score, an F4 score, or any other suitable technique without departing from the scope of one or more embodiments herein. Once trained on the training data, the AI / ML model may be tested on a set of evaluation data that the AI / ML model has not previously encountered. This helps ensure that the AI / ML model does not "overfit" such that it is able to recognize graphical elements in the training data well but does not generalize well to other images.

[0136] In some embodiments, it may not be known what level of accuracy the AI / ML model is capable of. Therefore, if the accuracy of the AI / ML model begins to decline when analyzing the evaluation data (i.e., the model performs well on the training data but begins to perform poorly on the evaluation data), the AI / ML model may be trained for more epochs on the training data (and / or new training data). In some embodiments, the AI / ML model is deployed only when a certain level of accuracy is achieved or the accuracy of the trained AI / ML model is better than that of an existing deployed AI / ML model.

[0137] In some embodiments, a set of trained AI / ML models can be used to complete tasks, such as employing an AI / ML model for each graphical element of interest, employing an AI / ML model to perform OCR, deploying yet another AI / ML model to identify proximity relationships between graphical elements, employing yet another AI / ML model to generate RPA workflows based on the output of other AI / ML models, etc. For example, this can collectively allow AI / ML models to achieve semantic automation.

[0138] Some embodiments may use transformer networks, such as SentenceTransformers TM , a Python library for state-of-the-art sentence, text, and image embeddings TMFramework. This transformer network learns associations between words and phrases with high and low scores. This trains the AI / ML model to determine what is close to the input and what is not. The transformer network can use not only word pairs / phrase pairs but also field lengths and field types.

[0139] Figure 8 8 is a flow chart illustrating a process 800 for training an AI / ML model according to one or more embodiments. Note that the process 800 can also be applied to other UI learning operations, such as for NLP and chatbots. At block 810, the process begins with training data, such as providing Figure 8 Labeled data is shown, such as labeled screens (e.g., with identified graphical elements and text), words and phrases, a "thesaurus" of semantic associations between words and phrases so that words and phrases similar to a given word or phrase can be identified, etc. The nature of the training data provided will depend on the goals to be achieved by the AI / ML model. The AI / ML model is then trained for multiple epochs at block 820 and the results are reviewed at block 830.

[0140] If the AI / ML model fails to meet the desired confidence threshold at decision block 840 (process 800 proceeds according to the "no" arrow), the training data is supplemented and / or the reward function is modified at block 850 to help the AI / ML model better achieve its goal, and the process returns to block 820. If the AI / ML model meets the confidence threshold at decision block 840 (process 800 proceeds according to the "yes" arrow), the AI / ML model is tested on evaluation data at block 860 to ensure that the AI / ML model has good generalization ability and that the AI / ML model is not overfitted relative to the training data. The evaluation data may include screens, source data, etc. that have not been processed by the AI / ML model before. If the evaluation data meets the confidence threshold at decision block 870 (process 800 continues according to the "yes" arrow), the AI / ML model is deployed at block 880. If not (process 800 continues according to the "no" arrow), the process returns to block 880 and further trains the AI / ML model.

[0141] Fig. 9 9 is a flow chart illustrating a process 900 of providing a conversion engine according to one or more embodiments. According to one or more embodiments, Figure 8 The process 800 and Fig. 9The process 900 performed in the embodiment of the present invention may be performed by a computer program that encodes instructions for a processor. The computer program may be embodied on a non-transitory computer readable medium. The computer readable medium may be (but is not limited to) a hard drive, a flash memory device, a RAM, a magnetic tape, and / or any other such medium or combination of media for storing data. The computer program may include a processor (e.g., Figure 5 510 of the computing system 500) to implement the coded instructions Figure 8-9 All or part of the process steps described in the present invention may also be stored on a computer-readable medium.

[0142] In general, process 900 can be thought of as an algorithm of a transformation engine that uses a model to extract, manipulate, and translate existing source data into well-structured translated data.

[0143] Process 900 begins at dashed block 910. Note that the dashed line of dashed block 910 indicates that this operation is optional. At block 910, the transformation engine performs mapping. The mapping operation includes when the transformation engine determines which data from a first location (e.g., a document, page, or table) needs to be pasted to one or more subsequent locations.

[0144] According to one or more embodiments, the mapping of the conversion engine includes mapping one or more source fields to one or more target fields. For example, the conversion engine may map a single first field to a single target field, the conversion engine may map a single first field to one or more target fields, the conversion engine may map multiple first fields to a single target field, and the conversion engine may map multiple first fields to multiple target fields. Examples of source fields and target fields may include, but are not limited to, text fields, radial buttons, drop-down menus, and other interface elements.

[0145] Go to Fig.10 , a mapping example 1000 is depicted according to one or more embodiments. The mapping example 1000 includes a first location 1010 and a second location 1020. The first location 1010 includes one or more source fields configured in a first format (e.g., a date, address, time, temperature, currency, etc. in a U.S. format). The second location 1020 includes one or more target fields configured in a second format (e.g., a date, address, time, temperature, currency, etc. in a European format). As indicated by the arrows between the first location 1010 and the second location 1020, the transformation engine can map one or more source fields to one or more target fields. For example, a name source field can be mapped to a first and last name target field, a birthday source field can be mapped to a birthday target field, a nationality source field can be mapped to a country target field, and a gender source field can be mapped to a sex target field.

[0146] return Fig. 9 , at block 920, the conversion engine selects a conversion option to convert source data of one or more source fields. The conversion option may define how the conversion will manipulate the source data into another format. The conversion option may be selected from one or more options. Note that the conversion option may be selected automatically or by user input. One or more options include, but are not limited to, date conversion options, address conversion options, currency conversion options, and temperature conversion options. In addition, one or more options include, but are not limited to, extracting subfields (e.g., extracting cities from long-form addresses or extracting surnames from full names), classifying text into a list of categories (e.g., sentiment classification, unacceptable language, message classification), spell checking, extracting keywords (e.g., extracting keywords from phrases), direct question and answer (e.g., closed book mode), and generating names (e.g., conditional generation). According to one or more embodiments, the source data may be in a first format. For example, the source data may include batch information copied from a first table configured in a first format containing one or more source fields.

[0147] At block 940, the transformation engine provides a field that prompts for input. Once the transformation engine maps the fields of the source and target, the transformation engine can generate fields and prompts on the interface that receive input and cause transformation. The prompt can include a tool tip to guide what query to put in the field. The input can be automatically placed in the field.

[0148] Go to Fig.11 , depicts interface 1100 according to one or more embodiments. Note that Figure 11-14 A set of continuous interfaces (ie, Fig.11 An interface 1100 is depicted, Fig.12 depicts an interface 1200, Fig.13 depicts an interface 1300, Fig.14 Interface 1400 is depicted).

[0149] Interface 1100 displays information area 1101, source area 1103, target area 1104, unmatched area 1105, cancel icon 1106, and paste data icon 1107. Information area 1101 may include a message, such as "We found that this is an invoice, but you can adjust the extraction by exploring different models." Cancel icon 1106 may cause the transformation engine to exit interface 1100. Paste data icon 1107 may cause the transformation engine to implement the transformation of the source data to the target (e.g., after mapping).

[0150] Source region 1103 identifies the first, initial, or previous document, form, or page analyzed by the conversion engine. Aligned with source region 1103 (e.g., which identifies a document titled "Invoice: Monoprice, Inc."), user interface 1100 depicts source boxes 1121, 1122, 1123, 1124, 1125, and 1126. Source boxes 1121, 1122, 1123, 1124, 1125, and 1126 represent portions of user interface 1100 that include source fields and, in some cases, source data. For example, source box 1121 may include a "supplier name" source field that includes "Monoprice, Inc." source data. Source box 1122 may include a "supplier address" source field that includes "1534 S. Ave, Townson, CA, 12345, US" source data. Source box 1123 may include an "invoice number" source field that includes "19583668" source data. Source box 1124 may include a "Billing Name" source field containing source data for "Hatch Company." Source box 1125 may include a "Shipping Address" source field containing source data for "Hatch Company 5600N Dr., City, CA, 54321." Source box 1126 may include a "Billing Address" source field containing source data for "PO Box 123456, City, CA, 54321, United States."

[0151] Target region 1104 identifies a separate, subsequent, or different document, form, or page to be analyzed by the transformation engine. Aligned with target region 1104 (e.g., which identifies a target titled "Invoice System"), user interface 1100 depicts target boxes 1131, 1132, 1133, and 1136. Target boxes 1131, 1132, 1133, and 1136 represent portions of user interface 1100 that include target fields that will ultimately receive transformed data. For example, target box 1131 may include a "Company Name" target field, target box 1132 may include a "Company Address" target field, target box 1133 may include an "Invoice Number" target field, and target box 1136 may include a "Billing Address" target field.

[0152] like Fig.111150, the transformation engine can map the fields of source boxes 1121, 1122, 1123, and 1126 to target boxes 1131, 1132, 1133, and 1136. Within the unmatched area 1105, the user interface 1100 depicts unmatched boxes 1144 and 1145. Unmatched boxes 1144 and 1145 represent aspects of separate, subsequent, or different documents, forms, or pages for which the transformation engine was unable to identify corresponding source fields for mapping. Unmatched box 1144 can include an "address" field (e.g., an additional address line field), and unmatched box 1145 can include a "bill to..." field.

[0153] In addition, the interface 1100 may include one or more conversion buttons, such as conversion button 1160. One or more conversion buttons may be provided for any of the source boxes 1121, 1122, 1123, 1124, 1125, and 1126. According to one or more embodiments, the user may cause the conversion engine to provide fields that prompt for input and cause conversion. For example, once the conversion engine maps the fields of the source and the target, the conversion engine may generate fields and prompts on the interface 1100 that receive input and cause conversion. For example, the user may select the conversion button 1160 within the interface 1100, which may cause the conversion engine to generate fields and prompts that receive input and cause conversion according to one or more embodiments. Figure 12-14 A set of consecutive interfaces 1200, 1300 and 1400.

[0154] Back to Fig. 9 As shown in sub-block 942, according to one or more embodiments, the conversion engine may provide the field within the user interface for display to the user on the screen. Fig.12 and interface 1200, providing the user with input fields 1220 and tooltips 1230 to guide the user as to what queries they can place. Tooltips 1230 may include text. For example, tooltips 1230 may include "Convert quantities to other units and explore comparisons with specific objects and corresponding quantities. For example: How many dollars is this equivalent to? For example: How many feet is that? For example: Is this in Johnson County? For example: USD, EUR. For example: MM / DD / YY".

[0155] Back to Fig. 9, as shown in sub-block 944, according to one or more embodiments, the conversion engine can receive input via the field to convert the source data into the converted data (as shown in sub-block 944). Note that the input can include a natural language query, such as a query entered by a user. According to one or more embodiments, the conversion engine can utilize natural language processing (e.g., using GPT-3's AI / ML model) to interpret the sentiment of the comments / queries posed by the user as positive or negative. In some cases, specific values ​​(e.g., city names, street names, building numbers, etc.) are used for natural language processing queries from GPT-3. Go to Fig.13 and interface 1300, providing a query 1320 (e.g., "Is this in Chestercopperpot County?"). The conversion engine can utilize natural language processing (e.g., using GPT-3's AI / ML model) to interpret the query 1320. Go to Fig.14 And interface 1400, after pressing the enter key, provides an answer 1420 (eg, "yes"). Answer 1420 can be placed into the corresponding target field.

[0156] Back to Fig. 9 At block 960, the transformation engine transforms the source data into transformed data based on the transformation options and the input. Note that the transformed data is in the second format.

[0157] According to one or more embodiments and as shown in sub-block 962, converting the source data may include providing the source data to a model having conversion options and inputs. In addition, as shown in sub-block 964, converting the source data may include obtaining the converted data from the model. The model of the conversion engine may be a natural language processing, such as GPT. The model may perform one or more operations to complete the conversion task. According to one or more embodiments, the model may understand a second / target format. In addition, converting the data may be based on a user command, or the data may be directly converted.

[0158] In an operation example, the model can generate parameters for the "convert()" function. This operation example can be applied to numeric and date conversions. For example, if the value and command include "value: 4pm" and "command: Tokyo time to New York" (or in the source data), respectively, the model of the conversion engine can respond with "result: convert('16:00', 'Tokyo', 'New York')". In another operation example, the model can directly generate answers for string values. This operation example can be applied to information extraction, question answering, classification, and other tasks that require semantic processing rather than numerical precision. For example, if the value and command include "value: This works well" and "command: Sentiment classification (positive / negative)", respectively, the model of the conversion engine can respond with "result: value('positive')". Note that value('positive') means that the text generated or extracted by the model does not need to be converted and can be returned. Go to Fig.15 , example code 1500 for a transformation engine, according to one or more embodiments. The example code 1500 includes seven (7) transformation examples implemented by the transformation engine.

[0159] According to one or more embodiments, the transformation engine may determine the scope of data transformation during mapping, such as whether advanced transformation or general transformation is required.

[0160] The conversion engine may use built-in conversion models (e.g., natural processing models) to perform general conversions. For example, built-in conversion models for general conversions include, but are not limited to, the phrase "USD to EUR", where the conversion engine applies the formula " <unit1>to <unit2>"or" <unit1>in <unit2>The transformation engine supports thousands of units, such as physical units, date formats, currency conversions with current exchange rates, and address subfield extraction (e.g., without using an AI / ML model using GPT-3). Based on the natural query for the transformation, the transformation engine's natural processing model understands whether the transformation is a date, currency, or temperature (e.g., a general transformation) and uses built-in transformers containing transformation expressions to transform date, currency, or temperature data.

[0161] The transformation engine can perform advanced transformations using natural language processing. According to one or more embodiments, the transformation engine overcomes the rough and cumbersome operation of traditional clipboard software by using natural language processing or natural processing models (e.g., AI / ML models using GPT-3). For example, the transformation engine can use GPT-3 to query "Is this located at 'XYZ'?" from the data based on the address, and can provide "yes / no" on the target form. Therefore, any question about input X can be submitted to the transformation engine, and the transformation can extract or classify input X.

[0162] If the second / target format is known, the conversion engine can automatically generate a natural language processing query for the conversion, otherwise the conversion query needs to be triggered manually. If the conversion engine matches the task to a known formula (e.g., "kilometers to meters"), a call to the conversion function can be automatically generated. The conversion engine's conversion function can be represented by "convert(value, input_format, output_format)", and the role of the conversion engine's big language model is to fill "value", "input_format" and "output_format" into the function. Note that the conversion engine can run the function to ensure that the conversion is numerically correct and then return the result.

[0163] According to one or more embodiments, when a user needs to use a natural language processing command to convert a unit, but the format of the natural language processing command is incorrect, and the unit cannot be converted due to incorrect command format (each conversion function has a command format). In this case, when the natural language processing command is incorrect, the conversion engine sends a request to GPT-3 to extract the correct parameters (i.e., values ​​and commands) of the command given by the user, and inputs the parameters to the correct function for conversion. That is, the conversion is not completed using GPT-3, but GPT-3 is used to extract parameters. If the user knows the function and its command for conversion, the conversion can be performed directly by calling the function (single call). Calling GPT-3 is only used to determine the correct parameters. Therefore, GPT-3 is used to provide a direct answer (city example) or call the parameters of the function (conversion example). The function is not limited to unit conversion, and any number of functions can be implemented, such as numbers, dates, and times. Use GPT3 to parse the intent and parameters of the function.

[0164] At block 980, the transformation engine provides the transformed data to one or more target fields. Fig.16 A transformation example 1600 is depicted according to one or more embodiments. The transformation example 1600 includes a source 1610, a transformation 1620 (eg, driven by GPT3), and a target 1630. Fig.16 As shown by arrow 1650 in , the transformation engine can provide transformed data from the source data of the source 1610 to the target 1630 based on the transformation 1620.

[0165] A computer program may be implemented in hardware, software or a hybrid implementation. A computer program may consist of modules that are in operable communication with each other and are designed to pass information or instructions for display. A computer program may be configured to run on a general purpose computer, an ASIC or any other suitable device.

[0166] It is readily understood that the components of the various embodiments as generally described and illustrated herein may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments as shown in the drawings is not intended to limit the scope of what is claimed, but is merely representative of selected embodiments.

[0167] The features, structures or characteristics described in this specification may be combined in one or more embodiments in any suitable manner. For example, references to "certain embodiments", "some embodiments" or similar language in this specification mean that a particular feature, structure or characteristic associated with an embodiment is included in at least one embodiment. Therefore, the phrases "certain embodiments", "in some embodiments", "in other embodiments" or similar language appearing in this specification do not necessarily refer to the same set of embodiments, and the features, structures or characteristics may be combined in one or more embodiments in any suitable manner.

[0168] It should be noted that references to features, advantages, or similar language in this specification do not mean that all features and advantages that may be realized should or are present in any single embodiment. Rather, language referencing features and advantages should be understood to mean that a particular feature, advantage, or characteristic described in connection with an embodiment is included in one or more embodiments. Therefore, discussions of features and advantages and similar language in this specification may, but do not necessarily, refer to the same embodiment.

[0169] In addition, the features, advantages and characteristics of one or more embodiments described herein may be combined in any suitable manner. Those skilled in the relevant art will recognize that the present disclosure may be implemented without one or more specific features or advantages of a particular embodiment. In other cases, additional features and advantages that may not be present in all embodiments may be recognized in some embodiments.

[0170] Those of ordinary skill in the art will readily appreciate that the present disclosure may be implemented with steps in different orders and / or with hardware elements in configurations different from the disclosure. Therefore, although the present disclosure is described based on these preferred embodiments, certain modifications, variations, and alternative configurations will be apparent to those of ordinary skill in the art while still being within the spirit and scope of the present disclosure. Therefore, in order to determine the limits and scope of the present disclosure, reference should be made to the appended claims.

Claims

1. A method performed by a transformation engine implemented as a computer program within a computing environment, the method comprising: selecting a conversion option to convert source data of a source field, wherein the source data is in a first format; Provides an input field for prompting input; Based on the conversion options and the input, converting the source data into converted data, wherein the converted data is in a second format; as well as The transformed data is provided to the target field.

2. The method of claim 1 , wherein prior to the selection of the conversion option, the conversion engine maps one or more source fields of a first document, form, or page including the source fields to one or more target fields of a second document, form, or page including the target fields.

3. The method of claim 2, wherein the source data comprises bulk information copied from the first document, form, or page, the first document, form, or page comprising the one or more source fields configured in the first format. The method of claim 2 , wherein the one or more target fields are configured in the second format. 5 . The method of claim 1 , wherein the transforming of the source data comprises providing the source data, the transform options, and the input to a model and obtaining the transformed data from the model.

6. The method of claim 1, wherein the input is automatically placed into the input field and causes a transformation engine to transform the source data into the transformed data.

7. The method of claim 1, wherein the conversion option is selected from one or more options including a date conversion option, an address conversion option, a currency conversion option, and a temperature conversion option.

8. The method of claim 1, wherein the input field is provided within a user interface for display to a user on a screen.

9. The method of claim 8, wherein the user interface includes a tool tip that provides guidance as to which query to place in the input field.

10. The method of claim 1, wherein the input comprises a natural language query.

11. A computer program product for a transformation engine, the transformation engine being stored as processor executable code on a memory of a computing environment and executed by at least one processor of the computing environment to cause operations within the computing environment, the operations comprising: selecting a conversion option to convert source data of a source field, wherein the source data is in a first format; Provides an input field for prompting input; Based on the conversion options and the input, converting the source data into converted data, wherein the converted data is in a second format; as well as The transformed data is provided to the target field.

12. A computer program product according to claim 11, wherein prior to the selection of the conversion option, the conversion engine maps one or more source fields of a first document, form or page including the source fields to one or more target fields of a second document, form or page including the target fields.

13. The computer program product of claim 12, wherein the source data comprises bulk information copied from the first document, form, or page, the first document, form, or page comprising the one or more source fields configured in the first format.

14. The computer program product of claim 12, wherein the one or more target fields are configured in the second format.

15. The computer program product of claim 11, wherein the transforming of the source data comprises providing the source data, the transformation options, and the input to a model and obtaining the transformed data from the model.

16. The computer program product of claim 11, wherein the input is automatically placed into the input field and causes a transformation engine to transform the source data into the transformed data.

17. The computer program product of claim 11, wherein the conversion option is selected from one or more options including a date conversion option, an address conversion option, a currency conversion option, and a temperature conversion option.

18. The computer program product of claim 11, wherein the input field is provided within a user interface for display to a user on a screen.

19. The computer program product of claim 18, wherein the user interface includes a tool tip that provides guidance as to which query to place in the input field.

20. The computer program product of claim 11, wherein the input comprises a natural language query.

Images