Content-based verification

Abstract

The methods, products, and devices implemented on an administrator user's end device include selecting a field on a page of a third-party application, defining a trigger event to identify that an end user has entered an input into the field, and defining an automation process to be executed in response to the trigger event, wherein the definition includes defining validation rules using free text in natural language, the automation process is configured to generate prompts to a generative artificial intelligence (AI) engine, the prompts include validation rules and instructions to determine whether the input conforms to the validation rules, and the definition includes defining a configuration for displaying the output from the generative AI on the page.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of Provisional Patent Application No. 63 / 530,184, titled "Large Language Model - based Rules In Digital Adoption Platforms", filed on August 1, 2023, which is hereby incorporated by reference in its entirety without disclaimer.

[0002] This disclosure generally relates to the use of large language models (LLMs) for input validation, particularly content - based input validation.

Background Art

[0003] A digital adoption platform may be an integrated software platform designed to facilitate and simplify the adoption and use of digital tools, applications, and software systems within an organization. The digital adoption platform provides interactive guidance, context assistance, and personalized training to users, enabling them to navigate and effectively utilize complex software interfaces and functions. By providing real - time, step - by - step guidance and performance analysis, the digital adoption platform enables businesses to improve user productivity, reduce training costs, and maximize the return on investment in digital initiatives.

[0004] Advances in artificial intelligence and natural language processing have led to the development of sophisticated models capable of understanding, generating, and processing human language on an unprecedented scale. For example, language models such as Large-Scale Language Models (LLMs) or Small-Scale Language Models (SMLs) constitute state-of-the-art machine learning models trained on vast amounts of text data, enabling them to understand and generate human-like text in various languages. These models use deep neural network architectures to learn patterns, semantics, and contextual information from text input, enabling them to perform tasks such as language translation, text summarization, sentiment analysis, and even creative writing.

[0005] By leveraging vast knowledge bases and linguistic proficiency, language models such as LLMs have the potential to transform a wide range of fields, particularly content generation, customer support, language comprehension, and information retrieval. Their diverse capabilities make LLMs invaluable tools for businesses and researchers seeking to leverage the power of natural language processing in these applications. Some notable examples of publicly available LLM products include ChatGPT®, BARD®, and BING® Chat. [Overview of the project] [Means for solving the problem]

[0006] One exemplary embodiment of the disclosed subject is a method for being performed on an administrator user's end device, the method comprising selecting a page element on a page of a third-party application, the page element comprising a field, the third-party application being executable on the administrator user's end device and multiple end devices of end users, the method further comprising defining a trigger event, the trigger event comprising identifying that an end user has entered an input into a field, the method further comprising defining an automation process to be performed in response to the occurrence of the trigger event, the automation process being configured to generate prompts to a generative artificial intelligence (AI) engine to incorporate at least input and validation rules, and the said Defining an automation process involves defining validation rules using free text in natural language, where prompts are configured to be generated to include a predetermined structure of static and dynamic parts, the dynamic part being configured to be populated with input from an end user in response to the firing of each trigger event, the static part being configured to include validation rules and instructions to determine whether the input conforms to the validation rules, the automation process is configured to send prompts to a generative AI engine and to obtain output from the generative AI engine in response to the prompts, the method further includes defining a configuration for displaying the results on a page, the results being determined based on the output from the generative AI engine, and the results indicating at least whether the input conforms to the validation rules.

[0007] Selectively, prompts are configured to instruct the generative AI engine to provide content-based feedback on the input, which includes suggestions on how to refine the input in a format that conforms to validation rules.

[0008] Selectively, a configuration for displaying results includes updating one or more characteristics of a page based on the results, the updating of one or more characteristics including at least one of the following: the color of a field border, the color of a field background, or a field highlight; and displaying the results as an overlay on the page, the overlay may be configured to be displayed on the page, the overlay is not part of a third-party application, and the overlay includes at least one of the following: a chat widget, a tooltip, a popup element, or a text field.

[0009] Selectively, the dynamic portion is configured so that context data is populated each time a trigger event is activated, and this context data includes data from the page.

[0010] Selectively, the data from the page includes the names of other fields on the page and at least some inputs into those other fields.

[0011] Selectively, data from a page includes validation rules for other fields on that page.

[0012] Selectively, other fields include form fields, and validation rules for other fields are defined to validate end-user input into form fields.

[0013] Selectively selecting the page elements, defining trigger events, defining automation processes, and defining configurations are performed via a digital adoption platform running on the administrator user's end device, the digital adoption platform being independent of third-party applications, and configured to enable administrator users to use the digital adoption platform to generate assistance layers to run on third-party applications on multiple end devices, the assistance layers being configured to help end users perform digital tasks.

[0014] Selectively, the prompt commands include pre-configured commands for the digital adoption platform that have not been defined by the administrator user.

[0015] Selectively, digital tasks include completing one or more forms in a third-party application.

[0016] Selectively, the assistance layer includes a validation tooltip defined for the field, and the validation tooltip is configured to include the validation rule.

[0017] Selectively, the method further includes defining to display a guidance message to the end user before the input is entered into the field, and selecting a guidance message from one or more predefined sets of validation tooltips, the one or more predefined sets of messages being past messages for fields defined by one or more users in the organization to which the administrator user belongs.

[0018] Selectively, the configuration for displaying results includes displaying the results as messages within a chat widget, which is overlaid on the page, inputs to fields are provided to the chat widget, and the automation process is configured to provide a compliant summary of the inputs in the chat widget to the fields.

[0019] Selectively, the generative AI engine may include a large-scale language model (LLM) engine or a small-scale language model (SLM) engine.

[0020] Another exemplary embodiment of the disclosed subject matter is a device including a processor and coupled memory, wherein the processor is located in an administrator user's end device. A step of selecting a page element on a page of a third-party application, wherein the page element includes a field, and the third-party application is executable on the end device of an administrator user and on multiple end devices of end users. A step of defining a trigger event, the trigger event including identifying that an end user has entered an input into a field, A step of defining an automation process to be executed in response to the occurrence of a trigger event, wherein the automation process is configured to generate prompts to a generative artificial intelligence (AI) engine to incorporate at least inputs and validation rules, defining the automation process includes defining validation rules using free text in natural language, the prompts are configured to be generated to include a predetermined structure of static and dynamic parts, the dynamic part is configured to be populated with input from an end user in response to the activation of each trigger event, the static part is configured to include validation rules and instructions to determine whether the input conforms to the validation rules, and the automation process is configured to send prompts to a generative AI engine and to obtain output from the generative AI engine in response to the prompts. A device adapted to perform the steps of: defining a configuration for displaying results on a page, wherein the results are determined based on the output from a generative AI engine, and the results indicate at least whether the input conforms to validation rules.

[0021] A further exemplary embodiment of the disclosed subject matter is a computer program product comprising a non-temporary computer-readable medium holding program instructions, wherein, when the program instructions are read by a processor, the processor causes the processor to execute a method on an administrator user's end device, the method comprising selecting a page element on a page of a third-party application, the page element comprising a field, the third-party application being executable on the administrator user's end device and multiple end devices of end users, the method further comprising defining a trigger event, the trigger event comprising identifying that an end user has entered an input into a field, the method further comprising defining an automation process to be executed in response to the occurrence of the trigger event, the automation process comprising generating AI to incorporate at least input and validation rules The method is configured to generate prompts to an engine, and defining the automation process includes defining validation rules using free text in natural language, the prompts are configured to be generated to include a predetermined structure of static and dynamic parts, the dynamic part is configured to be populated with input from an end user in response to the invocation of each trigger event, the static part is configured to include validation rules and instructions to determine whether the input conforms to the validation rules, the automation process is configured to send prompts to a generative AI engine and to obtain output from the generative AI engine in response to the prompts, the method further includes defining a configuration for displaying the results on a page, the results being determined based on the output from the generative AI engine, and the results are a computer program product that indicates at least whether the input conforms to the validation rules.

[0022] One exemplary embodiment of the disclosed subject is a method for being implemented on an end-user's end device, the method comprising displaying a third-party application and an assistance layer on the end-user, the assistance layer running on the third-party application, the method further comprising obtaining user input from the end-user to a field on a page of the third-party application, the method further comprising displaying a message on the page to the end-user, the message being obtained from the assistance layer, the message indicating that the content of the user input does not conform to the validation rules of the assistance layer, the message providing content-based feedback on the user input, and content-based feed The method includes suggestions on how to modify the content of user input in a form that conforms to validation rules, the assistance layer is configured to generate prompts to a generative artificial intelligence (AI) engine, send the prompts to the generative AI engine, and obtain content-based feedback from the generative AI engine, the prompts include a predetermined configuration of a static part and a dynamic part, the dynamic part is configured to be populated with user input each time a field is filled in by an end user, the static part is configured to include validation rules, and the method further includes obtaining the modified user input to the field, the modified user input is obtained following the display of the message, and is a method to be implemented on the end user's end device.

[0023] Selectively, before obtaining user input, a guidance message is displayed to the end user, which includes predefined text that guides the end user on how to fill in the fields, and this predefined text is provided by the assistance layer builder.

[0024] Selectively, the generative AI engine may include a large-scale language model (LLM) engine or a small-scale language model (SLM) engine.

[0025] Optionally, the dynamic part is populated with context data from the page, and the context data includes at least one of the names of other fields on the page, the validation rules for the other fields, or the input from the end user to the other fields.

[0026] Optionally, the assistance layer includes a validation tool tip defined for the field, and the validation tool tip includes the validation rule.

[0027] Optionally, the message is displayed within the chat widget of the assistance layer, the chat widget is overlaid on the page, and the user input to the field is provided via the chat widget.

[0028] Another exemplary embodiment of the disclosed subject matter is an apparatus including a processor and coupled memory, wherein the processor, on an end-user's end device, performs the steps of: displaying a third-party application and an assistance layer to an end-user, the assistance layer running on the third-party application; obtaining user input from the end-user to fields on a page of the third-party application; and displaying a message to the end-user on the page, the message being obtained from the assistance layer, the message indicating that the content of the user input does not conform to the validation rules of the assistance layer, and the message providing content-based feedback for the user input. Content-based feedback includes suggestions on how to modify the content of user input in a form that conforms to validation rules; the assistance layer is configured to generate prompts to a generative AI engine, send prompts to the generative AI engine, and retrieve content-based feedback from the generative AI engine, the prompts having a predetermined structure of static and dynamic parts, the dynamic part being configured to be populated with user input each time a field is filled in by an end user, and the static part being configured to include validation rules; and the device is adapted to perform the steps of: obtaining modified user input to a field, the modified user input being obtained following the display of the message.

[0029] A further embodiment of the disclosed subject is a computer program product comprising a non-temporary computer-readable medium holding program instructions, which, once read by a processor, causes the processor to execute a method on an end-user's end device, the method comprising displaying a third-party application and an assistance layer to the end-user, the assistance layer being executed on the third-party application, the method further comprising obtaining user input from the end-user to fields on a page of the third-party application, the method further comprising displaying a message to the end-user on the page, the message being obtained from the assistance layer, the message indicating that the content of the user input does not conform to the validation rules of the assistance layer, and Sage is a computer program product that provides content-based feedback to user input, the content-based feedback includes suggestions on how to modify the content of the user input in a form that conforms to validation rules, the assistance layer is configured to generate prompts to a generative AI engine, send the prompts to the generative AI engine, and retrieve content-based feedback from the generative AI engine, the prompts include a predetermined structure of static and dynamic parts, the dynamic part is configured to be populated with user input each time a field is filled in by an end user, and the method further includes the static part being configured to include validation rules, and retrieving the modified user input to the field, the modified user input being retrieved following the display of the message.

[0030] The subject matter disclosed herein will be better understood and recognized from the following detailed description made in relation to the drawings, in which corresponding or identical numbers or letters indicate corresponding or identical components. Unless otherwise specified, the drawings provide exemplary embodiments or aspects of the disclosure and do not limit the scope of the disclosure. [Brief explanation of the drawing]

[0031] [Figure 1] A schematic diagram of an exemplary flowchart of the method is shown, based on several exemplary embodiments of the disclosed subject matter. [Figure 2A] This section provides an exemplary execution of the verification tooltip through several exemplary embodiments of the disclosed subject matter. [Figure 2B] An exemplary verification tooltip is provided with several exemplary embodiments of the disclosed subject matter. [Figure 2C] An exemplary verification tooltip is provided with several exemplary embodiments of the disclosed subject matter. [Figure 3A] This section illustrates an exemplary process for defining a verification tooltip, using several exemplary embodiments of the disclosed subject matter. [Figure 3B] This section illustrates an exemplary process for defining a verification tooltip, using several exemplary embodiments of the disclosed subject matter. [Figure 4A] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4B] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4C] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4D] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4E] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4F] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4G]This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 4H] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 5A] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 5B] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 5C] This section illustrates exemplary scenarios of user interaction with deployed validation tooltips through several exemplary embodiments of the disclosed subject matter. [Figure 6] An illustrative flowchart of the method is shown, with some exemplary embodiments of the disclosed subject matter. [Figure 7] An illustrative flowchart of the method is shown, with some exemplary embodiments of the disclosed subject matter. [Figure 8] An illustrative flowchart of the method is shown, with some exemplary embodiments of the disclosed subject matter. [Figure 9] This section provides schematic diagrams of exemplary architectures in which the disclosed subject matter may be used, based on several exemplary embodiments of the disclosed subject matter. [Figure 10] A schematic diagram of an exemplary environment in which the disclosed subject matter may be used is shown, with some exemplary embodiments of the disclosed subject matter. [Modes for carrying out the invention]

[0032] One technical challenge addressed by the disclosed subject matter is assisting human users when performing digital tasks. In some exemplary embodiments, a digital task may include actions or activities performed by a user using a digital device or platform. Digital tasks may vary in complexity and purpose and include a wide range of actions performed in the digital realm. In some cases, performing a digital task may include multiple operations such as navigating a website, clicking links, accessing different pages, selecting page elements, entering data into data fields, and selecting options from dropdown elements. In some exemplary embodiments, a digital task may be performed via a digital platform such as a software application, desktop application, web-based application, operating system, or software-as-a-service (SaaS) application. It may be desirable to assist end users by performing digital tasks efficiently, appropriately, correctly, and in a timely manner.

[0033] In some exemplary embodiments, assisting a user with a digital task may be performed using one or more platforms, such as a Digital Adoption Platform (DAP). For example, WalkMe® may constitute a DAP. In some exemplary embodiments, the DAP may be designed to help an end user navigate and interact with digital assets. In some exemplary embodiments, the DAP may include an editor that enables a user to generate an assistance layer that can be run on a third-party system (also called a “Target System”) to assist an end user of the third-party system in performing a digital task. For example, the third-party system may include any software product, application, or website, but may be separate from the DAP, may not collaborate with the DAP, and may not make application programming interface (API) calls to each other.

[0034] In some exemplary embodiments, the DAP Editor may be used by users such as administrator (admin) users of an organization that has access to the DAP. For example, a client of the DAP may include a registered company (e.g., a bank) that has access to the DAP Editor service. In some exemplary embodiments, the DAP Editor may include a software platform designed to facilitate and simplify the adoption and use of digital tools, applications, and software systems within an organization. For example, an admin user may define a walkthrough for a third-party application through the DAP Editor, and the walkthrough may be distributed to the end users of the third-party application. In some exemplary embodiments, in addition to generating an assistance layer, the DAP may enable an admin user to perform one or more statistical analyses, identify patterns of end-user interaction with digital assets, etc. For example, based on a statistical analysis of user performance when using a DAP-based walkthrough, an admin user may extract useful insights to modify the walkthrough, improve parts of it, etc.

[0035] In some exemplary embodiments, an admin user may be enabled to design the assistance layer using DAP building blocks, which may be pre-configured, non-coded building blocks. For example, a DAP building block may include interactive GUI elements or widgets such as launcher widgets, buttons, tooltips, chat windows, balloon layouts, or any other graphical user interface (GUI) elements or controls with pre-configured properties. In some exemplary embodiments, the assistance layer may be generated using a non-code platform such as a DAP editor without requiring the admin user to provide or modify any coding. In many cases, the digital adoption platform may be a non-code platform that allows non-programmers to define automation, rules, etc., related to a selected target system, its elements, etc., using a simple user interface. For example, via the DAP editor, the admin may design interactive guidance, interactive walkthroughs, contextual assistance, personalized training, contextual prompts, etc., for end users, enabling them to navigate and effectively utilize complex software interfaces and functions. In other cases, the assistance layer may be designed via a non-DAP platform.

[0036] In some cases, the assistance layer may be designed for use by end users such as the organization's employees, customers, web users, or any other population segment. In some cases, the assistance layer may include interactive GUI elements configured to present data to users, allow users to activate automation processes, etc. As an example, using the DAP editor, an admin user may design the assistance layer to include desired widgets (e.g., walkthrough elements), define the behavior of each widget (e.g., automation processes), define a sequence of one or more trigger events (with or without branch conditions) configured to invoke that behavior, or do the same.

[0037] In some exemplary embodiments, automation processes may be designed by an admin user, for example, via the DAP editor, to help end users complete tasks, learn new features, overcome obstacles, or similar actions. For example, a launcher widget (also called a “launcher”) may be designed by an admin user to trigger a predetermined content presentation, such as in response to user interaction, a defined event, etc. As another example, an admin user may design a custom tooltip (or “ShoutOuts”) via the DAP editor that is configured to draw the end user’s attention to featured text or elements in order to help end users understand the function or significance of the elements. As yet another example, an admin user may design one or more tooltips (or “SmartTips”) via the DAP editor that are configured to appear when an end user hovers the cursor over a designated element, such as a button, icon, or link, and to provide additional information about the purpose or function of the associated element. As another example, an admin user may design a validation tooltip or launcher configured to validate end-user input to a text field via the DAP interface, for example, indicating whether the input to the field conforms to a given rule. For example, the validation tooltip may be configured to automatically validate end-user input when selected by an end-user, etc.

[0038] In several exemplary embodiments, additional aspects of the digital adoption platform include, in particular, U.S. Patent No. 9,922,008 dated March 20, 2018, entitled “Calling-Scripts Based Tutorials,” U.S. Patent No. 9,934,782 dated April 3, 2018, entitled “Automatic Performance Of User Interaction Operations On A Computing Device,” U.S. Patent No. 10,819,664 dated October 27, 2020, entitled “Chat-Based Application Interface For Automation,” U.S. Patent No. 10,620,975 dated April 14, 2020, entitled “GUI Element Acquisition Using A Plurality Of Alternative Representations Of The GUI Element,” and “Acquisition Process Of GUI Elements Using User These are described in U.S. Patent No. 10,713,068, titled “Input,” and are all incorporated herein by reference in their entirety for all purposes without causing disavowal.

[0039] In some exemplary implementations, a DAP may have one or more drawbacks. For example, its ability to assist users in performing digital tasks may be limited. For instance, a DAP may lack sufficient text analysis capabilities, text generation capabilities, etc., to provide content-based support and guidance. Overcoming such drawbacks may be desirable.

[0040] Another technical challenge addressed by the disclosed means is the need to adapt platforms such as the DAP platform to provide end users with a wider range of assistance operations. For example, it may be desirable to extend the DAP build blocks in the DAP editor to provide more sophisticated DAP build blocks, automation processes, trigger events, etc.

[0041] Another technical challenge addressed by the disclosed subject is improving the performance of field validation to determine compliance of user input with content-based validation requirements. In some exemplary embodiments, the DAP may provide validation tooltips for validating fields using one or more regular expressions or syntax constraints as DAP building blocks. In some exemplary embodiments, such validation tooltips may be limited to performing simple validations such as determining whether a field is mandatory, whether input to a field meets a required string length, whether input is in the correct language (as selected by the admin user), whether numeric input is within a specified range, whether input matches a defined format, whether a password meets complexity requirements, or determining similar rule-based decision criteria. It may be desirable to overcome these shortcomings and enable admin users to define validation tooltips with sophisticated content-based validation requirements and to be able to estimate compliance with them in real time.

[0042] For example, in some scenarios, a digital task may include submitting data, text, etc., in a digital or web-based format to a designated third-party system, website, application, etc. In some exemplary embodiments, the designated system may include an electronic interface designed to collect and organize specific information from an end user, such as an interactive GUI on a page presented within a software application or website, which includes designated fields and elements for capturing, processing, and storing user-provided data in a structured format. In some exemplary embodiments, the interactive GUI may be designed to allow the user to input data, make selections, and perform various actions directly on the screen.

[0043] For example, a user may have the task of filling out form fields presented by a third-party system page in accordance with the requirements, constraints, and semantic constraints of the third-party system. In some cases, instead of simply restricting user input to a number of characters or similar formal requirements, it may be desirable to restrict input to fields with semantic requirements, such as requiring the input to be of value, to answer a specific question, provide certain details, etc. In some cases, it may be desirable to extend the capabilities of validation tooltips to enable real-time estimation of whether a field's content-based requirement is compiled while the assistance layer is running on the end-user's end device. For example, instead of classifying in binary form whether a user's numerical input to a field is valid or invalid based on whether the input is within a specified range of values, it may be desirable to determine whether the content of the input matches a content-based requirement, such as whether the input is of value, valid, or consistent with other data records.

[0044] In some cases, significant difficulties can arise in determining compliance between user input and content-based requirements. Such decisions, at the very least, require a high level of text analysis capabilities, making it challenging to assess whether the field's content is valid, valuable, consistent with other available information, and consistent with policies. Overcoming these difficulties may be desirable.

[0045] Another technical challenge addressed by the disclosed subject is improving the performance of field validation to determine suggestions for modifying user input to be faithful to content-based validation requirements. It may be desirable to extend the capabilities of validation tooltips to enable meaningful feedback on input provided by end users to text fields, meaningful suggestions for modifying the input, etc. In some cases, generating text with meaningful suggestions for modification, meaningful feedback, etc., can be very difficult, as at least such tasks require a high level of text analysis and text generation capabilities. Overcoming these difficulties may be desirable.

[0046] Another technical challenge addressed by the disclosed subject matter is enabling clients of the DAP platform to customize the functionality of field validators. In some cases, different clients of the DAP may design assistance layers on third-party applications where the same text field may be used in different forms. For example, two different companies may use the same software application, such as SALESFORCE®, and design assistance layers with validation tooltips that run on SALESFORCE® to help end users perform digital tasks on SALESFORCE®. In this example, the two companies may have different requirements for the same field and may want to generate validation tooltips for the same field in SALESFORCE® that differ in their requirements and guidance. In some cases, the third-party application may also allow the customer to customize the form to meet their specific needs, imposing additional complexity and difficulty. For example, the form in the same third-party application (e.g., SALESFORCE®) may differ significantly among different customers. Overcoming these difficulties may be desirable, allowing clients to customize the functionality of field validators according to their needs, goals, and other factors.

[0047] It should be noted that, as used herein, the term "form" may mean any page of a third-party application that is rendered and contains at least one interactive GUI element, such as a text field or a dropdown element.

[0048] One technical solution provided by the disclosed subject matter may involve adapting a platform such as a DAP to provide content-aware DAP building blocks for input validation. In some exemplary embodiments, artificial intelligence (AI) language models such as large-scale language models (LLMs), small-scale language models (SLMs), and generative AIs may be combined with DAP building blocks such as validation tooltips to provide customized content-aware validation. For example, an LLM engine may be combined with validation rules and / or logic for a validation tooltip.

[0049] In some exemplary embodiments, the DAP build block may include widgets or GUI elements that can be used by an administrator user (also called a “builder” or “admin user”) to design an assistance layer configured to assist end users with digital tasks on a software application. For example, the widgets of the assistance layer may be displayed as an overlay on a third-party application. In some exemplary embodiments, the DAP build block may include layout modifications that can be performed within the GUI, for example, instead of or in addition to using widgets. For example, the DAP build block may be used to adjust the properties of GUI elements in the GUI.

[0050] In some exemplary embodiments, using DAP build blocks, an administrator user may generate an assistance layer from a selected type of build block, a defined position for each DAP build block on a third-party application page or layout, a selected automation process linked to each DAP build block (e.g., a validation process), a selected trigger event that invokes or starts each automation process, etc. For example, an administrator user may design and generate an assistance layer on top of a SALESFORCE® application or any other application, combination of applications, etc., to help a user use the application efficiently.

[0051] In some exemplary embodiments, the disclosed subject matter may be configured to adapt a platform such as DAP to include content-aware build blocks that utilize language models such as LLMs, for example, in order to provide a wider range of build blocks. In some exemplary embodiments, instead of configuring the operation of DAP build blocks such as validation tooltips according only to non-content-aware heuristics, rules, conditions, branches, etc., at least some DAP build blocks may be designed to provide content-aware capabilities. For example, content-aware capabilities may include text analysis capabilities, text generation capabilities, etc.

[0052] In some exemplary embodiments, language models such as LLM engines may include private LLMs, private tenants on cloud or other remote servers, publicly generative pre-trained transformers (GPTs), publicly trained LLMs on private datasets such as in-house knowledge bases, and on-premise LLMs. In some exemplary embodiments, DAP building blocks may utilize one or more technologies such as natural language processing (NLP) models, different machine learning (ML) models, AL models, and generative AI model buildings. In some exemplary embodiments, the LLM engine may be enabled to take text input and provide content-aware output based on the input. For example, the LLM engine may be enabled to analyze input text and generate output text according to instructions or directives in the input text.

[0053] In some exemplary embodiments, content-aware capabilities may be added to existing DAP build blocks, such as tooltips, by utilizing and / or collaborating with AI language models, such as LLMs. For example, an LLM infrastructure may be utilized by launcher widgets, tooltip widgets, or any other type of other DAP build block to assist end users with content-based digital tasks in a content-aware format. For example, an LLM infrastructure may be incorporated into the automation processes of a DAP build block. In some exemplary embodiments, LLM capabilities may be used to perform content-related tasks such as text analysis and text generation tasks, thereby improving the capabilities of the DAP build block and increasing the types and quality of available automation processes that can be incorporated into the DAP build block.

[0054] In some exemplary embodiments, combining an LLM infrastructure with DAP build blocks can enable the DAP build blocks to deliver content-based capabilities. In some exemplary embodiments, a validation tooltip may be designed to assist an end user in filling in a designated field (also called a “validated field” or “validation field”) on a page of a third-party application. In some exemplary embodiments, a validation tooltip may be designed to communicate or implement one or more LLM-based automation processes for the designated field, thereby leveraging LLM technology for the validation process. For example, using an LLM engine, a validation tooltip may be configured to include automation processes that utilize the LLM engine for one or more validation tasks. For example, based on the LLM engine, the validation process for a page element of a third-party application may be tailored to provide content-based feedback on user input to the page element, suggestions for content-based corrections to user input to the page element, and decisions regarding whether the input corresponds to stored data records. These content-based operations, which were previously impossible to perform, become possible by utilizing the LLM engine.

[0055] In some exemplary embodiments, to assist end users in properly completing forms, tooltip widgets may be designed and generated to utilize LLM technology to improve the process of validating end user input, according to various use cases. For example, LLM technology may improve the validation process by providing insights into the semantic characteristics of the submitted text, extracting action items from the text, and classifying free text into predetermined buckets.

[0056] In some exemplary embodiments, the DAP editor may be configured to provide content-aware capabilities to the validation tooltip. In some exemplary embodiments, the validation tooltip may be configured, designed, defined, etc., to incorporate LLM rules, allow an admin user to define LLM rules, etc. For example, the editor may provide a dropdown element from which an admin user can select one or more types of validation rules for the validation tooltip, such as LLM rules, regular expression rules, syntax constraint rules, etc. In some exemplary embodiments, the LLM rules may include, for example, validation rules that utilize one or more prompts to a language model such as LLM as part of the validation process.

[0057] In some exemplary embodiments, the LLM rule may represent a request from an admin user and may be embedded within the prompt to allow the LLM engine to determine whether user input to a field conforms to the admin user's request (LLM rule). In some exemplary embodiments, the prompt may be designed to compare a free-text natural language instruction, i.e., an LLM rule, provided by the admin user with real-time input from an end user. In some exemplary embodiments, the DAP may generate a prompt based on free text from the admin user and instruct the LLM engine to compare the user input with an LLM rule. In some exemplary embodiments, an LLM session may be invoked by generating a prompt according to input provided by the admin user, embedding real-time input from the end user into a field, and communicating the prompt to a locally deployed LLM, a remote LLM (e.g., on the cloud), etc.

[0058] In some exemplary embodiments, prompts may be designed to perform any other content-based tasks. For example, an admin user may choose to incorporate content generation instructions into a prompt to determine whether a field matches the admin user's first request and to generate text according to the admin user's second request. In another example, an admin user may provide multiple instructions for a content-based task as part of natural language input (e.g., to generate correction suggestions for an end user), and prompts may be generated to incorporate the requests. In some exemplary embodiments, a prompt to the LLM may instruct the LLM to perform content-based validation based on requests provided by the admin user, intended inputs for each field (e.g., which may be stored in a document or repository), etc.

[0059] In some exemplary embodiments, a prompt may be defined in the DAP to have a predetermined structure, for example, a structure including a static part and a dynamic part. In some exemplary embodiments, the static part may include a cross-client part defined in the DAP and used for all clients. For example, in the case of a validation tooltip, the cross-client part may instruct the LLM engine to perform validation on end-user input based on input from the admin user of each client, use a predetermined format, etc. In some cases, the LLM rule may include one or more non-validation instructions, such as formatting for the output of the LLM engine, text generation instructions, etc., in addition to validation-related instructions.

[0060] In some exemplary embodiments, the static portion may include a client-specific portion that can be defined by the admin user of each client. For example, the client-specific portion may indicate the desired output, specific validations intended to be performed by the LLM, desired text generation tasks, output formats, user segments of end users to which each request applies, user constraints on user segments, etc. In another example, the client-specific portion may indicate a dataset or content source from which input from end users must match in order to be validated, such as from which validation rules can be retrieved. For example, the content source may include a PDF document with a form request, a website, a repository, a library, etc.

[0061] In some exemplary embodiments, the dynamic portion of a prompt may include a portion that is populated with input from the end user each time a prompt is generated (for example, for an end user performing an assistance layer generated by an admin user). For example, the dynamic portion may include source data replicated from the GUI of a third-party application, for example, to be presented to the end user via the user device. In some exemplary embodiments, the source data may be defined to include input entered by the end user into a tooltip field, in addition to context data such as a portion of the text on the GUI display, or all the text on the rendered page of a third-party application, either alone or via the DAP. In some exemplary embodiments, the dynamic portion may be configured by the DAP or admin user to include context information for the field context, such as the field names of other fields on the same page, the values ​​provided to other fields by the end user, and the validation rules for other fields on the page. When providing context information, it should be noted that fields that are not filled in may be identified as having NIL, NULL, N / A values, etc.

[0062] In other cases, the verification tooltip may provide any other division into static cross-client parts, static client-specific parts, and dynamic parts. For example, the DAP editor may allow an admin user to adjust the static cross-client parts, making the entire static part client-specific.

[0063] In some cases, restricting the admin user's ability to modify static cross-client portions can enhance the security of the disclosed subject and the privacy of end users. For example, to ensure that end user privacy is maintained when running the LLM engine, the services provided by the LLM may be restricted to a defined domain. In some exemplary embodiments, instead of utilizing LLM capabilities in an unrestricted form, the LLM may be used under a restricted framework that is set up by the DAP and cannot be modified by the DAP's customers. For example, prompts to the LLM may be restricted to a predetermined prompt structure, a predetermined text portion, etc.

[0064] In some exemplary embodiments, in order to maintain user privacy and prevent the provision of personally identifiable information (PII) or sensitive data to an LLM engine (e.g., for an LLM not operated by the company), PII data may be removed from prompts, replaced with non-PII data, etc. In some exemplary embodiments, data may be anonymized before data from end users is provided to a public LLM. In some exemplary embodiments, end user privacy may be maintained by removing PII data.

[0065] In some cases, PII data may be identified and replaced using heuristic-based rules (removing or anonymizing data that matches email format, phone number format, amount, etc.), private language models, or internally operated language models. PII data may be identified on the end user's end device, trusted servers, etc. In some cases, in the case of highly sensitive PII data, prompts may be canceled and LLM validation may be blocked (e.g., non-LLM validation may be implemented instead). In some cases, PII data may be removed from prompts, replaced with non-PII data, etc. For example, sensitive data such as names may be replaced with common names, name tags, etc., to ensure that the text remains inclusive.

[0066] In some exemplary embodiments, in addition to restricting the LLM engine's access to end-user data, the DAP's access to such data may also be restricted. In some exemplary embodiments, to prevent data leakage from the end-user to the digital adoption platform, the digital adoption platform's backend may not have access to prompts generated from end-user input. Instead, a client-side agent running on each end device may generate prompts and send them to the LLM engine while protecting the privacy and confidentiality of the user data contained therein.

[0067] In some exemplary embodiments, the LLM engine may obtain prompts via a third-party chat GUI, API calls, system calls, or any other interface. In some exemplary embodiments, the LLM engine may process and / or analyze the provided prompts and generate output based thereon. For example, processing on the LLM side may include performing text analysis tasks, text generation tasks, NLP processing, semantic analysis, contextual analysis, etc. In some exemplary embodiments, the output from the LLM engine may be provided to an assistance layer, such as a validation tooltip, and extracted therefrom.

[0068] In some exemplary embodiments, the validation tooltip may generate output based on output obtained from an LLM engine. In some exemplary embodiments, the validation tooltip output may be generated based on one or more LLM sessions, one or more non-LLM operations, a combination thereof, etc. In some exemplary embodiments, an admin user may configure the format for generating output to the end user. For example, an admin user may define that the validation tooltip should generate output that includes output from one or more LLM engines, their instructions, their processed versions, a combination of LLM output and predetermined text, etc. In some cases, the output may incorporate part of the response from an LLM session, the entire response from LLM, etc.

[0069] In some exemplary embodiments, the validation tooltip may present the generated output according to one or more presentation configurations, such as those set by an admin user. In some exemplary embodiments, the generated output may be configured to appear on a page of a third-party application. For example, the output may appear as a message or queue within an overlay on the GUI, such as within a popup element, tooltip, window, balloon, text box, or chat widget. In other cases, the output may be displayed by adjusting the GUI layout without generating and displaying an overlay on top of the GUI, for example. In some exemplary embodiments, the admin user may be allowed to define a presentation configuration for displaying the output. For example, the admin may select a target location for displaying the output.

[0070] In some exemplary embodiments, the DAP editor may allow an admin user to select any other characteristics of the validation tooltip, such as when a validation rule should be invoked. In some cases, the admin user may define one or more trigger events designed to cause the tooltip's validation rule to be executed when identified. For example, trigger events may include determining that an end user has entered data into a specified field, that an end user has finished entering data, that an end user has moved away from a field after entering data, or that an end user has selected a different field after entering data.

[0071] In some exemplary embodiments, the DAP may allow an admin user or builder to define client-specific portions of prompts to the LLM engine, or define one or more dynamic portions of prompts to be extracted from the end-user's GUI. In some cases, one or more functions may not be editable by the admin user, for example, blocking the client's ability to adjust the cross-client portion of a prompt.

[0072] In some exemplary embodiments, after a validation tooltip is defined by an admin user, the defined validation tooltip may be compiled, processed, etc., and deployed on multiple end devices, for example, as part of the execution of the assistance layer. In some exemplary embodiments, the validation tooltip may be distributed to end devices, embedded in a digital task page, or made accessible to end devices in any other form. For example, the validation tooltip may be distributed independently or as part of the assistance layer.

[0073] In some exemplary embodiments, statistics on the performance of the validation tooltips may be collected, stored, evaluated, and analyzed by the customer who defined the validation tooltips, the DAP itself, the server that distributed the tooltips, etc. In some exemplary embodiments, the performance of the validation tooltips may be evaluated before or after deploying the validation tooltips, before or after adjusting the configuration of the validation tooltips, etc., by determining their effect on the end user's success in digital tasks that involve filling in fields on a form or other page.

[0074] It should be noted that, as referred to herein, validation tooltips may mean any widget or GUI control that can be used to provide text or non-text feedback in text input from a user. For example, a validation tooltip may include a launcher or any other DAP building block. While the disclosed subject matter is illustrated, for example, with respect to tooltips defined at the granular field level for a particular page element, it should be further noted that validation may be defined at any other level of granularity, for example, at the page level. For example, a DAP editor may allow an admin user to define validation rules that apply to the entire page, such as requiring that at least four fields of a form be filled in by an end user. Page-level validation may be defined via page-level tooltips, general DAP rules, etc.

[0075] One technical benefit of using disclosed subject matter is to assist human users in filling out forms or any other page fields. For example, by using disclosed subject matter, end users may be helped to fill out forms efficiently, properly, completely, and in a timely manner, which can improve end user productivity and engagement.

[0076] Another technical benefit of utilizing the disclosed subject matter is to provide improvements to the DAP platform by increasing the DAP platform's capabilities to provide an LLM-based verification process that can be implemented for DAP build blocks such as tooltip widgets. The disclosed subject matter further enables builders to generate tooltips with customized functionality that aligns with corporate policies and working styles, and to provide customer-specific functionality using non-code platforms such as DAP without requiring admin users to directly provide or modify code. It should be noted that the disclosed subject matter is not limited to a specific digital adoption platform and can also be implemented for non-DAP platforms.

[0077] Another technical benefit of using disclosed subject matter is that it helps users perform content-related digital tasks. For example, disclosed subject matter allows validation tooltips to automatically generate suggested corrections to end-user input based on text submitted to a form, determine compliance of input with a certain policy, and so on.

[0078] Another technical benefit of utilizing the disclosed subject matter is the provision of user-friendly solutions that can increase user engagement. In some cases, human-to-machine interaction may be enhanced by providing content-based feedback to the end user as they fill out a form, thereby allowing the user to easily determine the form requirements, the format in which to adjust their input to comply with the form requirements, etc. In some cases, to further enhance human-to-machine interaction, a chatbot may be used to communicate with the user in natural language until the required data for the form is obtained.

[0079] In some exemplary embodiments, this verification process can be set up and achieved simply by updating and configuring strings in the DAP editor, without any programming by an administrator user, and without requiring any recompilation, distribution, or updates of the underlying system. In some exemplary embodiments, the resulting verification tooltips may enable verification of user input requiring semantic and NLP processing, not limited to syntactic constraints.

[0080] Another technical benefit of utilizing the disclosed subject matter is that it paves the way for responsible AI that may be suitable for each company. Unlike other forms of generative AI, which may not necessarily be responsible, the disclosed subject matter provides a predictable, explainable, and secure LLM experience. Predictability may also be achieved by limiting the adjustable portion of the prompts, so that admin users may not be able to modify such portions. In some cases, the same prompt structure may be used for different automation processes, in multiple scenarios or use cases, for validation tooltips deployed by different customers, etc., thereby enabling a wide range of functionality within the same limited framework.

[0081] Another technical benefit of utilizing the disclosed subject matter is the provision of privacy preservation verification, which can assist end-users with digital tasks without providing end-users' PII data to the LLM and / or DAP backend.

[0082] The disclosed subject matter may provide one or more technical improvements to any existing technology and any technology that has been routine or conventional in the art. Additional technical challenges, solutions and effects will become apparent to those skilled in the art by examining this disclosure.

[0083] Herein, we refer to Figure 1, which shows an exemplary flowchart of the method with some exemplary embodiments of the disclosed subject matter.

[0084] In step 110, the administrator user may define trigger events via a DAP running on their end device. In some exemplary embodiments, a trigger event may include identifying when input is entered into a field by an end user. For example, the administrator user may select a field from a page of a third-party application (e.g., as a page element) and select a trigger event associated with that field. In some exemplary embodiments, the third-party application may run on the administrator user's end device, multiple end devices of end users, etc.

[0085] In some exemplary embodiments, an admin user may define trigger events that include an identification of when input is entered into a field by an end user. For example, the identification may include identifying when an end user has entered input into a field and a predetermined time threshold has elapsed since the user stopped entering input into the field (indicating that the end user has finished entering input into the field). Another example is when the identification may include identifying when an end user has entered input into a field and then moved toward a different page element or away from the field. Another example is when the identification may include identifying when an end user has entered input into a field and then selected another page element. In other cases, any other indication may be used to determine when an end user has provided input to a field, when they have finished providing input, etc.

[0086] In some cases, the admin user may define that, prior to a trigger event, the system should provide the end user with one or more guidance messages when it determines, for example, that a field is visible to the user, that the user has selected a field, or that the user has begun inserting data into a field. For example, the guidance message might guide the user on how to fill in the field before the user's input is validated. In some cases, the guidance message may be defined by the admin user or selected from one or more predetermined sets of validation tooltips. For example, one or more predetermined sets of messages may be past messages defined by one or more users in the organization to which the admin user belongs, defined by the DAP, etc.

[0087] In step 120, the automation process may be defined to be executed in response to the occurrence of a trigger event. In some exemplary embodiments, the automation process may be configured to prompt one or more LLM engines and to obtain output from the LLM engines in response to the prompts.

[0088] In some exemplary embodiments, the automation process may be configured to incorporate at least one or more validation rules configured to validate the input, generate prompts to produce guidance and assistance to adjust the input, etc. For example, while defining the automation process, the admin user may define validation rules using free text in natural language, for example, according to a desired validation policy, form request, etc.

[0089] In some exemplary embodiments, prompts may be configured to instruct the LLM engine to determine whether the input conforms to validation rules, to provide feedback on the input, or to suggest how to modify the input in a form that conforms to validation rules. For example, a prompt might ask the LLM whether input from an end user conforms to one or more validation rules defined by an admin user.

[0090] In some exemplary embodiments, the automation process may be configured to generate prompts in one or more defined forms. In some exemplary embodiments, the automation process may be configured to generate prompts that include a predetermined structure of static and dynamic parts. In some exemplary embodiments, the dynamic part may be configured to be all popularized invocations of trigger events, along with input from an end user, context data from a field on a page, etc. For example, the context data may include the names of other fields on the page, one or more inputs to other fields, validation rules for other fields on the page, etc. In some cases, other page fields may include fields on a form, and the validation rules for page fields may be defined (e.g., by an admin user or another entity) to validate end user input to page fields.

[0091] In some exemplary embodiments, the static portion may be configured to include text that remains immutable through trigger events on different end devices. For example, the static portion may be configured to include validation rules configured to validate input (e.g., defined in free text by an admin user). In some exemplary embodiments, the static portion may be configured to include one or more cross-client portions and one or more client-specific portions. For example, each client may be allowed to define its own client-specific validation rules, while one or more portions of the prompt may be cross-client and not configurable by an admin user.

[0092] In some exemplary embodiments, the automation process may be configured to provide a generated prompt to an LLM engine and to obtain output from the LLM engine in response to the prompt. In some exemplary embodiments, the automation process may be configured to generate results based on the output obtained from the LLM engine. Note that multiple LLM engines, non-LLM engines (e.g., those that perform heuristically defined operations), etc., may be used by the automation process. For example, the automation process may generate a message that incorporates at least a portion of the output from the LLM engine, one or more predetermined text portions, etc. In other cases, the message may be generated not by the automation process, for example, by a different process of a verification tooltip.

[0093] In step 130, the configuration for displaying the results on the page may be determined, defined, etc.

[0094] In some exemplary embodiments, the results may be generated to indicate at least whether the input conforms to the validation rules, to provide feedback to the input, to provide suggestions on how to modify the input in a form that conforms to the validation rules, etc. In some exemplary embodiments, the results may be determined based on one or more outputs from one or more LLM engines, non-LLM engines, etc. For example, the results may be determined based on the output from at least one LLM engine. In some exemplary embodiments, the results may be generated by an automation process, a validation tooltip engine, a DAP engine, etc.

[0095] In some exemplary embodiments, the results may be configured to appear as a message adjacent to a field, a change to the page layout, one or more overlays, etc. For example, the configuration for displaying the results may include updating one or more properties of the page, such as the color (e.g., the background, border, etc., of a field) or highlighting. In another example, the configuration for displaying the results may include displaying the results as an overlay on the page. In this example, the overlay may be a chat widget, tooltip, popup element, text field, message balloon, etc., which are not part of a third-party application.

[0096] In some exemplary embodiments, trigger events, automation processes, and display configurations may be defined by an administrator user as part of the definition of validation elements such as tooltips in a digital adoption platform that is unknown to the third-party application. In some exemplary embodiments, the digital adoption platform may be configured to allow an administrator user to assist an end user with digital tasks, such as filling out a form. In some exemplary embodiments, an administrator user may be able to use the digital adoption platform to generate an assistance layer to run on a third-party application. The assistance layer may be runnable on multiple end devices and may be configured to assist an end user with filling out fields in the third-party application.

[0097] In some exemplary embodiments, the assistance layer may define validation tooltips for fields, pages containing fields, etc. In some exemplary embodiments, the validation tooltips may be configured to include an automation process having one or more validation rules, for example, validation rules defined by an admin user.

[0098] In some exemplary embodiments, the digital adoption platform may or may not block the admin user from making certain changes to validation elements, such as prompts in an automation process. For example, the static portion of a prompt may be configured to include pre-configured instructions that are not defined by the admin user and cannot be modified by the admin user (for example, stating that the LLM engine should determine whether the input conforms to validation rules). For example, the pre-configured instructions may include cross-client instructions defined by the admin user of the digital adoption platform, rather than by the admin user of any client of the DAP.

[0099] In step 140, after the assistance layer has been generated to include at least a validation tooltip, the assistance layer may be executed on the end user's end device to assist the end user in filling out the form. For example, the assistance layer may be generated and executed according to the method shown in Figure 6.

[0100] Herein, we refer to Figure 2A, which shows an exemplary execution of the verification tooltip by some exemplary embodiments of the disclosed subject matter.

[0101] In some exemplary embodiments, as shown in Figure 2A, the GUI of a third-party application may include a text field 201 named “Name”. For example, text field 201 may be part of a form in a third-party application that may be required to be filled out by an end user as part of a digital task.

[0102] In some exemplary embodiments, the assistance layer defined using DAP may run on a third-party application on the end user's end device. For example, the assistance layer does not have to be defined according to the method of claim 1 and does not have to generate prompts to the LLM engine. In some exemplary embodiments, the assistance layer may run a validation tooltip for text field 201, defined to validate user input to text field 201 and provide the end user with guidance on how to properly fill in text field 201.

[0103] For example, the tooltip may be configured to provide a guidance message before input is provided to text field 201, in response to, for example, determining that text field 201 is visible to the user, or user interaction with an assistance layer overlay such as element 203. In this example, the guidance message may be displayed to the end user to guide the user on how to enter the correct information ("Please enter your name").

[0104] Herein, we refer to Figures 2B and 2C, which illustrate exemplary execution of the verification tooltip by some exemplary embodiments of the disclosed subject matter.

[0105] In some exemplary embodiments, as shown in Figure 2B, the GUI of a third-party application may include a text field 211 named "Email". For example, text field 211 may be part of a form in the third-party application, adjacent to text field 201, etc.

[0106] In some exemplary embodiments, the assistance layer defined by the DAP may run on a third-party application and may include a validation tooltip for text field 211 (e.g., separate from the tooltip for text field 201) defined for purposes such as validating user input to text field 211 and providing guidance on how to fill in text field 211. In contrast to the tooltip for text field 201, the tooltip for text field 211 may not be configured to provide guidance messages before input is provided to text field 211.

[0107] In some exemplary embodiments, after input is provided to text field 211 (e.g., trigger event), the tooltip for text field 211 may be configured to execute an automation process configured to validate the input. For example, the validation tooltip for text field 211 may not utilize prompts to the LLM engine and may therefore be configured to evaluate whether the user input matches one or more common expressions or syntax constraints. For example, when an end user enters a value into text field 211, the validation tooltip for text field 211 may use one or more heuristic rules (string comparisons) to determine whether the input matches a given format. In some exemplary embodiments, if the conditions are not met and the user input does not match a defined rule, the validation tooltip may display a pre-configured message to the end user indicating that the user input is incorrect, such as message 205 in Figure 2C. For example, message 205 may explain to the user how to use the correct format ("Format: Use myname@domain.com").

[0108] In some exemplary implementations, tooltips with automation processes defined simply using ordinary expressions or syntactic constraints may have one or more drawbacks. For example, tooltips may not perform any content-based analysis of the input, and may not be able to generate suggestions for users not directly scripted by an admin user. Note that scripted messages may relate to messages with pre-configured and fixed pre-written text.

[0109] In some exemplary embodiments, the editor of the DAP, such as a drag-and-drop editor, may be configured to allow an admin user (of the DAP client) to define an automated process that can perform content-based tasks. For example, Figure 3B shows a GUI of an editor that may be used to define an enhanced content-based validation tooltip, which may be enhanced with respect to the validation tooltips in Figures 2A-2C.

[0110] Herein, we refer to Figures 3A and 3B, which illustrate an exemplary process for defining a verification tooltip by some exemplary embodiments of the disclosed subject matter.

[0111] In some exemplary embodiments, a client-related administrator user may define, configure, and set up verification tooltips via the DAP editor. In some exemplary embodiments, the admin user may not be required to be a programmer or have any technical understanding to define verification tooltips, as the editor may include a non-code platform with a GUI that is easily operable by new users.

[0112] In some exemplary embodiments, the admin user may define one or more characteristics of a validation tooltip via the editor. For example, the admin user may define interaction conditions 321 to configure the interaction of the validation tooltip with the end user and the automation process for the validation tooltip. As another example, the admin user may define display conditions 323 to set when the tooltip message or other output should be displayed. As yet another example, the admin user may define appearance conditions 325 to set the appearance characteristics of the tooltip. As yet another example, the admin user may define selected elements 327 to set the attached page elements (e.g., associated text fields) to which the tooltip is related.

[0113] In some exemplary embodiments, via the editor, the admin user may set characteristics of the interaction condition 321, such as the stating guidance text of guidance 330, which may be configured to appear as a message associated with the page element before any input from the end user is received.

[0114] In some exemplary embodiments, an admin user may define one or more validation rules 301 to be executed on user input to determine whether the input is appropriate, correct, etc. In some exemplary embodiments, validation rules 301 may be defined using the DAP's rule engine, which may evaluate the rules and decide whether or not they should be retained. In some cases, when setting validation rules 301, the admin user may select rules from a given set of non-content-aware validation rules (e.g., email address format validation, telephone number format validation, etc.). In some exemplary embodiments, the admin user may be allowed to define custom rules via an editor using one or more common expressions, syntax constraints, etc. For example, the selection in update rule 303 may provide the user with a GUI for defining new custom rules and modifying existing rules.

[0115] In some exemplary embodiments, via the editor, the admin user may define at least one message 305 or other output that is displayed by a tooltip if user input does not conform to validation rules. For example, message 305 may correspond to message 205 in Figure 2C, and no other messages may be displayed to the end user regardless of the type of error made by the end user.

[0116] In some exemplary embodiments, the administrator may configure settings such as display conditions that determine when success and / or failure messages or instructions should be displayed. For example, an admin user may define via message display conditions 307 when message 305 should be displayed (e.g., when hovering over a text field, when entering input, etc.), when a success instruction should be displayed, etc. In another example, an admin user may define via success instruction 309 one or more visual or non-text instructions for validated inputs, a success message configured to be displayed for valid inputs, etc. In yet another example, an admin user may define via presentation configuration 311 where failure messages (message 305) should be displayed, where success instructions and / or messages should be displayed for the page element to which the tooltip relates, the appearance characteristics of message 305, the appearance characteristics of success instructions (e.g., size, font, color, etc.), etc.

[0117] In some exemplary implementations, administrators may configure validation rules for different user segments. In some cases, the digital adoption platform may identify active end users and select validation rules for end users with respect to specific forms / fields being used. The selection may be based on the segment to which the end user belongs. Segments may be based on the end user's organizational unit, role, geographical location, etc. In some cases, the digital adoption platform may use different validation rules for end users in different segments. However, it should be noted that in some cases, the rules may be uniform across different segments; for example, two different end users associated with two different segments may still be treated using the same validation rules.

[0118] In the scenario of Figure 3A, the administrator user may define the validation tooltips in Figures 2B and 2C by selecting "Email Address Format Validation" as the tooltip validation rule and defining that the message "Format: myname@domain.com" is displayed when the validation rule is violated. In other cases, any other message may be specified. For example, the user may specify the message "Please enter a valid email address, e.g., myname@domain.com".

[0119] In production, when an end user enters information into a field, validation rules may be applied, executed, etc., to determine whether the validation is successful. In some cases, if validation fails, message 305 may be displayed to the end user to notify them of the failure. In some exemplary embodiments, if validation is successful, a success indicator and / or message may be displayed.

[0120] In some exemplary embodiments, the DAP editor may be improved, modified, etc., to include more advanced, sophisticated validation rules. According to the disclosed subject, instead of using conditions based on propositional expressions that can be evaluated considering the values ​​of different variables, the rule engine may utilize LLM-based rules. For example, instead of simply defining validation rules using ordinary expressions and / or syntactic constraints, the DAP editor may incorporate language models such as LLM, natural language processing (NLP) models, different machine learning (ML) models, AI models, generative AI models, etc., to enable administrator users to utilize content-based rules for automation processes.

[0121] In some exemplary embodiments, content-based rules, also called "LLM rules" or "fill requests," may be validation rules that can be defined, selected, etc., by the admin user. In some exemplary embodiments, content-based rules may be selected from a set of content-based validation rules or automation processes that utilize the LLM engine to perform content-aware tasks. For example, the set of rules may be defined by other entities, the same admin for different fields, DAP, etc. In some cases, other settings for validation tooltips are automatically configured, such as by automatically configuring the respective result display configuration by selecting a preset automatic process from a set of preset automatic processes.

[0122] In some exemplary embodiments, content-based rules may include custom rules defined by the admin user. For example, the admin user may select the type of validation rule to be a content-based rule, such as by selecting "AI validation" from validation rule 333 in Figure 3B. In some exemplary embodiments, the admin may be enabled to provide free text via the DAP GUI, specifying validation rules or requests in natural language. As shown in Figure 3B, the admin user may specify LLM rules using free text in fields such as validation logic 331. The admin may define content-based rules using free text that explains when a value should be held as true and when it should be held as false. For example, the admin may write "Please provide a reason for churn" in validation logic 331, causing the LLM rule to require user input to provide a reason for churn.

[0123] In some cases, the admin may be allowed to use natural language to write complex content-based validation rules for text fields where end-user input must be solid. For example, a validation rule may have one or more conditions, branches, user segmentation (e.g., a first content-based rule for a first user segment, a second rule for a second segment, etc.), etc.

[0124] For example, a content-based rule may be defined as part of an automated process for a field where an end user is expected to specify the reason for churn. In this example, the admin user may define the LLM rule with free text such as, "Please provide accurate information about the reason for churn for this opportunity. If it is a technical or product-related reason, you must clarify how it could have been prevented." In some exemplary embodiments, and in other examples, any other text with any other validation rules may be provided by the admin user in addition to or instead of the above text. In some exemplary embodiments, the free text defining a custom rule may include any requirement at any level of complexity, any number of conditional branches, any semantic requirement, any level of detail regarding different scenarios, etc.

[0125] In some exemplary embodiments, free text specified by the admin user may be concatenated or incorporated into a predetermined structure of a prompt defined by the DAP. For example, the DAP may be configured to generate prompts for the LLM engine, which then queries the LLM engine to process input from the end user according to LLM rules specified by the admin user.

[0126] In some exemplary embodiments, a prompt may be defined to have a predetermined structure, for example, a structure including a cross-client static part (a request to the LLM engine to process input from the end user according to LLM rules specified by the admin user), a client-specific static part (LLM rules specified by the admin user), and an end-user-specific dynamic part (input from the end user). For example, the client-specific static part may be specific to the validation task intended to be performed by the LLM as defined by the admin. In some exemplary embodiments, the static part of the validation tooltip prompt generated by the admin user may remain constant or static throughout the execution of the tooltip on the end device, while the dynamic part may change dynamically according to the input the end user provides to the field.

[0127] In some exemplary embodiments, prompts may be defined to include validation rules, role-playing instructions, target field labels, information about other fields on the same page, page information, etc. For example, a prompt might be: "The following fields and values: "[FIELD1]"=[VALUE1], "[FIELD2]"=[VALUE2], ... "[FIELD N ]”=[VALUE N Determine whether the following rule is maintained for a completed form that has ]: Provide a response in JSON format {rule:x} if x is true or false. The LLM engine may evaluate the prompt and provide a response, for example, in a desired format. The response may indicate a semantic evaluation of the input's compliance with the validation rule.

[0128] As another example, when an end user enters a value for a field, a prompt may be generated by concatenating the input value for the field with cross-client static text and an LLM rule, resulting in a prompt such as: "Assuming the fill request for a field in the form is [FILL-REQUIREMENT], provide a grade 0-5 for the following response entered by the user, where 5 means the response fully complies with the above request, and 0 means it is completely irrelevant or has no value with respect to the above request. The user's response is [FIELD-VALUE]. Provide a grade [grade:x] in JSON format and include no description other than the JSON output." In this example, [FILL-REQUIREMENT] may be replaced with a defined LLM rule for each client, as defined by the admin user, and [FIELD-VALUE] may be dynamically populated from text input from the client's end user. In some cases, prompts may be configured to instruct the LLM to provide results in a specific format, e.g., JSON format or any other format. In other cases, prompts may be defined in any other form to represent the client's desired request in any other form.

[0129] In some exemplary embodiments, prompts may be generated to incorporate context data from the end user's end device, for example, to enable the LLM engine to perform context analysis. In some cases, to enable context analysis (e.g., as part of or separately from input validation), the admin user may configure the prompt to include context data in its dynamic portion. For example, instead of a dynamic portion of a prompt containing only input from the end user, the dynamic portion may be defined to include context information about the GUI that each end user interacts with, such as additional fields in the same GUI of a third-party application, the names of all fields on the page the end user interacts with, values ​​provided to other fields on the page next to the tooltip fields, validation rules for other fields, etc. In some cases, the admin user may define or select (from pre-configured options in the DAP) to add context data to a prompt, and this context data should be added, etc. For example, providing contextual data in prompts may allow the LLM engine to consider the context of user input, thereby improving the accuracy of the answer.

[0130] In some exemplary implementations, if a prompt is configured to incorporate contextual data, the LLM rule may or may not be related to the contextual data. For example, an LLM rule may be defined by an admin user to request output consistent with the values ​​of other page fields. As another example, an LLM rule may be requested by LLM to determine whether user input adds a value on top of information in other page fields. As yet another example, validation of user input may be required to be consistent with another field on the page the end user is interacting with. As yet another example, an LLM rule may not be related to contextual data, and the contextual data may be used by the LLM engine to improve accuracy.

[0131] For example, an LLM rule unrelated to contextual data could result in a prompt such as, "Assuming the fill-in requirement for a field in the form is "[FILL-REQUIREMENT]", the values ​​of the other fields in the form are [FIELD1-NAME]="[FIELD1-VALUE]", [FIELD2-NAME]="[FIELD2=VALUE]", ...[FIELDn-NAME]="[FIELDn-VALUE]", and provide a grade 0-5 for the following response filled in by the user, where 5 means the response fully complies with the above requirement, and 0 means it is completely irrelevant or has no value with respect to the above requirement. The user's response is "[FIELD-VALUE]", providing a grade [grade:x] in JSON format and including no description other than the JSON output."

[0132] As another example, LLM rules may be directly related to contextual data, such as when an LLM rule requires user input to be consistent with another page field. This could result in a prompt such as, for example, "Assuming the requirement for a field in a form is a value that must be a KPI fully relevant to the goal defined in the field named [OTHER-FIELD], the values ​​of the other fields in the form are [FIELD1-NAME]="[FIELD1-VALUE]", [FIELD2-NAME]="[FIELD2=VALUE]", ... "[FIELDn-NAME]"=[FIELDn-VALUE], and provide a grade 0-5 for the following response entered by the user, where 5 means the response fully complies with the above requirement, and 0 means it is completely irrelevant or has no value with respect to the above requirement. The user's response is "[FIELD-VALUE]". Provide a grade {grade:x} in JSON format and include no description other than the JSON output."

[0133] In some exemplary embodiments, after defining the prompt validation rules, their dynamic parts, etc., the prompt may be provided to one or more LLM engines during execution. In some exemplary embodiments, the LLM engine may be configured to process prompts obtained according to LLM rules defined by the admin. For example, according to the above example, the LLM engine may be configured to take context data into consideration and provide a grade based on whether the value entered by the end user conforms to the LLM rules. As another example, if the context data includes validation rules for other fields, the LLM engine may infer the intended function of each page element based on this and use this understanding to improve the accuracy of the output.

[0134] In some exemplary embodiments, in addition to instructing the LLM to validate end-user input, LLM rules may also instruct the LLM engine to perform text generation tasks, for example, as defined by an admin user. For example, LLM rules may incorporate text to request from the LLM engine, such as to provide suggested improvements to information entered by the end-user, to suggest alternative inputs to the end-user, to suggest modifications or corrections to input from the end-user, or to provide content-based feedback in the input. In another example, the prompt itself, rather than the LLM rule, may be configured to request a text generation task from the LLM engine.

[0135] For example, the prompt may be adjusted to state, "Assuming the fill requirement for a field in the form is "[FILL-REQUIREMENT]", provide a grade from 0 to 5 for the following response filled in by the user, where 5 means the response fully complies with the above requirement, and 0 means it is completely irrelevant or worthless with respect to the above requirement. The user's response is "[FIELD-VALUE]". Provide a grade, and if the grade is not 5, describe what is missing in the data to make it 5. Provide the response in JSON format {grade:x, missing_data:y}, and include no explanation other than the JSON response." According to this example, if the prompt is used for a field related to the end-user's reason for churn, the LLM engine may provide one or more responses in response to input from the end-user, such as {"grade":4, "missing_data": "It is not clear how this technical problem could have been avoided"}.

[0136] In some exemplary embodiments, this verification process, which uses prompts along with LLM rules, context data, text generation requests, etc., may be set up and achieved simply by updating and configuring strings in the DAP editor, without any programming being performed by an administrator user and without requiring any recompilation, distribution, updates, etc., of the underlying third-party system. In some exemplary embodiments, the resulting verification tooltips may enable content-aware verification of user input to fields requiring semantic and NLP processing, not limited to syntax requests, and to provide meaningful feedback, etc.

[0137] In some exemplary cases, in addition to field-specific LLM rules, one or more LLM rules may be defined at different levels of granularity. In some cases, validation rules may be defined at the field level, page level, website level, etc. For example, a tooltip or other widget may be defined at the page level, and its validation rules may include page-level LLM rules. In some exemplary cases, page-level validation rules may be defined for all page elements, subsets of page elements, etc. For example, an administrator user may define a page-level LLM rule for a specific page, form, etc., stating that "all responses in a form should contain at least three lines," thereby applying the rule to all page elements. As another example, a page-level validation rule may be defined by an administrator user to relate to multiple fields on a page, such as by defining an LLM rule, stating that "the user must fill in at least three goal fields, and for each goal, the description field identifies how success will be evaluated and the time frame for completing the goal." In this example, the page contains multiple goal fields and multiple description fields, and the end user is required to fill in at least three goal fields and associated description fields, thereby applying to a subset of page elements.

[0138] In some exemplary implementations, page-level validation rules may be applied in parallel after each field has been filled in by the end user, or after all fields have been filled in and / or the respective form has been submitted. For example, page-level LLM rules may be applied to each filled-in field to ensure that the information entered by the end user conforms to page-level rules. In another example, when the end user selects a control (e.g., a “Submit” button) to submit the form, all of the end user’s input may be validated at this stage before allowing the submit command to propagate. In other cases, validation rules may be applied in any other order, according to or regardless of their level of granularity.

[0139] In some exemplary embodiments, the verification tooltip may be designed by an admin user and then run on a third-party application rendered on the end user's end device. For example, the verification tooltip may run as part of an assistance layer configured to run on the third-party application and enhance it with additional functionality, content, markings, etc.

[0140] Herein, we refer to Figures 4A–4H, which illustrate exemplary scenarios of user interaction with deployed verification tooltips in some exemplary embodiments of the disclosed subject matter.

[0141] In some exemplary embodiments, page 400 of a third-party application is rendered and displayed to the end user, as shown in Figure 4A. In some exemplary embodiments, the third-party application may run with an assistance layer defined by the client (using DAP). In some exemplary embodiments, the assistance layer may be configured to enhance the GUI of page 400 to provide assistance with user input to fields on page 400 and content-based validation.

[0142] In some cases, the client defining the assistance layer may include an organization with access to the DAP platform, and the third-party application may include an application that is not associated with the DAP platform and potentially not associated with the client. For example, the client may include a bank, and the third-party application may include Microsoft Word®, which may be used by the bank's employees and / or end users, but the bank may not have access to the Microsoft Word® backend, its stored data, APIs, etc. In this example, an admin user may define an assistance layer configured to run on Microsoft Word® to assist the bank with bank-related digital tasks that use Microsoft Word® on behalf of the bank. In other cases, the third-party application may include the bank's own applications, which the bank may have full access and control over. In some exemplary embodiments, the assistance layer may be defined on the DAP or any other similar platform and may be designed to include at least one content-based validation tooltip.

[0143] In some exemplary embodiments, a content-based validation tooltip may consist of one or more guidance messages, such as guidance 330 in Figures 3A and 3B; one or more display conditions, such as message display condition 307 in Figure 3A; one or more success instructions, such as success instruction 309 in Figure 3A; and one or more validation rules, such as validation rule 301 in Figure 3A. In some exemplary embodiments, a validation tooltip may consist of one or more LLM rules, such as validation logic 331 in Figure 3B, which may include content-based rules that require text analysis capabilities, text generation capabilities, etc.

[0144] In some exemplary embodiments, page 400 may include a screenshot of an empty form with several free-text fields ("Generate Goals"). In some exemplary embodiments, the free-text fields may be intended to be filled out by the end user and submitted to a server, client, etc., of a third-party application. In some exemplary embodiments, page 400 may include at least a goal field 411 in which the end user is expected to describe a professional goal in natural language, and a description field 413 in which the end user is expected to provide a detailed description in a format in which the end user wishes to implement and evaluate the completion of the goal. In some exemplary embodiments, through the form on page 400, the end user may generate new goals, define goals, describe goals, categorize goals, provide additional information, and so on.

[0145] In some exemplary embodiments, where one or more guidance messages are defined for a tooltip, the validation tooltip may display one or more messages or visual cues before any input is submitted by the end user, during input submission, etc., to guide the end user in advance. In other cases, such as when no guidance messages or cues are defined, the validation tooltip may be designed to display one or more messages of a visual cue only after the input has been provided by the end user. In some cases, where multiple guidance messages are defined for each of multiple validation tooltips on a single page, the multiple guidance messages may be displayed sequentially, in the desired order of filling in the page fields, for example, by first displaying the guidance message for the first page element, and then displaying the guidance message for the second page element after the first page element has been interacted with by the end user, and so on. In other cases, two or more guidance messages, for example, all guidance messages defined for page elements that are visible to the end user, may be displayed simultaneously.

[0146] For example, guidance message 421 may be defined for a target field 411 and may be displayed before the end user fills in the target field 411, during the process, etc. Guidance message 421 may be displayed on or near the target field 411 as an overlay of a GUI element such as a tooltip widget or a balloon widget. In this example, guidance message 421 may inform the end user that the target field 411 is mandatory and must be filled in (for example by stating "Please fill in this field" or by any other terminology), how to fill in the target field 411, what conditions should be followed, etc. For example, guidance message 421 may be written by an admin user via the DAP editor.

[0147] In some cases, unfilled mandatory fields may be visually indicated in addition to or instead of the guidance message 421 by changing the color of one or more GUI elements associated with the target field 411, their borders, backgrounds, etc., to red or any other color, adding an asterisk symbol, etc. In some cases, instead of displaying the guidance message 421, one or more selectable widgets (e.g., indicated as a question mark) may be configured to display the guidance message 421 when selected or hovered over by the user. In other cases, one or more selectable widgets (e.g., indicated as a question mark) may be configured to display any other assistance data.

[0148] In some exemplary embodiments, as shown in Figure 4B, the end user may insert the text string “Take a world trip” as the desired goal into the goal field 411. In some exemplary embodiments, in response to user input, one or more LLM rules (defined by the admin user and not shown to the user) of the validation tooltip may be executed, a prompt may be generated and provided to the LLM engine.

[0149] In response to a prompt, the LLM engine may respond with a validation tooltip indicating that the input does not conform to LLM rules, relevant explanations, relevant guidance, etc. In some exemplary embodiments, the assistance layer may display an indication to the end user that the input is not compliant, for example, on or adjacent to the validation field. In some exemplary embodiments, in response to user input, the validation tooltip may display a message to the user, such as feedback message 423, as shown in Figure 4C, stating, "This response does not conform to the requirement because it does not describe a goal for improving performance or advanced skills." For example, the text of feedback message 423 may include an explanation, not written by the admin user, that is dynamically generated by the LLM engine to describe why the user input violated and was determined to be non-compliant with the validation rules. As another example, the text of feedback message 423 may be generated based on a combination of the LLM engine and a pre-configured set of responses.

[0150] In some exemplary embodiments, the end user may adjust the input in response to a feedback message 423, for example, as shown in the scenario in Figure 4D, and the LLM rule generating the validation tooltip may be re-executed. For example, in the scenario in Figure 4D, the end user may insert the text string "Improve coding skills" into the target field 411, and the validation rule may be executed on the provided value, generating a second prompt and providing it to the LLM engine (one or more LLM engines, the same or different as the one used for the string "Take a world trip"). In response to the prompt, the LLM engine may provide the validation tooltip with an indication that the input does not follow the LLM rule in the prompt, relevant explanation, relevant guidance, etc. For example, in response to user input, the validation tooltip may display a message to the user such as a feedback message 425 stating "The response is relevant but not specific enough," as shown in Figure 4D. For example, the feedback message 425 may be based on the output from the LLM engine and may not be manually scripted.

[0151] In some exemplary embodiments, the end user may modify the input in accordance with the received instructions and / or content-based feedback until the user input is validated by the validation rules, or until the LLM engine determines that it conforms to the LLM rules. For example, Figure 4E shows a scenario in which, in response to feedback message 425, the user updates the text string to say, "Improve coding skills by learning a new test framework." As with the first two text strings from the user, this text string may be processed by the LLM engine and compared to LLM rules, and the output from the LLM engine may be provided to a validation tooltip. For example, the LLM engine may indicate that the text string conforms to the validation rules.

[0152] In some exemplary embodiments, in response to determining that the validation rules have been followed, the validation tooltip may or may not provide a success instruction, for example, based on whether the admin user has defined a success instruction to be displayed to the end user. For example, in the scenario in Figure 4E, a success message "All good" may be displayed.

[0153] In some cases, the assistance layer may define that upon successful completion of a task associated with a tooltip, subsequent tooltips associated with the next field in the form's order may be triggered, and their respective messages may be displayed. For example, since the tooltip for the goal field 411 has been successfully executed, the tooltip for the description field 413 may be triggered, and one or more guidance messages may be displayed, if defined by the tooltip. For example, in the scenario in Figure 4E, a guidance message 433 stating "Please fill in this field" may be displayed near or above the description field 413. In other cases, any other messages may be defined by the admin user and displayed to the end user in place of guidance message 433. For example, Figure 4F shows replacing guidance message 433 with guidance message 434 stating "This is where you enter your goal details. How will you measure success? Are there multiple pieces to an overall goal that can be tracked throughout the year?", which may contain more concise guidance than guidance message 433 for filling in the description field 413.

[0154] Note that in some embodiments, instead of requiring the admin user to write guidance messages, the editor may allow the admin user to utilize existing guidance messages, for example, in a semi-automatic form. In some exemplary embodiments, existing guidance messages may be available for a particular client of the DAP based on guidance messages previously defined for that client. In some exemplary embodiments, existing guidance messages may be available across clients of the DAP based on guidance messages previously defined for different clients, such as those defined by entities associated with the DAP. In some exemplary embodiments, such legacy guidance messages may convey to the end user the purpose and meaning of each field. For example, guidance message 434 in Figure 4F may contain a legacy guidance message defined by a user other than the admin user and selected by the admin user to be executed.

[0155] In some cases, the DAP may automatically utilize legacy guidance messages as LLM rules, for example, by generating a prompt that asks the LLM engine whether the end-user input follows the legacy guidance message. For example, the following prompt may be generated: "The following guidance is included for this field [Legacy Guidance Message]. How does "[Input Field]" follow this guidance on a scale of 0 to 5?". In some cases, if multiple alternative guidance messages exist for the same field, such as for different user segments, the guidance message that matches the end user may be used for the prompt.

[0156] In some exemplary embodiments, the end user may attempt to fill in the description field 413, such as by following instructions in guidance message 433, guidance message 434, etc. For example, in the scenario in Figure 4G, the end user may enter the text string "Learn a new coding language" in the description field 413 as a description of the desired goal. In some exemplary embodiments, in response to user input, one or more validation rules of the validation tooltip for the description field 413 may be executed, a prompt may be generated and provided to the LLM engine (one or more LLM engines which may or may not correspond to the LLM engine used for the goal field 411).

[0157] In response to a prompt, the LLM engine may respond with a validation tooltip indicating that the input does not conform to LLM rules, relevant explanations, relevant guidance, etc. For example, in response to user input, the validation tooltip may display a message to the user such as feedback message 441, as shown in Figure 4G, stating, "The response only partially addresses the request. The response describes the goal of learning a new coding language, but does not provide details on how success is assessed." For example, feedback message 441 may be based on output from the LLM engine and may not be scripted by the admin user.

[0158] In some exemplary embodiments, the end user may modify the input according to the instructions and / or content-based feedback received until the user input is validated by the validation rules. For example, Figure 4H shows a scenario in which, in response to feedback message 441 or any other message, the user updates the input text string to say, "Learn a new coding language. Success will be evaluated by completing an online course and developing an application using the new language." This text string may be processed by an LLM engine, and the output from the LLM may be provided to a validation tooltip. For example, the LLM engine may indicate that the text string conforms to the validation rules.

[0159] In some exemplary embodiments, in response to deciding to follow the validation rules, the validation tooltip may or may not provide a success indicator for the validation tooltip in description field 413 to be displayed to the end user, based on whether, for example, the admin user defined it. For example, in the scenario in Figure 4H, the end user may see a success message that says "All good".

[0160] Herein, we refer to Figures 5A–5C, which illustrate exemplary scenarios of user interaction with deployed verification tooltips in some exemplary embodiments of the disclosed subject matter.

[0161] In some exemplary embodiments, validation tooltips may be deployed in relation to fields on a page of a third-party application, such as field 511 in Figure 5A. For example, field 511 may be intended by a third-party application to obtain detailed information from the user about a problem with the user's car. In some exemplary embodiments, if the user enters the text string "My car won't start" into field 511, the validation rules of the respective tooltips may be executed on it. For example, the execution of the rule may generate a prompt to the LLM engine, which instructs the LLM engine to determine whether the LLM rule has been broken or followed, to provide a grade of success for the user input (e.g., 5 means the response fully complies with the above request, 0 means it is completely irrelevant), and to provide feedback on the input from the end user.

[0162] For example, in the scenario of Figure 5A, the LLM engine may infer that the user input violates a validation rule and generate a feedback message, such as feedback message 521, which is obtained from the LLM engine and displayed as an overlay on the page. For example, feedback message 521 may be displayed in a widget (e.g., a GUI element), such as a callout balloon or tooltip, on or adjacent to field 511, similar to the messages in Figures 4A to 4H. In some cases, feedback message 521 may state "Grade: 4, Missing Information: No specific problem with the vehicle is specified," or any other message indicating that the input is not being followed, and provide suggestions for adjusting the input, etc.

[0163] In some exemplary embodiments, to enhance the user experience, the validation tooltip may use a chat widget, such as a chatbot, to communicate with the user via the LLM engine. For example, the chatbot may be deployed to repeatedly extract user input ("My car won't start") from field 511, obtain feedback from the LLM engine to the user input, and instead of displaying a message to the end user according to the response from the LLM engine, provide an output message and obtain input messages from the end user. In some exemplary embodiments, the chat widget may be used to chat with the end user using natural language, displaying previous user input and LLM-generated feedback as conversation messages in the chat widget.

[0164] For example, the chatbot 512 in Figure 5B may include a chat widget displayed as an overlay (e.g., a balloon element) on a page of a third-party application, deployed for field 511 (by a tooltip or assistance layer), or as a separate page, etc. For example, the chatbot 512 may be displayed on field 511 (e.g., hiding field 511), adjacent to field 511, etc. In some exemplary embodiments, the chatbot 512 may be configured to take a user message intended to be used as input for field 511, generate a prompt based on it, and provide feedback from the LLM engine as an answer in the conversation.

[0165] In some cases, prompts used for chatbot 512 may be adjusted to match a natural language interface. For example, instead of stating, in a non-natural language, "Grade: 4, Missing Information: No specific problem with the car is specified," chatbot 512 may provide a feedback message 523 stating, "Please also specify if you hear a sound when starting the car," or a similar message in natural language (e.g., corresponding to potential human conversation). For example, prompts may be adjusted to instruct the LLM engine to provide feedback in natural language, feedback that matches the chatbot's conversation, etc. In some exemplary embodiments, subsequent input from the user, such as "Yes, I hear a sound, but it is fainter, and the next time I start the car, I hear no sound at all," may be displayed to chatbot 512 and used to generate prompts to the LLM engine until it is determined that an LLM rule is being followed by the chatbot 512's conversation.

[0166] In some exemplary embodiments, when generating a prompt, the chatbot 512 may use the entire conversation as input (e.g., within the dynamic part of the prompt), and the prompt may instruct the LLM engine to determine whether the input conforms to LLM rules. In some cases, the prompt to the LLM engine may be generated to include each user message, a concatenated message of the entire conversation, etc. For example, each time the user provides input, validation rules may be executed not only on the last user input but also on the entire chat. For example, the prompt to the LLM engine may be: "The following information provided by the user: '[FIELD VALUE]', '[CHAT-RESPONSE1]', '[CHAT-RESPONSE2]', ... '[CHAT-RESPONSE kIt may also state, "Please generate a response that complies with the fill-in request "[FILL-REQUIREMENT]" based on ]." In other cases, the prompt may be generated to incorporate part of the conversation, the last input entered, any other part of the conversation, etc.

[0167] In some exemplary embodiments, if the conversation is not validated, feedback from the LLM engine may be provided as the next message in chatbot 512. In some exemplary embodiments, if the LLM engine determines that the conversation from chatbot 512 conforms to LLM rules, the LLM engine may be instructed to provide a summary of the conversation, details provided by the user in the conversation, etc., thereby automatically generating appropriate input for field 511 based on the conversation with the end user. For example, input 513 in Figure 5C may be generated by the LLM engine as appropriate input for field 511. For example, according to the scenarios in Figures 5A and 5B, input 513 may be generated to state, if it is determined that this data conforms to the LLM rules for the tooltip, "The car has not started, the sound has weakened, and now there is no sound at all when trying to start the car."

[0168] In some exemplary embodiments, input 513 may be provided to the end user for manual entry into field 511, or it may be automatically entered into field 511 by an automated process of a validation tooltip. For example, the automated process may use one or more acquisition processes to identify field 511 on the page and enter input 513 therein, potentially allowing the end user to modify the text of input 513.

[0169] In some exemplary embodiments, a tooltip may utilize a chat widget to obtain compliant input to a field on a page of a third-party application, for example, according to the method shown in Figure 8. In some cases, a tooltip defined at the page level may utilize the chat widget to obtain compliant input to multiple page fields via the same chat widget. In some cases, a tooltip defined at the field level may share a chat widget.

[0170] Here, refer to Figure 6, which shows a flowchart of the method according to the disclosed subject matter.

[0171] In step 600, a non-programmer human administrator user (also called a “builder” or “admin”) may run the editor software of the Digital Adoption Platform on an end device, e.g., a computer. In some exemplary embodiments, the admin may use the editor of the Digital Adoption Platform to configure the assistance layer from build blocks provided by the editor (e.g., DAP build blocks such as tooltips).

[0172] In some exemplary embodiments, the assistance layer may be configured to run on a form or any other page of the target system. In some exemplary embodiments, the target system may include a third-party system such as a web-based system, a native system, or a mobile system. In some exemplary embodiments, the assistance layer may be configured to enhance and improve the target system with additional functionality, perform data collection, provide additional data, etc. In some exemplary embodiments, the assistance layer may be configured to monitor, improve, and enhance one or more target systems involved in digital tasks related to the organization to which the admin belongs. For example, digital tasks may include cross-system business processes, single-system business processes, etc.

[0173] In some exemplary embodiments, the admin may generate an assistance layer that includes, in particular, at least one validation tooltip for selected fields on a form or any other page of the target system (e.g., a third-party application), among other building blocks.

[0174] In step 610, the admin may select a configuration for validation tooltips, such as one or more validation rules for user input to be inserted into the selected page field. In some exemplary embodiments, the editor may allow the admin to select validation rules from a pre-configured list of rules, write LLM rules using free text in natural language, etc.

[0175] In step 620, when setting up verification rules for verification tooltips, the verification rules may be stored by the digital adoption platform.

[0176] In some exemplary embodiments, the validation rules may be stored as part of an executable assistance layer that can run on multiple end devices. For example, multiple end users attempting to perform digital tasks on end devices (e.g., employees and / or customers of the organization, distinct from the admin) may be allowed to run the assistance layer on their end devices simultaneously.

[0177] In some exemplary embodiments, the assistance layer may be distributed to the end device, embedded in the page of the digital task, or otherwise made accessible to the end device. In some exemplary embodiments, the assistance layer may be implemented on the end device as a browser extension, a dedicated browser for the digital adoption platform, or client-side code in the web-based target system itself (e.g., an “include” directive configured to provide enhancements).

[0178] In step 630, the end user executing the target system and the assistance layer may reach a selected field on a page in the target system for which a validation tooltip is defined. In some exemplary embodiments, the assistance layer may monitor, enhance, and augment the target system based on which GUI elements of the target system are visible on the end device's screen, user input to the GUI, page addresses, etc. For example, in a web-based system, the URL to which a page is displayed may be monitored by the assistance layer and determined to have been reached when it matches a previously stored URL.

[0179] In step 640, one or more configurations or settings of the validation tooltip may be retrieved, executed, and activated from the DAP or assistance layer, for example, in response to identifying a tooltip trigger event. In some exemplary embodiments, the execution of the assistance layer may include at least the execution of the tooltip validation rules.

[0180] In some exemplary embodiments, trigger events may be identified when an end user reaches a field in a target system for which a validation tooltip is defined, when an end user hovers over or selects a field, or when an end user enters data into a field.

[0181] In some exemplary embodiments, inputs inserted by the end user into selected fields may be validated, for example, by searching for and executing validation rules in a validation tooltip, which are stored by step 620. For example, inputs may be validated using validation rules that utilize prompts to the LLM engine, for example, according to the method in Figure 7. In some exemplary embodiments, the results may be generated by a tooltip based on the output from the LLM engine and displayed to the end user in one or more messages, chat widgets, overlays, etc.

[0182] Here, we refer to Figure 7, which shows a flowchart of the method according to the disclosed subject matter.

[0183] In step 700, the data entered by the end user into the field may be obtained. In some exemplary embodiments, the end user may enter data into the field in the target system. In some exemplary embodiments, the field may include a field in which an admin user has previously defined a validation tooltip with validation rules. For example, the admin user may define validation rules according to step 610 in Figure 6. As another example, the admin user may define validation rules using ordinary expressions, syntax constraints, natural language free text, etc.

[0184] For example, an Opera admin user may define validation rules for a field as part of an assistance layer configured to help an end user purchase tickets for an opera performance or assist an end user with any other digital task. In this example, an end user may access one or more target systems (belonging to Opera or a third-party service) to purchase tickets for an opera performance while the assistance layer is running on the target system. In some exemplary embodiments, the assistance layer may monitor user interactions and determine when a trigger event defined by a tooltip has occurred, for example, when a user enters data into a field.

[0185] In step 710, if the field validation rules include LLM rules, the assistance layer may generate a prompt for the LLM engine in response to receiving data from the user. In some exemplary embodiments, the prompt may be generated based on the entered data, a given rule, or any other configuration of the validation tooltip. In some exemplary embodiments, the prompt may be designed to provide suggestions to the end user to enhance the input to the field, for example, to be used to determine whether the entered data conforms to the validation rules.

[0186] In some exemplary embodiments, the prompt may be generated to include static parts, dynamic parts, combinations thereof, etc. In some exemplary embodiments, the prompt may be generated to populate the dynamic parts with inputs to fields, other page data, etc. In some exemplary embodiments, the prompt may be generated to include at least the data obtained in step 700, which has been entered into a field. In some cases, the data may include text entered by the user into a field, in which case the text may be quoted or incorporated in the prompt, at least in part. In some cases, the data may include any other user input, characters, etc., and the prompt may be generated to incorporate such input accordingly.

[0187] In some cases, prompts may be generated to include contextual information that conveys the context of the page around the field, such as the field names and values ​​of other fields on the page, information about other inputs provided by the end user to the target system, information about validation rules for other fields on the page or the target system, and information collected from the target system. For example, the contextual information may include dynamic parts of different prompts, such as different prompts for each trigger event.

[0188] In some exemplary embodiments, prompts may be generated as a static part of the prompt, for example, to utilize instructions for validating LLM rules according to the dynamic part of the prompt, to provide predetermined feedback, to utilize a specific format for output, etc. For example, instructions may include one or more general descriptions of the task background, a preferred format for the answer, general instructions applicable to different validation tasks, etc.

[0189] In some exemplary implementations, after an admin user defines an LLM rule, the LLM rule may be incorporated into a prompt as part of the static portion and used statically for each end user. For example, all end users executing an assistance layer generated by the admin user would invoke a prompt with the same LLM rule in response to a field trigger event, but the dynamic portion may vary among different end users.

[0190] In some cases, instead of defining LLM rules from scratch, one or more LLM rules may be generated by selecting from a given set of LLM rules. For example, an admin user may select a pre-configured validation rule for a page field, such as "a valid email address from the form user@domain, where the domain is a valid domain name and has an active MX record on the domain name server" or "a valid email address compliant with RFC2822," which may be related to or translated into the prompt text. In this example, the text may be incorporated directly into the prompt or modified by the admin user.

[0191] In step 720, after the defined prompt has been generated with user data, the assistance layer may provide the prompt to the LLM engine. In some exemplary embodiments, the LLM engine may process the prompt and generate one or more outputs, responses, etc., based on the prompt.

[0192] In some cases, the LLM engine may be fine-tuned for one or more specific applications, digital tasks, formats, etc. For example, the LLM engine may be fine-tuned with specific role-playing instructions, such as "Assume you are an assistant ensuring compliance of user input to the system." In other cases, role-playing instructions may be included in the generated prompts without necessarily fine-tuning the LLM engine. In some exemplary embodiments, role-playing may be effective in providing contextualization (e.g., generating responses that match a desired context), expert emulation (e.g., generating responses that reflect domain-specific expert knowledge), enhanced engagement, target information (e.g., generating models for generating responses that specifically address the needs or questions of a given role), etc. As another example, the LLM engine may be fine-tuned to always use a specified format.

[0193] In some exemplary embodiments, based on a prompt, the LLM engine may generate one or more outputs, responses, etc. For example, the LLM engine may generate a response to a prompt indicating whether the data entered by the end user conforms to LLM rules, what is wrong with the data, etc. In some cases, the response may include additional data in addition to compliance instructions. For example, the response may provide messages to display to the end user in case of non-compliance, such as messages providing assistance and guidance to the end user, an explanation of why the data is not compliant, a suggested output response to the end user, and a message explaining how the user should update the data.

[0194] In some exemplary embodiments, if a prompt specifies a format for the response, or if the format is specified as part of a fine-tuning step, the LLM engine may provide the response in the specified format. For example, the response from the LLM engine may be provided in a format such as JavaScript Object Notion (JSON) or eXtensible Markup Language (XML).

[0195] In step 730, the assistance layer may determine, based on the output from the LLM engine, whether the input submitted by the end user conforms to the validation rules of the validation tooltip. In some exemplary embodiments, compliance may be determined based on the response from the LLM engine. For example, the LLM engine may be configured to provide instructions, grades, etc., indicating whether the user's data is compliant, and the validation tooltip evaluates the instructions to determine whether the data is compliant. In some exemplary embodiments, the method flow may proceed to step 750 if it is determined that the data from the end user conforms to the validation rules, and to step 740 if it is determined that the data violates the validation rules.

[0196] In step 750, if a compliant decision is made, one or more success indicators may be provided to the end user. For example, a success indicator may include output indicating successful validation, such as a color indicator, a format change, or a text indicator (predefined by the admin user or provided by the LLM engine). For example, a success indicator may include a callout balloon indicating, for example, "All good," "Content validation passed," "Your input is excellent," or "Your input meets our requirements," similar to Figure 4E.

[0197] In step 740, if a non-compliant determination is made, one or more failure indicators may be provided to the end user. For example, output may be provided to the end user indicating that the data is non-compliant using color indicators, formatting changes, text indicators (predefined by the admin user or provided by the LLM engine), etc. In some cases, if the data violates validation rules, the assistance layer may engage in on-screen interaction with the user, where the failure indicator may be displayed, mentioned, explained, etc. For example, the failure indicator may include red marking around the field, an asterisk symbol adjacent to the field, etc.

[0198] In some cases, in addition to or instead of failure indications, the assistance layer may serve end-user content-based feedback, such as an explanation of why the data is non-compliant and suggestions on how the end user can improve the data to make it compliant. For example, the content-based feedback may be similar to those shown in Figures 4C, 4D, 5A, and 5B.

[0199] In some exemplary implementations, failure instructions and / or content-based feedback may be served to the end user within the page layout, in one or more overlays on the page, etc. For example, failure instructions and / or content-based feedback may be served to the end user within a text message overlay, within a chat widget, etc.

[0200] When a chat widget, such as a mini-chat GUI or chatbot, is used to display messages to an end user, natural language conversation (e.g., using NLP processing) may be implemented between the chat widget and the end user to allow the end user to refine the data entered in the fields. For example, the first message from the chat widget may correspond to the failure instructions and / or content-based feedback described above, subsequent user input may be obtained from the end user via the chat widget and processed by the LLM engine, and subsequent responses from the chat widget may correspond, for example, repeatedly, to the output from the LLM engine. For example, Figure 8 may illustrate a use case for a chat widget.

[0201] In some exemplary embodiments, in response to the failure indication and / or feedback in step 740, the end user may adjust or update the data input, thereby redirecting the flow to step 700. In other cases, for example, if the end user disagrees with the feedback presented and determines that the LLM engine is incorrect, or for any other reason, the end user may not adjust the input, and the method may proceed to step 760.

[0202] In step 760, for example, after filling in all fields, some fields, etc., on a page, the end user may submit the data. For example, the previous step may be repeated for each field in a page or form where a tooltip with validation rules (e.g., LLM rules) is defined until all fields are properly filled in.

[0203] In some cases, data may be submitted only after all validation tooltips on the page have determined compliance with the values ​​provided in the fields by the end user who has validation rules. For example, the assistance layer may block the option to submit the form if validation rules are violated, such as by placing an overlay on the "Submit" button. In some cases, data may be submitted to the target system even if it is determined that one or more validation rules have been violated by user input. For example, if an end user does not want to change the input in a field, the end user may ignore or override the failure indications and / or feedback and submit the form even if one or more failure indications are displayed.

[0204] In some exemplary embodiments, the end user may choose to submit a form or page with at least some of its fields filled in by selecting a “Submit” button, by providing a voice command, or by activating a similar control.

[0205] In some cases, if an end user ignores a failure instruction, this incident may be recorded as an event (e.g., by the assistance layer) and provided to an admin user or another user (also called a reviewer) for review. In such cases, the admin user may review the event record, including the feedback served to the end user and ignored, and the user's input to each field. In other cases, in addition to or instead of manual review, automated filtering of override activity may be performed based on heuristics, etc. In some exemplary cases, the reviewer may manually review the event information to determine whether the end user's decision to override the LLM engine's judgment was appropriate, whether the input actually followed the validation rules, whether the LLM response was a missed detection, etc. In some cases, if the admin user estimates the input to be compliant, the admin user may, or may not, suggest edits to the validation rules, prompt configuration, etc., to ensure that future LLM feedback is more accurate. In other cases, such as when the input is actually incorrect and the LLM feedback is correct, the admin user may not need to adjust the system settings or may contact the end user to request corrected input.

[0206] In some exemplary embodiments, if the reviewer determines that the LLM feedback was a missed detection, the verification rule may be adjusted manually, automatically, etc. For example, automatic adjustment may be performed by configuring a prompt to the LLM engine to include the original verification rule, the response that was incorrectly determined to be non-compliant, and instructions to update the verification rule so that such a response will be compliant in the future. As an example, the prompt might be: "Consider the following fill request for the field named [FIELD-NAME]:[FILL-REQUIREMENT] and the following responses are the above requests [OVERRIDDEN-RESPONSE1],[OVERRIDDEN-RESPONSE2],…[OVERRIDDEN-RESPONSE n The prompt may be: "We propose an updated entry request so that it is considered to fully comply with [RESPONSE1],[RESPONSE2],..[RESPONSE m This also includes ].

[0207] Herein, we refer to Figure 8, which shows an exemplary flowchart of the method with some exemplary embodiments of the disclosed subject matter.

[0208] In step 800, chat widgets such as chatbots, mini-chat, or chat modules may be deployed and used for communication with the end user. For example, instead of communicating with the end user via messages displayed in response to each user input into defined fields, the communication and user input may be provided via the chat widget as part of a natural language conversation between the end user and the assistance layer. In some cases, the chat widget may be used in addition to or instead of the end user providing input to one or more fields. For example, some information may be inserted directly into page elements by the end user, while other information may be inserted via the chat modality.

[0209] In some exemplary embodiments, the chat widget may engage in natural language conversation with the end user. In some exemplary embodiments, one or more chat widgets may be used to iteratively analyze and provide feedback on input provided by the end user to a field until it is determined that the accumulated input conforms to the tooltip validation rules. For example, the chat widget may be used in a similar manner to the natural language conversation in the scenario of Figure 5B and disclosed in U.S. Patent No. 10,819,664 dated October 27, 2020, “Chat-Based Application Interface For Automation.”

[0210] In some exemplary implementations, the chat widget may be implemented as an overlay on a page of a third-party application (e.g., a form), or embedded within the page. In some exemplary implementations, the chat widget may be displayed adjacent to a field, on top of a field (e.g., partially or completely hiding the field), or in other exemplary implementations. In some exemplary implementations, the chat widget may be displayed as an overlay on a form, such as a hovering module displayed on the lower right side of the web page displayed to the user, in the center of the screen, in a non-anchored position (i.e., not remaining constant as the user scrolls the page), or in another corner of the page. In some cases, the chat widget may be implemented outside of the page, such as allowing an end user to fill in page fields without navigating to the page.

[0211] In step 810, one or more user inputs may be obtained into the chat widget. For example, instead of directly filling in the fields, the chatbot may use an automation process to obtain user input into the fields and to fill in the fields according to the user input when determining compliance with validation rules.

[0212] In some cases, a chat widget may be used for two or more fields on one or more pages. For example, instead of filling in fields on one or more pages, an end user may provide user input for a field to a chat widget, and the chat widget may converse with the end user until the user input conforms to the respective validation rules of validation tooltips defined for the two or more fields, and the compliant input (or a summary thereof, part thereof, etc.) may be provided to a tooltip for automatic population.

[0213] In some exemplary embodiments, the end user may provide input to fill in one or more fields on a page in a chat widget as a single chat message, or through a series of messages in a conversation with the chatbot. For example, the end user may instruct the chatbot to "Add a new lead named John Smith from NYC, email john@smith.com. He is interested in purchasing 10,000 units by September." In another example, data may be provided in stages, for example, as follows: User: Add a new lead Assistant: What is Reed's name? User: John Smith Assistant: What is John Smith's contact information? User: john@smith.com Assistant: What are John Smith's interests? User: We need to purchase 10,000 units by September. Assistant: Is that all, or is there any additional information I should add? User: That's all.

[0214] In this scenario, the chatbot may ask the user to provide information for all mandatory fields in the relevant form, as well as for non-mandatory fields, etc. In some exemplary embodiments, the chat widget may determine which fields are important based on statistics available on the digital adoption platform, and input for these fields may be requested by the end user. For example, the chat widget may select fields based on statistics showing that end users (generally or from user segments similar to the current end user) often input information in non-mandatory fields (e.g., exceeding a predetermined relative threshold such as 20%, 30%, 40%, etc.), fields where end users often encounter validation problems, fields where end users spend most of their time filling in information, etc. For example, the chat widget may be configured to suggest that the end user provide input in popular fields (generally, for similar end users, for end users who have provided similar information in other fields, etc.). As another example, the chat widget may provide a more detailed description of fields that have a relatively high validation failure rate (e.g., exceeding a given threshold) compared to other fields.

[0215] In step 820, after the end user provides input for one or more page elements, fields, etc., to the chat widget, the input may be validated. For example, the input may be validated in one or more LLM engines. In some cases, validation may be invoked considering one or more identified events, for example, when information is entered into the GUI, chat widget, etc. In some exemplary embodiments, to validate the input, the chatbot may provide the obtained information to one or more tooltips defined for the page elements. For example, the chatbot may provide the entire conversation, a part of it, contextual information, etc.

[0216] In some exemplary embodiments, if a chat widget relates to a field, the chat widget may provide a conversation in a tooltip for the field, and the tooltip may generate a prompt to the LLM engine to incorporate the conversation. In this scenario, the tooltip may send a prompt to the LLM engine and obtain a validation result from the LLM engine. If the validation result is negative, it indicates that the user input does not conform to the LLM rules, and content-based feedback from the LLM rules may be displayed to the user within the chat widget. In some exemplary embodiments, steps 810–820 may be repeated, allowing the end user to refine their input until it is determined that the conversation conforms to the LLM rules, in which case the method flow may continue to step 830.

[0217] In some cases, a chat widget may relate to multiple page fields. In such cases, the chat widget may interface with multiple field-level tooltips for each field, or it may interface with a single page-level tooltip defined for multiple fields.

[0218] In some exemplary embodiments, for field-level tooltips, each tooltip may retrieve a conversation from a chat widget, generate a prompt to the LLM engine to incorporate the conversation, send the prompt to the LLM engine, and retrieve validation results from the LLM engine. In some exemplary embodiments, due to non-compliant results, the tooltip may provide content-based feedback (from the LLM engine) to the chat widget to display feedback to the end user. In some exemplary embodiments, the end user may be allowed to respond to content-based feedback for each field until all LLM rules are followed. For example, each content-based feedback displayed to the end user may repeatedly call step 810 until no further content-based feedback is provided from the LLM for any field. The flow of the method may then proceed to step 830 in such cases.

[0219] In some exemplary embodiments, in the case of a page-level tooltip, the tooltip may generate prompts to the LLM engine in order to incorporate a conversation, send prompts to the LLM engine, and obtain validation results from the LLM engine. In some exemplary embodiments, the page-level tooltip may be configured to generate prompts containing separate LLM rules for each field, or to generate prompts containing page-level LLM rules that all fields must follow.

[0220] In some exemplary embodiments, the LLM engine may be used to analyze the content of a conversation and apply one or more validation rules to the content to determine compliance with those rules. For example, the LLM engine may determine which parts of the conversation relate to which page fields and determine the compliance of those parts with each LLM rule of the prompt. In other cases, the LLM engine may determine the compliance of the entire conversation with each LLM rule.

[0221] In some cases, instead of each tooltip generating a separate prompt for each field, a page-level tooltip may be configured to generate a prompt for multiple fields on a page that incorporates instructions on whether a field is mandatory, a field validation request, a field label, etc. In this scenario, the LLM engine may determine the compliance of all multiple fields at once and provide the corresponding output to the page-level tooltip. For example, the output may indicate that one or more first user inputs to each field comply with the LLM rule, one or more second user inputs to other fields do not comply with the LLM rule, content-based feedback for the second user input, etc. In such cases, the tooltip may provide content-based feedback to a chat widget, enabling the chat widget to display the content-based feedback to the end user.

[0222] In some cases, to enhance the level of content-based feedback, statistical information on each field may be provided to the LLM engine, such as informing the LLM engine which fields are more difficult for end users to fill out properly, what typical user errors are, etc. In some cases, the output of the validation process may be provided to the chat widget by an assistance layer, such as by extracting the output from one or more tooltips, or from the GUI, so that it is provided by one or more LLM engines.

[0223] In some exemplary embodiments, between each iteration of steps 810–820, new user input may be provided to the chat widget, and in response, one or more updated prompts (e.g., by one or more tooltips) may be generated and sent to the LLM engine for purposes such as determining compliance and providing content-based feedback.

[0224] In step 830, if it is determined that the conversation follows one or more LLM rules for one or more fields, and the input to the fields is not determined to be non-compliant, the LLM engine may be configured to generate one or more enhanced user inputs for each field.

[0225] For example, if the information from the chat is sufficient to comply with all fields, the LLM engine may be used to summarize the information into the respective field inputs that may be entered into each field. In other cases, any other text generation task may be performed to generate the field inputs, such as ensuring that each field input complies with its own LLM rule, or that all field inputs complies with page-level rules.

[0226] In step 840, the generated field inputs may be used to automatically populate each field. For example, one or more automation processes for each tooltip may take one or more elevated field inputs, and such inputs may be populated into each field via the GUI of the target system. In some cases, the automation process may be configured to populate fields in the GUI by simulating user interaction with the GUI, without relying on the application programming interface (API) of the target system. In other cases, the generated input data may be provided into fields in the GUI by any other means.

[0227] In some cases, for each field, the assistance layer may simulate interaction with the target system's GUI to select the field (e.g., to "focus" on the field) and to enter the generated value into the field. For example, user interaction with the target system's GUI may be simulated without user involvement to update the value. In other cases, the chat widget may provide input for each field to the end user, and the end user may be allowed to manually enter the respective input into each field (e.g., using copy and paste functionality).

[0228] In step 850, after filling in the fields on the page (e.g., drawing the form), the page may be submitted to the target system. In some cases, the form may be submitted via a submit button or another user interaction (e.g., a voice command). The form may be submitted by the assistance layer using simulated user interaction with the GUI to invoke the submit button, or by manual interaction by the end user, etc.

[0229] Herein, we refer to Figure 9, which shows schematic diagrams of exemplary architectures in which the disclosed subject matter may be used, based on some exemplary embodiments of the disclosed subject matter.

[0230] In some exemplary embodiments, the target system 900 may be any third-party target system, such as SaaS platforms ServiceNow®, Salesforce®, Microsoft Dynamics 365®, SuccessFactors®, Workday®, Liveperson®, Jira®, SharePoint®, NetSuite®, or TalentSoft®. For example, the target system 900 may include a banking application, a website listing opera performances, and so on.

[0231] In some exemplary embodiments, the target system 900 may define a page 902, such as a form having one or more fields 904, such as text fields, for performing banking transactions, purchasing tickets, etc. The page 902 may have a visual display that is shown to the end user on the screen of the end user's user device 930. In some cases, the target system 900 allows the end user to interact with the page 902 via the user device 930 in one or more different modalities. In some exemplary embodiments, the page 902 may allow the end user to enter information into fields 904 and send such information to the target system 900 for processing. In some cases, the page 902 may be associated with client-side functions, server-side functions, etc. As an example, the page 902 may be used to generate a new entity in the target system 900's database (e.g., a new "Lead" entity), update an existing entity (e.g., update the "Lead" entity), etc.

[0232] In some exemplary embodiments, user device 930 may run target system 900 and assistance layer 910 simultaneously. In some exemplary embodiments, assistance layer 910, as defined by DAP, may function as an intermediate layer between user device 930 and target system 900. For example, target system 900 may be a web-based system. In such a case, user device 930 may use a web browser to enable interaction with target system 900. In such a case, client-side code in the target system 900's code may directly call assistance layer 910 to perform its functions when the user accesses target system 900 using a browser. In addition, or alternatively, a browser extension may be installed which may inject client-side code for assistance layer 910 into a fetched web page, thereby calling assistance layer 910 to perform its functions. In other cases, the assistance layer 910 may be executed in any other form. In some exemplary embodiments, the assistance layer 910 may be separate from the target system 900, may not collaborate with the target system 900, may not make application programming interface (API) calls to the target system 900, may not have access to the backend of the target system 900, and so on.

[0233] In some exemplary embodiments, the assistance layer 910 utilizes one or more validation tooltips having field validators 912 to validate user input to each field 904. In some cases, some fields on page 902 may not have any corresponding tooltips having field validators 912. In addition, or alternatively, some fields may have multiple corresponding field validators 912, such as those relating to different segments of the end user. In such cases, the assistance layer 910 may be configured to select the relevant field validator 912 for each end user.

[0234] In some exemplary embodiments, the field validator 912 may utilize one or more LLM engines, such as the LLM engine 920, to evaluate whether user input to field 904 conforms to LLM rules. In some cases, the LLM engine 920 may be on-premises, in a private tenant in the cloud, in a public cloud, etc. Specific deployments may vary depending on confidentiality and privacy concerns related to the data sent to the LLM engine 920. In addition, or alternatively, multiple different LLM engines 920 may be deployed in different locations, enabling the use of different engines for different content, tasks, etc.

[0235] In some exemplary embodiments, an end user using user device 930 may input an input to field 904 (e.g., a trigger event) to cause the respective validators of field validator 912 to be executed. In some exemplary embodiments, prompts may be generated using input from user device 930 and LLM rules defined by an admin user (an admin user who defines the assistance layer 910 via DAP and is not associated with the target system 900) and provided to LLM engine 920. The LLM engine 920 may determine whether the input conforms to the LLM rules. In case of a violation of the LLM rules, the LLM engine 920 may provide content-based feedback, such as providing suggestions on how the input can be improved to conform to the LLM rules.

[0236] In some exemplary embodiments, the verification tooltip may retrieve results from the LLM engine 920 and display the output on the screen of the user device 930 based on the results.

[0237] In some cases, the chat GUI layer 914 may be used to communicate with the user of the user device 930, for example, according to the method shown in Figure 8. For example, the assistance layer may deploy the chat GUI layer 914 on the GUI of the target system 900, for example, as an overlay. In some cases, the end user may interact with the chat GUI layer 914 to provide input to one or more of the fields 904, and the chat GUI layer 914 may generate prompts in response and provide feedback to the input within the chat. As an example, the chat GUI layer 914 may include a chatbot used to communicate with the end user, using the LLM engine 920 to generate natural language messages, such as using the text input modality, using voice commands, etc.

[0238] In some exemplary embodiments, when an end user interacts with the chat GUI layer 914, the chat GUI layer 914 may utilize the LLM engine 920 to conduct a conversation with the user in natural language. In some exemplary embodiments, the LLM engine 920 may be used to validate user input according to one or more LLM rules, such as each field 904, the field validator 912 of field 904, etc., to provide content-based feedback for non-compliant input. For example, the chat GUI layer 914 may present the end user with one or more questions and prompt the end user to provide relevant information for all fields 904 on page 902. In some cases, the questions may focus only on mandatory fields 904 on page 902, on fields 904 with LLM rules, on all fields 904, etc.

[0239] In some exemplary embodiments, once the chat GUI layer 914 determines that each of the one or more inputs to field 904 conforms to the field validator 912, the chat GUI layer 914 may provide the input to the field validator 912. In some cases, the chat GUI layer 914 may process the input before providing it to the field validator 912, for example, by summarizing the conversation. For example, the field validator 912 may provide the input to a field automation process, either in its original or processed version, which may automatically populate field 904 by simulating user interaction with the GUI. In other cases, the chat GUI layer 914 may invoke another automation process to automatically populate field 904. In some exemplary embodiments, once the validated input has been provided to field 904, the form on page 902 may be submitted to the target system 900.

[0240] Herein, we refer to Figure 10, which shows schematic diagrams of exemplary embodiments in which the disclosed subject matter may be used, based on several exemplary embodiments of the disclosed subject matter.

[0241] In some exemplary embodiments, the environment 1000 may include a plurality of user devices 1040. The user devices 1040 may include a personal computer (PC), stationary computer, tablet, smartphone, etc., of each end user. The user devices 1040 may be connected to a computerized network 1020 such as a wide area network (WAN), local area network (LAN), wireless network, internet, intranet, etc.

[0242] In some exemplary embodiments, the user device 640 may deploy and run a third-party application, program, etc., such as the target application 1043. For example, the target application 1043 may include a SALESFORCE® application, Zendesk®, etc. In some exemplary embodiments, the target application 1043 may include a web page, web application, browser extension, mobile application, desktop application, etc. In some exemplary embodiments, the target application 1043 may display to the end user one or more screens that constitute a GUI, which may include one or more GUIs.

[0243] In some exemplary embodiments, the environment 1000 may include an administrator computer 1030 (referred to as the “admin device”) operated by an administrator user or another user with appropriate qualifications and permissions. In some exemplary embodiments, the administrator computer 1030 may be connected to a computerized network 1020.

[0244] In some exemplary embodiments, the administrator computer 1030 may deploy, run, etc., the platform 1033 and the target application 1043 (not shown). For example, the platform 1033 may include a software platform such as a DAP platform configured to allow a user to generate an assistance layer, for example, an assistance layer 1045, on the target application 1043. For example, the assistance layer 1045 may be generated to include a validation tooltip or any other validation widget for a third-party application such as the target application 1043. In some exemplary embodiments, an admin user may define one or more validation tooltips to be applied to each field of a form displayed by the target application 1043 via the platform 1033 on the administrator computer 1030.

[0245] In some exemplary embodiments, the verification tooltip may be defined to have multiple settings, configurations, etc., such as a trigger event that causes an automation process to run (indicating that data has been entered into a field), an automation process that performs verification of user input, and a display configuration for displaying the verification output.

[0246] In some exemplary embodiments, the defined verification tooltips in assistance layer 1045 may be made available to user device 1040. For example, assistance layer 1045 may be generated as a computer-executable program product, such as a script, software, browser extension, mobile application, web application, software development kit (SDK), shared library, dynamic link library (DLL), or SaaS, without limitation. In this example, assistance layer 1045 may be defined by an admin user and provided to user device 1040 by sending the program product to user device 1040. In another example, assistance layer 1045 may be made available to user device 1040 via an API call to a cloud or server that stores the defined widgets, for example, server 1010. As another example, the assistance layer 1045 may be made available to the user device 1040 through an update to an existing application or software agent deployed by the user device 1040.

[0247] In some exemplary embodiments, at least one user device 1040 may run the corresponding assistance layer 1045, target application 1043, etc., simultaneously, for example. In some exemplary embodiments, the assistance layer 1045 may run on the target application 1043 and enable the user of user device 1040 to receive content-based feedback on inputs to fields of the target application 1043, and to receive indications as to whether those inputs have been validated.

[0248] In some cases, the user device 1040 may run a software agent (not shown) associated with the assistance layer 1045. For example, the software agent may be configured to acquire the GUI appearing on the GUI of the target application 1043, communicate the acquired data to a server such as server 1010, and apply the assistance layer 1045 to the GUI according to the data from server 1010. In some cases, the software agent may correspond to one or more software agents disclosed in U.S. Patent No. 10,620,975, dated April 14, 2020, “GUI Element Acquisition Using A Plurality Of Alternative Representations Of The GUI Element,” which is incorporated in its entirety by reference without causing denial. As another example, the assistance layer 1045 may run independently of server 1010.

[0249] For example, a trigger event may be identified by an agent running on user device 1040, for instance, by monitoring the screen of user device 1040 and determining that the user has inserted data into a field of target application 1043, which has a defined validation tooltip in the assistance layer 1045. Identifying the data from the user may trigger the execution of various automation processes, such as a validation process including the development of LLM techniques for validating the data. The results or outputs of the validation process may be displayed to the user of user device 1040 according to the configuration of the assistance layer 1045, its tooltips, etc., thereby assisting the user in performing the digital task of filling in fields.

[0250] In some exemplary embodiments, the server 1010 may be connected to a computerized network 1020. The server 1010 may be directly or indirectly connected to an administrator computer 1030, a user device 1040, etc., via the computerized network 1020, etc. In some exemplary embodiments, the server 1010 and the administrator computer 1030 may or may not be implemented by the same physical device.

[0251] The present invention may be a system, method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0252] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or protruding structures in grooves having recorded instructions, and any suitable combination thereof. Computer-readable storage media, when used herein, should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0253] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmitters, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in the computer-readable storage medium within each computing / processing device.

[0254] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of computer-readable program instructions for personalizing the electronic circuit in order to carry out aspects of the present invention.

[0255] Aspects of the present invention are described herein with respect to flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0256] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device for manufacturing machines, such that instructions executed via the processor of a computer or other programmable data processing device produce means for performing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular way such that the stored instructions contain a manufactured article containing instructions that perform a mode of function / operation specified in one or more blocks of a flowchart and / or block diagram.

[0257] Computer-readable program instructions may be loaded into a computer, other programmable device, or other device in order to cause a computer execution process to occur, such that instructions executed in a computer, other programmable device, or other device perform a function / operation specified in one or more blocks of a flowchart and / or block diagram, by executing a series of operational steps in the computer, other programmable device, or other device.

[0258] The flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction containing one or more executable instructions for performing a specified logical function. In some alternative implementations, the functions shown in a block do not have to occur in the order shown. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or depending on the functions involved, the blocks may sometimes be executed in reverse order. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or operation or a combination of dedicated hardware and computer instructions.

[0259] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context clearly indicates otherwise. When the terms “comprises” and / or “comprising” are used herein, they specify the presence of the described features, integers, steps, operations, elements, and / or components, but it will be understood that they do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0260] In addition to the corresponding structures, materials, actions, and equivalents of all means or steps, the functional elements in the following claims are intended to include any structures, materials, or actions for performing a function in combination with other claimed elements specifically described in the claims. The description of the invention is provided for illustrative and explanatory purposes, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and alterations will become apparent to those skilled in the art without departing from the scope and spirit of the invention. The examples have been selected and described in order to best illustrate the principle and practical application of the invention and to enable those skilled in the art to understand the invention for various embodiments with various modifications to suit specific intended uses.

Claims

1. A method to be performed on an administrator user's end device, wherein the method is The method includes selecting a page element on a page of a third-party application, wherein the page element includes a field, and the third-party application is executable on the administrator user's end device and multiple end devices of end users, and the method further includes: The method includes defining a trigger event, the trigger event including identifying that an end user has entered an input into the field, and the method further includes: The method includes defining an automation process to be executed in response to the occurrence of the trigger event, wherein the automation process is configured to generate prompts to a generative artificial intelligence (AI) engine to incorporate at least the inputs and validation rules, defining the automation process includes defining the validation rules using free text in natural language, wherein the prompts are configured to be generated to include a predetermined structure of static and dynamic parts, the dynamic part is configured to be populated with the inputs from the end user in response to the activation of each trigger event, the static part is configured to include the validation rules and instructions to determine whether the input conforms to the validation rules, the automation process is configured to send the prompts to the generative AI engine and to obtain output from the generative AI engine in response to the prompts, and the method further includes A method for being performed on an administrator user's end device, comprising defining a configuration for displaying the results on the page, wherein the results are determined based on the output from the generating AI engine, and the results indicate at least whether the input conforms to the validation rules.

2. The method according to claim 1, wherein the prompt is configured to instruct the generative AI engine to provide content-based feedback to the input, the content-based feedback includes suggestions on how to modify the input in accordance with the validation rules.

3. The configuration for displaying the above results is: Updating one or more characteristics of the page based on the results, wherein the one or more characteristics include at least one of the field border color, the field background color, or the field highlight, and Displaying the results as an overlay on the page, wherein the overlay may be configured to be displayed on the page, the overlay is not part of the third-party application, and the overlay includes at least one of a chat widget, a tooltip, a popup element, or a text field. The method according to claim 1, comprising at least one of the following.

4. The method according to claim 1, wherein the dynamic portion is configured to populate context data each time the trigger event is activated, and the context data includes data from the page.

5. The method according to claim 4, wherein the data from the page includes the names of other fields on the page and at least some inputs to those other fields.

6. The method according to claim 4, wherein the data from the aforementioned page includes validation rules for other fields on the aforementioned page.

7. The method according to claim 6, wherein the other fields include fields of a form, and the validation rules for the other fields are defined to validate the end user's input to the fields of the form.

8. The method according to claim 1, wherein selecting the page elements, defining the trigger events, defining the automation processes, and defining the configurations are performed via a digital adoption platform running on the administrator user's end device, the digital adoption platform is independent of the third-party application, and the digital adoption platform is configured to enable the administrator user to use the digital adoption platform to generate assistance layers to run on the third-party application on the multiple end devices, the assistance layers being configured to assist the end user in performing digital tasks.

9. The method according to claim 8, wherein the command of the prompt includes a pre-configured command of the digital adoption platform that is not defined by the administrator user.

10. The method according to claim 8, wherein the digital task includes filling out one or more forms in the third-party application.

11. The method according to claim 8, wherein the assistance layer includes a validation tooltip defined for the field, and the validation tooltip is configured to include the validation rule.

12. Define that a guidance message be displayed to the end user before the input is entered into the field, The method according to claim 11, further comprising selecting the guidance message from one or more predefined sets of messages for the verification tooltip, wherein the one or more predefined sets of messages are past messages for the field defined by one or more users of the organization to which the administrator user belongs.

13. The method according to claim 1, wherein the configuration for displaying the results includes displaying the results as messages in a chat widget, the chat widget being overlaid on the page, the inputs to the fields being provided to the chat widget, and the automation process being configured to provide the fields with a summary of the compliant inputs in the chat widget.

14. The method according to claim 1, wherein the generation AI engine includes a large-scale language model (LLM) engine or a small-scale language model (SLM) engine.

15. A method to be implemented on an end-user's end device, wherein the method is The method further includes displaying a third-party application and an assistance layer to the end user, wherein the assistance layer runs on the third-party application, and the method further includes: The method includes obtaining user input from the end user to fields on the page of the third-party application, and the method further includes: The method includes displaying a message to the end user on the page, the message being retrieved from the assistance layer, the message indicating that the content of the user input does not conform to the validation rules of the assistance layer, the message providing content-based feedback to the user input, the content-based feedback including suggestions on how to modify the content of the user input in a form that conforms to the validation rules, the assistance layer being configured to generate a prompt to a generative artificial intelligence (AI) engine, to send the prompt to the generative AI engine, and to retrieve the content-based feedback from the generative AI engine, the prompt including a predetermined configuration of a static portion and a dynamic portion, the dynamic portion being configured to be populated with the user input each time the field is filled in by the end user, the static portion being configured to include the validation rules, and the method further includes This includes obtaining the corrected user input to the field, the corrected user input being obtained following the display of the message, A method for implementation on the end user's end device.

16. The method according to claim 15, wherein, before obtaining the user input, a guidance message is displayed to the end user, the guidance message includes predefined text that guides the end user on how to fill in the fields, and the predefined text is provided by the assistance layer builder.

17. The method according to claim 15, wherein the generation AI engine includes a large-scale language model (LLM) engine or a small-scale language model (SLM) engine.

18. A computer program product comprising a non-temporary computer-readable medium holding program instructions, wherein, when the program instructions are read by a processor, the processor causes the processor to execute a method on an end-user's end device, and the method is The method further includes displaying a third-party application and an assistance layer to the end user, wherein the assistance layer runs on the third-party application, and the method further includes: The method includes obtaining user input from the end user to fields on the page of the third-party application, and the method further includes: The method includes displaying a message to the end user on the page, the message being retrieved from the assistance layer, the message indicating that the content of the user input does not conform to the validation rules of the assistance layer, the message providing content-based feedback to the user input, the content-based feedback including suggestions on how to modify the content of the user input in a form that conforms to the validation rules, the assistance layer being configured to generate a prompt to a generative artificial intelligence (AI) engine, to send the prompt to the generative AI engine, and to retrieve the content-based feedback from the generative AI engine, the prompt including a predetermined configuration of a static portion and a dynamic portion, the dynamic portion being configured to be populated with the user input each time the field is filled in by the end user, the static portion being configured to include the validation rules, and the method further includes A computer program product that includes obtaining corrected user input to the aforementioned field, the corrected user input being obtained following the display of the message.

19. The computer program product according to claim 18, wherein the dynamic portion is populated with context data from the page, the context data includes at least one of the names of other fields on the page, validation rules for the other fields, or input from the end user to the other fields.

20. The computer program product according to claim 18, wherein the assistance layer includes a verification tooltip defined for the field, and the verification tooltip includes the verification rule.

21. The computer program product according to claim 18, wherein the message is displayed in a chat widget of the assistance layer, the chat widget is overlaid on the page, and the user input to the field is provided via the chat widget.

22. A device including a processor and coupled memory, wherein the processor is located in the end device of an administrator user. A step of selecting a page element on a page of a third-party application, wherein the page element includes a field, and the third-party application is executable on the administrator user's end device and on multiple end devices of end users. A step of defining a trigger event, the trigger event including identifying that an end user has entered an input into the field, A step of defining an automation process to be executed in response to the occurrence of the trigger event, wherein the automation process is configured to generate prompts to a generative artificial intelligence (AI) engine to incorporate at least the inputs and validation rules, defining the automation process includes defining the validation rules using free text in natural language, the prompts are configured to be generated to include a predetermined structure of static and dynamic parts, the dynamic parts are configured to be populated with input from the end user in response to the activation of each trigger event, the static parts are configured to include the validation rules and instructions to determine whether the input conforms to the validation rules, and the automation process is configured to send the prompts to the generative AI engine and to obtain output from the generative AI engine in response to the prompts. An apparatus adapted to perform the steps of: defining a configuration for displaying results on the page, wherein the results are determined based on the output from the generating AI engine, and the results indicate at least whether the input conforms to the verification rules.

Images