Toolcast Management System

Abstract

A system for creating and displaying toolcasts includes a processor and a toolcast management system executing on the processor in communication with an underlying system. The toolcast management system creates, displays, and updates at least one toolcast to provide media guidance to a user of the underlying system according to a toolcast management interface with the underlying system and an interaction between the user and the at least one toolcast. The toolcast management system includes an interface module that interfaces with the underlying system and detects at least one of objects, data, activities, and events to be recorded in a recording phase of the at least one toolcast, a toolcast creator that records, generates, and edits the at least one toolcast according to at least an output of the interface module, and a toolcast player that plays back the at least one toolcast while interacting with the user.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 183,662, filed May 4, 2021, and U.S. Provisional Patent Application No. 63 / 184,809, filed May 6, 2021, both of which are incorporated herein by reference.

[0002] Technical Field The present invention generally relates to the creation of assistive content elements, particularly for websites and applications.

Background Art

[0003] Website building systems (WBSs) are used by both beginners and experts to create interactive websites. Existing WBSs are based on visual editing models, and most WBSs typically provide multiple templates, which may include complete sample websites, website sections, single pages, or sections of pages.

[0004] Users of WBSs (also known as designers, contributors, contributing users, or site editors) can design websites, and end - users of the websites (the "users' users") can access the websites created by the users. End - users typically access the system in read - only mode, but the WBS (and the website) can allow end - users to make changes to the website, such as adding or editing data records, adding talkbacks to news articles, adding blog entries to blogs, etc. WBSs can actually permit multiple levels (i.e., three or more levels) of users and assign different permissions and functions to each level. Users of WBSs (in a full or partial online configuration, as described in more detail hereinafter) can register with the WBS server that manages users, their websites, and access by end - users.

[0005] A WBS can be a standalone system or it can be embedded within a larger editing system. A WBS can also be online (i.e., the application is edited and stored on a server), offline, or partially online (the website is edited locally but uploaded to a central server for publication). A WBS may use an internal data architecture to house the WBS-based site, which can organize the internal data and elements of the site being processed within the system. This architecture may differ from the external view of the site (e.g., seen by the end user). The architecture may also differ from how HTML pages typically sent to a browser are organized.

[0006] For example, the internal data architecture may include additional per-element properties within a page (such as author, creation time, access permissions, links to templates, and SEO-related information) that are related to editing and maintaining sites within the WBS but are not visible from the outside to end users (or even some editing users). The WBS may implement some of its functionality (including both editing and runtime functions) on a server or set of servers, and some of its functionality on client elements. The WBS may also dynamically decide whether some of its functionality runs on the server or on the client platform.

[0007] A WBS typically deals with the creation and editing of a visually designed application (such as a website) that consists of pages, containers, and components. Pages are displayed individually and may contain components. Components may include containers and atomic components. Figure 1 is a schematic block diagram of a website construction system (WBS) 2 that may be used to construct a website 3 according to some explicit embodiments of the present invention. WBS 2 may be used to construct, produce, edit, and / or generate a website 3 that may include pages 4 which may contain components 5 (e.g., text, images, videos).

[0008] WBS can also support a hierarchical structure of components using various types of container components, including atomic components (text, images, shapes, videos, etc.) and other components (e.g., standard containers, single-page containers, multi-page containers, gallery containers, etc.). Subpages contained within a container are called minipages, and each of them can contain multiple components. Some container components can only display one minipage at a time, while other containers can display multiple pages simultaneously.

[0009] Components may or may not have content. An example of the first category is a star-shaped component that has no internal content (but has color, size, position, attributes, and other parameters). An example of the second category is a text paragraph component, whose internal content includes the text itself, as well as information about the font, formatting, and layout (which are part of the content rather than just attributes of the component). Needless to say, this content may differ from instance to instance of the text paragraph component. Components that have content are often called fields (e.g., "text fields").

[0010] Pages can use templates, i.e., general page templates or component templates. Specific examples of templates include the use of an application master page containing components duplicated on all other standard pages, and the use of an application header or footer (which is repeated on all pages). Templates can be used for entire pages or for page sections. A WBS can provide inheritance between templates, pages, or components, which may include multi-level inheritance, multiple inheritance, and diamond inheritance (i.e., A inherits from B and C, and both B and C inherit from D).

[0011] The visual arrangement of components within a page is called layout. WBS may also support dynamic layout processing, which is a process in which editing of a given component (or other changes affecting a given component, such as externally initiated content changes) may affect other components, as further described in U.S. Patent No. 10,185,703, granted on 22 January 2019, entitled "Website Design System Integrating Dynamic Layout and Dynamic Content," shared by the applicant and incorporated herein by reference.

[0012] WBS can be extended using add-on applications such as third-party applications and their components (TPAs), list applications (e.g., as described in U.S. Patent Publication No. 2014 / 0282218, “WBS Integrating Data Lists with Dynamic Customization and Adaptation,” published September 18, 2014, shared by the applicant and incorporated herein by reference), and WBS configurable applications (e.g., WCAs, as described in U.S. Patent Publication No. 2020 / 0151266, “System And Method for Creation and Handling of Configurable Applications for Website Building Systems,” published May 14, 2020, shared by the applicant and incorporated herein by reference). These third-party applications and list applications can be added to and integrated into the designed website.

[0013] Such third-party applications and list applications can be purchased (or otherwise obtained) directly from third-party application vendors, through several distribution mechanisms, such as being pre-installed in the WBS design environment, from application stores (integrated into or outside of the WBS market store), for example.

[0014] Third-party applications can be hosted on the WBS vendor's own servers, the third-party application vendor's servers, or on a fourth-party server infrastructure.

[0015] WBS can also allow the addition of procedural code to some or all of the system's entities. Such code may be written in a standard language (such as JavaScript), an extended version of a standard language, or a language specific to the WBS itself. The executed code may reference APIs provided by the WBS itself or by external providers. The code may also reference internal structures and objects of the WBS, such as pages, components, and their attributes.

[0016] Procedural code elements can be activated via event triggers that may be associated with user activities (such as mouse movements or clicks, page transitions), activities associated with other users (such as a basic database or specific database record updated by another user), system events, or other types of conditions. The use of such procedural code elements is further described in U.S. Patent No. 10,209,966, granted on 19 February 2019, entitled “Custom back-end functionality in an online website building environment,” which is shared by the applicant and incorporated herein by reference.

[0017] The activated code may be executed within the client element of the WBS on the server platform, or a combination of the two, or by using a dynamically determined execution platform. Such a system is described in U.S. Patent Publication No. US2018 / 0293323, “System and Method for Smart Interaction Between Website Components,” published on 11 October 2018, which is shared by the applicant and incorporated herein by reference.

[0018] Typical site creation may be based on several models, including a visual editing model (where the user edits a previously created site), an automated site generation model, or a combination thereof as shown in Figure 2, as described in U.S. Patent No. 10,073,923, granted September 11, 2018, entitled “System and Method for the Creation and Update of Hierarchical Websites Based on Collected Business Knowledge,” shared by the applicant and incorporated herein by reference.

[0019] Throughout the specification, it will be understood that the acronym WBS may be used to represent a website construction system. Figure 2 shows a typical system 100 including a WBS2 that communicates with a client system and an external system 70 operated by WBS vendor staff 61, a site designer 62 (i.e., a user), and a site user 63 (i.e., a user of the user). The WBS2 may further include a WBS site manager 10, an object marketplace 15, a WBS RT (runtime) server 20, a WBS editor 30, a site generation system 40, and a WBS content management system (CMS) 50.

[0020] In a visual editing model, the user (designer) edits a site based on one or more website templates. A WBS provider may offer multiple site (or other) templates, each potentially containing a complete sample website, a website section, a single page, or a section of a page. While users may have the option to start with an empty site (essentially a "blank page" template), they typically begin with an actual site template.

[0021] WBS providers can offer site templates ranging from very general site templates (e.g., mobile sites, e-stores) to more specific site templates (e.g., law firms, restaurants, flower shops) and even highly specific site templates (e.g., commercial real estate law firms or Spanish tapas restaurants). Such templates are typically stored in a repository accessible to WBS users and are usually categorized by industry, subtype, or sector. Templates can also be created (and categorized) according to style, color range, or other parameters, rather than simply by industry. Site templates can be extended with additional (usually backend) functionality, services, and code to become a full-fledged vertical solution integrated with WBS.

[0022] A site can be converted / rendered to HTML for display. However, in some embodiments, components may be implemented and / or rendered using additional technologies such as locally installed or downloaded code, plugins, rendering frameworks (such as React), graphic display frameworks (such as SVG), or other methods.

[0023] Therefore, a user's initial experience when creating a site using the WBS visual editor is typically to select a template (e.g., according to style or industry type / subtype), possibly an empty template, and then edit the template in the visual editor, which includes editing content, logic, layout, and attributes. Such editing may involve (in particular) adapting the template and its elements to the user's business details. The user can then publish the modified site.

[0024] In the site generation model, the WBS generates an initial site for the user based on a selected template, which may be modified by filling in common elements of information, potentially allowing for follow-up editing of the generated site. This filling is necessary because various pieces of information (such as the company name or a description of the management team) are included in multiple locations within the template pages. Therefore, the user may need to change the company name in multiple locations throughout the template (for example).

[0025] Furthermore, some template elements (e.g., a general product page) may appear multiple times, with each instance displaying details of a different instance of a basic entity (e.g., different products offered on the site). These multiple instances may be manually specified (e.g., details of different people on a company's management team) or dynamically derived from an external database (e.g., product details from a "Products for Sale" database). Such configurations are often known as "repeaters."

[0026] Templates can also contain fields. For example, a WBS might allow a template designer to specify fields (also known as "placeholders") such as [CompanyName], [ProductName], and [ProductPrice] for inserting values ​​into the template. Users can also specify values ​​for fields defined in the template selected for the website.

[0027] WBS can allow users to input not only simple or complex values ​​(e.g., text and images), but also additional (non-field) information such as the selection of the page or site area to include, colors, style information, links, formatting options, website display options, and decorative elements (e.g., borders and backgrounds).

[0028] With the WBS, the user can also enter some of this additional information before selecting a template and use this information to assist in template selection (e.g., by narrowing down a set of proposed templates). For example, the user can select a specific general color scheme (e.g., pastel colors) or style (e.g., business / formal), and the system can then use this selection to narrow down the set of proposed templates.

[0029] The WBS can also display a series of views or questionnaires to allow the user to enter values or selections (both of the defined fields and the above additional information). The WBS can further create a connection (or binding) between a plurality of instance elements of a template (as described above in this specification) and an internal database or an external database that provides the data instances used to generate the displayed instances.

[0030] Once a template is selected and its fields and additional information are specified (e.g., through a questionnaire or through binding to a data source), the WBS can generate a website containing the combined information. The user can then publish the site (through the WBS or otherwise).

[0031] The WBS can support the SEO review of applications built into the WBS, as described in U.S. Patent Publication No. US2019 / 0026280, titled "System and method for integration of search engine optimization in website building systems," published on January 24, 2019, and shared by the applicant and incorporated herein by reference.

[0032] WBS may also perform semi-automated site creation using a different model described in U.S. Patent No. 10,073,923. In this model, the system may collect information about the user and the user's website requirements from multiple sources, including, for example, user-filled questionnaires, existing user presence (such as existing website or social media presence), industry sources (such as general trading websites), offline information, and an internal system repository that provides information about specific industries, such as basic template information for specific industries (e.g., lawyers, restaurants, plumbers, graphic designers) which may be refined for specific industries (e.g., distinguishing between real estate lawyers and personal injury lawyers).

[0033] WBS may also collect external information from other sites, both internal and external to WBS. Such information may influence choices such as surveys and layout elements provided, and suggested defaults. Such information may also be collected statistically or summarized to protect user privacy, anonymity, and legal rights (such as copyright), and typically does not expose information belonging to any single user. Such information may be found based on information provided by users, which may be direct (e.g., existing website addresses) or indirect (trade names and geographical addresses that can be used to find information about companies).

[0034] The collected information is analyzed and then placed in a repository of content elements, which are then used to present content from content elements and map them onto layout elements that combine layout elements to form the site. The process of mapping, selecting, and combining layout elements may be fully automated or semi-automated (i.e., involving user interaction).

[0035] To support the above functions, WBS may maintain a set of repositories typically stored on one or more servers or server farms. Such repositories may include various related repositories such as user information / profile repositories, WBS component repositories, WBS site repositories, business intelligence (BI) repositories, edit history repositories, and third-party application store repositories. The system may also include site / content creation related repositories such as survey type repositories, content element type repositories, layout element type repositories, design kit repositories, completed survey repositories, content element repositories, layout element repositories, rule repositories, and family / industry repositories. A description of these repositories is found in U.S. Patent No. 10,073,923. [Overview of the Initiative]

[0036] A preferred embodiment of the present invention provides a system for creating and displaying toolcasts. The system comprises a processor and a toolcast management system running on the processor that communicates with an underlying system, the toolcast management system creating, displaying, and updating at least one toolcast in order to provide media guidance to a user of the underlying system in accordance with a toolcast management interface with the underlying system and interaction between a user and at least one toolcast. The toolcast management system includes an interface module that interfaces with the underlying system and detects at least one of objects, data, activities, and events of the underlying system that will be recorded in the recording phase of at least one toolcast, a TC (toolcast) creator that records, generates, and edits at least one toolcast in accordance with the output of at least one interface module, and a TC (toolcast) player that plays back at least one toolcast while interacting with a user.

[0037] Furthermore, according to a preferred embodiment of the present invention, the basic system is integrated with a toolcast management system.

[0038] Furthermore, according to a preferred embodiment of the present invention, at least one toolcast is at least one of active tutorials, documentation, courseware, and in-user messaging.

[0039] Furthermore, according to a preferred embodiment of the present invention, the basic system is at least one of a website, a website building system, and a document editing system.

[0040] Furthermore, according to a preferred embodiment of the present invention, at least one tool cast is at least one of basic and composite.

[0041] Furthermore, according to a preferred embodiment of the present invention, the system comprises at least one of the following: a database for storing basic toolcasts and composite toolcasts; a maintenance module that maintains at least one toolcast when changes are made to the underlying system; a toolcast synchronizer that supports the synchronization of at least one toolcast with other related toolcasts when changes are made to the underlying system; an underlying system structure mapper that provides a mapping of the underlying system's structure, associations, events, and interfaces to a toolcast management system for at least one toolcast; and an ML (machine learning) manager that collects data from the underlying system and user data and provides machine learning training and support accordingly to TC creators and TC players.

[0042] Furthermore, according to a preferred embodiment of the present invention, the TC Creator includes: a Toolcast Storyboard Creator for generating a storyboard for at least one toolcast; a Generative Text Engine for generating text for the storyboard of at least one toolcast; an UNS (Underground Systems) Analyzer for analyzing the elements and site structure of the Underground Systems; a BI (Business Intelligence) Analyzer for analyzing business intelligence (BI) information collected from the Underground Systems; a CTC (Compound Toolcast) Creator for creating a composite toolcast including a hierarchy of basic toolcasts; a Text-Based Recorder for enabling at least one toolcast designer to create a text-based toolcast using a rich text editor; and a recording function for enabling at least one toolcast designer to record themselves. The system comprises at least one of the following: an audio-based recorder that enables conversion to a toolcast file; an association manager that creates toolcast associations in an interfaced configuration between the toolcast management system and the underlying system; a TC (toolcast) path optimizer that determines and optimizes the activity flow of at least one toolcast; an edit-time recommender that provides editing hints and options to the designer of at least one toolcast; a coverage analyzer that detects mismatches between the underlying system and at least one toolcast according to the toolcast definition of at least one toolcast; and a UNS (underlying system) / TC (toolcast) editor that provides integrated or combined editing of the underlying system together with at least one toolcast.

[0043] Furthermore, according to a preferred embodiment of the present invention, the TC player comprises at least one of the following: a TC activator that automatically enables the activation of at least one toolcast by the user; a remote user simulator that enables at least one toolcast to trigger responsive events on the underlying system by simulating user actions; a TC (toolcast) control bar display unit that displays a toolbar so that the user can control the display of at least one toolcast; an ACE (supporting content element) display unit that creates UI (user interface) elements added to or extended on top of the display by the TC control bar display unit; and a user interaction manager that enables interaction between at least one toolcast and the user during playback of at least one toolcast.

[0044] Furthermore, according to a preferred embodiment of the present invention, the interface module comprises at least one of the following: an integrated interface that interfaces with an integrated underlying system; a non-integrated interface that interfaces with an unintegrated underlying system; and a UNS (Underlying System) database interface that interfaces with an underlying system database to provide data for analysis by the TC Creator.

[0045] Furthermore, according to a preferred embodiment of the present invention, at least one toolcast is a user-adaptable toolcast.

[0046] Furthermore, according to a preferred embodiment of the present invention, the system also comprises an additional product generator that performs at least one toolcast to produce an alternative product, an app (application) store for sharing at least one toolcast via an app store, and a tooltip handler that adds at least one toolcast to an existing tooltip.

[0047] Furthermore, according to a preferred embodiment of the present invention, the maintenance module comprises at least one of the following: a TC (toolcast) introduction agent for analyzing the version of the underlying system and collecting initial information of the underlying system structure mapper; an information collection unit for collecting information on changes to elements of at least one toolcast; an update analyzer for receiving information from the information collection unit and determining which toolcast elements of at least one toolcast have been changed and are no longer functioning; a notification function for notifying at least one of the underlying system and the designer according to the output of the update analyzer; a TC (toolcast) adjustment unit for appropriately correcting non-functional elements of at least one toolcast; and a TC (toolcast) fallback application unit for transmitting the content of at least one toolcast through alternative means.

[0048] A preferred embodiment of the present invention provides a method which includes interface with a toolcast management system to an underlying system and create, display, and update at least one toolcast in order to provide media guidance to a user of the underlying system in accordance with the interaction between the user and at least one toolcast. Creating, displaying, and updating includes interface with the underlying system, detecting at least one of the objects, data, activities, and events of the underlying system which will be recorded in the recording phase of at least one toolcast, recording, generating, and editing at least one toolcast in accordance with the output of at least the detection, and playing back at least one toolcast while interacting with the user.

[0049] Furthermore, according to a preferred embodiment of the present invention, the basic system is integrated with a toolcast management system.

[0050] Furthermore, according to a preferred embodiment of the present invention, at least one toolcast is at least one of active tutorials, documentation, courseware, and in-user messaging.

[0051] Furthermore, according to a preferred embodiment of the present invention, the basic system is at least one of a website, a website building system, and a document editing system.

[0052] Furthermore, according to a preferred embodiment of the present invention, at least one tool cast is at least one of basic and composite.

[0053] Furthermore, according to a preferred embodiment of the present invention, this method further includes at least one of the following: storing basic toolcasts and composite toolcasts in a database; maintaining at least one toolcast when changes are made to the underlying system; supporting synchronization of at least one toolcast with other related toolcasts when changes are made to the underlying system; providing a mapping of the structure, associations, events, and interfaces of the underlying system to a toolcast management system for at least one toolcast; and collecting data about the underlying system and users, and accordingly providing machine learning training and support for recording, generating, and editing and playing back at least one toolcast.

[0054] Furthermore, according to a preferred embodiment of the present invention, recording, generating, and editing at least one toolcast includes at least one of the following: generating a storyboard for at least one toolcast; generating text for the storyboard for at least one toolcast; analyzing the elements and site structure of the underlying system; analyzing business intelligence (BI) information collected from the underlying system; creating a composite toolcast that includes a hierarchy of basic toolcasts; enabling at least one toolcast designer to create a text-based toolcast using a rich text editor; enabling at least one toolcast designer to record themselves and convert the recording into a toolcast file; creating toolcast associations in an interfaced configuration between the toolcast management system and the underlying system; determining and optimizing the activity flow of at least one toolcast; providing editing hints and options to at least one toolcast designer; detecting mismatches between the underlying system and at least one toolcast according to the toolcast definition of at least one toolcast; and providing integrated or combined editing of the underlying system together with at least one toolcast.

[0055] Furthermore, according to a preferred embodiment of the present invention, playing at least one toolcast includes automatically enabling user activation of at least one toolcast, enabling at least one toolcast to trigger responsive events on the underlying system by simulating user actions, displaying a toolbar so that the user can control the display of at least one toolcast, creating additional or extended UI (user interface) elements on top of the display by displaying the toolbar, and enabling interaction between at least one toolcast and the user during the playback of at least one toolcast.

[0056] Furthermore, according to a preferred embodiment of the present invention, interfacing with a base system includes at least one of the following: interfacing with an integrated base system; interfacing with an unintegrated base system; and interfacing with a base system database to provide data for analysis by recording, generating, and editing.

[0057] Furthermore, according to a preferred embodiment of the present invention, at least one toolcast is a user-adaptable toolcast.

[0058] The subject matter considered to be the present invention is specifically pointed out and explicitly claimed in the concluding portion of this specification. However, the present invention, with respect to both its configuration and method of operation, as well as its object, features and advantages, can be best understood by referring to the following detailed description with reference to the accompanying drawings. [Brief explanation of the drawing]

[0059] [Figure 1] This is a schematic diagram of a typical representation of a prior art website building system. [Figure 2]This is a schematic diagram of a system for creating and updating hierarchical websites using prior art. [Figure 3] A and B are schematic diagrams of a toolcast management system related to a basic system, built and operating according to the present invention. [Figure 4] This is a schematic diagram of a toolcast management system integrated into a website building system, built and operating according to the present invention. [Figure 5] These are schematic diagrams of the elements of the toolcast management system shown in Figures 3A and 3B, which are constructed and operate according to the present invention. [Figure 6] Figure 5 is a schematic diagram of the elements of the interface module, which is constructed and operates according to the present invention. [Figure 7] Figure 5 is a schematic diagram of the elements of the TC Creator, which is constructed and operates according to the present invention. [Figure 8] Figure 7 is a schematic diagram of the elements of a text-based recorder built and operating according to the present invention. [Figure 9] Figure 7 is a schematic diagram of the elements of the voice-based recorder built and operating according to the present invention. [Figure 10] Figure 7 is a schematic diagram of the elements of the CTC Creator and Editor, which are built and operate according to the present invention. [Figure 11] Figure 7 is a schematic diagram of the elements of the TC path optimizer, which is constructed and operates according to the present invention. [Figure 12] Figure 5 is a schematic diagram of the elements of the maintenance module constructed and operating according to the present invention. [Figure 13] Figure 5 is a schematic diagram of the elements of the TC player constructed and operating according to the present invention. [Figure 14] Figure 13 is a schematic diagram of the elements of the TC activator constructed and operating according to the present invention. [Figure 15] This is an exemplary screenshot of the interface for using Toolcast, a tool for a restaurant menu creation system built and operating according to the present invention. [Figure 16]Figure 15 shows a detailed interface illustrating a TC player control toolbar built and operating according to the present invention. [Figure 17] Figure 15 shows a detailed embodiment of an additional content element that is constructed and operates according to the present invention. [Modes for carrying out the invention]

[0060] For the sake of brevity and clarity in the examples, please understand that the elements shown in the drawings are not necessarily drawn to scale. For example, some dimensions of elements may be exaggerated relative to others for clarity. Furthermore, where deemed appropriate, reference figures may be repeated in the drawings to indicate corresponding or similar elements.

[0061] In the following embodiments for carrying out the invention, many specific details are given in order to provide a complete understanding of the invention. However, it will be understood by those skilled in the art that the invention can be carried out without these specific details. In other embodiments, well-known methods, procedures, and components are not described in detail so as not to obscure the embodiments of the invention.

[0062] Websites and applications typically consist of complex graphical user interfaces (GUIs) involving multiple pages (or application screens) containing multiple visual components. The applicant has observed that users of such applications often require assistance and guidance in using the underlying systems, such as designers who use website building systems to construct such websites and applications.

[0063] A highly beneficial solution to these needs is a personal teacher, instructor, or mentor who sits alongside the designer or user (or at least communicates with the user remotely via a screen-sharing system). One common alternative to a human teacher is a video segment explaining a specific interaction with the underlying system. However, such videos often lack interactivity and quickly become outdated as the underlying system changes.

[0064] The applicant further realized that the same benefits could be obtained through a toolcast management system that does not require a personal (human) instructor.

[0065] A toolcast management system can provide solutions through toolcasts, a media format that uses interaction with the underlying system as its foundation. The toolcast format can be used in combination with multiple technologies for creating, maintaining, and activating toolcasts using supporting content elements such as active tutorials, documentation, courseware, and in-user messaging. Media files and players combine interaction (and potentially integration) with the underlying system, synchronized narration, subtitles, or other supporting content elements. Toolcasts can also be adapted to the user according to user settings such as user parameters, settings, and attributes (such as the user's geographical location or browser type) stored by the underlying system.

[0066] The applicants also recognized that the advantages of such a toolcast management system, which integrates with / communicates with underlying systems such as websites, could include the ability to dynamically execute toolcasts (generating tool events, subtitles, and audio), as well as the ability to directly interact with users and perform actions on the website according to the state and data of the website itself. These may also include dynamically creating toolcasts based on user data.

[0067] The toolcast management system can create extended UI (user interface) elements on top of the underlying system, or merge them with the underlying system components in other ways, and can even suggest or autoplay toolcasts through (for example) the state of the website, interactions with the website, or interactions with the system through writing or conversation (including voice instructions, use of chatbots, guided questions, and user flow confirmation).

[0068] The toolcast management system can also detect what users "need" on a website through behavioral / BI (Business Intelligence) analysis, text chat, and conversations with users using machine learning technology.

[0069] Furthermore, the toolcast management system can create toolcasts by recording user actions on the website, and can automatically detect when descriptions of website functionality become outdated based on updates to the website and toolcast records.

[0070] Other applications of the toolcast management system allow toolcasts to be continuously updated by converting them into other media formats such as text, including images or videos, and can automatically generate toolcasts based on machine learning and text-to-speech technologies.

[0071] Finally, the toolcast management system can also handle responsive events on the website DOM that reflect remote user actions (clicks, drags, hovers, scrolls, edits, and creates elements) across multiple supported browsers.

[0072] It will be understood that the toolcast management system may consist of the toolcast itself (which is the data file that drives the system), a toolcast editor, a toolcast player, a toolcast deployment agent, and numerous additional elements, as will be described in more detail below in this specification. These elements together provide the functionality to execute the toolcast through smart playback. This interfaces with the underlying system and uses the underlying system elements and events to form display comments.

[0073] Furthermore, it should be understood that the architecture of the toolcast management system may allow users to actively participate in the interaction. For example, the playback (and flow) of a toolcast file may depend on specific actions performed by the user. In addition, users can stop the toolcast player at any point and continue using the underlying system from that point onward.

[0074] The toolcast management system can also provide resynchronization of toolcast files when the underlying system changes, in addition to online resynchronization. Resynchronization can be performed both online and offline, for example, when the execution timeline of the underlying system changes due to differences in communication performance, or when unplanned events, such as notification pop-ups occurring in the middle of a pre-planned interaction, affect the UI (user interface) of the underlying system.

[0075] The toolcast management system may also resynchronize when the display of the underlying system changes. For example, this can occur when the underlying system components are displayed differently due to changes in the display layout (dynamic layout, window size, portrait / landscape orientation, component movement due to responsive design considerations, different component variants being displayed, etc.).

[0076] The toolcast management system can further support the creation and editing of such toolcast files, for example, by recording user interactions with the underlying system, along with user-provided narration or titles. As will be described in more detail below, the recording does not consist of screenshots or screen video captures. Instead, the recording is object-oriented, capturing displayed objects, user interactions with them, and associated dynamics (e.g., motion) and timing information.

[0077] As previously described herein, the toolcast management system can be used in combination with the base system. (See Figures 3A and 3B.) Figures 3A and 3B show a toolcast management system 80 communicating with the base system 200 (Figure 3A) and a toolcast management system 80 integrated with the base system (Figure 3B), according to embodiments of the present invention.

[0078] It will be understood that the following discussion focuses on an integrated version of the Toolcast Management System 80 with a system or application that is supported through associated toolcast files manipulated by the Toolcast Management System 80. Such supported systems 80 may include systems such as websites or other web-based applications. In the case of web-based applications, it will be understood that the Toolcast Management System 80 can provide its functionality to designers as a service using a Work Breakdown Structure (WBS), so that designers can provide toolcasts to users of their own websites. These sites include both statically defined and dynamically generated sites.

[0079] Furthermore, it will be understood that the underlying system may be the WBS itself. This case includes, for example, the use of a toolcast management system 80 to provide training or documentation for elements of the WBS environment. In this scenario, the supported users may be website designers using the WBS, or website users using some aspect of the WBS made public to users (e.g., talkback editing functionality on a blog website). Next, refer to Figure 4, which shows the toolcast management system 80 integrated with the WBS 100 in Figure 2.

[0080] Toolcast can be applied to different areas of WBS100 (other than WBS Editor 30), such as management functions, dashboards, and vertical applications (as well as setup and configuration GUIs).

[0081] Furthermore, it will be understood that the underlying WBS elements can be constructed using the same WBS (i.e., a self-hosted WBS) or different WBSs. Thus, the toolcast management system 80 can be used in combination with such WBSs, and potentially at multiple levels (both service interface and provision).

[0082] Furthermore, the underlying system 200 may be a desktop application or a mobile application, regardless of whether it is pre-installed or dynamically installed or running.

[0083] The base system 200 may be any type of document editing system, such as a word processor or a code IDE (Integrated Development Environment). Such a base system 200 can support editing a single “document” (e.g., a word processor document or a web page) or multiple documents simultaneously. A toolcast running on the base system 200 can support a UI used to edit multiple documents (within a single toolcast).

[0084] It should be understood that the web-based sites and applications described herein may include those rendered via client-side rendering, server-side rendering, mixed rendering, or rendering provided by a framework (such as the React web development framework).

[0085] The Toolcast Management System 80 can also be applied to advanced application deployment environments such as wearable environments, immersive environments (virtual reality and augmented reality, etc.), and video-based applications.

[0086] Therefore, the toolcast management system 80 may include different embodiments and implementation details based on the type of the underlying system 200, as will be described in more detail below herein.

[0087] The following description will focus on the implementation of a toolcast management system 80 for websites, whether it is manually developed or WBS-based. As mentioned above in this specification, such sites may be based on a combination of templates at various levels (site, page, section, component set, or component level).

[0088] Next, refer to Figure 5, which shows the elements of the toolcast management system 80. The toolcast management system may comprise a TC (toolcast) creator 81, a database 82, an interface module 83, an UNS (Underground System) structure mapper 84, a TC (toolcast) player 85, a TC (toolcast) synchronizer 86, a maintenance module 87, an ML (machine learning) engine manager 88, as well as an additional product generator 89, an Appstore interface 8100, and a tooltip handler 8101. The ML engine manager 88 may further comprise a data acquisition unit 881 and multiple ML engines 882, and the additional product generator 89 may further comprise an automated execution unit 891. The functions of these elements will be described in detail below in this specification.

[0089] As will be explained in more detail below, it will be understood that the ML Engine Manager 88 can serve various elements of the Toolcast Management System 80.

[0090] The Tc Creator 81 can record, edit, and generate toolcasts. The database 82 can store toolcasts (including basic toolcasts and composite toolcasts, which are described in more detail below). The interface module 83 can provide interfaces depending on the type of the underlying system 200 (integrated / non-integrated). The UNS (Underlying System) Structure Mapper 84 can provide not only a mapping of the structure of the underlying system 200, but also associations, events, and interfaces used to affect the underlying system 200. The TC Player 85 can be used to play toolcasts and the TC Synchronizer 86. The TC Synchronizer 86 can be activated when changes are made to the underlying system 200 to support synchronization with the associated toolcasts. The maintenance module 87 can maintain toolcasts when changes are made to the underlying system 200. The ML Engine Manager 88 can provide machine learning training and support. The Additional Product Generator 89 can run toolcasts to generate alternative products. The app store interface 8100 can be used to allow created toolcasts to be shared via the app store. The tooltip handler 8101 can add toolcasts to existing tooltips. This may include tooltips created as toolcast activation UIs. The functionality of these elements is described in more detail below.

[0091] A typical usage flow may include the following steps: The designer or developer of the base system 200 can record and edit the toolcast of the base system 200 using the toolcast management system 80 (for example, through a hosted service that can provide toolcast services to various systems), as will be described in more detail below in this specification. This involves the developer or designer interacting with the base system 200 and the toolcast management system 80, which may be another toolcast developer who is not the base system developer.

[0092] Next, the designer of the base system 200 can store toolcasts locally with the base system 200 or within a hosted toolcast management system 80. Subsequently, the base system 200 can activate specific toolcasts on its own or based on user actions. Then, the TC player 85 can execute specific toolcasts and drive the base system 200 while interacting with the end user of the base system 200.

[0093] The toolcast management system 80 can interface bidirectionally with the underlying system 200. The types of interfaces and their possible implementations are described below in this specification.

[0094] It will be understood that the toolcast management system 80 can be used in different application areas and scenarios. The following descriptions of these areas are based on the context of a website designer using the toolcast management system 80 to benefit the users (visitors) of their website, but other use cases are also possible, such as WBS vendor staff 61 providing toolcasts to designers using WBS, or site owners 62 (whether WBS-based or not) providing toolcasts to third-party toolcast creators who provide toolcasts to visitors of their site and users of various systems (websites, WBS, or others) (described in more detail below).

[0095] One type of use for the Toolcast Management System 80 could be the development of interactive tutorials. It will be understood that the Toolcast Management System 80 can be used to create exploreable and interactive tutorials. Site designers may record site usage sessions (e.g., specific parts of the site). Site visitors can replay, pause, and interact with these recorded sessions (e.g., pause a tutorial and resume actual system usage from a given point in time where it was paused).

[0096] Such tutorials may be presented as multi-step instructions or wizards (e.g., pop-ups or on-screen instructions). Tutorials can also be presented in conjunction with displayed screen components, for example, by displaying supporting content elements near or alongside related elements.

[0097] Such tutorials can also function as courseware. These tutorials may include quizzes, logic based on user responses, scoring, and learning management features.

[0098] The toolcast management system 80 can also provide a "Do It With Me" (DIWM) tutorial. Under a DIWM scenario, the toolcast management system 80 may, as part of the tutorial section, suggest to the user that it perform an operation on the underlying system 200 on their behalf (for example, "Can you perform task X now, or let me do it for you?"). A tutorial session may include a mix of tutorial material delivery, tasks performed by the user, and DIWM tasks performed by the toolcast management system 80.

[0099] Another use of the Toolcast Management System 80 is the creation of interactive documentation. The Toolcast Management System 80 can be used to enhance a website by associating toolcast files with specific features or elements of the website. Such toolcast files can replace traditional help functionality, which is typically provided by static help files.

[0100] Another use of the toolcast management system 80 is the creation of editable video-like media files that describe the functions or procedures of the underlying system 200. It will be understood that such files may be defined in terms of UI elements of the underlying system 80, rather than screen captures.

[0101] The toolcast management system 80 can detect if video files are out of sync (for example, referencing site elements that have been deleted or otherwise modified) and can resolve this or, as will be described in more detail below, notify the toolcast creator. It will be understood that such videos can be used for many purposes, such as user support or marketing.

[0102] Another use of the toolcast management system 80 could be messaging related to the underlying system 200. The toolcast management system 80 can be used as a method of communication between people who need to communicate about the underlying system 200, such as developers, users, or people involved in system support, documentation, marketing, testing, etc.

[0103] These individuals often need to communicate and exchange information related to the operation sequence of the basic system 200, and may need to deliver additional information beyond the provided screenshots or screen video recordings.

[0104] The toolcast management system 80 may enable these users to quickly record and create toolcasts that include dynamic flows, which are normally difficult to communicate. Furthermore, communicated toolcasts that work in conjunction with the underlying system 200 may enable recipients to access the internal state of the underlying system 200 associated with the described flows of the underlying system 200. Such messaging may be synchronous (e.g., webinar-type streaming) or asynchronous (e.g., email / chat-type messaging, which may be delayed).

[0105] It will be understood that the above explanation refers to communication between people. However, such "people" may also include computerized nodes such as bots or other non-human communication actors.

[0106] It should also be understood that such messaging and communication can be multidirectional, such as from website designers to end users, from end users to end users, from designers to designers, from end users to designers, from end users / designers to WBS staff, and from WBS staff to end users / designers.

[0107] The toolcast management system 80 may also be used in conjunction with a visual editor (e.g., within a WBS such as the WBS editor 30) or another IDE. It should be understood that users of the visual editor or IDE can play back recorded toolcasts within the editor, essentially turning the IDE into an (interactive) media player. Because toolcasts actually interact with the editor of the underlying system 80 (or design environment), the actions of the toolcasts are "live" (not simply displaying a fixed recording).

[0108] Therefore, the user can stop the toolcast at any point and interact with the underlying system 200 (or its code) in the new state affected by the toolcast. At this stage, the user can use their tools and environment and learn from the work (and changes) done by the toolcast.

[0109] The toolcast management system 80 may enable toolcasts to continue, including, in some cases, adapting to changes made by the user, as will be further described below in this specification.

[0110] The toolcast management system 80 can also be used in conjunction with a structured communication system, such as customer support, as described in U.S. Patent Application No. 17 / 673,933, entitled "Structured Communication System Integrating Dynamic Display," filed on 17 February 2022, assigned to the assignee of the present invention, and incorporated herein by reference. Such a structured communication system can use toolcasts as a medium for interaction and communication between users and agents using the structured communication system.

[0111] The toolcast management system 80 can also provide additional uses of toolcasts other than those used with the TC player 85, as will be described in more detail below in this specification.

[0112] As described above in this specification, the toolcast data format is a media format that allows the toolcast management system 80 to record and replay the process of interacting with a website or application. This may be specific to a given website, or it may be generalized to a group of related (or all or partly similar) websites.

[0113] The Toolcast data format can be used as a file format or stored using a larger container or representation format (for example, within a video file container file using XML, JSON, or similar representations). The Toolcast data format can also be used as a data format stored within database fields or records, repositories, cloud storage (for example, within a cloud storage bucket), and object-oriented databases (OODBs).

[0114] Toolcast data formats can be hierarchical, with basic toolcasts (BTCs) implementing linear timelines corresponding to linear sequences of interactions, and higher-level composite toolcasts (CTCs) combining multiple BTCs to define the flow and routing between them.

[0115] The toolcast data file may include a set of support content elements (ACEs), a set of actions to be executed on the underlying system 200, a set of trigger definitions, flow management elements, and may also include additional displayable materials, additional types of media such as those displayed in a VR (virtual reality) / AR (augmented reality) environment, and interactive components or applications.

[0116] Supportive content elements (ACEs), such as audio and subtitles, may be displayed during playback, for example, during a toolcast-based training session. These ACEs may also contain elements that will be added to the website (such as annotations and highlightings), or values ​​used to influence the properties of those components.

[0117] A series of actions performed on the base system 200 (such as moving the cursor, clicking on elements, drawing near elements, and selecting UI elements on the base system 200) can affect the base system 200 as if the user were interacting with the website. These actions may include information added to or affecting the website, as will be further explained below.

[0118] A set of trigger definitions can be associated with website elements, operations, activities, and events (which may include both front-end events (visible or invisible) and back-end events). The toolcast management system 80 can listen to the underlying system 200 to detect events, states, and associated data, and can trigger specific actions by the TC player 85 (including information related to the underlying system 200 and the user as generally described herein).

[0119] Flow management elements can determine how actions are performed, and ACEs are displayed based on the timeline or detected website events. Flow management elements may use linear timelines (in the case of BTC), state charts, activity graphs (including multilevel graphs), or other mechanisms. The toolcast management system 80 may support branching and input processing in conjunction with some or all of the above flow management elements (e.g., timelines, state charts (e.g., those performed by state machines), and activity graphs).

[0120] Toolcast files can store some or all of event definitions using component-relative coordinates (so that the definitions are not affected even if the components are moved). Such component-relative definitions may apply to a specific component or parent container and may be based on anchors to the corners, edges, center, or other parts of the component. The anchors may be absolute or relative (for example, based on the size of the component). The toolcast management system 80 can further interact with layout systems used by the underlying system 200 (e.g., dynamic layout or responsive display systems) to adapt to non-trivial changes in the display created by such systems. Such changes may include, for example, changing the position, size, or placement of components according to the definition of various breakpoints in a responsive website. Such systems are described in U.S. Patent No. 10,185,703, entitled “Web Site Design System Integrating Dynamic Layout and Dynamic Content,” granted on 22 January 2019, and U.S. Patent No. 10,984,177, entitled “System and Method Providing Responsive Editing and Viewing, Integrating Hierarchical Fluid Components and Dynamic Layout,” granted on 20 April 2021, both of which have been assigned to the assignee of the present invention and are incorporated herein by reference.

[0121] As described above in this specification, in one possible embodiment, the toolcast management system 80 can use timelines (created, for example, by recording specific user interactions with website areas) to define BTCs. These BTCs form the basic elements of a CTC, including multiple interactions and branching structures that cover a broader area of ​​a given website.

[0122] Such larger CTCs can encompass interactions with an entire website or application, or cover a subset thereof. The toolcast management system 80 may enable the definition of a hierarchy of CTCs (providing a multi-level hierarchy).

[0123] It will be understood that toolcast data files can use many data formats, such as container formats, modified versions of existing media formats (such as video formats), or specialized data formats. Toolcasts may be physically attached to the relevant UI object definitions within the underlying system 200 (for example, as data blobs), or they may be stored separately.

[0124] Regarding ACEs, they can include many types of display media in conjunction with the underlying system 200. ACEs may include static media, such as text, subtitles, or associated images, or they may be inherently dynamic / temporal, such as audio used for narration or other purposes. Dynamic ACEs may also include different media types, such as video, but the primary interactive visual displays will typically be generated by the TC Player 85, which interacts with the website.

[0125] Additional displayable materials may include arrows or other screen highlights that point to or otherwise emphasize displayed components. Such elements may be added to the display as an additional layer, as further described herein, or integrated with the display of the underlying system 200.

[0126] Interactive components or applications may be part of the toolcast management system 80 (client and server), and may be operated and activated by the toolcast management system 80. They may also be part of the underlying system 200 (e.g., underlying WBS components) while continuing to be directed and started (or otherwise operated, exposed, or started) by the toolcast management system 80. Like other displayable ACEs described herein, they may operate as separate layers, either in combination with or separately displayed from the display components of the underlying system 200. As described above in this specification, the toolcast management system 80 can interface bidirectionally with the base system 200.

[0127] The toolcast management system 80 can receive information about the activities of the underlying system 200 via the interface module 83 using direct linking, detection, analysis, etc. The interface module 83 can detect the activities of the underlying system 200 that should be recorded in the toolcast recording phase, and can detect the activities of the underlying system 200 to activate toolcasts associated with specific objects, data, activities, and events of the underlying system 200.

[0128] The interface module 83 can also detect objects, data, activities, and events of the underlying system 200 that may affect the execution of a toolcast during its playback, as well as detect changes to the underlying system 200 that affect how the toolcast and the toolcast management system 80 play the toolcast (for example, for toolcast maintenance).

[0129] Conversely, when the toolcast management system 80 transmits information to the underlying system 200, the interface module 83 can simulate user actions within the underlying system 200 (e.g., keyboard strokes, mouse movements, and clicks) and provide direct manipulation of the underlying system 200 data structures (e.g., modifying the displayed browser DOM of a website).

[0130] The interface module 83 may also influence the operation of the underlying system 200, modify the display of the underlying system 200 (e.g., through annotations), and generally provide input to the underlying system 200. Refer here to Figure 6, which illustrates the elements of the interface module 83. The interface module 83 may further comprise an integrated interface 831, a non-integrated interface 832, and a UNS (Underlying System) database interface 833.

[0131] It will be understood that the interface does not interact directly with the underlying system 200, but rather with general system elements that work with the underlying system 200. For example, the toolcast management system 80 can interface (via interface module 83) with both a browser running the underlying website and a development framework / WBS runtime module running the website, but it does not interface with the underlying website itself. Interface module 83 can interface with multiple underlying systems 200, and therefore its sub-elements can support multiple targets using the integrated interface 831, as will be described in more detail below herein.

[0132] Furthermore, it will be understood that in some embodiments, the base system 200 may include (or otherwise adapted or modified) infrastructure and elements for interfacing with the toolcast management system 80. These base systems are known as integrated base systems.

[0133] In other embodiments, the base system 200 may not include the necessary infrastructure or elements for interfacing with the toolcast management system 80. In such cases, the toolcast management system 80 may need to interface with the base system 200 "from the outside" using, for example, external analysis, detection, or inserted hooks, as will be described in more detail below herein in relation to the non-integrated interface 832. These are known as non-integrated or "external" base systems 200.

[0134] In some embodiments, these two may be combined, for example, the basic system 200 may provide some interfaces but not others, or the basic system 200 may provide interfaces but not all of the necessary parameters, information, or options. The toolcast management system 80 may also include multiple interface methods, some of which are integrated and some which are not.

[0135] It will be understood that interfaces and integrations may be performed (whole or in part) at a software level separate from the actual underlying system 200. For example, the underlying WBS may provide the necessary interfaces for the toolcast management system 80 to process a website implemented on the WBS. This requires no changes or special code in the website itself (which is the actual underlying system 200). Other examples may include integrations performed at additional integration points other than the underlying website, such as with the operating system, drivers, browsers, or JS engine. This integration may also be performed with other development frameworks used for the development and deployment of the underlying system 200.

[0136] A partial alternative to integrating a WBS-based website with the underlying System 200 would involve actually integrating with the WBS used for the underlying website, but adding extra IDs, hints, metadata, or other relevant information to the website itself.

[0137] Therefore, by using such a method, users of the Toolcast Management System 80 can benefit from the integrated solution without requiring specific or substantial changes to the underlying website.

[0138] The toolcast management system 80 can also be implemented separately from the underlying system 200 (for example, in a different environment or on a different physical or virtual platform). For example, on a mobile device, the toolcast management system 80 may be implemented as a mobile application and the underlying system 200 as a website (or vice versa). The toolcast management system 80 can continue to interface with the underlying system 200 (in an integrated or non-integrated manner) via bridging mechanisms connecting the two implementation environments (e.g., local communication, local web services, an intermediate server hosted on a mobile device, a browser extension implementation, a local interface, etc.).

[0139] It will be understood that the toolcast management system 80 can interface with the integrated underlying system 200 in multiple ways. The integrated interface 831 can interface with the WBS used to build the underlying website (including specific web services or client-side integration points). The integrated interface 831 can use publicly available WBS server interfaces (APIs, web services, etc.). It may also use privately available interfaces. The integrated interface 831 may include a direct interface to the WBS server's internal document management system (which handles the storage and retrieval of web page definitions) or a WBS editor interface (which handles the editing of pages and components).

[0140] The integration interface 831 can also interface with the integration underlying system through interfaces added to the source code of the underlying website, web server, or application, and through additional parameters and hooks to the definition of tests (such as A / B or multivariate tests) used when deploying versions of the underlying system 200. Such tests are described in U.S. Patent No. 10,733,078, entitled “System and Method of Handling Complex Experiments in a Distributed System,” granted on August 5, 2020, and assigned to the assignee of this invention, and are incorporated herein by reference (when used in connection with the underlying WBS). In this scenario, it will be understood that there may be test-specific integrations and associations (e.g., associating one toolcast with test option A and another toolcast with test option A).

[0141] It will be understood that such integrations may include external website elements. That is, as described above herein, a website may include external elements such as external components, third-party applications, vertical products, subsystems, add-ons, or other elements. Such elements may include sub-elements (including front-end and back-end sub-elements) that may be hosted on WBS vendor servers, separate third-party servers, or cloud providers.

[0142] The integration interface 831 can also implement integration with external site elements, including the ability to handle toolcast triggers generated by such third-party elements (including the use of internal data, frontend, backend, or other sub-elements). External elements may also provide their own toolcasts (or sub-elements of toolcasts) associated with triggers. Such associations may be associated with triggers from the main site code or triggers from external site elements. Toolcasts on the main website may also include or reference such third-party toolcasts to create a complete toolcast flow.

[0143] The UNS database interface 833 may interface with the database of the underlying system 200, as will be described in more detail below herein, to provide data that will be used for analysis.

[0144] Here, we refer to Figure 7, which shows the elements of the TC Creator 81. The TC Creator 81 may further comprise a Toolcast Storyboard Creator 811, a Generative Text Engine 812, a UNS (Underlying Systems) Analyzer 813, a BI (Business Intelligence) Analyzer 814, a Text-Based Recorder 815, a Voice-Based Recorder 816, an Association Manager 817, a CTC Creator and Editor 818, a TC Path Optimizer 819, a UNS (Underlying Systems) / TC (Toolcast) Editor 8120, a Coverage Analyzer 8121, and an Edit Time Recommender 8122. The UNS / TC Editor 8120 may further comprise a Graph Analyzer 81201. The functions of these elements are described in more detail below in this specification.

[0145] The association manager 817 can create toolcast associations for interface configuration between the toolcast management system 80 and the underlying system 200 in both integrated and non-integrated setups, as shown in Figures 3A and 3B. It will be understood that this association can be explicit in the sense that the underlying system 200 is modified to directly activate the TC player 85 (e.g., through function calls, web services, additional specific components that activate toolcasts). Such activation code can be added to the website itself or be part of the WBS used to build the site (and can also be configured to make specific toolcast activation calls). Such code is usually necessary because the association (and underlying triggers) relate to internal site information (such as specific events in the underlying site data repository), and such events may be difficult to detect from the outside.

[0146] The association manager 817 can use the toolcast ID (or other naming or association mechanism) to determine which toolcast to activate. The underlying system 200 can pass these IDs as part of a trigger event, and the TC player 85 can recognize these IDs and activate the appropriate toolcast.

[0147] The toolcast management system 80 may also be used with “external” (non-integrated) underlying systems 200, such as hand-coded sites. This is relevant when the toolcast management system 80 cannot directly modify or influence the site and its generation process. For example, when creating toolcasts to sites where the user does not have editing privileges, or to applications that are not typical websites. Such applications (for web, mobile, or desktop environments) may be provided, for example, as binary installation kits from application stores.

[0148] It will be understood that the non-integrated interface 832 may use a different ("external") WBS that is not integrated with the toolcast management system 80, or it may use a different system that generates or edits code (e.g., HTML, CSS, and JavaScript code) that can be directly executed by a browser. This could be a WBS that outputs such code, or an editor (such as Adobe Dreamweaver CC® or Notepad++ with HTML support) that allows the user to directly edit such code.

[0149] The non-integrated interface 832 can also be interfaced using frameworks (such as Facebook, Inc.'s React) that allow users to create websites using models different from the standard HTML / CSS / JS model. Such frameworks may offer their own component architecture or language (such as React's JSX extension to JavaScript).

[0150] Another implementation is a site that uses pre-installed or downloaded code. This could be, for example, an integrated applet (such as a Java-based applet) that implements parts of the site, an interactive application for the entire site (including a mobile app), or a virtual machine-based component. Such code may also include code that uses native code, JIT-compiled code, interpreted code, or other execution methods.

[0151] It should be understood that the site creation method may affect how the toolcast management system 80 interfaces with the site and how toolcast associations can be defined.

[0152] The non-integrated interface 832 can access information from such external sites and (if applicable) parts of the underlying object architecture through the following means: The non-integrated interface 832 can access information by reading site code (HTML / CSS / JS or otherwise) directly from the host server; by reading auxiliary files (such as robots.txt files read by search engine spiders); by accessing the site via APIs or web services that provide access to stored pages; or by accessing the external (non-integrated) WBS through APIs or web services that it provides.

[0153] The non-integrated interface 832 can also access the traffic used to carry the evaluated website. This access can be achieved by passively listening to (and analyzing / decoding) network communications, or by agents embedded in servers, clients, intermediary machines (including those carrying CDNs (Content Delivery Networks)), or additional “intermediate agents” that analyze the relevant communications.

[0154] The non-integrated interface 832 may also use a specific browser (whether headless or not) that can "execute" the target website and intercept its (internal) object model or rendered code, implement elements of the associated toolcast management system 80 as a browser extension or a plugin to an existing browser, or analyze and modify the site's code at load time.

[0155] Another method the non-integrated interface 832 may use to access information may be by injecting new or additional code into the site at load time. The injected code could be, for example, a client-side agent (embedded with the site code) that interacts with the object model or rendering process. Alternatively, such an agent could interact with other system elements (such as the browser or the underlying operating system).

[0156] The non-integrated interface 832 can also have added code or hardware that can track visual displays generated by the site. An example would be an additional software agent on a mobile device that can read and analyze (at the operating system level) displays generated by a given mobile application that does not have "hooks" for a normal interface.

[0157] The non-integrated interface 832 can also operate based on the analysis of HTML, the browser object model, or both. Meanwhile, the UNS analyzer 813 can perform a preliminary analysis of the collected information to extract a higher-level model of the site elements and structure. Such a model may include aspects of structure (i.e., what the site is made of), layout-related, semantic, and behavioral aspects. In particular, the extracted model may include not only static information (e.g., the site structure at a given point in time) but also dynamic information (the site's dynamic behavior in response to various responses).

[0158] UNS Analyzer 813 can perform preliminary analysis using tools such as database rules, specific analysis scripts, and ML (machine learning) models, such as those described in U.S. Patent No. 11,275,815, entitled “System and Method for Integrating User Feedback into Website Building System Services,” granted March 15, 2022, co-owned by the applicant, and incorporated herein by reference. Furthermore, for sites created using a non-integrated WBS, specific rules, scripts, and ML models may be adapted to a specific WBS. Therefore, if a site is recognized as being built by a specific external WBS (e.g., WordPress), UNS Analyzer 813 may adapt its preliminary analysis to take into account specific elements of WordPress, such as its specific plugin architecture.

[0159] The UNS Analyzer 813 can further analyze site structure, flow, code (client and server), BI information, edit history, and other repositories. Analysis may include geometric analysis, semantic analysis, and usage analysis. The analysis can integrate information accessible via the Toolcast Management System 80 with the underlying system 200 (or the underlying system's WBS or development framework), as described herein. This may include various forms of site analysis, understanding, and learning, as described in U.S. Patent No. 11,275,815.

[0160] Based on this preliminary analysis, the non-integrated interface 832 can reconstruct the actual component structure of the underlying external WBS or framework (if such a thing was used). For example, the preliminary analysis could model and detect the use of complex galleries in an external WBS generation site (e.g., by recognizing specific generated HTML / CSS / JS) and reconstruct such galleries from their rendered subcomponents.

[0161] The analysis methods described above may also be used by TC Player 85 (where applicable) to guide the elements of TC Player that affect or modify the underlying website (as described in more detail below).

[0162] As mentioned above, for sites analyzed by UNS Analyzer 813 based on tracking the generated visual display, it should be understood that the analysis may be performed at the screen image level rather than at the HTML or browser object model level (for example, recreating the component structure "behind" the displayed screen even if this structure is not accessible, such as in mobile applications).

[0163] When using ML models, the ML engine manager 88 can retrain the models of the ML engine 882 on previously evaluated sites to adapt them to changes in the underlying external WBS (e.g.). The data acquisition unit 881 can collect data and user information from the underlying system 200, as will be described in more detail below herein.

[0164] External detection of internal events occurring within the underlying system 200 (e.g., to activate the TC player 85) by the toolcast management system 80 may, in some cases, be extended to the detection of server-side or other site-internal events. This detection may depend on the server provider (e.g., a cloud computing vendor) or site owner providing relevant access credentials to users of the toolcast management system 80, or, in some cases, allowing the installation of relevant server-side monitoring software (or access to such installed server monitoring software). Such access may be performed as a separate parallel channel to the primary communication between the client and server of the underlying system 200.

[0165] It will be understood that the Toolcast Management System 80 can be configured as a service (similar to other "X as a Service" platforms). In such a configuration, the Toolcast Management System 80 provider can create integrations with various other service providers (e.g., WBS provider, ISP, TPA provider, etc.). Such integrations can provide secure access through separate parallel channels, in parallel with the underlying System 200-related systems and services. This can be done without directly providing access credentials to the actual users of the Toolcast Management System 80 (i.e., those who create and edit toolcasts).

[0166] In such a "toolcast management system as a service" configuration, the actual functionality of the toolcast management system 80 can be implemented on top of the actual toolcast management system 80 server, other servers, the user's client machine, or a combination thereof.

[0167] As described above in this specification, all of the above interface methods can be used for multiple required interfaces. These required interfaces may include recording toolcasts, associating toolcasts with site elements, events, and triggers (including defining triggers), activating and replaying toolcasts (including interactions during toolcasts), pre- and post-processing of various toolcasts (such as optimization), and detecting underlying system changes that require modification or synchronization of toolcasts.

[0168] It will become clear that there are many ways to create and edit toolcasts. The following explanation describes possible steps for creating linear (timeline-based) BTCs or BTC sections, and for creating more complex CTCs that cover larger underlying systems (such as an entire website) or parts thereof. For such complex toolcasts, additional synthesis methods, such as statecharts, may be used to create the intricate toolcast. This explanation covers multiple creation methods, including creating template toolcasts or complete toolcasts, text-based recordings, and audio-based recordings.

[0169] Furthermore, it will be understood that a single BTC or CTC can be distributed across multiple locations (such as files, libraries, databases, repositories, servers, or the cloud). One example is a top-level CTC that includes (separately defined) reusable sub-CTCs that describe common components and templates included in the underlying system 200.

[0170] The following description generally focuses on embodiments having separate toolcast creation / editing phases and toolcast playback phases. However, in some embodiments, these phases may be mixed. For example, the TC Creator 81 may automatically generate new or modified toolcasts during toolcast playback and combine them within the playback flow (or modify the currently playing toolcast in another way). As another example, the TC Creator 81 may allow some users to enter editing during playback and modify the currently running toolcast or other toolcasts.

[0171] The Toolcast Storyboard Creator 811 can provide the automatic creation of toolcast templates. It should be understood that these toolcast templates, as used herein, are referred to as "storyboards" rather than "templates" to distinguish them from WBS templates. The Toolcast Storyboard Creator 811 can provide the creation of automatically generated (and semi-automatically generated) storyboards for toolcasts. Alternatively, the Toolcast Storyboard Creator 811 can also be used to create a complete (filled) toolcast based on the techniques and structures described herein.

[0172] One way the toolcast storyboard creator 811 creates toolcasts and storyboards is based on a "rehearsal" performed by the user. The toolcast storyboard creator 811 can enable the user to perform the "rehearsal" through the relevant parts of the underlying system 200. The toolcast storyboard creator 811 can then record the user activity and provide the user with a template storyboard that includes the relevant structure and its association with underlying system events.

[0173] Instead of using explicit "rehearsals," the interface module 83 can also track the actual operation of multiple periodic runs of the underlying system 200. This can be done by the underlying system 200 (or its execution infrastructure) within the toolcast management system 80 or through data collected as part of its operation. Based on this collected information, the TC path optimizer 819 can determine which routes use the underlying system 200, which elements are active, which interactions are permitted (and not permitted), etc. This is similar to navigation software (such as WAZE) that collects and analyzes traffic data to determine the actual underlying map and existing roads.

[0174] The Toolcast Storyboard Creator 811 can also provide the user with placeholder areas in the text to be filled in in order to create appropriate descriptive comments. Definitions of such areas may include information such as what should be written in them (e.g., a generated description of an element, flow, or function of the underlying system 200 described in this area), the appropriate amount of text or narration time for this area, and hints provided based on hints within the underlying system 200 or templates contained therein.

[0175] The placeholder area may further contain examples of text sections generated using a generative text engine (described in more detail below), parameters, models, and instantiated data used by machine learning (ML) models to populate specific placeholders (and common toolcasts).

[0176] In another embodiment, the toolcast storyboard creator 811 may operate based on site templates used by the underlying WBS. These may include page section templates, component templates, and so on. Such toolcast storyboards may be pre-generated and possibly optimized based on feedback from editing by the toolcast creator.

[0177] A generative text engine 812 (using technologies such as GPT3 or other ML-based natural language generation engines) can generate text for storyboards (either in whole or in sections within a rehearsal-based storyboard). A generative text engine 812 may be trained and run on material from different sources or based on an analysis of available information. Such sources and analyses may include information and explanations provided by users (via text chat, voice instructions, answers to generated questions, etc.), underlying templates used to generate the site, previous toolcasts created by the same users, toolcasts created for other parts of the website, and toolcasts created for the underlying templates or sites using them (subject to the anonymity, privacy, and intellectual property rights of the toolcast owners, where applicable).

[0178] The generative text engine 812 can also use the results of the UNS analyzer 813, which can analyze elements of the underlying system 200, or it can use the ML engine 882 to analyze information collected by the data collection unit 881 to determine how actual users use the site. Such analysis may include internal information (e.g., website DOM) and external information (e.g., screen images of displayed website screens) of the underlying system 200.

[0179] The generative text engine 812 can further use the results of the BI analyzer 814, which can analyze business intelligence (BI) information collected by the underlying WBS (or information collected by the underlying website hosting system). This may also include specific BI collected by the data collection unit 881 (for example, by listening to the underlying system 200), which requires (and collects) different levels of detail and may in some cases collect all mouse, keyboard, and other UI interactions.

[0180] The UNS Analyzer 813 can also analyze external information related to the base system 200, which may include articles, descriptions, specifications, and other technical information related to the base system 200.

[0181] The generative text engine 812 can also analyze user-provided instructions on which areas of the site to create toolcasts for and in what order, as well as additional feedback and answers provided by the user in response to questions provided by the generative text engine 812. Interaction with the user can take multiple forms, including the explicit use of surveys and forms, the use of text chat, the use of voice chat, and sensor feedback (e.g., gesture recognition).

[0182] The generative text engine 812 can also use user recording sessions from the TC creator 81 without additional information (the “rehearsal” described herein). Another source may be BI collected by the data acquisition unit 881 (working in conjunction with the interface module 83) describing end-user usage, events, and data paths through the underlying system 200. Such analysis by the UNS analyzer 813 may include, in particular, the detection of areas of the underlying system 200 where end-users are having difficulty using the underlying system 200 (e.g., based on an analysis of user behavior and the paths taken). These identified areas may serve as focuses to which toolcasts can be associated.

[0183] The generative text engine 812 can further use underlying user data stored in the base system 200, as well as information from support calls and support tickets associated with the base system 200. Such information may include information recorded or extracted from actual support sessions of the base system 200 (for example, using support staff feedback as "voice of the trainer").

[0184] It will be understood that the Toolcast Storyboard Creator 811 can also train an ML model that controls the generation of storyboard elements and action associations, as will be described in more detail below. Such feedback may include both changes made to the Toolcast proposed (by the designer) and feedback from site visitors (Toolcast users) regarding their use of the Toolcast. Feedback may also include feedback from additional (another) reviewers. The integration of feedback can create a feedback loop that further improves the quality of the generated Toolcast.

[0185] The Toolcast Storyboard Creator 811 can take additional steps to understand what the user "needs" to do in the underlying system 200 and the best way to convey the relevant knowledge to achieve these needs. Specifically, the Toolcast Storyboard Creator 811 can use an ML model via the ML Engine Manager 88 to break down the appropriate operation of the underlying system 200 into steps, guide the user through these steps, and, if possible, allow the user to perform the work themselves.

[0186] Therefore, the toolcast storyboard creator 811 may generate, modify, or manipulate toolcast segments as needed (including during playback using the user interaction module, as will be described in more detail below). The toolcast storyboard creator 811 may also dynamically reconnect, rearrange, or reconfigure existing BTCs based on user interaction with the toolcast. For example, if a user is quickly browsing a tutorial toolcast, the toolcast management system 80 may play the BTCs more quickly or use more concise or advanced BTCs instead of regular BTCs.

[0187] It will be understood that the functions of the toolcast storyboard creator 811 described herein are intended for the creation and editing of text-based toolcasts. However, in alternative embodiments, the toolcast storyboard creator 811 can provide equivalent functionality for voice-based editing (described in more detail below). In such embodiments, voice placeholders can be filled with voice recordings, and the generated text may be read by text-to-speech and modified so that some sections are replaced with recorded voice segments.

[0188] As discussed above in this specification, the toolcast storyboard creator 811 can also generate a complete toolcast that includes the flow definition (linear / BTC or graph / CTC) and the required media (text, audio, etc.).

[0189] Recording and editing BTCs will typically be linked to the UI and flow of the underlying system 200, as BTCs reflect sequences of UI actions and activities. The toolcast management system 80 can create and edit BTCs using known methods (such as defining BTCs via a standalone IDE). Alternatively, better results can be achieved through the underlying system 200 with an integrated BTC generation environment. Two such environments (text-based and voice-based) are described below in relation to the text-based recorder 815 and the voice-based recorder 816.

[0190] Furthermore, it will be understood that the recording / editing environment provides real-time feedback. Since both the text-based recorder 815 and the voice-based recorder 816 can run on the underlying system 200, the user can interact with the results immediately. For example, operations at the underlying site generate accurate site responses (e.g., responses that the user would receive under normal usage conditions).

[0191] Next, refer to Figure 8, which shows the sub-elements of the text-based recorder 815. The text-based recorder 815 may further comprise a rich content editor 8151, a text analysis engine 8152, a TC (Tool Cast) finalization module 8153, and a text decoder 8154.

[0192] Next, refer to Figure 9, which shows the elements of the audio-based recorder 816. The audio-based recorder 816 may further comprise an audio recorder 8161, an audio editor 8162, an audio postprocessor 8163, and an audio decoder 8164.

[0193] The text decoder 8154 and the speech decoder 8164 can decode text or speech entered by the user and, based on their interaction with the underlying system 200 and knowledge of the current user context, point to relevant elements or actions. The TC creator 81 can also provide toolcast-specific autocompletion. Such autocompletion can be at a higher level than that provided by typical IDEs and may include, for example, autocompletion of activities or WBS operations.

[0194] The text-based recorder 815 allows for the creation of toolcasts using a dedicated rich text editor as the primary editing environment. With its ability to create and edit interactive, video-like tutorials using text editing, TC Creator 81 becomes a virtually faster and easier solution compared to existing tools for creating training, documentation, and support materials.

[0195] It will be understood that the steps involved in creating a toolcast by a user (who may be a site designer / instructor / creator / audio specialist) may include creating text that will be used as the underlying training material using the Rich Content Editor 8151. This text may be delivered as narration, subtitles, or in other ways. This text may form the "storyboard" of the created toolcast.

[0196] The rich content editor 8151 can be used to associate system events with words in text, as well as to mark text attributes (e.g., bold or italics). These associated events may include events that drive the underlying system 200, such as mouse movements and clicks, or text input, or events that affect the screen display of the underlying system 200, including the highlighting of components or the display of additional elements (e.g., arrows).

[0197] The Rich Content Editor 8151 can also add text markers that control audio generated based on text. These can use markers defined in the SSML (Speech Synthesis Markup Language) standard, for example.

[0198] It will be understood that the UI of the Rich Content Editor 8151 can provide a graphical representation of these associations. For example, similar to how display attributes or links are represented in rich text, the UI may display a small image of the associated target location (e.g., the associated UI element) near the text. The background color or frame may represent the connection type. When the user hovers over or selects text linked to an association definition, the UI opens a popover allowing the user to edit the target location or association configuration. Further editing of the text may save additional information. For example, moving the text (e.g., via copy / paste) will also transfer any added augmentation elements.

[0199] It should also be understood that association definitions may include additional configuration information, attributes, and parameters.

[0200] The text-based recorder 815 may provide tools and environments to support the event association process. For example, the text-based recorder 815 may provide tools to identify the displayed website components that are the target of association.

[0201] For example, when creating a toolcast for the underlying system of a text editor, the user can highlight sections of text in the underlying text editor. In this scenario, the rich content editor 8151 can highlight relevant UI options within the underlying system 200, display a popup menu related to the underlying system 200 and highlight its options, and provide a list of possible and relevant UI actions for the underlying system 200.

[0202] The toolcast management system 80 can implement such tools (for example) using browser extensions or plugins, or as a function of the WBS used to implement or host the underlying website.

[0203] The text-based recorder 815 can provide an additional level of automation to this process. The text analysis engine 8152 can analyze the text of the provided storyboard and automatically suggest associations of relevant elements and actions (i.e., detect what we are talking about). For example, the storyboard might specify "[Press the Continue button]", and the analysis can create an association between this text segment and the action of clicking the relevant "[Continue] button" located in the appropriate place in the underlying system 200. Such associations may be performed entirely automatically or may require user approval.

[0204] It should be understood that the text analysis engine 8152 can analyze within the context of the current website display. Therefore, it can provide more specific and focused results than general-purpose text analysis (for example, only the currently displayed website elements may be relevant).

[0205] The text analysis engine 8152 can also benefit from separate analyses of the structure and flow of the underlying website (WBS-based or non-WBS-based), which may include front-end, back-end, and procedural elements. The text analysis engine 8152 can correlate the results of its analysis with the results of text-based analysis. For example, if a website requires that button A be pressed before button B, the text analysis engine 8152 can associate unspecified buttons (referenced in the text) with buttons A and B in order, i.e., correct situations where the user has not named the buttons they press in the text.

[0206] In the case of a WBS-based website, it will be understood that the ML engine 882 can be trained with common templates provided by the WBS using the ML engine manager 88. This training may include specific functions that associate a given site with the underlying template used to build it (since a site may contain elements from multiple templates). Thus, the text analysis engine 8152 can benefit from such training and information.

[0207] Furthermore, it will be understood that toolcast text representations can be edited using the Rich Content Editor 8151, just like editing regular rich text documents. Deleting, moving, and copying text segments may also change the associated events. Therefore, if a text segment is duplicated (for example), the set of matching sync actions will also be duplicated.

[0208] TC Creator 81 can also create additional toolcast elements. For example, the toolcast management system 80 can generate narration audio from text provided via a text-to-speech engine (or conversely, allow the user to dictate parts of the text via a text-to-speech engine, and in some cases integrate a voice-based editor, as described below, into a text-based editor).

[0209] Once the toolcast definition (including storyboard text and associated definitions) is complete, the TC finalization module 8153 can terminate the toolcast.

[0210] The TC finalization module 8153 can convert the text in a storyboard into speech (for example, via a text-to-speech engine) or into a set of subtitles placed at relevant locations on the screen (or both).

[0211] The TC finalization module 8153 can also calculate the appropriate timing of words within a script. The TC finalization module 8153 can run a toolcast in measurement mode and determine the start and end times of all words in the toolcast text converted to speech (to create a final timeline of toolcast events and audio / subtitles). In alternative embodiments, other speech units such as syllables or sentences may also be used.

[0212] Such execution may have a "variable speed," meaning the speed may be set by a reasonable speed of speech generation or by the execution time of the underlying system 200 (i.e., the speech must wait until the system has finished). It will be understood that such execution may be performed in a specific mode of the underlying system 200 that prevents persistent side effects from the execution (as will be described in more detail below in this specification).

[0213] It should also be understood that this finalization is separate from the toolcast publication process (described in more detail below). The TC finalization module 8153 may run on a client of the underlying system, a server, a separate server, or a combination thereof.

[0214] The voice-based recorder 816 may allow the user to record themselves and then convert that recording into a toolcast file so that it can be replayed. For some users, this flow may be more natural and easier to use than writing text. The resulting recording can also be converted to a toolcast rich text format for later editing with the text-based recorder 815.

[0215] Therefore, the user can record themselves while using the basic system 200. The voice recorder 8161 can record both audio and actions of the basic system 200. This can be enhanced with additional elements (such as visual highlighting added to the basic system 200).

[0216] Next, the speech editor 8162 can use speech-to-text conversion to convert the audio into text and obtain the start and end timings of each word (or other text section). The speech editor 8162 can convert raw user events into toolcast ready events associated with the element. The speech editor 8162 can generate a rich text representation of the recorded toolcast (along with the associated actions).

[0217] The toolcast management system 80 can also support audio-based recording using a two-phase method (similar to the use of the toolcast storyboard creator 811 described above). In this method, the base system 200 is recorded (without narration), and the narration is added in a separate follow-up phase. The audio-based recorder 816 can thus help the user plan the narration by providing preliminary timing information to the user.

[0218] It will be understood that the voice-based recorder 816 can provide voice-oriented editing. For example, the voice editor 8162 may allow the user to replace a given segment of a recording with a newly recorded segment "on top of" an existing voice segment.

[0219] The audio post-processor 8163 can provide related post-processing functions such as improving audio quality, removing or reducing "dead" segments, removing "slips of the tongue" such as "ah," and removing background noise. Such operations can be adapted to the presence of related actions and the functionality of the underlying system 200. That is, if system actions requiring audio gaps (including those that do not affect the display) are occurring in parallel, silent recording sections may not be automatically removed.

[0220] In this specification, it will be understood that the tools described above support the creation and editing of linear BTCs. The CTC creator and editor 818 can be used to create and edit CTCs containing one or more BTCs, or hierarchies of such CTCs, and to associate them with specific events and triggers occurring in the underlying system 200.

[0221] Here, we refer to Figure 10, which shows the elements of the CTC Creator and Editor 818. The CTC Creator and Editor 818 may further include a timeline determination unit 8181, a visual graph editor 8182, and a recording analyzer and separator 8183.

[0222] Furthermore, it will be understood that the CTC definition specifies the CTC flow and the events and triggers to which the CTC and BTC are associated (as described herein).

[0223] The timeline determination unit 8181 can determine how an action is performed based on the timeline or detected website events. This may affect higher-level CTCs (for example, determining when and how various elements of a CTC are executed) and may also affect the underlying BTCs (i.e., affecting how a particular BTC is executed).

[0224] A CTC definition may include a linear timeline (for both BTC and CTC), a status chart, an activity graph (including a multilevel graph), a script definition (using a script or programming language), or other mechanisms.

[0225] In one embodiment, a flow may be defined using one or more state charts. These state charts describe when and under what conditions toolcast elements are replayed, as will be described in more detail below herein. The visual graph editor 8182 may provide a visual state chart editor that enables editing of charts and integration with the underlying system 200.

[0226] The visual graph editor 8182 may include support for editing directed graphs, multilevel graphs (i.e., graphs in which nodes contain subgraphs), and multigraphs (where node pairs can have multiple connecting edges).

[0227] In another embodiment, the recording analyzer and separator 8183 can create a linear CTC from a single recording by intercepting clicks or meaningful actions and separating the linear recording into multiple BTCs connected by those actions.

[0228] Furthermore, it will be understood that in some cases the mapping between the navigation of the underlying system 200 and the CTC structure (e.g., the state chart graph structure) may not be clear or can not be determined automatically. Therefore, the visual graph editor 8182 can exchange information with the user to determine the correct structure and navigation. This can be done by presenting alternatives (which can be rejected, approved, or modified), asking one or more questions, or by other means.

[0229] In the case of BTC, the underlying graph may be simple (for example, a linear sequence of nodes with some entry points, exit points, or branches that skip certain subsequences). However, the same techniques and interfaces described for CTC may also be applicable to BTC editing.

[0230] In this specification, the recorders and editors described above will generally be understood to record user actions within the underlying system 200, for example, when using voice-based toolcast generation or when using a rehearsal method. Such recordings are typically direct; that is, they record actual user interactions with the underlying system 200 and translate these interactions into definitions of associated events, triggers, and timing information (to which toolcasts are associated).

[0231] However, such records may not contain the best way to perform the basic System 200 tasks as the user might imagine. Users may perform irrelevant or incorrect steps, miss necessary steps and return to them later (potentially confusing Toolcast viewers), or re-track their own steps.

[0232] Therefore, the TC path optimizer 819 can analyze the user's steps to create an optimized, canonical, or cleaned-up version of the user's activity. Such analysis can be implemented in various ways, including techniques from the field of code optimization (such as those used in compilers).

[0233] Here, we refer to Figure 11, which shows the elements of the TC path optimizer 819. The TC path optimizer 819 may comprise a description path generator 8191 and an adaptive engine 8192.

[0234] The TC Path Optimizer 819 uses the ML Engine 882 to analyze (and is trained on) the structure of the underlying system 200, the steps taken by users, the steps taken by end users of the same system (e.g., through collected BI information), and the steps taken by end users of other sites using similar templates (which are subject to the anonymity, privacy, and intellectual property (IP) rights of other users).

[0235] The ML engine 882 can further analyze hints provided by template designers (e.g., predefined per-template path definitions in the WBS configuration), content or explicit hints provided in other technical materials associated with the relevant sections of the underlying systems 200, and train on them. Such materials may include designs, specifications, documentation, training, or support that might create alternative, optimized, or simpler paths for performing the same function or flow.

[0236] The descriptive path generator 8191 can generate descriptive information / explanations about how the optimization was performed, i.e., how the proposed alternative flow (or changes made to the user-specified recoded flow) delivers results better or more optimized. Such explanations can be based on a mapping between the recorded flow and the modified flow.

[0237] The adaptive engine 8192 can adapt additional elements of the toolcast in relation to changes made to the recorded path. Such adaptations may affect the definitions of event triggers and ACEs. Transformations may include rearranging elements, removing redundant, unnecessary, or "orphaned" elements, and modifying the actual content of elements to fit the changed path (for example, through analysis and subsequent generation of various ACEs such as text, image, audio, or video content).

[0238] The adaptive engine 8192 can either directly implement the changes or create a change plan (e.g., presented as a task list) that is evaluated by the user, accepted, rejected, or further modified. The user's response in such cases can provide additional feedback for training the adaptive engine 8192.

[0239] It will be understood that the integrated UNS / TC editor 8120 can provide integrated or combined editing of a given underlying system 200, along with its toolcast. Such integrated editing may help the designer of the underlying system 200 to maintain synchronization between the two. The UNS / TC editor 8120 may be equipped with a graph analyzer 81201.

[0240] Furthermore, it will be understood that the relationship between the base system 200 and the toolcasts is many-to-many. A single base system 200 can have multiple associated toolcasts (for example, for different components and subsystems). Such toolcasts may, but do not necessarily, be integrated under a top-level CTC "umbrella".

[0241] A single toolcast (e.g., for common components) can be associated with multiple underlying systems 200 (e.g., multiple underlying websites using the same template, using the same components, or multiple instances of the same underlying system 200).

[0242] As described above in this specification, toolcasts are typically objects of graph structure (more precisely, multilevel graphs). This applies to both CTCs (which can be described using graph-based state charts) and BTCs (which may also include graph representations).

[0243] The basic system 200 can also be described as a graph (or possibly a multilevel graph), and the graph analyzer 81201 can determine the interconnections between the displayed pages, and possibly between subareas or components of a single page (where the flow is defined).

[0244] It will be understood that navigation between nodes in the graph of the underlying system 200 may be explicit or implicit. For example, explicit navigation may be based on direct hyperlinks between different pages of the underlying website. Implicit navigation may be based on navigation targets (e.g., URLs) generated based on analysis, calculations, search results, or external database information.

[0245] For explicit navigation, the graph analyzer 81201 can determine graph arcs based on a static analysis of the underlying system 200 (e.g., from wired "href" links within the underlying website). For implicit navigation, the graph analyzer 81201 can determine graph arcs of the underlying system 200 (e.g., web page transitions) using a dynamic (runtime) analysis of the displayed components, user flow, and other information collected at runtime.

[0246] Both the base system 200 and the toolcast graph may be separate, that is, they may consist of multiple disjoint subgraphs. This can occur in the base system 200 if it includes multiple websites presented based on external parameters (e.g., time period, user segmentation detected based on geography, user type, etc.).

[0247] The integrated UNS / TC editor 8120 may enable simultaneous editing of the underlying system 200 and associated toolcasts, i.e., a combination of editing functions of the underlying system 200 (e.g., the underlying WBS visual editor) and toolcast editing (e.g., the CTC graph / status chart editor). Integrated TC / UNS editing may be based on matching between graphs in the underlying system 200 and TC graphs, allowing for parallel editing of two graphs. Therefore, changes made to one graph can be reflected in the other graph, for example, based on an analysis of similarity (homomorphism) between the graphs. Furthermore, if an area is changed in the underlying element of a node in one graph (e.g., a page on a website), matching nodes in other graphs may be marked for maintenance (e.g., matching toolcasts for that page).

[0248] The integrated UNS / TC editor 8120 can also propose automatic associations between the underlying system 200 and toolcasts that can be rejected, confirmed, or edited by the user (via the association manager 817).

[0249] The coverage analyzer 8121 can detect areas where mismatches exist between the underlying system 200 and the toolcast, and point them out to the user so that the user can correct the situation, for example, by modifying either the underlying system 200 or the toolcast. These may include areas with no coverage, situations where toolcast coverage overlaps, areas where the underlying system 200 and the toolcast are out of sync, and the use of similar sub-elements with different toolcast coverage. The coverage analyzer 8121 can check the toolcast coverage of pages, components, associated procedural code, and other units of the underlying system 200. It can also determine what is missing from the toolcast. The coverage analyzer 8121 can use analysis of the toolcast text or audio to detect mentioned elements that are not covered in the toolcast definition.

[0250] The Edit Time Recommender 8122 can provide toolcast designers with editing hints and options. The Edit Time Recommender 8122 may be trained based on user activity of the underlying system 200 (or its underlying development framework or WBS) and the toolcast management system 80.

[0251] An example of the use of the integrated UNS / TC editor 8120 is the ability for users to edit toolcasts, add toolcast nodes, and instantly associate them with the appropriate nodes and events within the underlying system 200. The visual graph editor 8182 can also be integrated with the various toolcast recorders described herein (such as the text-based recorder 815 and the voice-based recorder 816).

[0252] It will be understood that one issue the toolcast management system 80 needs to address is how to maintain toolcasts when the underlying system 200 is changed. This includes what to do when a toolcast is activated and the underlying system 200 no longer matches any references defined in the toolcast at any level. This can happen, for example, if necessary maintenance operations are not performed (correctly or not at all). Such changes can result from changes to the underlying system 200 itself, or from changes to the deployment and runtime environment of the underlying system 200 (e.g., changes to the underlying operating system, or changes to dynamically linked code used by the underlying system 200).

[0253] It will be understood that the underlying system 200 is often developed (and deployed) using a development framework. This development framework may be a framework based on, or providing, continuous integration (CI) and continuous deployment (CD) services. In the following description, the WBS can be considered a category of development framework.

[0254] The maintenance module 87 can solve the maintenance and matching problems described above and can be integrated with the development framework of the underlying system 200. In this way, the maintenance module 87 can automatically recognize when toolcasting should be interrupted based on an analysis of changes to the underlying system 200 (e.g., changes to its layout, flow, content, or data).

[0255] Such integration can be achieved in any of the ways described herein, such as through a special interface (e.g., a web service) or by using a deployment agent or plugin of the toolcast management system 80 integrated into the development framework.

[0256] To address potential toolcast interruptions, the toolcast management system 80 can provide elements that offer fallback, notification, and interruption severity detection, as will be further detailed below in this specification. This can be done in both the TC creator 81 and the TC player 85.

[0257] Next, refer to Figure 12, which shows the elements of the maintenance module 87. The maintenance module 87 may further include a TC (Tool Cast) deployment agent 871, an update analyzer 872, an information gathering unit 873, a notification unit 874, a TC (Tool Cast) adjustment unit 875, and a TC (Tool Cast) fallback application unit 876.

[0258] The TC deployment agent 871 can be added to an element of the underlying system 200 within the development framework (for example, as an add-on module). The TC deployment agent 871 may also be a generic agent that can only be configured via the application ID (or other metadata) of the associated underlying system 200. The TC deployment agent 871 can fetch additional configuration information related to the toolcast project from, for example, a server providing the toolcast management system 80 service.

[0259] The TC deployment agent 871 may be activated when changes to the underlying system 200 are about to be deployed. The update analyzer 872 can perform analysis to detect toolcast elements that are broken or not working for any other reason. The analysis may provide information about the level of disruption in the toolcast. The disruption may be related to a specific BTC or it may affect a large portion of the toolcast.

[0260] The update analyzer 872 can operate at the structural level, that is, it can analyze whether elements within the underlying system 200 corresponding to a toolcast have been changed holistically. This analysis may include when and where a given structural change occurred, as well as the conformity of IDs and timestamps. The analysis may also analyze whether the toolcast describes a given underlying system 200 element that has been changed or deleted, and its content and intent. The update analyzer 872 can also perform recognition and analysis of the content of the toolcast and the underlying system 200. In certain cases where the toolcast management system 80 is integrated with a WBS hosting an underlying website, the information gathering unit 873 can interface with the WBS's document management subsystem to obtain direct information about what has changed on the underlying website. Alternatively, the information gathering unit 873 can obtain different versions of the underlying site (along with information in WBS object format) and compare the versions to detect changes.

[0261] The second scenario described above might relate to a situation where, for example, a toolcast attached to site element A describes site element B (which does not have a directly associated toolcast). For instance, it might be more logical to describe B in combination with A, which is semantically related to B (rather than B itself). In such a scenario, the update analyzer 872 can provide the necessary information.

[0262] It will be understood that the update analyzer 872 can be supported by hints provided by the developer. For example, by defining in the toolcast configuration information of A that this toolcast is related not only to A but also to B, the update analyzer 872 can determine, based on this analysis, which toolcast elements (CTC or BTC) should be modified and in what way.

[0263] The notification unit 874 may notify the underlying system 200 or the toolcast developer of such updates. This may include generating explanatory media related to the relevant toolcast.

[0264] The TC adjustment unit 875 can provide automatic correction or adjustment (e.g., adjustment of position hook or timing information) to the tool cast itself.

[0265] The TC fallback application unit 876 can apply fallback to some or all of the toolcast elements. The TC fallback application unit 876 can also disable toolcast regeneration for some or all of the elements of the underlying system 200 and shut down the toolcast management system 80 (for example, until the toolcast is updated).

[0266] The TC fallback application 876 can use alternative technology to transmit relevant content through alternative means such as a regular video describing the site section of the relevant underlying system 200.

[0267] The UNS analyzer 813 and / or update analyzer 872 can also analyze the state mapping information of the underlying system 200 used by the TC player 85, as will be described in more detail below herein.

[0268] Developers of the Basic System 200 will understand that the Basic System 200 development framework allows for the management of multiple versions and configurations of the Basic System 200. These may include serial versions (chronologically consecutive), parallel configurations (multiple coexisting variants), and provisional versions (still under development).

[0269] The toolcast management system 80 can maintain toolcasts that match multiple versions and configurations, for example, based on version / configuration IDs.

[0270] Such a parallel configuration of the underlying system 200 can include A / B tests or multivariate tests that provide multiple versions of the underlying code. Since some A / B tests may be open for a considerable period, the developers of the underlying system 200 can create multiple toolcast elements for multiple A / B test branches (for example, UNS would show a different TC element for each test variant). In fact, the toolcasts themselves can be the subject of A / B testing, allowing us to test which toolcast provides a better use of the relevant section of the underlying system 200.

[0271] In these cases, the TC Creator 81 and TC Player 85 may interface with a test management system (such as the one described in U.S. Patent No. 10,733,078) that processes the basic system 200.

[0272] Furthermore, it will be understood that such parallel configurations on the underlying system 200 may also include multiple language versions (translations) of toolcasts. The toolcast management system 80 may support multiple matching translations (including text translation and event relocation) of a given toolcast, for example, based on a translation UI extension that allows input of multiple language versions of displayed ACEs (including multiple all media types).

[0273] Here, we refer to Figure 13, which shows the elements of the TC player 85. The TC player 85 may include a TC (Tool Cast) activator 851, a remote user simulator 853, a TC (Tool Cast) control toolbar display unit 854, an ACE display unit 855, and a user interaction manager 856. Here, we refer to Figure 14, which shows the elements of the TC activator 851. The TC activator 851 may further include an activation information collection unit 8511, an activation information analyzer 8512, and a runtime recommender 8513.

[0274] The TC Activator 851 can provide explicit activation of the toolcast by the user (for example, by pressing a specific button), and can also provide automatic activation of the toolcast. That is, the TC Activator 851 runs in the background and automatically activates the toolcast as described below herein.

[0275] The TC player 85 can dynamically execute toolcasts on the user's underlying system 200 based on the user's state and data, as well as triggers and events in the underlying system 200. This execution may include activating individual toolcasts, or activation and entry using specific entry points (such as graph nodes like state chart nodes) within a particular multi-node CTC.

[0276] The TC activator 851 may be fully automated or may require user confirmation. The latter may be particularly relevant when multiple toolcasts are activated (or otherwise made available) for a specific underlying system 200 area, or under specific conditions and system states.

[0277] When making a decision to activate or navigate a toolcast, the TC activator 851 may take into account information collected by the activation information collection unit 8511 from multiple sources, including the status of the underlying system 200, including the currently displayed page, internal state information, and what is displayed at any given moment, as well as BI information, site data (including design, layout, and site data), and any internal databases and repositories of the underlying system 200 (or the system hosting it), including data about a specific user.

[0278] The TC activator 851 may also take into account the user's state, which is determined from the available information. In an integrated system, this information may include user-related information collected separately by the activation information collection unit 8511 from the underlying system 200, including user information not related to a specific website (e.g., integrated WBS information about the user in general, not just the specific website being hosted).

[0279] It will be understood that the activation of a toolcast by the TC activator 851 can be performed based on associations (as described herein). Such associations may also include activation conditions and rules. These conditions and rules can activate a toolcast based on various events occurring within or outside the underlying system 200 (e.g., the user's website), as well as data from the underlying system 200 analyzed by the activation information analyzer 8512. These may include user actions such as pressing a button, hovering the mouse near a component, or clicking a link from the underlying system 200, such as a flow within the underlying system 200. Clicking may open a site page in which the toolcast is activated. The link may be a general link within the site or a specific link related to the toolcast.

[0280] Toolcast activation can also be performed through an API, web service, or other interface type available to the underlying system 200. Through interface module 83, the TC activator 851 can implement its own logic for when to trigger and activate toolcasts, separate from any logic and associations implemented by the toolcast management system 80.

[0281] In the case of the basic system 200 which is a website, the activation information analyzer 8512 can analyze the DOM and status information of the website (as well as the information of the basic system 200 or user data), and derive information that can be converted into toolcast nodes and opened.

[0282] The TC activator 851 can further activate through predefined initial nodes that can be associated with points within the basic system 200, or from the results of the activation information analyzer 8512 that examines the screen from external visual analysis in addition to internal / DOM analysis. Many such analysis algorithms are described in U.S. Patent No. 11,275,815.

[0283] Activation can be based on proactive analysis regarding the basic system 200 or saved thereby, such as the analysis of user behavior by the activation information analyzer 8512 like "whether the user is stacking or having problems in a specific basic system 200 area", or server-side information regarding the basic system 200 (therefore, it can activate "shadow" toolcasts that are not normally activated).

[0284] The activation information analyzer 8512 can also determine activation, user-related metrics, parameters, and events. This may include the analysis of user metrics such as voice-based sentiment analysis and voice-based triggers. These may be based on direct voice commands or voice analysis indicating stress, frustration, etc.

[0285] Activation can also be triggered, for example, by detections related to the user via sensors, gesture analysis, etc., by other events or conditions that can be detected via the above-described toolcast management system 80 / basic system 200 interface, or by interaction with the basic system 200, or by the toolcast management system 80 itself (including any mouse, keyboard, voice, or other interactions).

[0286] The runtime recommender 8513 can select which CTC to activate. The runtime recommender 8513 may be rule-based or ML-based and may use the associations and other information and events described hereinabove.

[0287] The runtime recommender 8513 can also explicitly recommend to the user (rather than the internal flow engine) which toolcast is optimal to execute at a given point in time. The runtime recommender 8513 can use the underlying condition specifications for each toolcast. That is, it can use what conditions are required for a toolcast to be playable with respect to the state of the basic system 200, the presence of the required basic system 200 structure, and the user's previous actions (such as moving to a specific URL). Information collected and generated by the runtime recommender 8513 (such as a list of toolcasts that may be executed at a particular point in time) can be used to create a toolcast content table for a "Tell me more" display.

[0288] It will be understood that the TC player 85 can navigate within the toolcast (both BTC and CTC) based on the user state and events within the underlying system 200. This user state is defined based on information extracted from the underlying system 200 (and possibly the underlying infrastructure hosting the underlying system 200) by the activation information collection unit 8511 and the UNS analyzer 813. Such information may include the underlying system 200 information being displayed (e.g., site DOM). This information may include specific details such as the currently displayed URL / page, the elements being displayed, and the location of content within each element (e.g., scrollbar position).

[0289] This information may also include data available through separate parallel channels, and data exposed by the underlying system 200 (from the client or server side), including data extracted from the underlying system 200 and analyzed by the activated information analyzer 8512, as described herein (with respect to the interface between the toolcast management system 80 and the underlying system 200, whether integrated or not).

[0290] It will be understood here that some of the analysis performed by the activation information analyzer 8512 described above can be performed in advance. For example, instead of performing the analysis at runtime, the TC deployment agent 871 can perform an analysis of the deployed version of the underlying system 200 and collect initial information for the state mapping model each time a change to the underlying system 200 is deployed.

[0291] Based on such runtime or pre-run analysis, the UNS structure mapper 84 can create and manipulate mapping models to map the states and data of the underlying system 200 (such as complex DOM trees) to clearly defined user states in state charts.

[0292] The UNS structure mapper 84 can use a mapping model to derive user states (as defined above herein) and convert them into system states. The UNS structure mapper 84 can map collected information (DOM, user location, data from the underlying system 200, or others) to states in a state chart. The TC activator 851 can then be instructed to traverse the states in the state chart based on the mapped user events and data and activate a toolcast based on the derived system state. The UNS structure mapper 84 can then generate outputs from the activated toolcast, including multiple output types and effects, as will be described in more detail below herein.

[0293] The UNS structure mapper 84 can also implement a state chart integrated into the CTC (and forming the underlying graph structure of the CTC). The toolcast management system 80 can also implement a separate state chart (for example, one related to the underlying system 200 and not part of the CTC, but still driving a separate toolcast regeneration).

[0294] It will be understood that the toolcast management system 80 may use alternative structures to map the state and data of the underlying system 200, user activity, and toolcast flow. These alternative structures may include graph structures and the use of branching and conditions.

[0295] As described above in this specification, a toolcast definition may include various “outputs” that are manipulated, displayed, or activated by the running toolcast. These typically include the generation of events on the underlying system 200 (such as keyboard or mouse events) and the display of ACEs (such as subtitles or audio). ACEs may include additional media types, including static and dynamic media.

[0296] These outputs can affect the underlying system 200 by replicating the effects of user-initiated events. The TC player 85 can activate these effects through the interface module 83, as described herein. In particular, the remote user simulator 853 can enable toolcasts to trigger responsive events on the underlying website DOM as if a remote user were performing them (clicking, dragging, hovering, scrolling, editing, and creating elements).

[0297] In general, the remote user simulator 853 can trigger events by simulating normal user actions (e.g., mouse clicks), thus minimizing the effort required to integrate the TCMS with the underlying system 200. For example, the remote user simulator 853 can dispatch DOM events to relevant elements, which typically react as if the dispatched event had been triggered by a normal user.

[0298] It will be understood that the TC player 850 can provide various UIs in the form of toolbars to provide user input to both the TC creator 81 and the TC player 85, as described above herein. The TC control toolbar display unit 854 can display a toolbar for the user to control the display of tool casts as needed. Hereinafter, refer to Figures 15, 16, and 17 illustrating such toolbar interfaces.

[0299] Figure 15 shows TC Player 85 running a toolcast for a restaurant menu building system, with the toolcast toolbar A displayed at the bottom of the page.

[0300] Figure 16 is a magnified view of toolbar A showing control buttons D such as pause, play, and volume, which are shown on the left, with an image of the virtual trainer avatar B (a person representing the "voice" of the toolcast, the current subtitle ("Help me set up the restaurant"), and an exit button C to exit the toolcast system) next to it.

[0301] Next, we refer to Figure 17, which shows a different embodiment of Toolbar A. This version shows an additional drawing (ACE) F placed on top of the screen of the base system 200 by the TC Player 85. In this case, the drawing is in the Toolcast toolbar area, but it could also be added elsewhere on the screen of the base system 200. The drawing includes “Pause” and “Play” labels and a chef’s hat F next to the relevant toolbar buttons. It will be understood that there is also a Toolcast-specific cursor G to guide the Toolcast viewer and point to relevant elements. In this embodiment, the Toolcast cursor is labeled “Maya” to provide the viewer or user with an anthropomorphic representation of the Toolcast virtual trainer.

[0302] The integrated foundational system 200 will reveal that the toolcast management system 80 has two key advantages. First, the TC deployment agent 871 is integrated with the development framework or WBS of the foundational system 200, ensuring reliable detection of any toolcast failures. Second, if it is difficult to simulate required user events, the interface module 83 can inject the remote user simulator 853 into the foundational system 200 (or its underlying framework) to replace these difficult-to-simulate actions.

[0303] The ACE display 855 can create additional or extended UI elements on top of the underlying website or modify existing elements. These changes can be based on the use of "floating elements" (optionally using an additional display "layer") that are added on top of the view of the current underlying system 200. This enables the use of the screen of the underlying system 200 to function as a smart whiteboard where users of the toolcast management system 80 can add markers, arrows, text, etc.

[0304] The user interaction manager 856 can manage the interface between the TC player 85 and the user during the execution of the toolcast, such as receiving instructions from the user and processing additional elements such as pop-up questionnaires that require input during the live toolcast.

[0305] It will be understood that the TC player 85 can respond to each user interaction (e.g., question and answer).

[0306] The user interaction manager 856 may provide a "reaction mode" in which the editing of the actions (such as actions / reactions) of the TC elements can be performed in a way that is integrated with the modified versions of the relevant underlying system 200 elements.

[0307] In such a mode, the user interaction manager 856 can change the display of specific underlying system 200 elements to reflect the current actions associated with those elements. The user can also select an action and associate it with a given underlying system element 200.

[0308] For example, when the TC player 85 clicks on an element with a yellow border, if that element leads to another toolcast step, the element can be changed. Also, if clicking on the area of this element stops the toolcast, the element with a red border may be changed.

[0309] TC Player 85 can also modify existing elements within the underlying system 200. Such modifications may include highlighting, hiding, or changing the attributes or settings of elements. For the underlying website, such modifications may occur at the DOM level.

[0310] For the underlying website hosted by WBS, the toolcast management system 80 can support additions and modifications at the component level. Furthermore, the underlying WBS, as a recipient of such effects and actions, may provide additional services such as applying dynamic layouts and responsive editing to new or modified components (as described in U.S. Patents 10,185,703 and 10,984,177).

[0311] It should be understood that such website modifications can be adapted to different rendering methods of the website, as described above in this specification.

[0312] Furthermore, it will be understood that the underlying system 200 operates not only without the toolcast management system 80, but also typically operates in conjunction with the toolcast management system 80. Therefore, navigating and operating the underlying system 200 can result in several persistent consequences, such as changes to the database and requests made to connected systems.

[0313] However, in some cases, users of the toolcast management system 80 may want to interact with the underlying system 200 in a way that limits the persistent side effects arising from the operation of the underlying system 200. Such cases include when performing a "rehearsal" of a toolcast to gather information for building a follow-up toolcast, or when editing a toolcast. For example, if a user edits a toolcast that explains the purchase process of a product multiple times, they may not want to order multiple copies of the product to be used in the demonstration.

[0314] Users may also wish to limit persistent side effects during the toolcast finalization phase (described herein) in which the toolcast management system 80 performs a run of the toolcast to collect timing information, and during the playback of the toolcast to study the operation of the underlying system 200 with substantial side effects (and to use rewind or forward on the toolcast).

[0315] In some toolcasts, the issue of side effects is mitigated because the toolcast requires the user to activate the operation (which causes side effects) themselves. TC Player 85 can guide the user and prompt them to make changes, but the user can choose whether or not to do so.

[0316] For other toolcasts, the toolcast management system 80 may use many technologies that can be implemented at the underlying system 200 or the underlying WBS / development framework level. These technologies may include the use of the underlying system that implements a special mode in which changes and side effects made by the underlying system are not permanently saved. The underlying system may save such changes to its database during a toolcast session (otherwise the toolcast session would lack context).

[0317] Other technologies may include implementations of parallel databases used when performing toolcasts instead of the usual databases of the underlying system, and implementations of underlying transaction systems with rollback / undo functionality.

[0318] The basic system 200 can interact with additional related systems. These related systems may also need to support coordinated rollback / undo across multiple related systems, using the techniques detailed above in this specification.

[0319] As described above in this specification, the toolcast management system 80 can utilize multiple ML (machine learning) engines 882 for analysis, recommendation generation, transformation, and other needs.

[0320] As described above in this specification, the ML engine manager 88 can collect data and train multiple different ML models to suit different needs and different ML-based engines. The ML engine manager 88 can generate new models (based on the collected training data) and evaluate their performance against existing models. The training data may include collected information as well as information about actions taken based on various recommendations.

[0321] It is understood that an ML model can be any model suitable for the task or activity performed by each ML-based engine. Machine learning models are known in this art and are typically some form of neural network. The term refers to the ability of a system to recognize patterns and provide conceptual solutions based on existing algorithms and datasets. The more a system is trained, the more knowledge it develops.

[0322] Basic ML models can be learning models (supervised or unsupervised). Examples of such algorithms include prediction (e.g., linear regression) algorithms, classification (e.g., decision trees, k-nearest neighbors) algorithms, time series prediction (e.g., regression-based) algorithms, association algorithms, clustering algorithms (e.g., K-means clustering, Gaussian mixture models, DBscan), or Bayesian methods (e.g., Naive Bayes, Bayesian model averaging, Bayesian adaptive trials), image-to-image models (e.g., FCN, PSPNet, U-Net), sequence-to-sequence models (e.g., RNN, LSTM, BERT, autoencoders), or generative models (e.g., GANs).

[0323] Alternatively, ML models may implement statistical algorithms such as dimensionality reduction, hypothesis testing, one-way analysis of variance (ANOVA), principal component analysis, combined analysis, neural networks, support vector machines, decision trees (including random forests), ensemble methods, and other techniques. Other ML models may be generative models (such as generative adversarial networks or autoencoders) that generate TC definitions and elements.

[0324] In most embodiments, an ML model may undergo a training or learning phase before being released into the product or runtime phase, or it may start operating with a model from an existing system or model. During the training or learning phase, the ML model may be tuned to focus on specific variables to reduce the error margin or otherwise optimize its performance. The ML model may initially receive input from a wide variety of data, such as the collected data described herein.

[0325] In another embodiment, where appropriate for a particular task, one or more ML models may be implemented in rule-based systems such as expert systems or hybrid intelligent systems incorporating multiple AI technologies. These are described in the Wikipedia articles entitled “Rule-Based System” and “Hybrid Intelligent System” at en.wikipedia.com.

[0326] It will also be understood that the toolcast management system 80 can provide additional functionality based on toolcasts. Such functionality is outside the scope of normal use of toolcasts via the TC player 85.

[0327] The toolcast itself operates in conjunction with the underlying system 200 (at both the BTC and CTC levels), providing input to the underlying system 200 and reviewing and analyzing its operation. However, the generated product can operate standalone and does not need to be linked to the running underlying system 200.

[0328] In particular, the automated execution unit 891 can automatically execute toolcasts (or parts thereof) to generate a large number of possible alternative products (or artifacts). These products may have linear or nonlinear / branching structures and may or may not be interactive.

[0329] At the CTC level, the automated execution unit 891 can arrange various CTC sub-elements in an appropriate order for linear representation. This may be based on (for example) an analysis of the content of the CTC elements or hints contained within the CTC elements. The automated execution unit 891 may also use information and hints provided by the underlying website directly, or based on such information and hints from the underlying template used to build the website.

[0330] It will be understood that this automated execution may require the underlying system 200 to provide a set of inputs (e.g., UI interactions) to the underlying system 200 on behalf of the user. Such inputs may be added to the input sequence covered by an existing toolcast. These inputs may be generated based on the collected toolcast user (replay) responses (as described above). These inputs may also be generated by an ML model that analyzes the collected inputs to recommend the correct way to proceed with the toolcast.

[0331] The generated product may include, for example, regular videos for documenting the basic system 200 or providing training on how to use it. Such videos can be played using a regular video file player (such as VLC with VideoLAN). The generated product may also include interactive videos using a different interactive video player. Such interactive videos may be based on the use of "hot regions" that, for example, allow clicking within the video to branch within the video or to external media outside the video.

[0332] Other generated products may include standalone websites or applications (e.g., mobile or desktop) that can provide a visual simulation of the underlying system 200 or training on the underlying system 200 without requiring access to the actual underlying system 200.

[0333] Standalone, playable data files may also be included, such as timing, text, and image content that provides interactive tutorials run by a dedicated player. The player still does not require the existence or any integration of the underlying system 200 to function, and may further include analysis and coverage maps, for example, maps showing which parts of the underlying system 200 code (or screens or objects) are "covered" by the toolcast content.

[0334] It will be understood that the additional toolcast-based artifacts described herein may also require maintenance if the underlying system 200 is modified, similar to the maintenance of the toolcasts described above. In such cases, the above mechanism (for handling changes to the underlying system 200) can be applied to these additional toolcast-based artifacts.

[0335] It should be understood that the generated products may include offline products such as written documents generated based on the toolcast content, in addition to the displayed content of the underlying system 200.

[0336] The app store interface 8100 can support the provision and distribution of toolcasts via app stores (including in-app purchases). App stores can provide toolcasts as individual, sellable objects. App stores can also provide toolcasts as add-ons to existing applications sold on the relevant app store.

[0337] The tooltip handler 8101 can add toolcasts to existing tooltips, including those created as toolcast activation UIs, providing a simple way to extend apps using toolcasts. These tooltips are sometimes referred to as live tooltips.

[0338] Live tooltips look the same as regular tooltips, but they have a corresponding "Learn More" play button, and the toolcast starts immediately in the same context. This rapid replacement allows for a way to integrate toolcasts into the system with minimal UI / UX (user experience) effort.

[0339] It will be understood that toolcasts can also be recorded by third-party content creators or "influencers" and subsequently triggered in conjunction with the use of the Basic System 200. This could be done, for example, by using a special link. In this way, the link to the website would open both the basis of the Basic System 200 and the toolcast recorded by the third-party content creator.

[0340] As part of this system, vendors of both the Toolcast Management System 80 and the Underlying System 200 may employ various measures to ensure the quality, functionality, security, scope, and appropriateness of the toolcasts provided. Vendors of the Toolcast Management System 80 or the Underlying System 200 may also disqualify (in whole or in part) toolcasts that are fraudulent, do not properly describe the Underlying System 200, are of low quality, infringe on intellectual property rights, and may cause security risks (e.g., through links to inappropriate external websites).

[0341] Therefore, the toolcast management system can be used in conjunction with underlying systems such as website building systems to create and execute toolcasts that provide help, events, subtitles, etc., based on user data and the state of the underlying systems, and can also handle direct interaction with users.

[0342] Unless otherwise specified, as is evident from the preceding discussion, discussions throughout this specification using terms such as “processing,” “computing,” “calculating,” and “decision” are understood to refer to actions and / or processes of any type of general-purpose computer, such as client / server systems, mobile computing devices, smart appliances, cloud computing units, or similar electronic computing devices that manipulate and / or convert data in the registers and / or memory of a computing system to other data in the memory, registers, or other similar information storage devices, transmitters, or display devices of a computing system.

[0343] Embodiments of the present invention may include apparatus for performing the operations described herein. This apparatus may be specifically constructed for a desired purpose, or it may include a computing device or system having at least one processor and at least one memory, which are typically selectively activated or reconfigured by a computer program stored in a computer. The resulting apparatus, when instructed by software, can transform a general-purpose computer into the elements of the present invention described herein. The instructions can define the apparatus of the present invention operating on a desired computer platform. Such computer programs may be stored in computer-readable storage media, including, but are not limited to, optical disks, magneto-optical disks, any type of disk, read-only memory (ROM), volatile and non-volatile memory, random-access memory (RAM), electrically programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, disk-on-key, or other types of media that can be connected to a computer system bus and are suitable for storing electronic instructions. Computer-readable storage media may be implemented in cloud storage.

[0344] Some general-purpose computers may include at least one communication element to enable communication with data networks and / or mobile communication networks.

[0345] The processes and representations presented herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used with programs written according to the teachings herein, or it may be advantageous to construct devices more specifically designed to perform the desired methods. Desired structures of these various systems will become apparent from the following description. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that various programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0346] While certain features of the present invention have been illustrated and described herein, those skilled in the art will find that numerous modifications, substitutions, alterations, and equivalents will arise. It will therefore be understood that the aim is to cover all such modifications and alterations that fall within the true spirit of the present invention.

Claims

1. A system for creating and displaying toolcasts, Processor and A toolcast management system that runs on a processor communicating with an underlying system, wherein the toolcast management system creates, displays, and updates the at least one toolcast in order to provide media guidance to the user of the underlying system in accordance with a toolcast management interface with the underlying system and interactions between the user and at least one toolcast, The tool cast management system is equipped with, An interface module that interfaces with the aforementioned basic system and detects at least one of the objects, data, activities, and events of the basic system which will be recorded in the recording phase of at least one toolcast, A TC (Toolcast) Creator that records, generates, and edits the at least one toolcast according to any of the at least one object, data, activity, and event detected, A TC (Toolcast) player that plays the at least one toolcast while interacting with the user, An ML (machine learning) engine manager configured to train one or more machine learning models using the underlying system and data collected from the user, and to apply the models to dynamically adapt the creation and playback of the at least one toolcast, The system comprising the above.

2. The system according to claim 1, wherein the at least one toolcast is at least one of active tutorials, documentation, courseware, and in-user messaging.

3. The system according to claim 1, wherein the basic system is at least one of a website, a website building system, and a document editing system.

4. A database for storing basic tool casts and composite tool casts. When the aforementioned basic system is modified, a maintenance module maintains the at least one toolcast. A toolcast synchronizer that supports the synchronization of at least one toolcast with other related toolcasts when the basic system is modified, and A basic system structure mapper provides, for at least one toolcast, a mapping of the structure, associations, events, and interfaces of the basic system to the toolcast management system. The system according to claim 1, further comprising at least one of the following.

5. The aforementioned TC Creator is, A toolcast storyboard creator that generates a storyboard for at least one toolcast, A generative text engine that generates text for the storyboard of the at least one toolcast, A UNS (Basic Systems) analyzer analyzes the elements and site structure of the aforementioned basic system. A BI (Business Intelligence) analyzer that analyzes business intelligence (BI) information collected from the aforementioned basic system. A CTC (Compound Tool Cast) creator that creates composite tool casts that include a hierarchy of basic tool casts. A text-based recorder that enables the designer of at least one toolcast to create a text-based toolcast using a rich text editor, A voice-based recorder that enables the designer of at least one toolcast to record himself and convert the recording into a toolcast file. In the interfaced configuration between the toolcast management system and the underlying system, an association manager creates associations between toolcasts. A TC (Toolcast) path optimizer determines and optimizes the flow of activity for at least one of the aforementioned toolcasts. An edit-time recommender that provides editing hints and options to the designer of the at least one toolcast, A coverage analyzer that detects a mismatch between the underlying system and the at least one toolcast according to the toolcast definition of the at least one toolcast, and A UNS (Basic System) / TC (Toolcast) editor that provides integrated or combined editing of the aforementioned basic system along with at least one toolcast, The system according to claim 1, comprising at least one of the following.

6. The aforementioned TC player is A TC activator that automatically enables the activation of the at least one toolcast by the user, A remote user simulator that enables at least one toolcast to trigger a responsive event on the underlying system by simulating the user's actions. A TC (Toolcast) control bar display unit that displays a toolbar so that the user can control the display of the at least one toolcast, An ACE (Supportive Content Element) display unit that creates UI (User Interface) elements added to or extended on top of the display by the TC control bar display unit, and A user interaction manager that enables interaction between the at least one toolcast and the user during the playback of the at least one toolcast, The system according to claim 1, comprising at least one of the following.

7. The system according to claim 1, wherein the at least one toolcast is a user-adaptable toolcast.

8. A toolcast management interface with the underlying system, and in accordance with the interaction between the user and at least one toolcast, the creation, display, and updating of the at least one toolcast in order to provide media guidance to the user of the underlying system. This includes creating, displaying, and updating the above, Interface with the basic system via the interface module, The interface module is used to detect at least one of the objects, data, activities, and events of the underlying system that will be recorded in the recording phase of the at least one toolcast. Recording, generating, and editing the at least one toolcast via the TC (Toolcast) Creator according to any of the at least one object, data, activity, and event detected, Playing the at least one toolcast via a TC (Toolcast) player while interacting with the user, Using the aforementioned basic system and the data collected from the user, one or more machine learning models are trained via an ML (machine learning) engine manager. Applying the aforementioned model to dynamically adapt the creation and playback of the at least one toolcast, Computer implementation methods, including those mentioned above.

9. The method according to claim 8, wherein the at least one toolcast is at least one of active tutorials, documentation, courseware, and in-user messaging.

10. The method according to claim 8, wherein the basic system is at least one of a website, a website building system, and a document editing system.

11. To save basic tool casts and composite tool casts to a database. When the aforementioned basic system is modified, maintain the aforementioned at least one toolcast. To support synchronization between at least one toolcast and other related toolcasts when changes are made to the aforementioned basic system, To provide a mapping of the structure, associations, events, and interfaces of the underlying system for at least one of the toolcasts, The method according to claim 8, further comprising at least one of the following.

12. Performing the recording, generation, and editing of the aforementioned at least one toolcast is: To generate a storyboard for at least one of the toolcasts, To generate text for the storyboard of the at least one toolcast, To analyze the elements of the aforementioned basic system and the site structure, Analyzing business intelligence (BI) information collected from the aforementioned basic system, To create a composite toolcast that includes a hierarchy of basic toolcasts, The designer of at least one toolcast can create text-based toolcasts using a rich text editor. The designer of at least one toolcast can record himself and convert the record into a toolcast file. Creating toolcast associations in an interfaced configuration between the toolcast management interface and the underlying system, Determine and optimize the activity flow of at least one toolcast. To provide the designer of the at least one toolcast with editing hints and options, To detect a mismatch between the underlying system and the at least one toolcast according to the toolcast definition of the at least one toolcast, and To provide integrated editing or combined editing of the aforementioned basic system together with the aforementioned at least one toolcast, The method according to claim 8, comprising at least one of the following.

13. The playback of the at least one toolcast is To automatically enable the activation of at least one toolcast by the user, The at least one toolcast can trigger a responsive event on the underlying system by simulating the user's actions. Displaying a toolbar so that the user can control the display of at least one toolcast, By displaying the toolbar, UI (user interface) elements are created that are added to or extended on top of the display, and To enable interaction between the at least one toolcast and the user during the playback of the at least one toolcast, The method according to claim 8, comprising at least one of the following.

14. The method according to claim 8, wherein the at least one toolcast is a user-adaptable toolcast.

Images