System and method for integrating user feedback into web building system services

Abstract

To provide a website building system (WBS) for building a website based on feedback, a feedback system, and a method.SOLUTION: A WBS includes: a processor implementing a ML feedback-based proposal module; and a database storing at least the websites of a plurality of users of the WBS and components of the websites. The module includes a plurality of per activity AI units and a feedback system. Each per activity AI unit supports one or more specific activity related to the WBS and includes at least one ML model suitable for the activity supported by its per activity AI unit. The feedback system provides a plurality of different kinds of feedback from the users for updating the ML model and analyzes the feedback to determine which one of the at least one ML model to update.SELECTED DRAWING: None

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This patent application claims the benefit of priority from U.S. Provisional Patent Application No. 62 / 853,191, filed May 28, 2019, U.S. Provisional Patent Application No. 62 / 970,034, filed September 25, 2019, and U.S. Provisional Patent Application No. 63 / 027,369, filed May 20, 2020, all of which are incorporated by reference in their entireties.

[0002] The present invention relates generally to web building systems, and more particularly to feedback-based site creation. [Background technology]

[0003] Website building systems are used by both novices and experts to create interactive websites. Existing website building systems are based on a visual editing model, and most website building systems typically provide multiple templates with templates possibly including complete sample websites, parts of websites, single pages or parts of pages.

[0004] Users of the website building system (also called designers, subscribers, subscriber users, or site editors) may design websites, and end users of the website ("user's users") may access the websites created by the users. End users typically access the system in a read-only mode, but the website building system (and the website) may allow end users to perform changes to the website, such as adding or editing data records, adding talkbacks to news articles, adding blog entries to blogs, etc. The website building system may allow multiple levels of users (i.e., three or more levels) and assign different permissions and capabilities to each level. Users of the website building system may be registered with a website building system server, which manages the users, their websites, and access by end users.

[0005] The WBS may be a standalone system or may be embedded within a larger editing system. It may also be online (i.e., the application is edited and stored on a server), offline or partially online (the website is edited locally but uploaded to a central server for publication). The website building system may use an internal data architecture to store the site based on the website building system, and this architecture may organize the internal data of the site and the elements within the system that are processed. This architecture may be different from the external view of the site (e.g., as seen by an end user) and is also typically different from the way HTML pages are organized that are sent to the browser.

[0006] For example, the internal data architecture may include additional properties for each element in a page (such as author, creation time, access permissions, links to templates, SEO (Search Engine Organization) related information, etc.) that are relevant for editing and maintenance of the website building system's site, but are not visible from the outside to end users (or even to some editing users). The website building system may realize part of its functionality (including both editing and runtime functionality) on a server or set of servers and part of its functionality on client elements. The website building system may also dynamically decide whether to execute certain functions on the server or on the client platform.

[0007] A website building system typically handles the creation and editing of visually designed applications (such as websites). In some website building systems, such as the one provided by Wix.Com, an application is composed of pages, containers and components. Pages are displayed separately and may contain components. Components may contain containers as well as atomic components.

[0008] A website building system may support hierarchical arrangement of components using various types of container components, including atomic components (text, image, shape, video, etc.), as well as other components (e.g., normal containers, single-page containers, multi-page containers, gallery containers, etc.). Subpages contained within a container component are called minipages, and each minipage may contain multiple components. Some container components display only one minipage at a time, while others display multiple minipages simultaneously.

[0009] These components may have little content, or they may have internal content. An example of the first category is a star component, which has no internal content (although it has color, size, position, attributes and other parameters). An example of the second category is a text paragraph component, whose internal content includes the inner text, along with font, formatting and layout information (which are not attributes of the component, but are also part of the content). Of course, this content may vary from one instance of the text paragraph component to another. Components that have content are often called fields (e.g., "text fields").

[0010] A page may use a template, a generic page template or a component template. Specific cases of templates include the use of an application master page that contains components that are replicated on all other regular pages, and the use of an application header or footer (repeated on every page). The website building system may provide inheritance between templates, pages or components, which may include multi-level inheritance, multiple inheritance and diamond inheritance (i.e. A inherits from B and C, and both B and C inherit from D).

[0011] The visual arrangement of components within a page is called a layout. The website building system may also support dynamic layout operations in which editing a given component (or other changes affecting it, such as externally driven content changes) may affect other components, as further described in U.S. Patent No. 10,185,703, issued Jan. 22, 2019, entitled “Website Design System Integrating Dynamic Layout and Dynamic Content,” which is commonly owned by the applicant and incorporated herein by reference.

[0012] The website building system may be extended using third party applications and components thereof, list applications (such as those described in U.S. Patent Application No. US2014 / 0282218, entitled “Website Building System Integrating Data Lists with Dynamic Customization and Adaptation,” published September 18, 2014, which is co-owned by the applicant and incorporated herein by reference), and website building system configurable applications (such as those described in U.S. Patent Application No. 16 / 683,338, entitled “System and Method for Creation and Handling of Configurable Applications for Website Building Systems,” filed November 14, 2019, which is co-owned by the applicant and incorporated herein by reference). These third party applications and list applications may be added and integrated into the website being designed.

[0013] Such third-party applications and listing applications may be purchased (or obtained) through several distribution mechanisms, such as being pre-included in the website building system's design environment, from an application store (integrated with the website building system's market store or external to it) or directly from the third-party application vendor.

[0014] Third party applications may be hosted on the website building system vendor's own servers, on the third party application vendor's servers or on a fourth party's server infrastructure.

[0015] The website building system may also allow procedural code to be added to some or all of the system's entities. Such code may be written in a standard language (such as JavaScript), an extended version of a standard language, or a language specific to the particular website building system. The executed code may reference APIs provided by the website building system itself or by external providers. The code may also reference internal structures and objects of the website building system, such as pages, components and their attributes.

[0016] Elements of procedural code may be activated via event triggers, which may be associated with user activity (such as mouse movements or clicks, page transitions, etc.), other user-related activity (such as an underlying database or a particular database record being updated by another user), system events, or other types of conditions.

[0017] The activated code may be executed within a client component of a website building system, on a server platform, using a combination of the two, or a dynamically determined execution platform. Such a system is described in U.S. Patent Publication No. US2018 / 0293323, published October 11, 2018, entitled "System and Method for Smart Interaction Between Website Components," which is commonly owned by the applicant and is incorporated herein by reference.

[0018] Exemplary site creation may be based on several models, including a visual editing model (wherein a user edits a previously created site) and an automated site generation model or a combination thereof, as shown in referenced FIG. 1, and is described in U.S. Pat. No. 10,073,923, entitled “System and Method for the Creation and Update of Hierarchical Websites Based on Collected Business Knowledge,” issued on September 11, 2018, which is commonly owned by the applicant and incorporated herein by reference.

[0019] It will be understood that throughout this specification the acronym WBS may be used to denote a website building system. Figure 1 shows a system 100 comprising a typical website building system 5 in communication with client systems operated by WBS vendor staff 61, site designers 62 (i.e. users), site users 63 (i.e. users of users) and external systems 70. The website building system 5 may further comprise a WBS (Website Building System) site manager 10, an object marketplace 15, a WBS RT (Runtime) server 20, a WBS editor 30, a site generation system 40 and a WBS content management system (CMS) 50. It will be understood that the elements of Figure 1 may function as described in U.S. Pat. No. 10,073,923.

[0020] In the visual editing model, a user (designer) edits a site based on one or more website templates. A website building system provider may provide multiple site (or other) templates, as described above. A user typically starts with an actual site template, although they have the option to start with an empty site (essentially a "blank page" template).

[0021] A website building system provider may offer site templates ranging from very generic (e.g., mobile sites, e-stores) through more specific (e.g., law firms, restaurants, florists) to very specific (e.g., commercial real estate law firms or Spanish tapas restaurants). Such templates are typically stored in a repository accessible to users of the website building system and are typically categorized according to business type, subtype or industry. Templates may also be created (and categorized) according to business type, but also style, color range or other parameters. Site templates may be extended by additional (typically back-end) features, services and code to become fully-fledged vertical solutions integrated with the website building system.

[0022] Thus, a user's initial experience in creating a site using a website building system's visual editor typically involves the user selecting a template (e.g., according to style or industry type / subtype) and then editing the template in the visual editor, including editing the content, logic, layout and attributes. Such editing may include adapting the template and its elements to the specifics of the user's business. The user may then publish the modified site.

[0023] Under the site generation model, a website building system generates an initial site for a user based on a selected template, which is then modified by filling in common information elements and possibly allowing follow-up editing of the generated site. This filling is required when various information (such as the business name and a description of the management team) is included in multiple places within the pages of the template. Thus, a user may need to change the business name (for example) in multiple places throughout the template.

[0024] Furthermore, some template elements (e.g. a generic product page) may appear multiple times, with each instance displaying details of a different instance of the underlying entity (e.g. a different product offered on the site). Such multiple instances may be specified manually (e.g. details of different people in a company's management team) or may be dynamically derived from an external database (e.g. product details from a "products for sale" database). Such configurations are often known as "Repeaters".

[0025] Templates may also include fields. For example, a website building system may allow a template designer to specify fields (also called "placeholders") for inserting values ​​into the template, such as {CompanyName}, {ProductName}, {ProductPrice}, etc. A user may also specify values ​​for fields defined in a template selected for a website.

[0026] The website building system may allow users to enter not only simple or complex values ​​(e.g., text and images), but also additional (non-field) information such as selection of included pages or website areas, colors, style information, links, formatting options, website display options, decorative elements (such as borders and backgrounds), etc.

[0027] The website building system may also allow the user to enter some of this additional information before selecting a template, and use this information to aid in the selection of the template (e.g., by narrowing the set of suggested templates). For example, the user may select a particular generic color scheme (e.g., pastel) or style (e.g., business / formal), and the system may use this selection to narrow the set of suggested templates.

[0028] The system may also display a series of views or questionnaires to allow the user to enter values ​​or selections (for both the defined fields and the additional information described above). The system may further create connections (or linkages) between the template's multi-instance elements (as described herein above) and internal or external databases that provide the data instances used to generate the displayed instances.

[0029] Once a template has been selected and its fields and additional information specified (e.g., through a questionnaire or through connections to data sources), the website building system may generate a website containing the composite information, and the user may then publish the site (through the website building system or otherwise).

[0030] The website building system may perform semi-automatic site creation using different models described in U.S. Patent No. 10,073,923. Under this model, the system collects information about the user and the user's website requirements from multiple sources that may include, for example, a questionnaire completed by the user, existing user presence (such as existing websites or social media presence), industry sources (such as general trade websites), offline information, and internal system repositories that provide information about specific industries, such as basic template information for specific industries (lawyers, restaurants, plumbers, graphic designers, etc.) that may be subdivided for specific industries (e.g., differentiating between real estate lawyers and personal injury lawyers).

[0031] The system may also collect external information from other sites, both internal and external to the system. Such information may influence, for example, surveys offered and layout element selections, suggested defaults, etc. Such information may also be collected on an aggregate or summary basis, typically to not reveal information belonging to any single user and to protect users' privacy, anonymity, and legal rights (such as copyright). Such information may be identified based on information provided by the user, which may be direct (e.g., existing website addresses) or indirect (business names and geographic addresses that can be used to identify information about the business).

[0032] The collected information is analyzed and placed into a repository of content elements, which are then mapped to layout elements that present the content and combine the layout elements to form the site. The layout element mapping, selection and composition process may be fully automatic or semi-automatic (i.e., involving user interaction).

[0033] To support the above-mentioned functions, a website building system typically maintains a set of repositories stored on one or more servers or server farms. Such repositories may typically include various related repositories such as a user information / profile repository, a WBS component repository, a WBS site repository, a business intelligence (BI) repository, an edit history repository, a third-party application store repository, etc. The system may also include site / content creation related repositories such as a questionnaire type repository, a content element type repository, a layout element type repository, a design kit repository, a completed questionnaire repository, a content element repository, a layout element repository, a rules repository, a family / industry repository, etc. A description of these repositories may be found in U.S. Pat. No. 10,073,923. Summary of the Invention

[0034] Thus, according to a preferred embodiment of the present invention, there is provided a website construction system (WBS) and a method implemented thereon. The system includes a processor implementing a machine learning feedback based suggestion module and a database storing at least a plurality of user websites and components of the websites of the WBS. The module includes a plurality of per activity AI units and a feedback system, each unit supporting one or more specific activities associated with the WBS and providing at least one system recommendation to users of the WBS related to the specific activities. Each per activity AI unit includes at least one machine learning model suitable for the activity supported by the per activity AI unit. The feedback system provides a plurality of different types of feedback from users for updating the at least one machine learning model. The feedback system analyzes the feedback and determines which of the at least one machine learning model to update.

[0035] Further, in accordance with a preferred embodiment of the present invention, the feedback system includes an implicit feedback handler that at least analyzes the user's edit history, determines what further activities the user performs on the website and / or within the WBS, and generates implicit feedback therefrom for training a corresponding machine learning model.

[0036] Furthermore, in accordance with a preferred embodiment of the present invention, the feedback system also includes an explicit feedback handler that at least analyzes the user response to the at least one system recommendation, determines how the user responded to the at least one system recommendation, and generates explicit feedback therefrom for training a corresponding machine learning model.

[0037] Further, in accordance with a preferred embodiment of the present invention, the WBS also includes an editor that operates with the suggestion module. The task includes at least one single-component task in the editor. The single-component task may be image resolution enhancement, face detection, portrait segmentation, object segmentation, image cropping, image enhancement, logo creation, or site text generation.

[0038] Further in accordance with a preferred embodiment of the present invention, the tasks include at least one multi-component task for improving site functionality, which may be component grouping, component group labeling, component ordering, object analysis, object transformation, desktop-to-mobile transformation, website import, template replacement, responsive editor support, or alternating design suggestions.

[0039] Further, in accordance with a preferred embodiment of the present invention, each activity-based AI unit includes an interaction generator configured to provide the user with at least one recommendation related to each activity based on the output of the at least one machine learning model.

[0040] Furthermore, in accordance with a preferred embodiment of the present invention, the at least one machine learning model is a model suitable for the task and selected from one or more of the following model types: supervised, unsupervised, prediction algorithms, classification algorithms, clustering algorithms, association algorithms, time series prediction algorithms, image-to-image models, sequence-to-sequence models, and generative models.

[0041] Further, in accordance with a preferred embodiment of the present invention, the feedback system also includes at least one of a response evaluator for evaluating a response quality of a feedback response from a user, a user evaluator for evaluating a user quality in providing feedback, a vendor handler for analyzing at least feedback from vendor staff of the WBS, and a community handler for analyzing at least feedback from a community of users.

[0042] Furthermore, according to a preferred embodiment of the present invention, the suggestion module updates at least one of the machine learning models periodically and based on user activity, which can be whenever a user makes changes to the website or whenever a user publishes the website.

[0043] Furthermore, in accordance with a preferred embodiment of the present invention, the explicit feedback handler receives multiple object types from multiple interaction formats.

[0044] Additionally, in accordance with a preferred embodiment of the present invention, the implicit feedback handler receives information collected from within the WBS, which may be information about the nature of the components and / or business, user and site information.

[0045] Further, in accordance with a preferred embodiment of the present invention, the at least one machine learning model is a plurality of models for the plurality of WBS tasks, and in one embodiment, a first model of the plurality of models interacts with a second model, e.g., the first model assists the second model or the first model provides transfer learning for the second model.

[0046] Furthermore, in accordance with a preferred embodiment of the present invention, a first model of the plurality of models receives input from a different collection of users than a second model, for example the collection being defined as a country, a community, a group or a user profile.

[0047] Furthermore, in accordance with a preferred embodiment of the present invention, the feedback system infers a user profile for a particular user from one of the particular user's editing history and information about the particular user's website.

[0048] Furthermore, in accordance with a preferred embodiment of the present invention, the interaction generator provides one of a plurality of interactions as a function of a website user's parameter of the edit history.

[0049] Furthermore, in accordance with a preferred embodiment of the present invention, user quality is defined as a function of at least one of the content of the feedback, the user's profile, the user's editing history, and at least one past score of the user for past responses.

[0050] Furthermore, in accordance with a preferred embodiment of the present invention, the user evaluator makes the user ineligible to provide further feedback if the user quality is below a predefined threshold.

[0051] Furthermore, in accordance with a preferred embodiment of the present invention, when the at least one single-component task is image cropping, its machine learning model is an image cropping model that infers that areas of the image covered by overlapping components are non-important areas of the image.

[0052] Finally, in accordance with a preferred embodiment of the present invention, the WBS also includes a site generation system that receives indications of non-important areas of the background image from at least one associated activity unit AI unit and places layout elements on the non-important areas. [Brief description of the drawings]

[0053] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification, however, the invention, both as to organization and method of operation, together with its objects, features, and advantages, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a website construction system according to the prior art. [Diagram 2]FIG. 1 is a schematic diagram of a system for integrating online machine learning feedback into the services of a website building system constructed and operative in accordance with an embodiment of the present invention; [Diagram 3] FIG. 3 is a schematic diagram of elements of a machine learning feedback-based suggestion module that forms part of the system of FIG. 2. [Figure 4] FIG. 3 is a flow chart diagram of a typical user workflow for the system of FIG. 2. [Diagram 5] FIG. 4 is a schematic diagram of the training and continuing set-up of one activity unit AI unit forming part of the module of FIG. 3. [Figure 6] FIG. 6 is a flowchart diagram of an exemplary method implemented by one type of machine learning model forming part of the AI ​​unit of FIG. 5 for the tasks of layout understanding and component grouping. [Figure 7] FIG. 4 is a schematic diagram of elements of an edit task handler that forms part of the module of FIG. 3; [Figure 8] FIG. 4 is a schematic diagram of elements of a site functionality updater that forms part of the module of FIG. 3; [Figure 9A] FIG. 13 is a flowchart diagram of an alternative method for component grouping utilizing a computer vision based ML model. [Figure 9B] FIG. 13 is a flow chart diagram of an alternative method for component grouping utilizing an ML model based on triplet analysis. [Figure 9C] FIG. 13 is a flowchart diagram of an alternative method for component grouping utilizing an ML model based on a combination of triplet analysis and computer vision. [Figure 10] FIG. 2 is a pictorial diagram of a hierarchical grouping process performed on a page. [Figure 11] FIG. 11 is a schematic diagram of the hierarchy of the page of FIG. 10. [Figure 12A] FIG. 1 is a plot of group labels for the same web page defined at three levels of the hierarchy. [Figure 12B]FIG. 1 is a plot of group labels for the same web page defined at three levels of the hierarchy. [Figure 12C] FIG. 1 is a plot of group labels for the same web page defined at three levels of the hierarchy. [Figure 13A] FIG. 13 is an illustration of a labeling user interface useful in understanding the labeling process for grouping. [Figure 13B] FIG. 13 is an illustration of a labeling user interface useful in understanding the labeling process for grouping. [Figure 14] FIG. 2 is a pictorial diagram of how a given layout can be displayed on multiple available screen widths.

[0054] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the size of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] In the following detailed description, numerous specific details are given to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the present invention.

[0056] Applicant has recognized that the process of building a website can be difficult for a novice builder or designer, especially one with limited technical knowledge. Novice designers can face problems both at the layout level (such as placing objects on a page) and at the editing level (such as image enhancement). Moreover, Applicant has recognized that even the most expert website designers can be challenged by certain tasks, such as image enhancement and logo creation or other functions that cannot be performed manually.

[0057] The applicant has also recognized that the above-mentioned difficulties may be overcome by having a website building system that provides designers / WBS users with a set of machine learning / artificial intelligence based services to support website building and maintenance workflows (such as image cropping support for images used on the website). The WBS may utilize different types of feedback to train the underlying machine learning models, utilizing different object types, feedback types and interaction formats, as well as collected information available from within the website building system and external information that may be analyzed accordingly. The results of these data collection / analysis / feedback interactions may then be integrated into the site design workflow.

[0058] Applicant further recognized that the WBS may utilize certain models to determine from the results of the above analysis whether feedback from a particular user (or the user itself) is trustworthy and whether the feedback response needs to be weighted accordingly before it is used.

[0059] In the field of website building systems, machine learning models / artificial intelligence may provide different solutions. Designing, creating and maintaining a website involves various tasks. These tasks are at different levels of abstraction and deal with different types of objects. Some of these tasks deal with media files (images, audio, video, etc.) or text files / objects, while some deal with higher level objects (parts of a page, the design and layout of a single page or a collection of pages, etc.). Some are very mechanical or technical, while some involve a high degree of creativity or skill.

[0060] Applicant recognized that a machine learning model should be designed for each specific activity, initially trained using vendor training data, and may be further enhanced using feedback from both implicit and explicit data from the WBS user (designer) and his website and from both implicit and explicit data about the WBS user's website. Implicit feedback may include designer behavior information (edit history, business intelligence, general user information, etc.), and explicit feedback may be based on questions posed to the designer regarding suggestions or suggestions (which of the proposed photos of the Taj Mahal is clearer?) at any appropriate point during the website design, creation, or maintenance. Models may be standalone, i.e., task-specific, or may operate as subroutines for other models.

[0061] Applicant further recognized that functions and tasks associated with the automated process of site creation (e.g., site generation, site import, conversion from other systems, conversion between different display platforms, etc.) could also be enhanced using the same data collection / analysis / feedback interactions.

[0062] Using user feedback to improve the performance of machine learning models is known in the art and is a form of reinforcement learning. One example is the classification of emails into spam and non-spam emails (e.g., U.S. Patent Publication No. US2017 / 0222960A1 by Microsoft Technology Licensing LLC). In the case of spam emails, the user feedback is typically fairly simple. Emails can be classified as spam or non-spam (or "light spam" in some systems, i.e., spam that is less noticeable and less irrelevant than the majority of spam emails). The simplicity of the response makes it easier to obtain user feedback than in cases that require more complex feedback.

[0063] Another common example is a web search engine that tracks the actual search results selected by users and uses them to refine further searches. The results selected by users provide training data for the search engine's machine learning models.

[0064] A machine learning task in this invention involves editing or transforming a given object X to generate a new or modified object Y. For example, we take an image X and convert it into a modified image Y through editing, cropping, attribute changes (brightness, contrast, etc.), quality enhancement, etc.

[0065] The resulting object Y may be much more complex than the modified version of the image, and may be a very different object or set of objects than object X. For example, in an object segmentation task applied to image X, the result of the task may be a composite Z that includes a version of image X that shows only the physical objects (detected in image X) on a modified background. However, composite Z may also include a set of high-level product details, including elements such as object contour coordinates in the original image, product metadata, product classification, etc. (e.g., based on matching the detected physical objects against an e-commerce product catalogue).

[0066] Other categories of tasks may include suggesting a given object or element to be added. This may be purely system-initiated or based on user interaction. For example, the WBS may suggest an image (e.g., from the system's repository or other stock images) that would match well with the user's text in a particular site portion. Conversely, the WBS may suggest text to match the image. Thus, the WBS may add a web element rather than simply editing an existing one.

[0067] As another example, a WBS may provide users with the ability to perform template replacement. Users (designers) often build sites based on templates provided by a WBS vendor. Users may perform substantial modifications to the underlying template, including design and layout changes, insertion and deletion of pages, page elements and site parts, as well as input of user-specific site text, media and data. The user may later wish to switch to a different template, which would require the user to map the changes (layout and content) to the new template and reapply them accordingly. The machine learning tools of the present invention may provide high-level editing tools to perform this task, and may also provide automated and machine learning-based tools, as described below. In such cases, the object being processed is not just an image (or other media file), but the entire site.

[0068] It will be appreciated that the WBS may provide a variety of tools for performing these tasks, ranging from simple task-specific tools (e.g., image croppers, image attribute modifiers) to media editing tools (e.g., image editors) to full-scale page editing or similar tools (template replacement applicators). These tasks may occur at various stages of the site creation workflow, for example, during a preliminary media gathering and uploading stage, immediately prior to site / page editing, during or in conjunction with site or page editing, or as part of a post-editing or publishing process.

[0069] Reference is now made to FIG. 2, which illustrates a system 200 for integrating online machine learning feedback into the services of a website building system, constructed and operative in accordance with an embodiment of the present invention. As described herein above, for a WBS system, there are various tasks that may benefit from machine learning models, and each of these models may benefit from user feedback. The system 200 may integrate a feedback loop into the workflow of the WBS to make it easy (and desirable) for the user to provide the required response. Furthermore, the feedback loop may extract relevant feedback from user activity. The system 200 may aim to make the user feedback interaction part of the normal WBS interaction required by the system to complete the required task.

[0070] System 200 may include elements of system 100 (along with any sub-elements) as described in U.S. Patent No. 10,073,923, along with an ML feedback-based suggestion module 300. The ML feedback-based suggestion module 300 may interact with multiple portions of website building system 5, in particular with WBS editor 30, site creation system 40, and CMS 50, as described in more detail below.

[0071] It will be understood that the following discussion refers to applications created by designers using WBS5 and accessed as websites by end users, although the system 200 may be applied to other categories of online applications accessed using specific client software (dedicated or not). End users may access these websites using client software on ordinary personal computers, smartphones, tablet computers, and other desktop, mobile, or wearable devices. Alternatively, the system 200 may be applicable to systems for generating mobile applications, native applications, or other application types, as described in U.S. Patent No. 9,996,566, entitled “Visual Design System for Generating a Visual Data Structure Associated with a Semantic Composition Based on a Hierarchy of Components,” issued on June 12, 2018, which is co-owned by the applicant and incorporated herein by reference.

[0072] Furthermore, the following discussion focuses on websites operated by a website building system provider. It will be understood that the following discussion refers to processing of imported images and their integration into the edited site. However, the system 200 may process other objects or sets of objects, such as images with additional information (cropping, embedded objects, identified faces / portraits / objects, etc.), text objects (fields or embedded text type objects), other components, collections of components, page layouts, complete web pages, and complete sites.

[0073] Reference is made to FIG. 3, which illustrates the elements of the ML feedback-based suggestion module 300. It will be appreciated that the ML feedback-based suggestion module 300 may enhance all forms of editing processes and WBS functions by suggesting possible enhancements to the current task of the WBS user 61 or 62, which may be derived from feedback previously received from the community of the WBS users 61 or 62. The module 300 may comprise one or more activity-based AI units 310 (as described in more detail below), each specifically designed for its associated editing task or WBS system task. Each activity-based AI unit 310 may be a module for enhancing a particular editing task or WBS system task, and may generate a particular interaction UI for a particular activity to which the WBS user 61 or 62 may respond. Each activity-based AI unit 310 may comprise an interaction generator 315 and at least one ML model 317 trained for a particular activity or task. Alternatively, multiple activity unit AI units 310 may share one interaction generator 315 and / or the unit 310 may include multiple ML models 317, and the explicit feedback handler 320 may also operate together with one of the interaction generators 315.

[0074] The ML feedback-based suggestion module 300 may further include an edit task handler 370 that provides task-specific suggestions from the associated AI unit 310 to the WBS user 61 or 62 for single object processing, and a site capability updater 400 that provides such suggestions for multiple object processing. Each of the edit task handler 370 and the site capability updater 400 may communicate with either the WBS editor 30 or the site generation system 40, as appropriate, as described below with respect to Figures 7 and 8.

[0075] Additionally, the ML feedback based suggestion module 300 may include a feedback system 305, including explicit feedback handlers 320, implicit feedback handlers 340, community handlers 380, and vendor feedback handlers 390, that provides user feedback to the ML model 317 to improve its functionality based on what the WBS users 61, 62 have done. The ML model 317 may be updated or retrained at any appropriate time, such as during operation, periodically, or upon triggers such as when a user changes or publishes their website.

[0076] As described in more detail below, the feedback system 305 may analyze the feedback received from users and determine which machine learning models to train.

[0077] The explicit feedback handler 320 may further include an explicit feedback analyzer 325 and a user analyzer 328. The implicit feedback handler 340 may further include an implicit feedback analyzer 345. The functionality of these elements is discussed in more detail below. Alternatively, although not shown in FIG. 3 for simplicity, the analyzers 325, 328, and 345 may operate within the activity-based AI unit 310, especially when they are highly specific to the activity identified by the particular activity-based AI unit 310. As mentioned above, the system 200 may use machine learning models to support users in performing various tasks. The system 200 may integrate user feedback collection (both explicit and implicit) into the WBS workflow and the various tasks themselves. The system 200 may utilize the provided feedback, along with additional information known to the WBS, to improve the ML model 317. Such information may include additional user and site related information, along with current and historical information about the user, the user's media, and the user's website. Such information may include, for example, user parameters (geography, experience, professional design state, etc.), user website parameters (e.g., template used or hints from the template), and additional information collected. This information may be stored in any of the repositories in CMS 50, as described in more detail below.

[0078] Reference is now made to Figure 4, which illustrates a typical user workflow of the system 200. In stage 1 (user preparation), a WBS user selects, typically via the WBS editor 30, the objects (images, layout pages, etc.) to be processed in a given task. Alternatively, the user may upload an image (after editing) to be used in the current website. The result is object A, selected for processing.

[0079] In stage 2 (system recommendations), the edit task handler 370 or site feature updater 400 may receive object A and provide it to an associated AI unit 310 whose ML model 317 may analyze and determine a set of "recommendations" B1-Bn that may improve (or provide processing parameters or additional information, as described further below) object A. It will be appreciated that the recommendations may take multiple forms, as discussed in more detail below. The ML model 317 may also recommend adding appropriate elements, such as images from an image repository in the CMS 50, generated text, etc.

[0080] In stage 3 (user feedback interaction), the interaction generator 315 may display the object A to the user along with its versions B1-Bn suggested via the associated handlers 370 or 400. The user then decides or derives a version C through interaction with the handlers 370 or 400. This decision or derivation provides explicit feedback to the system 200.

[0081] In stage 4 (further editing, interaction and site design), the user further edits version C (if necessary) to create version D which is integrated into the site S published by site generation system 40. This further editing provides implicit feedback to system 200. Such editing may include editing of the object itself (version C), the nature of the object (whether and how the object has been integrated into the website), operations performed on the object, information extracted from the object, or other edits related to the object (even if they do not directly change it).

[0082] It will be appreciated that the suggestion B may not be limited to the actual modified object (or set of objects). The output may also include, for example, a set of parameters used to process A to create the modified object B. For example, the image enhancement task AI unit 310 may suggest values ​​for a particular filter (e.g., brightness, contrast, etc.) rather than the new image B itself. The user may then modify these parameters. Similarly, in an image cropping task (or various segmentation tasks), the associated AI unit 310 may provide cropping parameters (X / Y range) that the user may modify, rather than providing the resulting cropped image.

[0083] Recommendation B may also be a set of specific editing steps (or instructions) that can be applied to object A to create a modified object B. The system 200 may allow the user to modify these steps rather than providing the results of their execution, or the activity unit AI unit 310 may itself generate these steps instead of a completed solution. When processing the editing steps, the ML feedback-based suggestion module 300 may utilize machine learning models from the fields of natural language processing and natural language generation, which may provide better results for the sequence of steps. Such natural language processing models may be used in addition to or instead of regular models.

[0084] It will be appreciated that the suggestions and feedback interactions in stages 1 and 2 can take multiple forms. For example, the ML feedback-based suggestion module 300 may suggest B, which may be, for example, a modified version of the image. The user may accept or reject the suggested B. The derived version C is then either B (if the changes are accepted) or A (if the changes are rejected).

[0085] In another implementation, the interaction generator 315 may display B together with a set of parameters for modification or other processing of A (e.g., cropping rectangle coordinates, face rectangle coordinates, or luminance filter settings). The user may accept these parameters (and the presented B) or may modify these parameters (e.g., move / resize cropping rectangle) to create a new version C. The ML feedback-based suggestion module 300 may also apply model-based generation of a sequence of processing steps or instructions.

[0086] The ML model 317 may also suggest modified versions of A (e.g., modified versions of an image). In this scenario, the system 200 may provide a native editor that allows the user to further modify B to create a preferred version C.

[0087] The ML feedback-based suggestion module 300 may also generate (and provide) multiple versions B1-Bn from which the user may select one (to determine version C). The ML feedback-based suggestion module 300 may also allow the user to reject all options (to generate additional options), merge options, or specify generation parameters for different versions (and re-run stage 1 to generate a new set of solutions).

[0088] The ML feedback-based suggestion module 300 may also utilize both implicit and explicit feedback, as described above, or may utilize only implicit feedback or only explicit feedback, as desired.

[0089] It will be appreciated that the system 200 may also combine the above feedback interaction options, allowing (for example) the user to accept, reject or edit a given proposed solution (or set of solutions). The system 200 may also provide specialized editing tools or environments that recognize the presence of multiple "layers" (original A, proposed B and currently edited C) or multiple solutions (B1...Bn). The feedback interaction tools or environments are not limited to simple editing environments (for image / layout editing or otherwise) but may also interact with the original activity unit AI unit 310 that generated B or utilize additional auxiliary models for the specific feedback interaction task.

[0090] For example, the system 200 may provide (for image-related tasks) the use of specific brush or region marking tools that allow some parts of proposal B (defined by brush shapes / masks or marked regions) to revert to values ​​in prior model A, i.e., to essentially "undo the brush."

[0091] Conversely, the system 200 may provide a similar brush or region-marking tool that can be utilized on a copy of A to apply a solution B suggested by the model to specific regions, essentially a “brush redo.” Such a brush may be useful, for example, when a user is not entirely satisfied with the proposed recommendation B, but still wants to use specific portions of B in creating C.

[0092] Similarly, the system 200 may provide an option to mark specific regions of A and activate or apply the activity-unit AI unit 310 to them. Such an option may be available, for example, in an object recognition task where the relevant activity-unit AI unit 310 fails to detect objects in some regions. This option may allow the user to activate the relevant AI unit 310 for the regions marked by the user that may be used in addition to or instead of the proposed solution (either fully or in specific regions) and may provide further hints (serving as training information) to the AI ​​unit 310. The system 200 may use the original model, may use the original model with specific execution parameters, or may use a different model that is particularly suitable for detecting objects in the regions marked by the user.

[0093] When reviewing multiple proposed layouts, system 200 may provide tools for merging sub-regions from different models. System 200 may utilize specific machine learning models to merge multiple layout elements into one coherent layout, and may do so using dynamic layout techniques, as described in U.S. Patent No. 10,185,703, which is commonly owned by the applicant and incorporated herein by reference.

[0094] It will be appreciated that the system 200 may also provide comparison tools, for example, allowing a user to overlay two or more versions of an object (A, B or B1...Bn and the current C) and flip between them or visually compare them to assist the user in editing C.

[0095] The system 200 may provide multiple different feedback interactions (or user interfaces) for different users, or may provide multiple interactions with the same user, which may depend on the object itself (e.g., simple vs. complex object), the results from the interaction generator 315 (i.e., the proposed solution), user parameters (e.g., the user's experience level, the number of times the user has performed this particular task, etc.), site parameters, or other WBS parameters.

[0096] System 200 may then retrain activity-based AI unit 310 using pair [A,C] as an example of a desired result C based on the original object A, or the parameters used to generate C, similar to the parameterized filter example above. Subsequently, system 200 may collect additional information (including, for example, D's information, WBS information, and user activity information) and may use this information for further training, as discussed in more detail below.

[0097] Alternatively, instead of training existing models with additional training data, system 200 may generate new models (based on the collected training data) and evaluate their performance against existing models, as described in more detail below.

[0098] The ML model 317 may be any model appropriate for the task or activity to be realized by each activity unit AI unit 310. Machine learning models are known in the art and are typically some form of neural network. The term describes the ability of a system to recognize patterns and provide solution concepts based on existing algorithms and data sets. The more it is trained, the more knowledge it develops.

[0099] The ML model 317 may be a learning model (supervised or unsupervised). As specific examples, such algorithms may be prediction (e.g., linear regression) algorithms, classification (e.g., decision trees, k-nearest neighbors) algorithms, time series prediction (e.g., regression-based) algorithms, association algorithms, clustering algorithms (e.g., K-means clustering, Gaussian mixture models, DBscan), or Bayesian methods (e.g., Naive Bayes, Bayesian model averaging, Bayesian adaptive trials), image-to-image models (e.g., FCN, PSPNet, U-Net), sequence-to-sequence models (e.g., RNN, LSTM, BERT, autoencoders), or generative models (e.g., GANs).

[0100] Alternatively, the ML model 317 may implement statistical algorithms such as dimensionality reduction, hypothesis testing, one-way ANOVA (Analysis of Variance) testing, principal component analysis, joint analysis, neural networks, support vector machines, decision trees (including random forests), ensemble methods, and other techniques. Other ML models 317 may be generative models (such as GANs (Generative Adversarial Networks) or autoencoders) for generating solution elements (such as elements and media objects of a website building system).

[0101] In most embodiments, the ML models 317 may undergo a training or learning phase before being released to the production or runtime phase, or may begin processing with models from existing systems or models. During the training or learning phase, the ML models 317 may be tuned to focus on specific variables, reduce error margins, or otherwise optimize their performance. The ML models 317 may initially receive inputs from a wide range of data, such as website building system CMS data, site information (including components, structure, layout, formatting, and other information), external information, user information, collected editing history, collected business information, etc.

[0102] In other embodiments, when appropriate for a particular task, one or more of the ML models 317 may be implemented by a rule-based system, such as an expert system incorporating multiple AI techniques or a hybrid intelligent system, as described in the Wikipedia articles entitled "Rule-Based System" and "Hybrid Intelligent System" at en.wikipedia.com.

[0103] During runtime, the trained ML models 317 may provide suggestions for improvement that their associated interaction generators 315 may provide to the WBS users 61 or 62. As described above, at any appropriate time, the ML models 317 may be updated with information from any or all of the feedback handlers 370, 340, 380, or 390.

[0104] Reference is made to FIG. 5, which illustrates the training and continuing setup of one activity-based AI unit 310. As described herein above, each machine learning model 317 may be per activity, and the ML models 317 may have sub-models and interact with each other. The models 317 may also be trained with different levels of data, such as data from all users (“universal data”), data from a community of users (“community data”), or data from a single user. It will be appreciated that the existence of multiple models 317 for multiple tasks and / or multiple groupings of users allows for multi-task learning and benefits from analysis of commonalities and differences between different tasks. Furthermore, the models 317 may support each other, for example, a face detection model and an object detection model may provide support (including via transfer learning) to an image cropping model or a portrait / object segmentation model. This interaction between the models 317 may also enable the system 200 to provide supporting information or explanations to its proposed solutions, for example, indicating the underlying detected faces and objects that support a given image cropping solution, as described herein below in more detail.

[0105] Reference is made to Fig. 6, which illustrates an exemplary method implemented by one type of ML model 317 for the tasks of layout understanding and component grouping. The ML model 317 of Fig. 5 may be RNN (Recurrent Neural Network) or LSTM (Long Short-Term Memory) based and may initially extract the web elements to be processed (step 210). In step 212, the ML model 317 may extract relevant features of the web elements or components, such as geometric properties (position, size and depth on the page), type and other metadata (text font and size, image embeddings, content (or parts thereof), links to other components, template information, edit history, associated code, etc.).

[0106] These features are then processed by a multi-layered RNN / LSTM model to generate embeddings for each component. These embeddings are then clustered into groups (step 214). Pages may be labeled by manually grouping elements that need to be kept together when rendering the site in a new aspect ratio (such as a mobile view) or a different target display. These labeled pages comprise the ground truth on which an ML model 317 is trained via supervised learning to better identify groups of elements. The RNN / LSTM model is trained using a contrastive or triplet loss function, whereby the ground truth group labels are processed into an adjacency matrix (the target signal) and the model's output embeddings are used to generate a distance matrix between element pairs (the predicted signal). Because each site is different, in step 216 the resulting layout definitions (or other classifications) may be clustered by a clustering model to define the best groups to use for the site being processed.

[0107] It will be appreciated that the system 200 may utilize multiple sources of training data and feedback, as shown in FIG. 5 (to which reference is now made again), to support the building and training of machine learning models for various WBS tasks. These are available for ongoing online training as well as initial training of the models. Training data may include explicit training by the WBS vendor staff 61 (e.g., via an interactive UI) and training via mass data import (via a suitable import interface). This import may be from an external data source (such as an existing repository of images in the CMS 50 associated with matching cropping area definitions). It may also be training data derived from existing user material in the WBS (e.g., those cropping area definitions created by users along with images used in the WBS). An implicit feedback handler 340 may provide this data to each ML model 317, as described in more detail below.

[0108] Initial training may also come from staff external to the WBS vendor staff 61, such as hired external staff (perhaps subject matter experts / professionals) or crowdsourced staff (using a crowdsourcing platform such as Amazon's Mechanical Turk).

[0109] Training may also come from existing users of the WBS via the system 200 in the form of initiated interactions requesting specific feedback on specific data samples as processed by the explicit feedback handler 320. These may be data samples specifically selected to train a particular ML model 317. The system 200 may also utilize active learning techniques to select interaction strategies that provide the highest training value, and the implicit feedback analyzer 345 may also analyze user information and user website information to find the best users to query, as described in more detail below. It will be appreciated that the implicit feedback analyzer 345 may take into account information maintained regarding a user's past responsiveness, performance, and answer quality. The analysis may also take into account the user's proficiency in performing a particular task (for which a particular model is trained). Such proficiency analysis may be based on (for example) an analysis of the user's website. Additionally, the implicit feedback may include other activities within the WBS 5, such as save / discard decisions, publishing actions, designer site browsing patterns, etc.

[0110] The system 200 may also encourage users to provide monetary or other bonuses for feedback, such as premium package upgrades for use with the system or other in-system products (e.g., enhanced website templates, higher capacity limits, etc.).

[0111] It will be appreciated that the above-mentioned sources are in addition to the training information collected overall by the implicit feedback handler 340 and the explicit feedback handler 320, as described in more detail below. Additionally, it will be appreciated that for the training cases described above, the vendor staff 61 providing the training may provide additional labeling or feedback information that is not required as part of the normal feedback interaction.

[0112] As mentioned above, the system 200 may extract feedback information from additional sources other than the user feedback interactions themselves, and such information may be more indicative, representative or appropriate than the normal feedback provided by the user. This inferred feedback is referred to as "implicit feedback" in contrast to "explicit" feedback that is explicitly provided. Furthermore, it will be appreciated that the implicit feedback may be generated by the implicit feedback handler 320 at any suitable time, such as periodically, whenever the user makes changes to the website, or whenever the user publishes the website. The implicit feedback handler 320 may collect the changes, analyze them, and provide them to the associated AI units 310 to update their models appropriately.

[0113] For example, the system 200 may assist the user in importing an image by providing an enhanced version of the image using an image enhancement ML model 317, which may provide automatic enhancement of image brightness, contrast, etc. The user may be asked whether he prefers the original image or the enhanced image, and the user's decision may be used to train the model.

[0114] Alternatively, the system 200 may provide the user with specific editing tools to modify the applied brightness, contrast, and other modifications (also known as "filters"). The system 200 may provide model-based default settings for these editing tools, and the user may modify these settings. The user-modified settings may then be used as user feedback to train the image enhancement ML model 317.

[0115] However, the user may continue editing the site and may further modify the imported images during ongoing site editing. The user may also re-import the original image and modify some of the parameters (e.g., via an image editor built into the WBS editor 30) or import a different image altogether (i.e., drop the original imported image). The system 200 may analyze the new image and use it to provide additional or updated training information for the associated model 317. For example, the implicit feedback analyzer 345 may analyze the luminance levels in the final modified image and the selected initial image (or selected initial luminance value) and use that information about the luminance changes to retrain the image enhancement ML model 317. The implicit feedback handler 340 and its analyzer 345 may also extract additional features for training that are not directly related to image content, such as images used on important pages (e.g., the site home page), images used on less important pages, or images not used at all. Similarly, the implicit feedback analyzer 345 may analyze the placement of an image on the page in which it appears (e.g., as "above the fold," "below the fold," or as an arbitrary image) to determine its importance, all of which may serve as model features for the ML model 317.

[0116] As another example, at an appropriate time, the Explicit Feedback Analyzer 325 may process an existing page A to determine an improved layout B suitable for that page (e.g., by modifying the placement, size, and type of components). The analysis may include reviewing the components of the selection, multiple possible layouts (e.g., B1-B4) presented to the user, and the user's final selection (e.g., B3). The user's explicit selection is explicit feedback. The Explicit Feedback Analyzer 325 may utilize all the explicit feedback or only a portion of it, and may merge or summarize the changes to provide better input for retraining the ML model 317.

[0117] As described herein above, the user may subsequently edit the "accepted" C version (which was B3) and perform additional changes to create a subsequent version D of the page. The implicit feedback analyzer 345 may track these subsequent changes and use the D version as an additional feedback source for retraining the corresponding ML model 317. If the user edits page B3 before accepting it, the explicit feedback analyzer 325 may analyze that information as implicit feedback.

[0118] It will be appreciated that the implicit feedback analyzer 345 may analyze the differences between the C version of the page (the B3 version initially selected by the user) and the D version (the one that was finally published) to isolate the specific differences that are related to the model and features used by it. Such analysis may include component difference analysis, for example, using the differences to train changes to components present in B3. Such analysis may also include attribute specific analysis, i.e., only considering attributes such as position and size and not other attributes. The implicit feedback analyzer 345 may then provide the results of the analysis to the appropriate ML model 317 for retraining.

[0119] Moreover, it will be appreciated that such analysis can also be performed by the implicit feedback analyzer 345 for other types of tasks and models available within the WBS.

[0120] Implicit feedback may also be collected from sources external to the site, such as feedback generated for other sites of the same designer (user) or similarly located users (e.g., a professional graphical designer in Japan), as described in more detail below.

[0121] Further, it should be appreciated that implicit feedback may be collected at various points during the site creation workflow: it may be collected online, i.e., as soon as a user modifies a site element having a configuration that includes machine learning model-based edits, or it may be collected based on specific events, for example, each time a processed site element is edited or replaced.

[0122] Implicit feedback may also be collected at the end of an editing session, assuming that the user has created a more finalized version of the web page that better reflects the user's understanding. This assumption may not be correct in some cases, as the user may end their editing session without finalizing the page (or a given object therein) and attempt to continue working in the next editing session.

[0123] It may also be collected when the site is published, since this is the “final” version of the webpage and the objects in it are likely to be published multiple times with multiple versions. For example, in the improved page layout example above, this may be the best point to sample implicit changes. In this scenario, implicit feedback collection may help train the model with the correct values ​​even if (for example) the user changes their mind.

[0124] The system 200 may also collect implicit feedback for multiple tasks and models simultaneously, since a given site may contain multiple objects generated using multiple ML models 317. Further edits (which generate feedback) may involve multiple such objects.

[0125] It will be appreciated that for many types of systems and tasks, a universal ML model 317 may be applicable. This may be the case, for example, when the decisions made by all users in the system's target audience are similar. One example is classifying email as spam or non-spam, which is typically uniform for all people reviewing a given email. In this situation, it is rare for one person to classify a given email as spam and another person to classify it as non-spam.

[0126] Such a model may be used for all users, but may be augmented with additional model features that provide sufficient discrimination between different user categories. For example, features including the user's language and the language of the email may be highly relevant for spam detection, since unprompted emails in a language different from the user's language are very likely to be spam.

[0127] In general, it will be understood that for many of the tasks performed in a WBS system, the desired outcome is not universal. The desired outcome for a given user may depend heavily on local or personal preferences, community-related design preferences (e.g., country, region, designer, professional experience / knowledge, industry, etc.). For example, professional and non-professional designers often have different preferences regarding how some of the tasks (site design or image enhancement) should be performed. As another example, the preferences of a designer who specializes in law firm websites may be significantly different from the preferences of a designer who specializes in music performance websites.

[0128] Therefore, training a single universal activity-based AI unit 310 (for any particular task) with user feedback may not provide optimal results for all users, and it may be better to train separate models for communities with completely separate preferences.

[0129] However, the system 200 may utilize a universal activity-based AI unit 310 with a universal ML model 317 for some tasks. The universal AI unit 310 may utilize additional information available in the WBS, i.e., the system 200 may utilize not only user feedback but also additional existing or analyzed WBS information (as described above with respect to the implicit feedback analyzer 345) to train the universal ML model 317. The implicit feedback analyzer 345 may also extract features based on user parameters (geography, experience, professional design state, etc.), user website, user activity, and additional information collected (available in the WBS CMS 50 or elsewhere). A suitable activity-based AI unit 310 is best to handle cases common to all communities / users.

[0130] System 200 may also utilize additional group-based or community-based ML models 317 for different populations (e.g., for all professional developers, for all lawyer website designers, for designers who specialize in Japanese design, etc.). System 200 may train associated ML models 317 and associated group-based ML models 317. The training data may include additional WBS information (as described herein above) along with collected user feedback. System 200 may also utilize personal models 317 (per user and per task) for tasks whose solutions are highly individualized.

[0131] In fact, groups may overlap, and thus the system 200 may train multiple (group-specific) models for each given collected feedback. The community handler 380 may first split the training flow (i.e., send training data to multiple models) and then integrate the recommended solutions generated by multiple ML models 317 in multiple ways, for example, creating a merged solution and selecting the best solution among these provided by multiple activity-per-activity AI units 310. The definition of a group may be unique for a given task (and its associated models) or may be shared among multiple tasks.

[0132] The community handler 380 may allow groups to be manually managed (i.e., adding and removing group definitions and associated models as needed). Alternatively, the community handler 380 may automatically identify sub-segments of the general population that have similar tastes (based on discriminatory models or other techniques such as clustering analysis). To determine which users belong to which communities, the community handler 380 may infer a user profile for a particular user from the user's editing history and / or information about the user's website. The user profile may also include business intelligence about the user's website. For example, an art website accessed primarily from Japan is most likely (but not necessarily) Japanese-style art.

[0133] The community handler 380 may also publish some group definitions to the user community and further allow users to declare their affiliation with one or more groups. For example, the system 200 may support a "Baroque design style" group (with associated activity unit AI unit 310) and allow users who want to promote Baroque design style to join the group. Furthermore, the community handler 380 may avoid publishing some groups, such as "Professional Users" and "Non-Professional Users" groups, that are automatically assigned to users based on the declared status or analysis of the user's site.

[0134] As discussed above, integrating feedback into the WBS workflow improves the probability that users will provide feedback for a particular model and improves the quality of the feedback provided. This improvement can result from the integration of either explicit or implicit feedback.

[0135] In particular, for some tasks, feedback interactions may be fully integrated with the WBS workflow such that users transparently provide feedback when performing a task (e.g., by accepting or rejecting a proposed solution). Thus, the response rate to such feedback interactions can be as close to 100% as possible, and users are likely to provide the correct response (given their preferences and knowledge at the time).

[0136] However, in some cases, the user may not be able to provide feedback or may provide inaccurate feedback. For example, for an image enhancement task where a corresponding ML model 317 may provide initial settings for image filters, the user may select junk values ​​(e.g., with filter parameters having a range of values ​​from 0 to 100, set all values ​​to values ​​close to 0 or close to 100). As another example, in an object detection or segmentation task, the user may mark regions that do not contain relevant objects (or do not contain any objects at all).

[0137] Such failure to provide feedback may be a localized problem with a particular task or a recurrent problem with some users, e.g., some users are “repeat offenders” who do not provide useful feedback most of the time (due to indifference, lack of understanding, negligence, or any other reason). As an example, a user may have a strong preference (for whatever reason) to “do their own thing”. Such a user may disqualify several model suggestions B1-Bn by rating all suggestions as bad. Alternatively, such a user may provide any low-quality or erroneous feedback (e.g., marking a face segmentation cropping rectangle that does not contain any face). The user may then proceed to use different versions of the target object, which may or may not be similar to any of the proposed Bn solutions.

[0138] In such cases, the system 200 should detect such users and generally, instead of using their explicit feedback, should completely ignore their (incorrect) explicit feedback and use the implicit feedback (collected from their sites).

[0139] This problem may be more pronounced in cases where the output of ML model 317 and the desired feedback do not visually contribute to the objects in the edited website. For example, an embodiment of system 200 may include a face detection model that is used for internal purposes (e.g., as an input to other models, such as determining image regions of interest in an image cropping model) but does not directly affect the objects in the edited website. System 200 may still collect feedback for the face detection model, for example, by detecting faces in imported images and asking the user to correct face rectangles detected in these images. In such cases, the user is likely to provide inaccurate or low-quality feedback without visually harming the website.

[0140] For example, this problem may occur when the ML feedback-based suggestion module 300 initiates a feedback interaction on non-task data as part of an active learning system, where the ML feedback-based suggestion module 300 seeks feedback on data / objects that are not related to the actual user task. In such a scenario, the ML feedback-based suggestion module 300 may not be able to collect any implicit feedback and may only utilize (possibly invalid or erroneous) explicit feedback. The ML feedback-based suggestion module 300 may also handle such cases of missing or erroneous feedback by utilizing one or more auxiliary models. For example, one or both of the explicit feedback analyzer 325 and the activity-based AI unit 310 may utilize a response evaluator 330 and / or a user evaluator 332, which form part of the feedback system 305, as described in more detail below. These units may evaluate the quality of the feedback response and the user (respectively). For example, the response evaluator 330 may analyze the quality and relevance of a response from a given user. Theoretically, a given user may provide excellent feedback on photo corrections (if he is a professional photographer), but provide poor quality / inaccurate feedback on layout-related issues (if he is a very bad graphic designer).

[0141] The response evaluator 330 may compare the user's feedback with feedback provided by other similarly located users (e.g., as determined by clustering analysis). If the user feedback is "extremely far from other clusters," the user may be defined as an undesirable outlier. The response evaluator 330 may also search for whether the user uses extreme values ​​in settings where these values ​​are meaningless (e.g., setting brightness to 100% in photo correction parameters). Similarly, the response evaluator 330 may review the output based on the user's settings / definitions. If the output is evaluated as very poor using an external (possibly objective) evaluator, such as the layout quality evaluator 47 described in U.S. Patent No. 9,747,258, entitled "System and Method for the Creation and use of Visually-Diverse High-Quality Dynamic Layouts," issued on August 29, 2017, which is co-owned by the applicant and incorporated herein by reference, the user may be defined as an outlier.

[0142] The user evaluation component 332 may utilize a similar type of analysis on previous answers of the same user rather than comparing to a group or community of users.

[0143] It will be appreciated that both evaluators 330 and 332 may take into account user feedback and its quality (both explicit and implicit). Additional features may be derived through specific analysis and rules that look for bad responses (as described above), such as responses that use threshold filter settings or responses that have user-specified face detection rectangles that do not contain a face image.

[0144] It should be appreciated that both evaluators 330 and 332 may also take into account other parameters such as user response history, user ratings (professional designers vs. regular designers), user skill level, the complexity of the user's site, user parameters (geography, etc.) or site parameters, etc. In particular, evaluators 330 and 332 may evaluate user responses / feedback to user choices or final site versions (such as the "C" and "D" level versions above) in previous uses of the same task (e.g., does the current user feedback match past feedback for the same task?).

[0145] In one embodiment, both evaluators 330 and 332 may also utilize some or all of human evaluation (labeling) or feedback, e.g., via interaction with WBS vendor staff 61 or crowd-sourced evaluators. Such human labeling may be useful, for example, for initial training of auxiliary models.

[0146] It will be appreciated that these auxiliary evaluators 330 and 332 may also evaluate the user's skill level as it relates to a particular task, related tasks, overall site design (e.g., how good is the website created by this designer), or other factors and features extracted from the WBS information. As noted above, "quality" and "skill" are not absolute indicators, since the "best" result for a given task may depend on the user community, culture, skill level, etc.

[0147] Both evaluators 330 and 332 may also evaluate "soft" parameters of the interaction. Such parameters may include the time it took the user to respond, the level of certainty of the response (e.g., did the user select a response and change it before pressing "OK"), and / or biometric parameters of the interaction (e.g., mouse movements, hand movements, eye movements, measurable biometric / biological user parameters or other parameters).

[0148] As described herein above, the Explicit Feedback Analyzer 325 or the activity-based AI unit 310 may apply the Response Evaluator 330 to evaluate the user's feedback (for validity, accuracy, etc.) and determine if and how the response should be used (e.g., for training), if any. The Response Evaluator 330 may use the feedback ratings to disqualify feedback data whose ratings are below a certain threshold. The Response Evaluator 330 may also use the ratings to assign weights to particular training data samples (i.e., user feedback) when working with the underlying model that supports the weight assignment.

[0149] The user evaluator 332 may evaluate the user himself and provide an ongoing evaluation of whether the user is likely to provide useful feedback. The activity-based AI unit 310 may invoke the response evaluator 332 before entering into a feedback interaction to determine whether to provide a feedback interaction or perhaps no feedback interaction or model solution (e.g., for a user who consistently rejects solutions provided by a particular model).

[0150] It will be appreciated that because system 200 trains its ML model 317 by user feedback, model 317 should generally follow changes in user tastes and preferences. However, system 200 may detect that user feedback (both explicit and implicit) may reflect an increasing loss of accuracy in the solutions generated by ML model 317 (requiring the user to implement more significant changes to suggested solution B to obtain the desired result). It may also be detected when the user consistently rejects suggested results, even without editing these results.

[0151] Such loss of accuracy can result from a combination of errors introduced into the ML feedback-based suggestion module 300 during design and development, changes in the tastes and preferences of particular user groups or segments, and global changes in user tastes and preferences, including changes that require analysis of features not previously provided by the activity-based AI unit 310. Such global changes are known as concept drift.

[0152] As described herein above, the results of the model may depend on the classification of users into groups or segments, and the particular preferences of each group by the community handler 380. However, both the preferences of particular groups and the exact assignment of users to groups may change over time.

[0153] The activity-based AI units 310 may be able to correct some of the above over time. However, some issues (such as implementation bugs) may not be self-correcting. In particular, the activity-based AI units 310 may not be able to detect and handle cases where the definitions of features they use need to be changed (such as whether the definitions of certain features should be changed or whether new features should be created).

[0154] A concrete example of this is a user community that splits over time into subpopulations with very different preferences for some tasks: if new features that distinguish these subpopulations need to be added, a particular activity-based AI unit 310 may not be able to recommend such changes or even prescribe such new features on its own.

[0155] Vendor feedback handler 390 (FIG. 3) may address all of the above by reporting key metrics and performance from system 200 to WBS vendor 61 staff. Such reports may provide necessary feedback to WBS vendor staff and may enable them to initiate human-guided corrective action as necessary. Vendor feedback handler 390 may also provide additional information to WBS vendor 61, such as alerts and statistics (possibly including user / site / object information related to specific issues).

[0156] As described herein above, the ML feedback-based suggestion module 300 may also utilize feedback interactions as part of editing related tasks. It will be appreciated that there are several application areas centered around processing only a single object / component (such as an image) and additional objects related to the single object (such as a cropping rectangle specification or a product descriptor for a product identified in the image).

[0157] For some of the tasks, all of the feedback collection techniques already mentioned above (e.g., based on positioning of generated images at the site) may be relevant to each application area, as described herein below, but there may also be specific activity-based AI units 310 and implicit feedback collection techniques.

[0158] The edit task handler 370 (FIG. 3) may use this collected information as appropriate to enhance the edit tasks provided by the WBS editor 30. Reference is now made to FIG. 7, which illustrates the components of the edit task handler 370. The edit task handler 370 may comprise an image resolution enhancer 371, a face detector 372, a portrait segmenter 373, an object segmenter 374, an image cropper 375, an image enhancer 376, a logo creator 377, and a text generator 378. It will be appreciated that each component of the edit task handler 370 may work with its own activity-based AI unit 310 in the ML feedback based suggestion module 300 and may utilize both explicit and implicit feedback information.

[0159] It will be appreciated that when a user uploads small images to a larger component that contains them, the images may be pixelated and noisy. To solve this problem, the image resolution enhancer 371 may utilize the super-resolution AI unit 310 to upscale the pixels of the image, once enlarged, to create a smooth, non-pixelated result. The image enhancer 371 may also combine elements of deblocking and upscaling.

[0160] The image resolution enhancer 371 may trigger the super-resolution AI unit 310 under several conditions, such as upon import, whenever a container is resized during page editing, or when the image resolution enhancer 371 detects a resolution difference (e.g., when the required resolution is significantly greater than the input resolution, e.g., four times greater).

[0161] It should be appreciated that the image resolution enhancer 371 may utilize multiple feedback interactions such as any of the following actions or a combination thereof: The image resolution enhancer 317 may display the original image using a set of images generated by one or more models and allow the user to select which image to use. These multiple images may be generated by multiple activity-based AI units 310, multiple algorithms, and multiple settings for a particular model / algorithm. The image resolution enhancer 371 may also display the image generated by the model and allow the user to edit it and coordinate with the explicit feedback handler 320 accordingly.

[0162] It will be appreciated that for implicit feedback, the image resolution enhancer 371 may utilize subsequent versions of the image (e.g., from subsequent versions on the site or from published versions). The system 200 may limit such utilization to cases where the user has further modified the image after the initial feedback interaction. The image resolution enhancer 371 may also utilize additional sources of information for the model, such as image placement, user parameters, and image editing history (as described herein above), in corresponding cooperation with the implicit feedback handler 340.

[0163] The face detector 372 may recognize the appearance of one or more faces in a given image. The face detector 372 may perform detection only, or may include face recognition (i.e., determining the identity of a detected face image). In this scenario, the face detection AI unit 310 may generate a frame (e.g., a rectangle) surrounding the detected face, but may also generate additional information (e.g., a more precise surrounding curve or polygon).

[0164] It will be appreciated that feedback interactions for the face detector 372 may allow the user to accept, reject or edit frames surrounding detected faces via the explicit feedback handler 320. Editing may include allowing the user to remove frames or add new frames (i.e., manually identify additional faces) as well as allowing the user to move / resize frames to correct detection.

[0165] One problem with the face detection AI unit 310 is that the detected faces are not always directly used in the page editing process (thus reducing the quality of the feedback). As mentioned above, the system 200 may utilize the auxiliary ML model 317 to verify that the marked area (e.g., modified by the user) contains a face so that bad feedback can be disqualified. Alternatively or additionally, the ML model 317 may utilize implicit or explicit feedback collected in a follow-up process to determine whether the user clipped or cropped the image according to the face rectangle.

[0166] The output of the face detector 372 may typically be used as an input to an image cropper 375, as described in more detail below. If the face detector 372 detects a face with a probability above a minimum threshold, the output may be used as an indicator for redirection to a portrait segmenter 373, as described in more detail below.

[0167] The portrait segmenter 373 may remove background from portrait images to help users create homogenous images containing multiple portraits. Such images may be required for part of a website such as "Our Team" which may be part of a manually edited site or an automatically generated site. In this scenario, the portrait segmentation AI unit 310 may be trained to distinguish people from the background, including processing full body images. The portrait segmentation AI unit 310 may generate a surrounding frame (rectangle) or a more accurate surrounding curve / polygon.

[0168] It will be appreciated that the portrait segmenter 373 may include editing tools that allow the user to define or edit the segmentation boundary. Such tools may (for example) allow the user to utilize a brush to clean up areas of the photo so that the only thing not brushed is the foreground portrait. The tools may also allow the user to directly interact with the segmentation polygon by moving / inserting / deleting nodes of the polygon. It will be appreciated that the tools may provide feedback interaction.

[0169] The portrait segmenter 373 may then process the detected portrait and the remainder of the image ("background") in multiple ways, such as by graying out the existing background to make the portrait stand out when viewed, by replacing the background (similar to a video blue screen) with a simpler one such as a solid color, a simple gradient or pattern, and / or by identifying a matching image to be used as an alternative background. The auxiliary background image selection AI unit 310 may be trained to select such images based on the layout of the detected portraits, the user's preferences, and other factors of the user's site. In particular, the suggested photos may be selected to match the position of the portrait in the original image. The portrait segmenter 373 may provide multiple such potential background images for selection.

[0170] The portrait segmentation unit 373 may also provide tools that allow the user to edit the new combined [image+portrait] or change the background image or pattern. These may be part of explicit user feedback during normal site editing or follow-up editing (providing additional implicit feedback). This feedback may be used to train the primary portrait recognition AI unit 310 together with the auxiliary background image selection AI unit 310.

[0171] The object segmenter 374 has functionality similar to the portrait segmenter 373 and may be part of the WBS or may otherwise be integrated into an electronic commerce system that may operate in conjunction with the WBS.

[0172] The object segmenter 374 may recognize multiple object types (possibly using multiple specialized object segmentation AI units 310 for different object classes). Such recognition may be performed at multiple levels, such as recognizing the outline of an object, recognizing an object type (e.g., a T-shirt), recognizing a specific object (a T-shirt made by vendor X) or a specific attribute of an object (a blue-white colored T-shirt in size medium from series Y). Such recognition may also require a catalog of objects and attributes for specific object identification and attribute detection.

[0173] The detection may be automated or may be user-assisted (e.g., by the user pointing to a specific object to be recognized or by providing an approximate outline as a starting point for detection), and the object segmenter 374 may be used to directly locate a product page that includes multiple fields on a given product page.

[0174] For the object segmentation portion 374, the feedback process may include feedback provided by the user identifying objects or specifying attribute values ​​and tags, as well as interactions (such as changing segmentation rules) as described for the portrait segmentation portion 373. Additional feedback may be generated by edits made by the user to the generated product page.

[0175] The image cropper 375 may provide automatic cropping of images to select "important" or "interesting" portions of images for use in an edited page. It will be appreciated that these concepts (important / interesting) may be subjectively defined, and the image cropper 375 may implement user-specific image cropping AI units 310 in addition to a general, universal image cropping AI unit 310. The image cropper 375 may utilize image cropping AI units 310 that are specific to an industry or other subgroup. For example, for city street images, the sub-area importance ratings for a fashion design website may be quite different from those used for a transportation planning website.

[0176] The image cropper 375 may create a "heat map" of the image and recommend the best cropping rectangle for the image based on the main focus and objects in the photo. The image crop AI unit 310 may utilize inputs from other AI units 310 as described above for face detection, portrait segmentation, and object segmentation (including identifying object types and selecting more relevant or important objects). The image crop AI unit 310 may display inputs from these supporting AI units 310 (e.g., displaying a rectangle enclosing a detected face) as part of the image crop preview (described in more detail below) to assist the user in their decisions and feedback.

[0177] The image cropper AI unit 310 (and part of the underlying supporting model) may also take into account the expected use of the cropped image (in a manually edited or automatically generated site), which may be, for example, information related to the design and layout of the target page, or information related to the e-commerce product page that contains the final image.

[0178] The image cropper AI unit 310 may generate multiple possible regions. This may allow the user to select the best region to utilize or support systems and scenarios that support multiple cropped sub-images (e.g., by creating multiple image gallery components from multiple cropped sub-images). Even in this latter scenario, the user may still want to select a subset of the multiple proposed regions to use. The multi-region solution may also be relevant in the context of a site generation system that can utilize multiple (and possibly correlated) portions of an image in different locations in a proposed layout.

[0179] The image cropper 375 may display a preview of the cropping options in context, for example within a particular product page that contains the cropped image, and allow feedback interaction (and editing) to take place within this preview.

[0180] For feedback interaction, the image cropper 375 may display a preview of the image along with the suggested cropping rectangle. The display may be standalone or may be created in context, for example, within an e-commerce product page that contains the cropped image (showing multiple versions of the page), and may allow feedback interaction (and editing) to take place within this preview. The image cropper 375 may then allow the user to move or resize the suggested frames, select specific frames, add frames, or remove frames.

[0181] The image cropper 375 may subsequently collect follow-up (i.e., implicit) feedback based on the actual use of the image and any changes made to its zooming and cropping as the user edits the page.

[0182] The image cropper 375 may also collect feedback from users who manually place overlapping WBS elements (e.g., text or small images) in areas of the image used as background. In this scenario, the image cropper 375 may infer that the overlapping areas are areas of less importance and use that information to train its image cropping model 317. Such feedback can be collected from images cropped by the image cropper 375, but also from images that have been manually cropped (or not cropped at all) outside of the system 200. Such feedback can serve as a source for large-scale initial supervised learning for the image cropper AI unit 310, even before automatic cropping facilities are made available to users.

[0183] It will be appreciated that multiple users may treat the same image (or very similar images) differently, for example selecting different areas of the image in which to place other objects. This may be somewhat arbitrary (e.g., due to different users having different priorities or preferences) or may be related to various parameters, users or work scenarios. The image cropper 375 may provide specific parameters or users / scenario as additional features for training the learning algorithm.

[0184] Further, it will be appreciated that the image cropping AI unit 310 may also be utilized outside the realm of normal image cropping. For example, the site generation system 40 (as described above) may utilize the image cropping AI unit 310 in selecting which layout elements should overlap an image. Thus, the image cropping AI unit 310 may perform an initial analysis to determine important and non-important areas of a background image. The results of this analysis may be provided to the site generation system 40 and used to determine which layout elements to use in generating the site and where to place them (e.g., so that they overlap non-important portions of the background image). Alternatively, the image cropping AI unit 310's models may be integrated with ML models or rule-based engines used by the site generation system 40 (so that cropping analysis and site generation are performed together).

[0185] The image enhancement unit 376 may provide the user with the option to enhance the user's image by automatically changing (as one example) the brightness, contrast, saturation, and other characteristics of the user's image. In general, the user interface of the WBS editor 30 may allow the user to specify values ​​for brightness, contrast, saturation, sharpness, etc., via a number of levers, rotary buttons, etc. The image enhancement unit 376 may provide an "auto" option (for each setting, a combination of settings, or a combination of all settings) and may allow the user to correct the settings that were generated / modified either during the initial import of the image or during normal page or image editing (e.g., a pop-up user interface (UI)).

[0186] It will be appreciated that the user feedback collected for the image enhancement section 376 may be based on changes made by the user to various settings. The image enhancement section 376 may train the image enhancement AI unit 310 based on user changes at any point in time, such as when using automatic settings (for example) or when using manual settings. The image enhancement section 376 may also omit boundary values ​​of user settings from the training data when these values ​​are clearly wrong (such as 100% brightness) or as determined by the user evaluation model 332.

[0187] It will be appreciated that for the image enhancement unit 376, the interaction generation unit 315 may provide suggested settings for various scenarios, or it may directly modify the underlying image.

[0188] The logo creator 377 may provide the site creator with automatically generated logo suggestions. The logo creator 377 may utilize the logo creation AI unit 310 to create a logo based on information known about the user (text, images, etc.) from, for example, information collected by the site creation system 40, and information collected from user responses, analytics, and internal and external sources. Such information may include, for example, the business name, slogan, photos related to the business, existing offline logos, or logos used on existing non-WBS sites. Thus, the logo suggestions may include text (including specific fonts, sizes, and other parameters), images, graphic art, etc. The generated logo may combine multiple such elements.

[0189] The logo creator 377 may typically generate multiple suggestions and allow the user to evaluate them, for example by rating each suggestion separately, or (preferably) by rating pairs of suggestions relatively (i.e., one of the pair looks better). These ratings and the final selection made by the user may then be used to train the logo creation activity unit AI unit 310.

[0190] It will be appreciated that there may be differences between the scores provided by the designer and the scores provided by the user, and thus the data and training provided by actual users may be more representative than the initial training by professional designers.

[0191] The text generation and editing portion 378 may include a text generation AI unit 310 that may provide a structured generative editing environment for the media and text elements embedded in the site's components. One example is a video editing environment that provides a set of recommended video segments along with tools to modify the segments, embed information in the segments, concatenate segments, add video effects and transitions. A further example is an audio editing environment that provides similar functionality when processing audio segments.

[0192] A further example may be a structured text editing environment similar to that described in U.S. Patent Publication No. 2019 / 0163728, entitled “System and Method for the Generation and Editing of Text Content in Website Building Systems,” published May 30, 2019, which is co-owned by the applicant and incorporated herein by reference. Such an environment may provide a set of text elements or “mini-templates” that can “operate together” to form a text element. These can be used to edit portions of site text such as “about us,” “our products,” “terms of use,” team biographies, specific product descriptions, etc. The “mini-templates” may contain placeholders that can be filled using information available to the system (such as the site’s business name, address, product listings, or other details available in the site generation system 40 or an underlying database).

[0193] The text generation and editing unit 378 may also include a specialized editor that provides image creation based on a set of suggested photo or video elements (e.g., cartoon character images and backgrounds) and manipulation processes. Modification and embedding processes may be provided at multiple levels, such as adding audio or captions to simple video segments, modifying text, audio and sub-images viewed in intermediate video segments (as may be performed when translating a video from one language to another), and actually specifying the movement and behavior elements of characters and objects in complex video segments to create parameterized video segments.

[0194] The text generator / editor 378 may collect feedback from the activities of multiple users in creating and editing these segments. The collected feedback (and training data) may also include user characteristics. For example, users in one domain may have different preferences compared to users in other domains.

[0195] Based on the collected feedback, the text generation and editing portion 378 (together with its text generation and editing AI unit 310) may decide what material and options to provide to the user (e.g., which text templates or segments, which video sub-segments, which treatments to suggest, etc.), and may filter and / or rate the possible options. In general, it will be appreciated that for implicit feedback, the system 200 may utilize feedback information collected by the follow-up activity AI unit 310, such as image cropping or portrait segmentation models, as described above.

[0196] As mentioned above, the editing task handler 370 and its elements may provide support for tasks that process a single object. This functionality may be used to support processing previously performed using a specialized rules engine that uses a predefined set of rules generated from a study of the problem domain. These rules are difficult to build and even more difficult to maintain. It will be further understood that the system 200 may use its machine learning based solution instead of a rules engine, or may combine the two techniques and integrate the output from both sources. For example, an AI-based method may detect a group of web elements and a rules-based method may determine how to position them in a mobile view based on their various parameters.

[0197] Additionally, the system 200 may utilize an existing rule-based system to provide initial supervised training to the underlying activity unit AI unit 310. Such initial training can be performed by utilizing the rule engine on collected sites (e.g., randomly generated sites along with other unprocessed WBS / non-WBS sites) as well as utilizing existing outputs of the rule engine (e.g., existing site analysis and site transformation results).

[0198] The editing task handler 370 may also provide the user with certain editing or markup tools that may enable the user to review or edit the source material to be processed (analyzed or transformed). Such tools may enable the user to provide additional hints or related information that may provide additional input to the rule-based system or ML model 317.

[0199] It will be appreciated that the edit task handler 370 may execute the ML feedback based suggestion module 300 to enhance the analysis of existing WBS information, typically by looking at the layout and other information on the page currently being edited. Such analysis may result in the creation and modification of auxiliary structures (such as defining component groupings or high-level layout descriptions), but typically does not directly affect the visible site page.

[0200] The different analyses may utilize multiple features extracted from the layout and components, including features reflecting the results of semantic analysis or other types of analysis (e.g., following the types of analysis detailed in U.S. Patent No. 10,176,154, entitled “System and Method for Automated Conversion of Interactive Sites and Applications to Support Mobile and Other Display Environments,” issued on January 8, 2019, and U.S. Patent Publication No. 2018 / 0032626, entitled “System and Method for Implementing Containers which Extract and Apply Semantic Page Knowledge,” published on February 1, 2018, which are commonly owned by the applicant and are incorporated herein by reference).

[0201] Reference is now made to Figure 8, which illustrates elements of a site functionality updater 400 that may provide support for the task of processing multiple WBS components together. The site functionality updater 400 may include a component grouper 401, a component group labeler 402, an object parser 403, a component orderer 404, an object converter 405, a desktop-to-mobile converter 406, a WBS importer 407, a response editing supporter 408, a template replacer 409, a manual site adapter 410, and a design recommendation 411.

[0202] It will be appreciated that, similar to the edit task handler 370, each element of the site feature updater 400 may work with an associated activity-based AI unit 310 of the ML feedback-based suggestion module 300. Each activity-based AI unit 310 may be universal, community-based, or user-based. The elements of the site feature updater 400 may utilize the ML model 317 adapted for natural language understanding, with the WBS components as symbols provided to the model. Such symbols may be generated directly by the WBS or may be the result of an underlying analysis to identify layouts and components in a non-WBS website (e.g., when analyzing a non-WBS site to be imported into the WBS or analyzed for extracted information). Such language-related models may include, for example, Recurrent Neural Networks (RNNs) and similar machine learning models.

[0203] It will also be appreciated that language association models typically treat information as sequences of input symbols that are one-dimensional in nature, whereas typical web page structures are two-dimensional and can be considered three-dimensional when considering the z-coordinate of the display order. Web page structures can become more complex, especially when container hierarchies need to be analyzed.

[0204] The Site Function Updater 400 component may utilize an algorithm to determine the "natural order" of elements in the layout, thereby converting the 3D+ layout data into a sequence of 1D symbols. Such an algorithm may be similar to that described in the Supernode Creation and Ordering section of U.S. Patent No. 10,176,154.

[0205] It should be appreciated that the ML feedback-based suggestion module 300 may include attributes of the original component as features of the generated sequential symbols processed by the activity unit AI unit 310. Such attributes may include geometric attributes (such as the original X / Y coordinates, the height and width of the component and its z-order) as well as other attributes of the component.

[0206] It will also be appreciated that each element of site feature updater 400 may enable the user to provide explicit or implicit feedback (as described above). For example, in a component grouping analysis task, the user may provide an initial labeling of the groups, and provide labeling of the generated analysis (e.g., component groupings) as to which are incorrect or suboptimal.

[0207] It will also be appreciated that the analysis task provided by the site functionality updater 400 may be a stand-alone task, but may also be the first stage of a conversion task, as described in more detail below.

[0208] One such task is component grouping. Similar to the system described in U.S. Patent Application Publication No. 2018 / 0032626, component grouper 401 may support persistent grouping of objects and provide multiple functions based on such grouping, such as special behaviors for editing operations (e.g., resizing and inserting, deleting, or editing subelements) or special connections to underlying data repositories.

[0209] The component grouper 401 may utilize the component grouping AI unit 310 to detect such groups and understand their layout, or may utilize an ML model 317 similar to that shown in Figure 6. The result of such detection is a component grouping specification that may be "flat" (i.e., consisting of a single set of group specifications) or hierarchical (i.e., consisting of a hierarchy of groups and supergroups).

[0210] Reference is made to Figures 9A, 9B and 9C which show other methods of the component grouper 401 using ML models based on computer vision, triplet analysis and a combination of triplet analysis and computer vision. Each process may start with extracting web elements (step 210) similar to the method of Figure 6.

[0211] For the computer vision model, the method may continue by generating images of pages containing the web elements extracted in FIG. 9A (step 220) and providing the resulting site images to a trained object detection neural network. For training, each page is rendered as an image whereby each element type is given a color and bounding boxes are manually drawn around each element. The model learns to draw bounding boxes around groups of elements (step 222). This is similar to how image object detectors are trained to draw bounding boxes around objects in images. These bounding boxes indicate groups of elements and no further clustering is required. These images are then used to train the neural network via supervised learning (bounding box regression). The trained neural network may then generate a set of bounding boxes (for potential groups) around elements of the page (step 222) that may be processed into component groups (step 224), as described in more detail below. It will be appreciated that the training input may include site images and their associated group definitions. The model in Figure 9B is called the Triplet Model and involves training a model to classify groups of three elements into one of three categories: 1) all elements are in the same group, 2) all elements are in different groups, or 3) two of the three elements are in the same group (step 234). The model uses relationship geometric features between the elements (e.g., distance, overlap, intersection, union) to classify the triplets. These features are extracted in step 232. Once all the triplets have been classified, the groups are reconstructed in post-processing by clustering the elements that are most frequently classified as belonging together using DBScan clustering (step 236).

[0212] As shown in Figure 9C, computer vision and triplet methods may be combined. First, the computer vision process of Figure 9A may be used and its groups provided as input to the triplet method of Figure 9B. This may provide a more accurate grouping of elements. The combination of the two methods allows a computer vision model to identify high-level groups, and implements a triplet model on each high-level group to better detect lower-level groups within the high-level groups.

[0213] 9C may be implemented on a particular container, typically starting with the page as the top node, and recursively for each component node in the containment hierarchy. For each node (e.g., a container with X subelements within it), step 230 creates each possible triplet from these X elements (i.e.,

number

[0214] Reference is now made to Fig. 10, which illustrates a hierarchical grouping process. As shown, page A shows pairs of photos and associated captions (such as a person's photo and name) that should be grouped together ([b,e], [c,f], and [d,g]). Using its associated component grouping AI unit 310, the component grouper 401 (Fig. 8) may group the components into three groups (k, l, and m) as shown on page B. Furthermore, the component grouper 401 may group the three pairs together (on page A) to form a "Our Team" type group (j) (on page B), as defined by the hierarchy shown in Fig. 11 for pages A and B referenced herein.

[0215] It will be understood that a grouping hierarchy may differ from or even contradict a "natural" page containment hierarchy: for example, a grouping hierarchy may include subtrees made up of elements that technically exist at the same container level (i.e., are all siblings) and therefore have no predefined hierarchical relationship between them.

[0216] Furthermore, group definitions may cross container boundaries. For example, a page designer may have originally placed the three photographs (b, c, and d) from Figure 9 in one container, and the three text captions (e, f, and g) in a second, separate container to aid in editing and alignment. However, (content-related) groupings may cross these container boundaries, as discussed above.

[0217] The component grouping AI unit 310 may be trained to utilize container hierarchy information (when available as features) and also to “break” container hierarchies when appropriate (e.g., based on human labeling).

[0218] The component grouper 401 may also integrate predefined grouping rules and methods (e.g., similar to the supernode creator described in U.S. Pat. No. 10,176,154). The component grouper 401 may apply such rules and methods together with its component grouping AI unit 310 and may provide initial training data to the component grouping AI unit 310.

[0219] The component grouper 401 may also train its component grouping AI unit 310 to provide layout / group understanding, for example, it may assign specific types to groups at all levels. Such types may be defined as being in the “WBS component world” (list, matrix, image+caption, image gallery, etc.) or in the “real world” (person details, team members, products offered, etc.). The system 200 may provide a relevant classification of both types, possibly including a hierarchy of types. Such training may also be provided by a labeling tool that allows the user to label groups by their type. This group understanding serves as an additional layer of information to be determined in addition to group detection as described above.

[0220] Accordingly, the component grouper 401 may provide the user with suggested grouping definitions (possibly including the suggested group types described above) and collect explicit feedback, for example, by allowing the user to accept a group definition, reject a group definition, choose among multiple grouping suggestions, edit the provided definitions, and suggest the user's grouping definitions or modifications (e.g., merging, splitting, reordering, or re-parenting of groups).

[0221] The component grouper 401 may collect follow-up implicit feedback from subsequent operations performed during editing. In particular, the WBS editor 30 may provide group and ungroup operations, which may provide additional feedback. The component grouper 401 may also collect feedback based on analysis of other operations, such as component insertion / removal and operations performed on sets of elements. For example, if a text field and an image field are typically modified together, this may indicate that they are interconnected (e.g., the text field describes the content of the image field) and should be grouped if possible.

[0222] The component group labeler 402 may support the user in labeling groups of components in a (possibly hierarchical) manner. Such labeling may include both the composition of the group (i.e., what components the group contains) and the semantics of the group (i.e., what the function of the group is, e.g., "team member", "picture+caption", etc.). These labels may then be used to train other ML models 317 for group identification and understanding.

[0223] Reference is made to Figures 12A, 12B and 12C, which show group labels for the same web page defined in a three level hierarchy. Figure 12A shows a high level hierarchy of components, such that the components of the page have very general component labels such as title, headline and article. Figure 12B shows a more detailed hierarchy of components, where columns are added to the previous set of component types with associated labels. Figure 12C shows a highly detailed hierarchy of components, where text and images are added to the previous set of component types with associated labels.

[0224] The component group labeler 402 (FIG. 8) may label groups at various levels, such as portions of pages, entire pages, sets of pages, or entire sites. The component group labeler 402 may be used to label sites developed within a WBS or from external sites (developed using other WBS or non-WBS technologies or platforms). The component group labeler 402 may consist of multiple tool variations or embodiments aimed at performing labeling for multiple supported platforms.

[0225] For sites created or hosted by a WBS, the component group labeler 402 may be integrated with the WBS runtime server 20. It may therefore support the labeling process with complete information about the components displayed, their layout, their state, their content, and any additional information available to the WBS (including historical information such as edit history).

[0226] The component group labeler 402 may also be integrated with the user's activity at the site and site changes that occur during a session, thus "following" the user and the site and synchronizing labeling activities with site changes. The component group labeler 402 may, for example, follow user activity that causes parts of a page to change, so that previously labeled areas are hidden or made invisible and new page areas, components, component versions, or configurations become visible. The component group labeler 402 may modify the grouping rules as a result. The component group labeler 402 may also follow changes to a site made, for example, due to user edits (e.g., resizing components, moving or affecting the layout of other components) or due to non-user modifications (components that reflect changes made by other users, or dynamically changing data embedded in components). It may also follow changes to a site caused by a responsive layout, for example, when the site switches between multiple layout versions depending on the width (or other parameters) of the display area.

[0227] The component group labeler 402 can also affect the functionality and behavior of the WBS. For example, the component group labeler 402 may instruct the WBS runtime server 20 to display components as they are processed by the user in a modified, alternate, simplified, frame-only, or another mode that makes labeling easier (e.g., by removing visual elements or clutter or by clarifying relationships between various components).

[0228] The component group labeler 402 may also use information available in other ways (e.g., from the WBS) to support or provide the user with possible hints and instructions, such as adding specific markings, modifications, animations or other UI elements (e.g., handlers or inter-element connections) to components that may form a group.

[0229] As another example, the component group labeler 402 may indicate to the user the order in which components were added (or edited), or mark components that were added or edited during a given period or session. Such information may be directly available to the tool (based on edit history information) or may be generated based on data analysis.

[0230] For sites implemented on an external WBS platform or non-WBS sites, the component group labeler 402 may be implemented (for example) as a browser add-on or extension, and may act as an additional layer of UI that is displayed along with the underlying site display, allowing users to label groups (and group hierarchies) on a displayed page for easy reference, as shown in Figures 13A and 13B. Figure 13A shows the labeling UI, and Figure 13B shows the labeling UI once several components have been selected for grouping.

[0231] In this scenario, the component group labeler 402 may also interact with the browser and the displayed page document object (DOM) structure, for example, by parsing the DOM (at page load, in real time, or otherwise). This interaction may enable the component group labeler 402 to synchronize any group marking activity with changes to the displayed page.

[0232] The component group labeler 402 may also interact with the underlying DOM of the displayed page by making modifications to the DOM to provide hints to the user performing the groupings (similar to the changes made to the page UI through interaction with the WBS, as described above). Alternatively, the component group labeler 402 may provide additional UI indications and hints through other means, such as through an additional (possibly semi-transparent) layer displayed on top of the displayed page.

[0233] Thus, the component group labeler 402 may support marking both WBS and non-WBS sites. It may use additional information available to it (e.g., as a result of parsing it, extraction from the WBS or DOM structure, etc.) as additional features to be added to the label provided by the user (in both cases).

[0234] In a typical scenario, the component group labeler 402 may be used internally by the WBS vendor 61 to train the activity unit AI unit 310 that can then be deployed to regular (external) users. However, the WBS may also deploy the system 200 to external users in scenarios where the user is required to provide grouping feedback for a particular site. For example, when a user wishes to create a site in a WBS using the site generation system 40 and the user provides details of additional related sites (e.g., the user's own or competitor sites), information extracted from such sites can be utilized by the site generation system 40 when generating or modifying the site for the user. As part of the interaction, the user may be asked to point to or mark elements of the provided site. Such markings may include grouping information or provide information about them that the site generation system 40 can provide to the component group labeler 402 to be utilized as part of the group labeling process.

[0235] The component group labeler 402 may also utilize the grouping data to provide loop-back integration with the group labeler AI unit 310. In this manner, the component group labeler 402 may send the page (or a portion) to be reviewed to the group labeler AI unit 310, and may receive initial grouping suggestions from the interaction generator 315. The component group labeler 402 may then present these suggestions (along with the page to be displayed) to the user. The user may then review these suggested grouping instructions (including editing them or creating different groupings), and the component group labeler 402 may provide the groupings to the group labeler AI unit 310 for training.

[0236] The component group labeler 402 may compare work performed by multiple people (labelers) on the same page or component set to see if they agree on how to define groupings for a given page or component set. Such a comparison may be based, for example, on the value of an adjusted Rand Index. This index measures the similarity between different grouping definitions. Presumably, if multiple people recommend the same (or similar) labeling, this labeling will be more accurate or otherwise more reflective of what the best grouping should be.

[0237] The object analyzer 403 may detect specific objects or patterns within a site, such as logos. This is particularly relevant when analyzing non-WBS sites to detect logos (e.g., for use in the site generation process). Such analysis tasks may also include detecting "user-configured" logos within existing WBS sites that are not marked or designated as logos.

[0238] The object parser 403 aims to detect a set of components that together form a logo. It will be appreciated that a user may construct a logo by combining multiple text, image and graphic elements (e.g., a star) without any indication that these elements form a logo (or any type of grouping). Detecting such logos can be particularly difficult when parsing non-WBS sites where components are also not directly specified, but rather their specification must be detected in the underlying HTML code. The object parser 403 may also include detecting a single component that is (alone) the logo of the parsed site.

[0239] The object analyzer 403 may collect feedback on the logo detection task through an accept / reject interaction with the user when presenting the detected logo. The object analyzer 403 may collect follow-up feedback by detecting whether the logo was used in a published version of a site created based on the analyzed site (by direct conversion or site generation), or by detecting whether the logo was moved to the top area of ​​a mobile version of the site or added to a header that appears on multiple pages of the site. It will be appreciated that such processing may provide further feedback on how acceptable (or useful) the detected logo is to the user.

[0240] As described herein above, the system 200 may utilize an ordering algorithm (such as that detailed in U.S. Pat. No. 10,176,154) to define a linear order among the displayed components arranged in the three-dimensional layout. Although the linear order is typically defined based on the natural reading order of the page, such an order may be difficult to define or ambiguous, and other possible orders may exist. Such a linear order may be required, for example, to utilize a language-related activity unit AI unit 310 on the component set, or to utilize an AI unit 310 using a symbol sequence-based ML model 317. Such ordering may also provide the basis for additional system functionality (and provide input to additional task-specific AI units 310), such as converting an existing layout to mobile or different display parameters, providing responsive editing and display capabilities, providing alternative views of the site pages that rely on a given component order, such as reading the site for blind users as part of the system's accessibility support.

[0241] The component ordering unit 404 may be initialized using an explicit component ordering analysis (as described above) and may utilize the ordering AI unit 310 for component ordering, which may be further trained through feedback from users (possibly including feedback from end users of a website created by a WBS user or a designer). The component ordering unit 404 may collect feedback by providing a UI that may allow attaching order numbers to components. Alternatively, the component ordering unit 404 may have a simple “Oops” button available to the end user whenever the model provides an incorrect order. The Oops button may be combined with a cursor to point to the location of the error (e.g., the location of the misplaced component). Such a button may be used, for example, by a blind user to cause the system 200 to reread a set of misordered components using a different order.

[0242] The object transformer 405 may transform a collection of objects, such as components in a web page with their layout and other associated information (a source object set), into a modified target object set, typically based on a preliminary analysis of the source object set, such as the grouping and ordering analysis described above.

[0243] It will be understood that the following description of object transformer 405 is in terms of web page transformation, however the same processing may be applied to portions or subsets of pages (e.g., the contents of a given container), sets of pages (including complete websites), any collection of components, and other non-web components or sets of elements.

[0244] It should be understood that the transformation can be performed at multiple levels, such as when both the source object set and the target object set are the same website page (e.g., in a WBS) and the transformation involves changing several components, layouts and attributes. An example is changing the components and layout in a desktop version of a page to generate a mobile version of the page.

[0245] Another level is when the source and target object sets are typically different pages on the same platform (e.g., a WBS), and the object converter 405 may transfer content, layout and design information from the source object set to the target object set. One example is processing a site built based on a particular template and transferring the content, layout and design (and any changes made to the original source template) to a new target template.

[0246] A further level is when the source and target object sets are pages on different platforms. One example concerns importing pages into a WBS from a different WBS or non-WBS site. Unlike the above-mentioned type of conversion task, in this type of conversion, the source and target object sets are actually "different languages" and the set of available components and attributes may differ substantially between the source and target platforms. In fact, when converting from a regular (HTML / JS) web page, the source page may not have any component definitions (unlike the pages defined in the WBS of the present invention).

[0247] The object transformation section 405 may utilize an object transformation AI unit 310 adapted to the domain of natural language processing, including both parsing and generative models (possibly through the ordering of various components). Some of these tasks (and associated facilities and models) are described in more detail below.

[0248] The object transformation AI unit 310 may be trained using matched pairs of source and target object sets. Such pairs may be provided by the WBS vendor staff 61 and others mentioned above. Such pairs may also be collected using implicit feedback data extracted from users who correct the initial solutions provided by the system 200. The object-based transformation AI unit 310 may also be trained using other explicit and implicit forms of feedback and interactions, as described in more detail below.

[0249] It will also be appreciated that existing WBSs often need to retarget sites built for the desktop to different screen sizes (with different display window sizes) such as those found on tablet and mobile devices. Such tablet and mobile devices do not have standard screen sizes, and many sizes exist on the market. Transformations made to the page to fit different screen sizes are necessary and may involve rearranging components to avoid horizontal scrolling and minimize vertical scrolling. It may also include changing component types (e.g., changing a grid gallery to a mobile-friendly slider gallery), changing the layout and other attributes of components, adding components (e.g., mobile navigation bars), dropping components (e.g., decorative components not needed in the mobile version), and other modifications.

[0250] Existing systems are typically based on analysis of page layout and elements and the use of a rules engine to provide layout and other page modifications. One such system is described in U.S. Pat. No. 10,176,154.

[0251] The desktop-to-mobile conversion part 406 may use a desktop-to-mobile conversion AI unit 310 that can be trained using the corresponding rule engine on existing sites. It may also be trained using previously converted sites (which may have been edited by a designer to improve the proposed rule-based version). It may also be trained based on newly converted sites, i.e., based on changes made by a user to the converted site using the model.

[0252] Other training approaches may utilize analysis-based page transformation algorithms aided by an appropriate activity-based AI unit 310. For example, U.S. Patent No. 10,176,154 describes such an analysis-based algorithm that incorporates grouping of related components (e.g., photo and caption pairs) and hierarchical determination of the reading order of the components of a page. This application describes multiple methods for analyzing pages and determining such grouping and ordering.

[0253] The WBS importer 407 may parse a non-WBS site and create a matching WBS site. This may be a conversion of a site from a source WBS to a target WBS, or a conversion of a normal (HTML-based) site to a site in the target WBS. In the first instance (WBS to WBS), the main problem is the mapping of components and attributes from the source WBS to the target WBS. In particular, the two WBSs may use different data models, architectures and / or concept sets. For example, one of them may allow a hierarchy of pages to be created, while the other may utilize different mechanisms to connect related pages. One of the WBSs may define some capabilities (such as page routing menus) using components and layout conventions, while the other WBS may express these capabilities via data tables or procedural conventions.

[0254] In the second case (HTML to WBS), the WBS import AI unit 310 needs to detect complex components from within HTML, because the HTML representation of such complex components may consist of multiple HTML elements that may appear unrelated. For example, a single WBS video player component may be realized in HTML using multiple frames, video display windows, multiple buttons (stop, start, pause, ...) and other controls. The video player element may be distributed in a sub-hierarchy of multiple HTML elements and composited with other page elements. The model should be able to recognize such combinations of HTML elements and map them to a single video player component (with appropriate attributes and layout information).

[0255] The WBS import 407 may utilize multiple WBS import AI units 310. In particular, an initial WBS import AI unit 310 may provide an analysis of the hierarchy of elements at the site. The WBS import AI unit 310 may also analyze the context of elements in the hierarchy to recognize (for example) that a given sub-level represents an "about us" element at the site. Such analysis may be expressed using semantics similar to those for layout elements as described in U.S. Patent No. 10,073,923.

[0256] The second WBS import AI unit 310 may process an analysis of the site's layout as well as an understanding of additional characteristics of the site, such as the site's responsive behavior (i.e., what are the screen width breakpoints at which the site changes its element configuration to provide a responsive design), as described in more detail below. An exemplary responsive design and editing system is described in commonly owned U.S. patent application Ser. No. P-25078-US, filed on even date herewith, entitled "System and Method Providing Responsive Editing and Viewing, Integrating Hierarchical Fluid Components and Dynamic Layout," which is incorporated herein by reference.

[0257] Reference is made to Fig. 14 which shows how a given layout is displayed on multiple available screen widths. As shown, a given layout S1 consisting of five components a-e is displayed on four different screens having widths (W1-W4). For widths W1 and W2, components a-e are laid out in a similar manner with component a at the top, components b-d in a single row below component a, and component e at the bottom. For the much narrower width W3, component a remains at the top, followed by component b, followed by components c and d in a single row, and component e at the bottom. For the very narrow width W4, only components a-d are shown in a single column.

[0258] In FIG. 14, widths W1-W4 are divided into ranges by two breakpoints bp1 and bp2. As can be seen, a change in width may cause components a-e to change size, position, relative relationships, and in some cases even be removed from the layout (such as component e at width w4). The WBS importer 407 may analyze the site to determine if such breakpoints are already defined for the site and what breakpoint values ​​should be used (predefined or new). The WBS importer 407 may also analyze any existing responsive behaviors (which may be mapped to WBS versions) and determine the best responsive behavior to use.

[0259] It will be appreciated that a responsive site can be viewed as a site with a set of breakpoint ranges with separate layout and component configurations for each range. Additionally, each range may have its own dynamic layout rules that may control the behavior of the components (e.g., moving or resizing) as the display size changes with the range (or other changes occur, such as changes in component content).

[0260] The WBS importer 407 may use one form of the site (e.g., desktop only) to provide an offline copy of the original site as input to the activity-based AI unit 310. The second activity-based AI unit 310 may require some interaction with the real site, for example, to test the real site under multiple display configurations (which may be performed by the real site or its offline copies).

[0261] For the WBS importer 407, feedback may be provided as described above (training on the converted sites, implicit user feed resulting from model result corrections, etc.). The system 200 may also provide a dedicated imported sites editor to provide implicit feedback immediately after the import process is complete. Such a specialized editor may display additional feedback, for example, hierarchy analysis results, and allow the user to edit the hierarchy in ways not provided by the WBS editor 30.

[0262] As mentioned above, responsive site structures are described in the context of importing sites into a WBS, in which context the system 200 may also import existing responsive site definitions from non-WBS sites.

[0263] However, responsive site processing may also involve directly editing sites defined within a WBS. This may include editing existing responsive sites as well as adding responsive behaviors (including breakpoints and configurations) to previously non-responsive sites. It should be understood that editing a responsive site may include editing (modifying, adding, deleting, splitting, or merging) breakpoint range definitions. Such editing may also include editing components and their layout or configuration for a particular breakpoint range, affecting changes that the system 200 intends to apply to the layout or configuration (with appropriate modifications) for other breakpoint ranges.

[0264] As an example of the above, a user may make a change (such as adding a new component) to the layout of a particular breakpoint range. The system 200 may apply the change to the layout of other related breakpoint ranges (if, when, and with related adaptation). Assume (for example) that a given WBS site has three configurations (for the three breakpoint ranges), which could be two desktop configurations (A and B) and one mobile configuration (M). If a component is added to configuration A, the system 200 may automatically add the component (with modified size and position) to configuration B and support adding a mobile version of the component to configuration M.

[0265] The response editing support 408 may use the changes and edits made by the user to one website configuration or the other to train its associated response editing AI unit 310. In an alternative embodiment, the interaction generator 315 may generate multiple alternatives to apply the user's changes made to one configuration on the other configuration. The user may be prompted to select which alternative to use. The response editing support 408 may use this feedback to train its response editing AI unit 310. It will be appreciated that the response editing support 408 may operate in conjunction with the site generation system 40, the editor 30, or both, as appropriate.

[0266] As mentioned above, the system 200 may provide users with the ability to perform template substitution. Users (designers) often build sites based on templates provided by the WBS vendor 61. Users may perform substantial modifications to this underlying template, including design and layout changes, insertion and deletion of pages, page elements and site portions, as well as input of user-specific site text, media and data. The user may later wish to switch to a different template, which will require the user to map the changes (layout and content) to the new template and reapply them appropriately (in a template conversion session).

[0267] Template substitution may or may not be possible in some cases. For example, sites created utilizing site generation system 40 are highly structured and typically based on a hierarchy of layout elements (as described in U.S. Pat. No. 10,073,923). Editing of such sites may be restricted such that the highly modular layout elements are preserved; for example, system 200 may prevent a user from moving components from one layout element to another. Furthermore, layout elements may be associated with underlying content elements that are not part of the visual design and thus are not visually changed during site editing (e.g., changes are typically limited to changes to the content).

[0268] It should be understood that the situation may differ from a free-edited website, i.e., a website created using a conventional WBS editor 30. The source and target templates may not initially correspond, and existing similarities between the two templates may be destroyed through changes made by the user. Thus, exact mapping may not be possible.

[0269] The template replacer 409 may provide a machine learning based solution to this problem. The template replacer 409 may provide a partial solution, mapping some of the content, layout, and other changes made to the source template while omitting content that cannot be mapped to the target template.

[0270] The template replacement unit 409 may also utilize hints added to templates by their creators (e.g., staff at the WBS vendor 61) that may convey additional information regarding the equivalence of template elements. For example, the WBS vendor 61 may mark different parts of different templates with semantic markers (e.g., marking a given set of components with the marker “our company history”). These markers may be comparable to sets of content elements / layout elements as defined in U.S. Pat. No. 10,073,923, or may be different sets of group types that may be detected by the template replacement unit 409 for template replacement. Other modules in the site functionality update unit 400, such as the component grouper 401, the component group labeler 402, and the component ordering unit 404, may assist the template replacement unit 409 in determining group types. The resulting template replacement may also be similar (and share elements) to the transformations performed by the object transformer 405 and the WBS importer 407.

[0271] The template replacement unit 409 may include a special editor that provides additional indication of what content / layout was successfully transferred (and how it was transferred) and what was not transferred, and allows the user to correct and complete the transfer results. This editor may also be integrated into the WBS editor 30 (e.g., as a task list, wizard, or workflow system).

[0272] It will be appreciated that the template replacement AI unit 310 used by the template replacement component 409 may be trained with pairs of sites that are converted between templates. The template replacement AI unit 310 may also be trained with changes made to sites using the WBS editor 30, along with follow-up corrections made to sites that have been converted using the dedicated editor described above. The template replacement AI unit 310 may also be trained with a set of templates for which content equivalence hints have been specified (as described above), with no particular content inserted or with partial particular content inserted.

[0273] Furthermore, it will be appreciated that training template replacement AI unit 310 can be difficult since there are very few conversion sessions associated with each template pair. As the number of templates increases, the number of possible template pairs (source site template and target site template) can also increase. Template replacement unit 409 may generalize and determine what processing is required given good examples. Template replacement unit 409 may utilize, for example, the following approaches or a combination thereof: Template replacement unit 409 may train a separate activity unit AI unit 310 for a given template pair with all conversion sessions for the given pair, may train a separate model for each group of “similar” pairs (as specified by WBS vendor 61), or may train a single model for all pairs together.

[0274] The collected training data may still include features that identify the source and target templates and their parameters and attributes (e.g., business domain and general style, as well as more detailed features). Training a model for each group of similar pairs, or even for all pairs (with associated pair information features), may be preferable since in many cases there may be similar pairs even in multiple domains.

[0275] For example, the WBS vendor 61 may create a set A of elegant style templates (e.g., for fitness studios and clothing stores) with similarities between all templates in A. A subsequent WBS vendor 61 may similarly create a set B of formal style templates (for the same or different business domains). Then, a user may convert some sites created with templates from group A to use with templates from group B. Such conversion may typically be done within the same business domain (i.e., converting templates of fitness studio A to templates of fitness studio B). In this scenario, the template conversion AI unit 310 may benefit from being trained with information collected from multiple conversion sessions between templates of group A and templates of group B, typically with added features such as “group name” (A, B or other) and “business domain”. This may be much better than training a separate model for each specific pair [A member, B member] actually used.

[0276] The template replacement unit 409 may also utilize multiple activity unit AI units 310 for separate functional areas (e.g., one for analyzing the site hierarchy and structure and one for analyzing the layout and additional attributes), as described above for the WBS import unit 407.

[0277] As described above, sites created using site generation system 40 may be highly structured. One major advantage of this highly structured layout element hierarchy is that the site may be easily regenerated after its underlying data is modified. System 200 may update associated content elements, re-evaluate content element-layout element associations and layout element placements, and regenerate the layout element hierarchy and resulting pages as necessary. On the other hand, typical (manually edited) sites utilize less rigid component hierarchies. Their structure has evolved over a series of editing sessions, for example, during the creation of the underlying template and editing of the actual website. Such sites cannot simply be regenerated when the underlying data changes.

[0278] Thus, a user may wish to convert a manually edited site into a format adopted by the site creation system 40. The manual site adaptor 410 may utilize one or more activity unit AI units 310 to discover groups in the WBS component hierarchy and determine the semantic connections between components. The manual site adaptor 410 may then match these groups against an existing set of site creation system structures (including content elements and visual site parts) and build a matching site creation system form of the site.

[0279] Similar to the functionality of the WBS Importer 407, the Manual Site Adaptation AI unit 310 may be trained on existing site pairs (pre- or post-conversion sites). It may also be trained based on subsequent modifications made by the user to the post-conversion version (e.g., via a site generation system-specific editor).

[0280] The design recommendations unit 411 may enable a user to select a set of components, query the WBS, receive alternative options for the selected set layout / design, select a preferred layout / design option, and apply the preferred layout / design option to the selected set of components. The design recommendations unit 411 may integrate suggested alternative quality ratings and diversifications similar to those described in U.S. Patents 9,996,566 and 9,747,258.

[0281] It will be appreciated that the design recommendations 411 may base its offer on layouts collected from multiple sources (other sites in the WBS, pre-built templates, non-WBS sites, etc.) or built layouts that are created based on the components in the selected set. Furthermore, when applying the preferred layout / design to the selected set of components, they may not be an exact match and the system 200 may be required to extend or modify elements of the proposed or preferred layout / design to create a match.

[0282] It will be further appreciated that the design recommendation section 411 may benefit from the techniques described above by including a design recommendation AI unit 310 supporting the following functions and receiving user feedback collected from the user's interaction with the system. This process may be performed in several stages.

[0283] Phase 1 is layout collection. As mentioned above, the activity-based AI unit 310 may support import from multiple sources, including understanding and analyzing the imported layouts (such as the ML model 317 associated with the WBS importer 407 as described above).

[0284] The design suggestion unit 411 may implement and train a specific layout extraction AI unit 310 to create the suggested layout. Such a specific layout extraction AI unit 310 may provide better results than a full-blown site import model because the layout extraction AI unit 310 only extracts abstract layout information (more abstract or generalized) and does not extract complete component information or actual component content. For example, when importing a layout that includes a video component, the full layout extraction AI unit 310 may identify the exact video used and its display attributes, whereas the design suggestion AI unit 31 only needs the video component frame position and size and any associated video controls included (as the actual video clip is replaced by the one specified by the user).

[0285] The design recommendation AI unit 310 may be trained by user feedback collected from follow-up edits. The design recommendation component 411 may classify the follow-up changes because not all changes indicate an error in the original extraction. For example, the design recommendation AI unit 310 may interpret a given combination of HTML elements in the imported site as an image gallery, which the user may subsequently change to a video gallery. This change may indicate an error in the original understanding of the inspected site or a decision by the user to change a component type in the proposed layout. The design recommendation component 411 may realize such changes in classification, for example, based on how many users made this change (e.g., if multiple users made the same change to the layout, it is more likely that the original extraction was incorrect).

[0286] Stage 2 is selection of layout suggestions and interaction with the suggested layouts. In this post-query stage, the design suggestions component 411 may select, rank and present design alternatives to the user. The design suggestions component 411 may utilize the selection and ranking AI unit 310 to support these functions.

[0287] Such a selection ranking AI unit 310 may be trained using examples of selection and ranking based on rating and diversification algorithms such as those described in U.S. Patent Nos. 9,747,258 and 9,996,566. The selection ranking AI unit 310 may be further trained based on input from a user. Thus, the features used may include the original set selected by the user passed to the query. The features may include semantic signatures (as described in U.S. Patent Nos. 9,747,258 and 9,996,566) describing the component set, as well as the actual component data (including type, layout, content and attributes). The features may further include additional parameters and features such as user parameters and site parameters (as described above). Such features allow the design recommendation AI unit 310 to adapt to specific user and community preferences, such as local preferences for more colorful or more formal designs.

[0288] Other input characteristics may further include the actual selection made by the user, including additional alternatives that may have been confirmed by the user but not selected, and the resulting level of usage of the page (e.g., whether and when it was published).

[0289] Stage 3 is the adaptation of the selected component set to the selected result layout. It will be appreciated that such adaptation may be performed utilizing an adaptation AI unit 310 trained for the task. The adaptation AI unit 310 may be initially trained based on component matching and adaptation algorithms. Thereafter, it may be trained based on user feedback that is implicitly collected from changes made by the user to the adapted layout.

[0290] Thus, alternative designs may be implemented using one or more activity-based AI units 310 to support the required functionality as well as the import, query, matching, ranking and diversification algorithms of a given component. Such activity-based AI units 310 may be further trained using feedback collected from the user during further sessions.

[0291] Therefore, machine learning / artificial intelligence models may be built into standard WBS functions and editing tasks. These models can be trained and continually updated using feedback based on both explicit and implicit information.

[0292] Additionally, the above description has focused on websites operated by a website building system provider (implementing system 200). However, system 200 may be implemented with additional types of websites and other non-web digital creations. These may include, for example, entire websites and portions of websites (e.g., a portion of a website page) or one or more portions of a website page, websites designed for normal desktop computer viewing, mobile websites and tablet-optimized websites, websites created by a website building system that is operated externally (i.e., not by the website building system vendor), websites that run locally on a local server installed on a user's machine, and websites that act as a UI and are operated within other systems (including embedded systems and applications) (or any combination of these).

[0293] Other types of websites and other non-web digital creations may also include websites or other displayed visual compositions hosted within larger systems, including (for example) pages hosted within social networks (such as Facebook), blogs, portals (including video and audio portals that support user-customizable pages, such as YouTube® channels), etc. This may include other types of remotely accessible online presences that are not considered websites.

[0294] Other types of websites and other non-web digital creations may also include interactive (e.g., mobile) applications, including hybrid applications (combining locally installed elements with remotely extracted elements) and non-interactive digital creations such as emails, newsletters, and other digital documents.

[0295] Unless otherwise indicated, and as will be apparent from the above description, discussions throughout this specification utilizing terms such as "processing," "operating," "calculating," "determining," and the like, are understood to refer to the actions and / or processes of any type of general purpose computer, such as a client / server system, mobile computing device, smart appliance, cloud computing unit, or similar electronic computing device, that manipulates and / or transforms data in the registers and / or memory of a computing system to other data in the memory, registers, or other such information storage, transmission or display device of the computing system.

[0296] An embodiment of the present invention may include an apparatus for carrying out the processes herein. The apparatus may be specially constructed for the desired purposes or may comprise a computing device or system, typically having at least one processor and at least one memory, selectively activated or reconfigured by a computer program stored in the computer. The resulting apparatus, when instructed by software, may transform a general-purpose computer into an inventive element as described herein. The instructions may define the inventive device to operate with a computer platform where it is desired. Such a computer program may be stored on a computer-readable storage medium, such as any type of disk, including, but not limited to, optical disks, magnetic optical disks, read-only memory (ROM), volatile and non-volatile memory, random access memory (RAM), electrically programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, disk-on-key, or any other type of medium suitable for storing electronic instructions and capable of being coupled to a computer system bus. The computer-readable storage medium may also be implemented in a cloud storage device.

[0297] Some general purpose computers may include at least one communication element to enable communication with data networks and / or mobile communication networks.

[0298] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the desired method. The desired structure for a variety of these systems will appear from the description below. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0299] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims

1. A website building system (WBS), a database for storing user websites and website components; a processor implementing a machine learning feedback based suggestion module; The machine learning feedback based suggestion module: A plurality of activity unit AI units; A feedback system, The plurality of activity unit AI units support a particular WBS activity and provide system suggestions to a user, and a single unit of the plurality of activity unit AI units includes at least one dynamically updatable machine learning model; the particular WBS activity comprises at least one multi-component task for processing a website; the at least one dynamically updatable machine learning model is trained using different object types and feedback types; The feedback system comprises: (a) Implicit feedback regarding editorial activities within the WBS; (b) user suggestions and explicit feedback on the suggestions; (c) input from a rule engine that employs rules generated from learning the problem domain; dynamically updating the at least one dynamically updatable machine learning model using A feedback system comprising:

2. The feedback system comprises: an implicit feedback handler that analyzes at least an edit history of the user to determine future activities that the user will perform on the website and / or within the WBS and generates the implicit feedback for training the at least one associated dynamically updatable machine learning model; 2. The WBS of claim 1 .

3. The feedback system comprises: analyzing at least a response of the user to the at least one system suggestion to determine how the user responds to the at least one system suggestion; and an explicit feedback handler configured to generate the explicit feedback for training the at least one dynamically updatable machine learning model associated therewith. The WBS of claim 2.

4. The implicit feedback handler receives information collected from within the WBS; The information includes a layout of components and at least one of business information, user information, and site information. The WBS of claim 2.

5. At least one particular activity further includes a single component task for processing a website within the WBS; the single component task includes at least one of image resolution enhancement, face detection, portrait segmentation, object segmentation, image cropping, image enhancement, logo creation, and site text generation; 2. The WBS of claim 1 .

6. the at least one multi-component task includes at least one of component grouping, component group labeling, component ordering, object analysis, object transformation, desktop-to-mobile transformation, website import, template replacement, responsive editor support, and iteration design suggestions; The WBS of claim 1.

7. the at least one dynamically updatable machine learning model is a model suitable for the task and is selected from one or more of the following model types: supervised, unsupervised, prediction algorithms, classification algorithms, clustering algorithms, association algorithms, time series prediction algorithms, image-to-image models, sequence-to-sequence models, and generative models. The WBS of claim 1.

8. The feedback system comprises: A response evaluation unit for evaluating a response quality of a feedback response from the user; a user evaluation unit for evaluating a user's quality in providing feedback; a vendor handler for analyzing at least said WBS vendor staff feedback; and a community handler for analyzing feedback from at least a community of users.

2. The WBS of claim 1 .

9. A method for a website building system (WBS), comprising: storing the user's website and website components; having at least one dynamically updatable machine learning model relating to a particular activity; providing a user with system suggestions related to the particular WBS activity comprising at least one multi-component task for processing a web site; training the at least one dynamically updatable machine learning model using different object types and feedback types; (a) Implicit feedback on editing actions within the WBS; (b) User suggestions and explicit feedback on the suggestions. (c) input from a rule engine employing rules generated from learning the problem domain; dynamically updating the at least one dynamically updatable machine learning model using A method for providing the above.

10. The step of dynamically updating the at least one dynamically updatable machine learning model comprises: analyzing at least the editing history of the user to determine future activities that the user will perform on the website and / or within the WBS; generating implicit feedback for training the at least one dynamically updatable machine learning model; The present invention is characterized in that it comprises 10. The method of claim 9.

11. The step of dynamically updating the at least one dynamically updatable machine learning model comprises: analyzing at least a user response to the at least one system suggestion to determine how the user responds to the at least one system suggestion; generating explicit feedback for training the at least one dynamically updatable machine learning model; The present invention is characterized in that it comprises The method of claim 10.

12. At least one particular activity further includes a single component task for processing a website within the WBS; the single component task includes at least one of image resolution enhancement, face detection, portrait segmentation, object segmentation, image cropping, image enhancement, logo creation, and site text generation; Characterized in that 10. The method of claim 9.

13. the at least one multi-component task includes at least one of component grouping, component group labeling, component ordering, object analysis, object transformation, desktop-to-mobile transformation, website import, template replacement, responsive editor support, and alternate design suggestions; 10. The method of claim 9.

14. the at least one dynamically updatable machine learning model is a model suitable for the task and is selected from one or more of the following model types: supervised, unsupervised, prediction algorithms, classification algorithms, clustering algorithms, association algorithms, time series prediction algorithms, image-to-image models, sequence-to-sequence models, and generative models.

10. The method of claim 9.

15. The step of generating explicit feedback includes receiving information collected from within the WBS; The information includes a layout of components and at least one of business information, user information, and site information. The method of claim 13.

16. The step of dynamically updating the at least one dynamically updatable machine learning model comprises: evaluating a response quality of a feedback response from the user; evaluating user quality of the feedback; analyzing feedback from at least the vendor's staff regarding said WBS; and analyzing feedback from at least a community of users.

10. The method of claim 9.

Images