Analyzing web pages to facilitate automated navigation

Abstract

The invention discloses analyzing web pages to facilitate automated navigation. Embodiments are described herein for analyzing existing interactive websites to facilitate automatic engagement with those websites with minimal effort from hosts of those websites, for example by an automatic assistant or via other user interfaces. For example, in various implementations, techniques described herein may be used to abstract, verify, maintain, summarize, extend, and / or distribute individual actions and "traces" of actions that may be used to browse various interactive websites. In addition, techniques are described herein for utilizing these actions and / or traces to automate aspects of interactions with third party websites. For example, in some implementations, the techniques described herein may enable a user to engage with an automatic assistant (via a spoken or typed dialog session) to interact with a third-party website without the user directly visually interacting with the third-party website and without the third party implementing their own third-party agent.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980063632.3, with a filing date of May 9, 2019. Technical Field

[0003] The present disclosure relates to analyzing web pages to facilitate automatic navigation. Background Art

[0004] A human may engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," "virtual assistant," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands, queries, and / or requests (collectively referred to herein as "queries") using free-form natural language input, which may include spoken utterances and / or typed free-form natural language input that is converted to text and then processed. The automated assistant may perform various types of processing on the natural language input, such as natural language processing, syntactic processing, semantic processing, etc., in order to identify and respond to the user's intent.

[0005] An automated assistant may interact with a third-party software agent, or "third-party agent," in order to respond to some types of user intent that the automated assistant itself cannot fully respond to. For example, a provider such as a restaurant, a ride-sharing service, or the like may design its own third-party agent that is configured to interact with the automated assistant's application programming interface ("API"), for example, so that the automated assistant can act as a liaison between the user and the third-party agent. However, this development paradigm may be slow to scale. Each provider must first create a third-party agent that is compatible with the automated assistant's API. This may require more resources and / or technology than many providers may have at their disposal. In addition, many providers have already built interactive web pages and / or mobile applications ("apps") that users can interact with to generate various goods and / or services. Requiring these providers to design another interface that is compatible with the automated assistant may result in duplication of effort, at least from the provider's perspective. Summary of the invention

[0006] Described herein is a technique for analyzing existing interactive websites (including individual HTML, XML, CSS, and / or JavaScript web pages that form those websites) to facilitate automatic engagement with those websites, for example, through an automated assistant or via other user interfaces with minimal effort from the hosts (e.g., vendors) of those websites. For example, in various embodiments, the techniques described herein may be used to abstract, verify, maintain, summarize, extend, and / or distribute individual actions and "scripts" (i.e., sequences of actions) of actions that can be used to browse various interactive websites. Additionally, described herein is a technique for automating various aspects of interaction with third-party websites using these actions and / or traces. For example, in some embodiments, the techniques described herein may be employed to enable a user to engage with an automated assistant (via verbal or typed dialogue sessions) to interact with a third-party website without the user visually interacting directly with the third-party website and without the third party implementing their own third-party agent.

[0007] Assume that a user requests an automated assistant to use a certain vendor to perform a certain action (e.g., purchase an item or service). Assume further that a particular vendor has not yet created a third-party agent that can interact directly with the automated assistant, but the vendor does have an interactive website with a series of web pages, each of which has one or more interactive elements that can be operated by the user to purchase items or services. Using the techniques described herein, the first web page (i.e., the start web page) of the interactive website can be analyzed and / or processed to generate a current user state associated with the user. This current user state can be represented in various ways such as semantically rich feature vectors, embeddings in latent space, etc. The current user state can also include other information, such as one or more parameters provided by the user (e.g., as part of their natural language input), one or more preferences of the user (e.g., preferred pizza toppings or crust styles), previous actions performed by the user (e.g., how the user got here, previous searches from the user, previous requests to the assistant, active topics of conversations with the automated assistant, previous user states, etc.), etc. In some implementations, the current user state may also include one or more of what will be referred to herein as a "universal identifier matrix" that includes contextual clues about each interactive element of the current web page (e.g., radio buttons, input fields, submit buttons, search buttons, drop-down lists, seating charts, etc.). The universal identifier matrix will be described in more detail shortly.

[0008] In various embodiments, the current user state may be compared to a plurality of past user states contained in one or more databases, for example, to identify one or more candidate user states. These past user states may represent user states generated during previous interactions with the same website or different websites, and may include information similar to that included in the current user state. These past user states may also include related actions corresponding to interactive elements acted upon during previous interactions, such as checking one or more radio boxes, selecting a particular item on a drop-down menu, filling in a particular input field with particular parameters, and the like.

[0009] Candidate user states can be identified based on a variety of criteria. For example, in some embodiments, candidate user states can be identified from a larger superset of past user states based on a measure of similarity between past user states and the current user state (e.g., Euclidean distance, similar universal identifier matrix, etc.). In some embodiments, only those past user states that have a similarity measure that satisfies a certain similarity threshold with the current user state can be considered as candidate user states (i.e., in order to reduce the search space from all past user states, which can grow over time). One benefit of using a similarity measure is that a candidate user state is likely to become similar to the current user state. Therefore, an operation that is successfully initiated on a web page for generating a candidate past user state is likely to be successful for the current user state.

[0010] Once the candidate user states are identified, in some embodiments, the candidate user states can be ranked, for example, based on a measure of similarity with the current user state, based on the similarity between their corresponding universal identifier matrices and the universal identifier matrix of the current web page on which the current user state is generated, etc. Then, the highest-ranked candidate user state can be selected, and one or more actions associated with the highest-ranked state can be performed, for example, to automatically (i.e., without explicit user input) interact with one or more interactive elements of the current web page. If one or more performed actions prove unsuccessful, then in some embodiments, the second-highest-ranked candidate user state can be selected, and one or more of its associated actions can be performed, and so on. In some embodiments, the success or failure of a particular candidate user state can be used as a signal, for example, to move forward among other candidate user states to rank the candidate user state.

[0011] The successful execution of the action associated with the candidate user state can correspond to whether the user has interacted with the web page to enter the value into all the required interactive elements and pressed "submit". In the web page context, the user's browser will be directed to the next web page in one or more scripts of the web page forming the interactive website, and the user may be required to provide additional input at one or more additional interactive elements. Similarly, in the context of the present disclosure, after the successful execution of the action associated with a candidate user state, the current user state can be updated based on the next web page in the script of the interactive web page of the interactive website. Then, the process can be repeated. The candidate user state that is most similar to the updated current user state (that is, corresponding to when the user arrives at the second web page using the browser) can be identified. These updated candidate user states may or may not be ranked, and actions associated with one or more of the updated candidate user states can be performed until the action associated with one of the updated candidate user states is successful. This process can continue until the transaction is completed, that is, until the automated assistant has successfully navigated the entire script through the interactive website and purchased the desired item or service.

[0012] Additionally or alternatively, in some embodiments, the process of automatically navigating a website using the techniques described herein can be similar to a reinforcement learning operation. Using reinforcement learning, the current state is checked according to a strategy (e.g., a neural network or other statistical model) to select from a plurality of candidate actions. Similarly, using the techniques described herein, the current user state can be checked according to one or more strategies to select one or more actions to be performed. For example, in some embodiments, once the current user state is generated, all verified actions (the process of verifying actions and scripts will be described shortly) can be available. However, there may be a large number of potential actions, and the vast majority of them may be irrelevant to the current user state (e.g., actions related to buying movie tickets may not be relevant in the context of trying to buy a shared bike). Therefore, in various embodiments, there may be a strategy that effectively narrows the list of available actions to those actions that make sense for the current user state. For example, in some embodiments, an applicable strategy can rank the available actions based on the suitability of the available actions to the current user state. The highest ranked actions can be tried first (e.g., in the order of their ranking) until one or more of the actions performed are successful (e.g., navigating to the next web page of the script).

[0013] On an interactive web page (e.g., HTML, XML, etc.), an HTML or XML node (e.g., an interactive element) can usually be identified by a human from the set of words and / or images surrounding it. Otherwise, it would be difficult for the user to identify what to read or where to click after the page has been scrolled up or down slightly. In other words, each interactive element of a web page has its own "visual" or "geometric" context.

[0014] The techniques described herein utilize these geometric contexts so that scripts can be adapted to new websites and / or updated by previously analyzed websites. As previously noted, in some embodiments, a "universal identifier matrix" can be generated for each interactive element in a web page (the current web page under consideration and / or the web page used to generate past user states). The universal identifier matrix for a given interactive element can include data indicating both the given interactive element (e.g., its type, such as a drop-down menu, input box, radio button, etc.) and other visible content of the web page immediately surrounding the given interactive element when the web page is rendered (e.g., as it would be rendered in a web browser application). In other words, the universal identifier matrix provides a "visual" or "geometric" context for each interactive element of a web page, similar to the visual / geometric context that a human would see / understand. The term "universal" refers to the independence of the universal identifier matrix from the underlying document object model ("DOM") tree.

[0015] There are various benefits to segmenting interactive elements into a universal identifier matrix. While web pages can change over time, individual interactive elements and their geometric context within the rendered web page (e.g., what text or other DOM elements they appear near, below, above, etc.) can remain relatively constant. Thus, when a web page changes, the same interactive elements can be positioned at their new locations and / or in their new arrangements based on the contextual clues provided by their corresponding universal identifier matrices. The same actions can then be performed on these same interactive elements as were performed prior to the change to the web page.

[0016] Universal identifier matrices can take various forms. In some embodiments, they can be defined as a set of <position, text> pairs associated with a given interactive element. <position> can be an orientation relative to the center of a bounding rectangle (or other shape) centered on the interactive element of interest. Each element surrounding the interactive element of interest can be classified by its position relative to the interactive element. For example, <(-10,10): "first", "name"> can refer to the text "first name" located ten pixels (or other distance units) to the left and ten pixels (or other distance units) above the interactive element of interest. When comparing two universal identifier matrices constructed in this manner, the distances between the individual pairs can be calculated to determine the overall similarity between the two universal identifier matrices.

[0017] Another less elaborate approach is to define the universal identifier matrix as a grid of n cells (e.g., three by three or nine cells), where the central cell contains the interactive element of interest, and the surrounding cells include the pieces of web page content immediately surrounding the interactive element of interest. A universal identifier matrix constructed from such a grid can scale linearly with the number of cells in the grid.

[0018] As mentioned previously, a database of past user states (and their corresponding actions) can be used to compare with the current user state. These past user states can be generated or "learned" in a variety of ways. For example, they can be generated based on interactions between actual human users and web pages (i.e., a corpus of past user states can be bootstrapped from a limited number of human interactions). Additionally or alternatively, they can be generated based on automatic interactions with web pages.

[0019] However, as previously mentioned, web pages may change over time, e.g., due to the addition of new features, aesthetic updates, etc. Therefore, in various embodiments, past user states may be periodically "validated" or "tested" to ensure, for example, that they are still used as part of a valid script through a corresponding interactive website. For example, a uniform resource locator ("URL") associated with a previously learned web page may be used to periodically (e.g., hourly, daily, weekly, etc.) reload the web page. One or more universal identifier matrices previously generated from a prior version of the web page may be compared with the universal identifier matrix generated from the updated web page, e.g., to develop a mapping between a previously known universal identifier matrix and the universal identifier matrix of the updated web page. If the web page has changed substantially enough, then an appropriate mapping may not be created, and the past user states associated with the URLs may be marked as no longer valid (or simply discarded from the database in some cases).

[0020] On the other hand, if the mapping between the old universal identifier matrix and the new universal identifier matrix is ​​successfully identified, then in various embodiments, various actions associated with the past user states can be performed to determine whether they are still valid with respect to the updated web page. If the same or similar actions are still executable, the updated web page can be considered verified. In some cases, a new user state can be generated for the updated web page and stored in a database for future use. If the same or similar actions are no longer successfully executable, the past user state associated with the URL can be marked as no longer valid (or simply discarded from the database in some cases).

[0021] In some embodiments, when selecting an action to be performed for a given candidate state, another criterion may be considered. For example, in some embodiments, only those actions associated with a universal identifier matrix that can match the universal identifier matrix of the current web page may be performed, or may be performed before other potential actions. Additionally or alternatively, in some embodiments, when a successful navigation or "trace" is detected by a series of web pages forming a website (whether the successful trace is generated automatically by a human or using the technology described herein), the trace may persist in association with the action associated with the individual user state generated from the trace, for example. Those actions may thereafter be considered "verified" and may be favored over other actions for which verification has not yet occurred or at least has not occurred frequently or recently.

[0022] As previously noted, the input elements of the web page can be operated by the user to provide parameters such as pizza toppings, cinema / airplane seat selection, shared bike destination, etc. Many of these parameters can be provided in advance by the user when engaging with the automated assistant. For example, the user can say something like "OK Assistant, order me a thin crust pepperoni pizza from Hypothetical Pizza Shop". Assuming that the hypothetical pizza shop has no active third-party agent to interact directly with the automated assistant, the techniques described herein can be used to automatically navigate the interactive website of the hypothetical pizza shop to purchase pizzas on request. However, assume that the user does not provide the parameters required to complete the order. These missing parameters can be obtained in various ways.

[0023] In some embodiments, aspects of a user's profile, such as the user's preferences (e.g., learned from the user's history of computing interactions) or other information associated with the user's profile can be used to fill in the blanks. For example, a website for a hypothetical pizza shop may require a delivery address. Rather than soliciting this information from the user, the user's address can also be automatically determined, for example, based on his or her profile and / or based on location coordinates provided by a device carried by the user. In some embodiments, the user's address can be made into his or her current user status. This information can then be used to automatically fill in the address field of the hypothetical pizza shop's website without the user having to manually provide such information.

[0024] However, some missing parameters may not be automatically filled in based on the user's profile and / or history. For example, assume that the user tends to use a variety of different payment methods to purchase goods or services. The user may not want the automated assistant to "guess" which payment the user wants to use to pay for the pizza, or the user's preferred payment method may not be accepted by the hypothetical pizza shop. Alternatively, there may be other required parameters for ordering pizza that the user has not specified and that cannot be identified based on her profile (e.g., interest in specials, does she want a drink, etc.). In some embodiments, the automated assistant can pause to traverse the website of the hypothetical pizza shop in order to solicit these missing parameters from the user, for example on an on-demand basis. However, the user may still not need to physically interact with the website - the automated assistant can do that on behalf of the user.

[0025] In some embodiments, a method performed by one or more processors is provided, the method comprising: receiving natural language input from a user at an input component of a computing device, wherein the natural language input is directed by the user to an automated assistant operating at least in part on the computing device; analyzing the natural language input to identify the user's intent and one or more parameters associated with executing the user's intent; obtaining a web page including one or more interactive elements, wherein the web page is operable to execute the user's intent and the one or more interactive elements are operable to input one or more of the parameters; determining a current user state associated with the user, wherein the current user state is determined at least in part based on content of the web page; identifying one or more candidate past user states from a database of past user states based on the current user state user state, wherein each of one or more candidate past user states includes one or more candidate actions for interacting with one or more of the interactive elements of the web page, wherein the one or more candidate actions have been previously verified as being capable of executing at least a portion of the intent; selecting a particular candidate past user state from the one or more candidate past user states based at least in part on a first criterion; selecting a particular candidate action from one or more candidate actions associated with the selected past user state candidate based at least in part on a second criterion; executing the selected candidate action using one or more of the parameters to interact with one or more of the interactive elements of the web page; and providing, by the automated assistant, natural language output as audio or visual output to the user indicating that at least a portion of the user's intent has been executed.

[0026] In various embodiments, the method may further include identifying a webpage based on one or more of a natural language input, a user's intent, or one or more parameters. In various embodiments, each of the one or more candidate actions may have been previously verified as compatible with the same webpage. In various embodiments, each of the one or more candidate actions may have been previously verified as compatible with a different webpage operable to perform the corresponding intent.

[0027] In various embodiments, the first criterion may include that the current user state and the selected candidate past user state have a maximum similarity measure. In various embodiments, determining the current user state may include, for each of one or more interactive elements of the web page, generating a universal identifier matrix, the universal identifier matrix including data indicating both the corresponding interactive element and other visible content of the web page immediately surrounding the corresponding interactive element when the web page is rendered. In various embodiments, identification may include comparing the one or more universal identifier matrices generated from the web page with one or more other universal identifier matrices associated with each of the one or more candidate past user states. In various embodiments, identification may include, for each given candidate past user state in the one or more candidate past user states, comparing the one or more universal identifier matrices generated from the web page with one or more other universal identifier matrices corresponding to one or more candidate actions associated with the given candidate past user state.

[0028] In various embodiments, the method may further include ranking one or more candidate actions associated with a given candidate past user state based on the comparison. In various embodiments, the universal identifier matrix may include one or more tuples, wherein each of the one or more tuples may include a piece of content from a web page and a geometric position of the piece of content relative to an underlying corresponding interactive element used to determine the universal identifier matrix. In various embodiments, the universal identifier matrix may include a two-dimensional grid of cells, wherein a central cell in the two-dimensional grid of cells may correspond to an underlying corresponding interactive element used to determine the universal identifier matrix, and one or more of the two-dimensional grid of cells surrounding the central cell may correspond to other pieces of content from the web page.

[0029] In another aspect, a method implemented using one or more processors may include: determining an identifier for a web page, wherein the identifier was previously verified to indicate that the web page was last known to be operable to perform a user's intent, and wherein the web page was last known to include a first set of one or more interactive elements known to be operable to input one or more parameters associated with the intent; retrieving an updated version of the web page based on the identifier; analyzing the updated version of the web page to identify a second set of one or more interactive elements; generating an updated set of corresponding universal identifier matrices based on the second set of one or more interactive elements identified from the updated version of the web page, the corresponding universal identifier matrices each including data indicating both corresponding interactive elements of the second set and other visible content of the web page immediately surrounding the corresponding interactive elements when the web page is rendered; determining a mapping between the updated set of universal identifier matrices and a known set of universal identifier matrices generated based on a first set of one or more interactive elements last known to be included with the web page; based on the mapping, attempting an action previously performed on the first set of one or more interactive elements on one or more corresponding interactive elements of the second set; and selectively classifying the identifier of the web page as verified or invalid based on one or more results of the attempt. In various implementations, a verified web page identifier can be used to facilitate natural language-based automatic navigation through a website that includes a web page associated with the verified web page identifier.

[0030] In various embodiments, the identifier of the web page may include a uniform resource locator ("URL"). In various embodiments, each given universal identifier matrix of the update set may include one or more tuples, wherein each of the one or more tuples may include a piece of content from the updated version of the web page and a geometric position of the piece of content relative to an underlying interactive element used to determine the given universal identifier matrix. In various embodiments, each given universal identifier matrix of the update set may include a two-dimensional grid of cells, wherein a central cell in the two-dimensional grid of cells may correspond to an underlying interactive element used to determine the given universal identifier matrix, and one or more of the two-dimensional grid of cells surrounding the central cell may correspond to other pieces of content from the updated version of the web page. In various embodiments, the verified web page identifier may be utilized by an automated assistant to facilitate natural language-based automated navigation through a website that includes a web page associated with the verified web page identifier.

[0031] In another aspect, a method implemented using one or more processors may include: obtaining an unknown web page including one or more interactive elements, wherein the unknown web page is operable to perform at least a portion of a purchase and the one or more interactive elements are operable to input one or more parameters for performing at least a portion of the purchase; generating a corresponding first set of one or more universal identifier matrices based on the one or more interactive elements of the unknown web page, wherein each universal identifier matrix of the first set includes data indicating both the corresponding interactive element of the unknown web page and other visible content of the unknown web page immediately surrounding the corresponding interactive element when the unknown web page is rendered; identifying one or more candidate past user states from a database of past user states based at least in part on the first set of universal identifier matrices, wherein the one or more candidate past user states are Each corresponding candidate past user state in the selected past user states includes one or more candidate actions for interacting with one or more interactive elements of a known web page used to generate the corresponding candidate past user state, and wherein a given candidate past user state in the one or more candidate past user states is identified based on a comparison of a first set of universal identifier matrices and a second set of one or more universal identifier matrices associated with the given candidate past user state; one or more candidate actions associated with one or more of the candidate past user states are attempted on one or more of the interactive elements of the unknown web page; and based on one or more results of the attempts, the unknown web page is selectively classified as verified or invalid; wherein the verified web page can be used to facilitate natural language-based automatic navigation through a website that includes the verified web page.

[0032] In various embodiments, the verified web page may be made available to an automated assistant to facilitate natural language based automated navigation through a website including the verified web page. In various embodiments, the method may further include generating a new user state in response to classifying the unknown web page as verified, wherein the new user state may be generated based on the unknown web page, and adding the new user state to a database of past user states.

[0033] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the foregoing methods.

[0034] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0036] Figure 2A , 2B , 2C, 2D, 2E, 2F, 2G, 2H and 2I depict examples of a series of interactive web pages forming an interactive website.

[0037] Figure 3A , 3B and 3C depict how the disclosed technology can be used to at least automatically navigate Figure 2A -I An example of a sequence of interactive web pages.

[0038] Figure 4A and Figure 4B Describes how the disclosed technology can be used to at least automatically navigate Figure 2A Another example of a sequence of interactive web pages of -I.

[0039] Figure 5 One example of how the techniques described herein may be used to automatically interact with a given interactive web page using natural language is schematically depicted in accordance with various implementations.

[0040] Fig. 6A and 6B Depicts an example of how a universal identifier matrix may be generated from interactive elements of a web page according to implementations disclosed herein.

[0041] Figure 7 and Figure 8 Depicted is example pseudo code for implementing various types of universal identifier matrices in accordance with various implementations.

[0042] Fig. 9 Depicted is a flow chart illustrating an example method according to implementations disclosed herein.

[0043] Fig.10 Depicted is a flow chart illustrating another example method according to implementations disclosed herein.

[0044] Fig.11 Depicted is a flow chart illustrating another example method according to implementations disclosed herein.

[0045] Fig.12 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0046] Now turn to Figure 1 , illustrates an example environment in which the techniques disclosed herein may be implemented. The example environment includes one or more client computing devices 106. Each client device 106 may execute a respective instance of an automated assistant client 108, which may also be referred to herein as the "client portion" of the automated assistant. One or more cloud-based automated assistant components 119, which may also be referred to herein as the "server portion" of the automated assistant, may be implemented on one or more computing systems (collectively, "cloud" computing systems) communicatively coupled to the client devices 106 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 114.

[0047] In various embodiments, an instance of automated assistant client 108, through its interaction with one or more cloud-based automated assistant components 119, may take the form of a logical instance of an automated assistant 120 that appears from the user's perspective to be something that the user can engage with in a human-computer conversation. Figure 1 108 is depicted with dashed lines in FIG. 10. It should therefore be understood that each user who engages with an automated assistant client 108 executing on a client device 106 may actually engage with his or her own logical instance of automated assistant 120. For the sake of brevity and simplicity, the term "automated assistant," as used herein as if "serving" a particular user, will refer to the combination of an automated assistant client 108 executing on a client device 106 operated by a user and one or more cloud-based automated assistant components 119 (which may be shared among multiple automated assistant clients 108). It should also be understood that in some embodiments, automated assistant 120 may respond to requests from any user, regardless of whether that user is actually "served" by that particular instance of automated assistant 120.

[0048] One or more client devices 106 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which may include a visual sensor in some cases), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. Some client devices 106, such as stand-alone interactive speakers (or "smart speakers"), may take the form of assistant devices that are primarily designed to facilitate conversations between a user and an automated assistant 120. Some such assistant devices may take the form of a stand-alone interactive speaker with an attached display, which may or may not be a touch screen display.

[0049] In some embodiments, the client device 106 may be equipped with one or more visual sensors 107 having one or more fields of view, but this is not required. The visual sensor 107 may take various forms, such as a digital camera, a passive infrared ("PIR") sensor, a stereo camera, an RGBd camera, and the like. One or more visual sensors 107 may be used to capture image frames (still images or videos) of the environment in which the client device 106 is deployed. These image frames may then be analyzed, for example, by a visual cue module 1121 to detect user-provided visual cues contained in the image frames. These visual cues may include, but are not limited to, gestures, gazes toward specific reference points, facial expressions, predefined movements made by the user, and the like. These detected visual cues may be used for various purposes, such as invoking the automated assistant 120 and / or causing the automated assistant 120 to perform various actions.

[0050] As described in more detail herein, the automated assistant 120 participates in a human-computer conversation session with one or more users via the user interface input and output devices of one or more client devices 106. In some embodiments, the automated assistant 120 may participate in a human-computer conversation session with a user in response to a user interface input provided by the user via one or more user interface input devices of one of the client devices 106. In some of those embodiments, the user interface input is explicitly directed to the automated assistant 120. For example, a user may verbally provide (e.g., type, speak) a predetermined invocation phrase, such as "OK, Assistant" or "Hey, Assistant". When spoken, this verbal input may be captured by the microphone 108 and may cause the automated assistant 120 to begin actively listening or monitoring typed text. Additionally or alternatively, in some embodiments, the automated assistant 120 may be invoked based on one or more detected visual cues, either alone or in combination with a verbal invocation phrase.

[0051] In some embodiments, the automated assistant 120 may participate in a human-computer dialogue session in response to the user interface input even when the user interface input is not explicitly directed to the automated assistant 120. For example, the automated assistant 120 may examine the content of the user interface input and participate in the dialogue session in response to the presence of certain terms in the user interface input and / or based on other clues. In many embodiments, the automated assistant 120 may utilize speech recognition to convert the speech from the user into text, and accordingly respond to the text, for example, by providing search results, general information, and / or taking one or more responsive actions (e.g., playing media, launching a game, ordering food, etc.). In some embodiments, the automated assistant 120 may additionally or alternatively respond to the speech without converting the speech into text. For example, the automated assistant 120 may convert the speech input into an embedding, into an entity representation (which indicates one or more entities present in the speech input) and / or other "non-text" representations, and operate on such non-text representations. Therefore, the embodiments described herein as operating based on text converted from the speech input may additionally and / or alternatively operate directly on the speech input and / or other non-text representations of the speech input.

[0052] Each of the client computing device 106 and the computing device operating the cloud-based automated assistant component 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by the client computing device 106 and / or by the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented, for example, as a computer program running on one or more computers coupled to each other over a network in one or more locations.

[0053] As noted above, in various implementations, client computing device 106 may operate automated assistant client 108 or the “client portion” of automated assistant 120. In various implementations, automated assistant client 108 may include voice capture module 110 and / or visual cue module 1121. In other implementations, one or more aspects of voice capture module 110 and / or visual cue module 1121 may be implemented separately from automated assistant client 108, for example, by one or more cloud-based automated assistant components 119. For example, in Figure 1 There is also a cloud-based visual cue module 1122 that can detect visual cues in image data.

[0054] In various embodiments, the voice capture module 110, which can be implemented using any combination of hardware and software, can interface with hardware such as a microphone 109 or other pressure sensor to capture an audio recording of the user's speech. Various types of processing can be performed on this audio recording for various purposes.

[0055] In various embodiments, the visual cue module 1121 (and / or cloud-based visual cue module 1122) can be implemented using any combination of hardware or software and can be configured to analyze one or more image frames provided by the visual sensor 107 to detect one or more visual cues captured in and / or across one or more image frames. The visual cue module 1121 can employ various techniques to detect visual cues. For example, the visual cue module 1121 (or 1122) can use one or more artificial intelligence (or machine learning) models that are trained to generate outputs indicating user-provided visual cues detected in the image frames.

[0056] The client device 106 may also have other applications installed, such as a web browser 111 and / or a message exchange client 113. The message exchange client 113 may take various forms. In some embodiments, the message exchange client 113 may take the form of a short message service ("SMS") and / or multimedia message service ("MMS") client, an online chat client (e.g., instant messaging, Internet relay chat, or "IRC", etc.), a messaging application associated with a social network, etc. In some embodiments, the message exchange client 113 may be implemented within a web page rendered by the web browser 111. In various embodiments, the message exchange client 113 may provide an interface for a user to engage in a typed or spoken human-computer conversation with the automated assistant 120, either as a one-to-one conversation or as a multi-participant conversation in which the automated assistant 120 can "participate". In some embodiments, the web browser 111 may be specially designed with, for example, a microphone button or other user interface elements operable to invoke the automated assistant 120, so that the user can issue voice commands to assist in the operation of the web browser 111.

[0057] As previously mentioned, the voice capture module 110 can be configured to capture the user's voice, for example, via the microphone 109. Additionally or alternatively, in some embodiments, the voice capture module 110 can be further configured to convert the captured audio into text and / or into other representations or embeddings, for example, using speech-to-text ("STT") processing techniques. Additionally or alternatively, in some embodiments, the voice capture module 110 can be configured to convert text into computer-synthesized speech, for example, using one or more speech synthesizers. However, in some cases, because the client device 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the voice capture module 110 local to the client device 106 can be configured to convert a limited number of different spoken phrases, particularly phrases that invoke the automated assistant 120, into text (or into other forms, such as lower dimensional embeddings). Other voice inputs can be sent to a cloud-based automated assistant component 119, which can include a cloud-based text-to-speech conversion ("TTS") module 116 and / or a cloud-based STT module 117.

[0058] The cloud-based TTS module 116 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some implementations, the TTS module 116 can provide the computer-generated speech output to the client device 106 for direct output, for example, using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the speech capture module 110, which can then convert the text data into computer-generated speech that is output locally.

[0059] The cloud-based STT module 117 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the voice capture module 110 into text, which can then be provided to the intent matcher 135. In some embodiments, the cloud-based STT module 117 can convert the audio recording of the voice into one or more phonemes, and then convert the one or more phonemes into text. Additionally or alternatively, in some embodiments, the STT module 117 can employ a state decoding graph. In some embodiments, the STT module 117 can generate multiple candidate text interpretations of the user's speech. In some embodiments, the STT module 117 can weight or bias a particular candidate text interpretation higher than other candidate text interpretations depending on whether there are visual cues detected at the same time.

[0060] The automated assistant 120 (particularly the cloud-based automated assistant component 119) may include an intent matcher 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some embodiments, one or more of the one or more modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. In some embodiments, to protect privacy, one or more of the components of the automated assistant 120 (such as the natural language processor 122, the TTS module 116, the STT module 117, etc.) may be at least partially implemented on the client device 106 (e.g., excluded from the cloud).

[0061] In some embodiments, the automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-computer conversation session with the automated assistant 120. The automated assistant 120 can provide the responsive content for presentation to the user as part of the conversation session (e.g., over one or more networks when separate from the user's client device). For example, the automated assistant 120 can generate responsive content in response to free-form natural language input provided via the client device 106. As used herein, free-form input is formulated by the user and is not limited to a set of options presented for selection by the user. The free-form natural language input can be spoken (and captured by the microphone 109) and / or typed (e.g., typed into one or more interfaces provided by one or more applications, such as the message exchange client 113).

[0062] As used herein, a "conversation session" may include a logically self-contained exchange of one or more messages between a user and automated assistant 120 (and in some cases, other human participants). Automated assistant 120 may distinguish between multiple conversation sessions with a user based on various signals such as: time elapses between sessions, user context (e.g., location, before / during / after a scheduled meeting, etc.) changes between sessions, one or more intermediate interactions between the user and a client device other than the conversation between the user and the automated assistant are detected (e.g., the user switches applications for a while, the user leaves and then returns to a standalone voice-activated product), a client device locks / sleeps between sessions, a change in client device used to interface with one or more instances of automated assistant 120, etc.

[0063] The intent matcher 135 may be configured to determine the user's intent based on input provided by the user (e.g., vocal utterances, visual cues, etc.) and / or based on other signals such as sensor signals, online signals (e.g., data obtained from a web service), etc. In some embodiments, the intent matcher 135 may include a natural language processor 122 and the aforementioned cloud-based visual cues module 1122. In various embodiments, the cloud-based visual cues module 1122 may operate similarly to the visual cues module 1121, except that the cloud-based visual cues module 1122 may have more resources at its disposal. In particular, the cloud-based visual cues module 1122 may detect visual cues that may be used by the intent matcher 135 alone or in combination with other signals to determine the user's intent.

[0064] Natural language processor 112 may be configured to process natural language input generated by a user via client device 106 and may generate annotation output (e.g., in text form) for use by one or more other components of automated assistant 120. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 106. The generated annotation output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0065] In some embodiments, the natural language processor 122 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the natural language processor 122 may include a lexical module that can separate individual words into morphemes and / or annotate morphemes, for example, by their class. The natural language processor 122 may also include a part-of-speech tagger that is configured to annotate terms by their grammatical roles. For example, the part-of-speech tagger can mark each term by its part of speech, such as "noun", "verb", "adjective", "pronoun", etc. In addition, for example, in some embodiments the natural language processor 122 may additionally and / or alternatively include a dependency parser (not depicted) that is configured to determine syntactic relationships between terms in the natural language input. For example, a dependency parser can determine which terms modify other terms, the subject and verb of a sentence, etc. (e.g., a parse tree)—and such dependencies can be annotated.

[0066] In some embodiments, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some embodiments, a knowledge graph may include nodes representing known entities (and in some cases, representing entity attributes) and edges connecting nodes and representing relationships between entities. For example, a "banana" node may be connected (e.g., as a child) to a "fruit" node, which in turn may be connected (e.g., as a child) to a "product" and / or "food" node. As another example, a restaurant called "Hypothetical Café" may be represented by a node that also includes attributes such as its address, the type of food served, business hours, contact information, etc. The "Hypothetical Café" node can, in some embodiments, be connected via edges (e.g., representing a child-parent relationship) to one or more other nodes, such as a "restaurant" node, a "business" node, a node representing the city and / or state in which the restaurant is located, and the like.

[0067] The entity tagger of the natural language processor 122 may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to a class of entities such as people) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve the specific entity.

[0068] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not depicted) that is configured to group or "cluster" references to the same entity based on one or more context clues. For example, a coreference resolver may be utilized to resolve the term "there" in the natural language input "I liked Hypothetical Cafe last time we ate there" to "Hypothetical Cafe".

[0069] In some implementations, one or more components of the natural language processor 122 can rely on annotations from one or more other components of the natural language processor 122. For example, in some implementations, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. Additionally, for example, in some implementations, a coreference resolver can rely on annotations from a dependency resolver when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 122 can determine one or more annotations using relevant prior inputs and / or other relevant data outside of the particular natural language input.

[0070] Intent matcher 135 may use various techniques to determine the user's intent, for example, based on output from natural language processor 122 (which may include annotations and terms of natural language input) and / or based on output from visual clue modules (e.g., 1121 and / or 1122). In some embodiments, intent matcher 135 may be able to access one or more databases (not depicted) that include, for example, multiple mappings between grammars, visual clues, and response actions (or more generally, intents). In many cases, these grammars may be selected and / or learned over time and may represent the most common intents of the user. For example, a grammar "play <artist>"[play <artist>]" maps to an intent that invokes a response action that causes music by <artist> to be played on a client device 106 operated by the user. Another syntax "[weather|forecast]today" may be able to match user queries such as "what's the weather today" and "what's the forecast for today?".

[0071] In addition to or in lieu of grammars, in some embodiments, the intent matcher 135 can employ one or more trained machine learning models, alone or in combination with one or more grammars and / or visual cues. These trained machine learning models can also be stored in one or more databases and can be trained to recognize intent, for example, by embedding data indicative of a user's utterance and / or any detected user-provided visual cues into a latent space and then determining which other embeddings (and therefore, intents) are closest, for example using techniques such as Euclidean distance, cosine similarity, etc.

[0072] Such as "play <artist>As you can see in the example grammars, some grammars have slots that can be filled with slot values ​​(or "parameters") (e.g., <artist>). Slot values ​​can be determined in a variety of ways. Often the user will actively provide slot values. For example, for the syntax "Order me a <topping>pizza", the user may very well say the phrase "order me a sausage pizza", in which case the slot <topping>Additionally or alternatively, if the user invokes a syntax that includes slots to be filled with slot values, automated assistant 120 can solicit those slot values ​​from the user without the user actively providing the slot values ​​(e.g., "what type of crust do you want on your pizza?"). In some embodiments, the automated assistant 120 can provide a query based on the information provided by visual cue module 112. 1-2 The detected visual cues fill the slots with slot values. For example, a user can say something like "ORDER me this many catbowls" while holding up three fingers toward the visual sensor 107 of the client device 106. Or, a user can say something like "Find me more movies like this" while holding a DVD case of a particular movie.

[0073] The fulfillment module 124 can be configured to receive the predicted / estimated intent and associated slot values ​​(whether actively provided by the user or solicited from the user) output by the intent matcher 135 and execute (or "parse" or "fulfill") the intent. In various embodiments, the fulfillment (or "resolution" or "execution") of the user's intent can cause, for example, various fulfillment information (also referred to as "response" information or "resolution information") to be generated / obtained by the fulfillment module 124. As will be described below, in some embodiments the fulfillment information can be provided to a natural language generator ("NLG" in some figures) 126, which can generate a natural language output based on the fulfillment information.

[0074] Because intent can be fulfilled (or "resolved") in a variety of ways, the fulfillment (or "resolved") information can take a variety of forms. Assume that a user requests pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" The user's intent can be determined as a search query, for example, by intent matcher 135. The intent and content of the search query can be provided to fulfillment module 124, which can be used as a search query. Figure 1 As depicted in , the fulfillment module 124 may communicate with one or more search modules 150 configured to search for response information in a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.). The fulfillment module 124 may provide data indicative of the search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to the search module 150. The search module 150 may provide response information, such as GPS coordinates or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon". This response information may form part of the fulfillment information generated by the fulfillment module 124.

[0075] Additionally or alternatively, fulfillment module 124 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from intent matcher 135, and trigger a response action. Response actions may include, for example, ordering goods / services, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such implementations, fulfillment information may include slot values ​​associated with the fulfillment, a confirmation response (which in some cases may be selected from a predetermined response), etc.

[0076] The natural language generator 126 can be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some embodiments, the natural language generator 126 can be configured to receive as input fulfillment information associated with the fulfillment of the intent, and generate natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 126 can receive information from other sources such as third-party applications (e.g., required slots) that can be used to compose natural language output for the user.

[0077] Various aspects of the present disclosure may be implemented in whole or in part by a website navigation engine 128. Typically, the website navigation engine 128 may be configured to analyze websites, particularly constituent web pages of websites, in order to facilitate automatic browsing through all or part of these websites. More specifically, the website navigation engine 128 may analyze web pages and their constituent interactive elements to generate abstractions of these web pages to generate user states. These user states may capture the underlying functionality of web pages so that if these web pages are changed, they may still be automatically browsed. User states are also similar to previously verified / known websites based on their underlying functionality, thereby enabling automatic navigation through hitherto unknown websites.

[0078] Website navigation engine 128 may access a plurality of databases and / or indexes, such as web page database 127. Web page database 127 may include one or more databases of HTML, XML, PDF, or other documents that are commonly available on the Internet using web browser 111 or other applications and searchable using various search engines (not depicted).

[0079] The website navigation engine 128 may also access a database of past user states 129. The database of past user states 129 may store past user states and corresponding actions that may have been executed (or may potentially be executed) when those past user states were active. The sequence of interactive web pages that form the interactive website may be represented in the database of past user states 129 as a "script" of past user states and corresponding actions. The corresponding actions may be actions that can be executed to interact with the interactive elements of the web pages. Interactive elements may include, but are not limited to, radio buttons, toggle switches, check boxes, text input fields, drop-down menus, sliders, and any other user interface elements that may use technologies such as HTML, XML, CSS, JavaScript, etc.

[0080] In some implementations, each script of past user states / corresponding actions may correspond to a particular website or web page identifier (e.g., a uniform resource locator or "URL," a domain name, an IP address, etc.). For example, one script may be associated with a pizza ordering website, and another script may be associated with a website that a user may operate to order various services (e.g., rides, lawn service, housekeeping, haircuts, reservations, other types of appointments, etc.).

[0081] As will be described in more detail in the near future, in some embodiments, scripts of past user states / corresponding actions may be "verified," for example, by a human, or automatically by the website navigation engine 128 using the techniques described herein. "Verifying" a script means determining that the script is capable of performing the specific intent or purpose of the interactive website to which the script corresponds. The process of verifying a script may occur periodically (e.g., every N seconds, minutes, hours, days, weeks, etc., where N is a positive integer), on demand, etc.

[0082] User states can be generated based on a variety of information and can take a variety of forms. As a non-limiting example, past user states (more generally "user states") can be data structures generated based at least in part on the content of a particular web page, such as semantically rich feature vectors, embeddings, etc. In other words, past user states can at least partially represent an abstraction of a web page. In some embodiments, the web page abstraction can include semantic and / or syntactic information extracted from the web page, and / or a data abstraction representing interactive elements found on the web page, which in some embodiments takes the form of a "universal identifier matrix" described herein. In various embodiments, the universal identifier matrix generated from the interactive elements of the web page can be made from past user states generated from the web page and / or represented as candidate actions associated with past user states.

[0083] Various other data may also be made into a user state, including actions previously performed during a sequence or "trace" of actions and / or parameters utilized in performing those actions. As an example, take a user state generated from the second (or third or beyond) consecutive web page in a sequence of web pages forming an interactive website. This continuous user state ("continuous" meaning not an open user state associated with the interactive website) may include information extracted from the second consecutive (i.e., current) web page in the sequence, as well as information about previous past user states (e.g., indicating previous web pages of the interactive website) and previous actions taken in association with those previous past user states. For example, if an action of populating an interactive element of a previous web page with parameters such as "pepperoni" was performed in a previous past user state, the previously provided parameters may be made into a continuous user state.

[0084] Additionally, in some embodiments, various information about the user, such as may be obtained from the user's profile, may be included in the user status. For example, various user preferences, such as a preferred payment method, whether payment information should be solicited from the user on each transaction, the user's preferred seating orientation (e.g., aisle / window on an airplane, section of a movie theater or sporting event, etc., as will be described in more detail shortly), a preferred price range, or various other product or service specific parameters (e.g., vegetarian or vegan, smoker or non-smoker, etc.) are made into the user status. Other user profile information that may be included in the user status includes, but is not limited to, appointments on the user's electronic calendar (e.g., to automatically eliminate movie showtimes if the user has a conflict), the content of electronic communications (such as emails, text messages, etc. (including metadata associated therewith), social media posts (e.g., status updates, uploaded pictures, etc.), etc.

[0085] User preferences may be determined in a variety of ways. In some embodiments, they may be set manually, e.g., using a graphical user interface, or by the user informing the automated assistant 120 of their preferences. Additionally or alternatively, in some embodiments, some user preferences may be learned over time, e.g., based on user interactions with an interactive website, user interactions with the automated assistant 120, and the like. For example, assume that when purchasing goods or services using a web browser 111, the user always clicks through confirmation screens relatively quickly (e.g., faster than would normally be required to fully consume any disclaimers or other language presented). This may be taken into account when determining that the user may be comfortable with screens that are automatically navigated through by, e.g., the website navigation engine 128 and / or the automated assistant, without explicit input from the user.

[0086] The techniques described herein may even take other signals into account when determining whether to automatically navigate through confirmation pages. For example, assume that when operating the web browser 111, the user tends to click through these pages more quickly when the amount of money spent by the user is below a threshold, but tends to stay on these confirmation pages when the amount of money spent is above the threshold. This threshold may be taken into account when determining whether to automatically navigate through a particular confirmation web page without seeking explicit user input, for example, by the website navigation engine 128.

[0087] Figure 2A -I depicts an example of a sequence or "trace" of web pages that may be interacted with by a user (not depicted) on a client device 206 to purchase items (in this case, two movie tickets) from a fictitious ticket purchasing website fakebooking.com. The client device 206 takes the form of a smart phone or tablet with a touch screen in this example, but this is not intended to be limiting. In these examples, the user operates the web browser 111 of the client device 206 to interact with the interactive website, but other applications could be used instead. For example, another application such as the messaging exchange client 113 may provide a "webview" feature in which it enables a user to interact with web pages loaded into the application's webview. Figure 2A -I, the microphone icon can be operated by the user to invoke the automated assistant 120 (or at least speech recognition) so that the user can give verbal commands.

[0088] Assume that the user starts by providing natural language input "Tickets for 'Hypothetical' on Friday". In response to this request, Figure 2A In, client device 206 receives and loads the first web page that provides the showtimes of the requested movie on the requested date. In this example, the requested movie is being played at two theaters, with three showtimes at each theater.

[0089] Assume that the user selects the "20:45" showing at "The First Theater," for example by clicking on a time (which may be an optional link). The next web page that is loaded is depicted at Figure 2B In. Figure 2B In the example above, the user is informed that he or she has selected a "fake" movie at the first theater on September 21 at 8:45 (the same as 20:45), and that the movie lasts 124 minutes. The user is now faced with two options: log in to an existing account, for example using fakebooking.com, or book as a guest. Assuming the user logs in as a guest, the user is then presented with Figure 2C 's web page.

[0090] exist Figure 2C In the example, the web page provided is a type of web page referred to herein as a "confirmation" web page. In the confirmation web page, once the user has time to consume any disclaimers or information presented on the confirmation page, the user is asked to select some interactive elements ( Figure 2C ). Figure 2C For example, the user is informed that a fee of $0.76 will be charged for each ticket purchased. If the user selects the "Continue" button, the user may be presented with Figure 2D The web page depicted in .

[0091] exist Figure 2D In the example, a new interactive web page is provided to the user, which includes a drop-down menu that allows the user to select how many tickets he or she wants to purchase and the type of ticket (e.g., adult, child, youth, senior, etc.). Each type of ticket has a different price. Figure 2D , the user has not yet interacted with the drop-down menu. Therefore, the "Next" button is inactive as indicated by the shading. Once the user selects at least one ticket, this button may become activated. For example, in Figure 2E In the example above, the user has selected two adult tickets. Therefore, the "Next" button is now activated and the total order value is also depicted. Once the user selects his or her desired tickets and operates the "Next" button, she is taken to the Figure 2F The web page depicted in .

[0092] Figure 2F Another confirmation web page is depicted. At this point the user is informed that the ticket is non-refundable and non-exchangeable. The user is also provided with other general information regarding children entering the theater, etc. Once the user clicks the "Confirm" button, he or she may be presented with Figure 2G 's web page.

[0093] exist Figure 2G In, a web page is presented to a user including a seating chart as one or more interactive elements with which the user can interact to select a seat. Figure 2G , the user has already selected two seats, as indicated by the black boxes. White boxes represent seats that are still available, while shaded boxes represent seats that are unavailable. Once the user selects a seat (e.g., Figure 2E The user can select the same number of seats as the ticket selected in the ticket list) and then operate the "Select" button at the bottom to move to the next screen.

[0094] exist Figure 2H In the payment process, the user is presented with multiple interactive elements, primarily text input fields, with which the user can interact to provide payment information. Once the user enters valid payment information, the user can be taken to Fig.2I 's web page. Fig.2I The web page of is another final confirmation page summarizing the user's purchase and asking for confirmation from the user for the last time. Once the user clicks "Confirm", the transaction can be completed. The user can receive the ticket in various forms, such as in a text message or email, in the mail, etc.

[0095] As from Figure 2A -I, the user is required to interact with nine different interfaces (web pages) in order to purchase the desired merchandise. This process can be cumbersome, especially if the user is physically disabled or otherwise unable to interact with the visual interface (e.g., because she only has a stand-alone interactive speaker available, because she is driving, etc.). Therefore, the techniques described herein, including techniques performed by aspects of the automated assistant 120 and / or the website navigation engine 128, make it possible to automatically navigate through web pages such as Figure 1 and the interactive website depicted in FIG. 2 , even if the owner of the website has not yet created a third-party agent (e.g., Figure 1 150) to allow the automatic assistant 120 to do the same.

[0096] Figure 3A -C describes how browsing can be enabled in whole or at least in part Figure 2A -I, an example of automating the same interactive website depicted in Figure 1 to make it easier for a user to purchase goods and / or services. FIG. 3A to FIG. 3C In the Figure 2A -I, but this is not intended to be limiting. For example, it can be implemented on a stand-alone interactive speaker equipped with a touch screen display and acting as an assistant device. Figure 3A -Interfaces and interactions depicted in C.

[0097] exist Figure 3A , the user may have clicked the microphone and spoken or typed (e.g., into messaging client 113) the natural language input "'Hypothetical' movie Louisville Friday, two tickets." In various embodiments, this natural language input can be processed by various components of automated assistant 120 to identify the user's intent (buy movie tickets) and one or more parameters that can be used to execute that intent (e.g., movie=Hypothetical, date=Friday, September 21, num_tickets=2).

[0098] Based on this data, the automated assistant 120 can retrieve one or more web pages from the same interactive website fakebooking.com and begin automatically navigating at least a portion of the interactive website using the techniques described herein. The user's request does not include all of the necessary parameters for purchasing movie tickets. For example, the user did not state a specific time, theater, or desired seat. Therefore, in various embodiments, the automated navigation of the fakebooking.com website can pause at a selected web page / interface to allow the user to provide any missing information.

[0099] exist Figure 3A In, for example, automated assistant 120 provides showtimes for the requested movie on the requested date. Figure 2A ,exist Figure 3A In , the user can select the desired showtime to proceed. Figure 3B For example, the user taps the microphone and speaks the phrase "8:45 at 1 st theatre (8:45 in the first theatre)". Alternatively, the user may have selected a link to the desired time similarly as above. In any case, the automatic navigation of the interactive website can be resumed as far back as possible until additional missing information is needed.

[0100] The user does not specify which seats he or she wants. Figure 3B In the first sentence of the sentence, the automated assistant 120 provides the output "Please select your seats". Below that, the automated assistant 120 provides a simulated or replicated Figure 2G An interactive seat selection interface with the same interactive elements of the interactive web page depicted in FIG. Once the user selects the same seat, a button listing the selected seat may appear below (it may also simply say "Confirm" or "Next"). Clicking this button may resume automatic navigation to Figure 3C The interface depicted in .

[0101] exist Figure 3C In the example, automated assistant 120 asks the user to confirm the details of the purchase. This information may be compiled from various parameters provided by the user and / or from the underlying website (e.g., Fig.2I ). The user confirms by verbally replying "Yep". The automated assistant 120 replies "Enjoy the show" and may then provide some helpful links, such as a button for adding the event to the user's calendar, a deep link for opening a navigation app with preloaded directions, etc.

[0102] In some embodiments including this example, the user is asked to confirm the purchase. However, this is not always necessary. In some embodiments, if it is sufficiently certain that the user expects to complete the purchase, the purchase can be carried out automatically. For example, if the user frequently makes a purchase of a specific type and always confirms the purchase almost without delay, the end user can stop being asked to confirm the purchase by the user. Additionally or alternatively, in some embodiments, factors such as the price paid, the type of product and / or other signals can be considered when determining whether to require the user to confirm the purchase. For example, if the price paid is lower than a certain threshold value that is speculated to be just in the user's spending comfort zone, the purchase can be automatically confirmed without further user input. Or, if there are other signals that confirm the purchase, such as a calendar entry stating that the user will participate in the event for which the user is buying a ticket, an email to a friend who says something similar, or the user has purchased other items related to the event (e.g., a hotel room, a backstage pass, etc.), the purchase can be automatically confirmed.

[0103] Figure 4A -B depicts Figure 2A -I's interactive website with automatic navigation in Figure 3A -C is an alternative implementation that can make the user experience smoother to a greater extent. In this example, the same client device 206 is depicted. However, Figure 4A -B does not necessarily require a display interface for the interaction between the user and the automated assistant 120. For example, instead of displaying Figure 4A Instead of the showtimes depicted in FIG. 1 , the automated assistant 120 may audibly output different theaters and showtimes. Thus, it may be possible to implement the system using an assistant device that does not even include a display, such as a stand-alone interactive speaker or a vehicle computing system (which may have a screen but may not use it to avoid distracting the driver). Figure 4A The interaction depicted in -B Figure 4A -B.

[0104] Figure 4A and Figure 3A Once the user selects a theater and showtime as depicted in 4B, in various embodiments, automated assistant 120 may present a card that may be selected to complete the transaction. Or, in the alternative, if Figure 4A -B's interaction is purely auditory, then automated assistant 120 may audibly output the details and ask the user for confirmation.

[0105] exist Figure 4A -B, the seat selection table is skipped. This can be due to various reasons. For example, the theater or showtime selected by the user may be open seating, in which case the seats cannot be reserved, but instead are first come first served. Alternatively, in some embodiments, the user's preference for seats can be determined according to the user's profile, for example based on manual input by the user or the user's seat preferences learned over time. Assume that the user always chooses seats between the last three rows and five rows, and only selects the middle part of the theater. Further assume that in the past when these seats were not available, the user did not buy a ticket or bought the desired seats for different showtimes. These signals can be used to learn the user's seat preferences. Therefore, the automated assistant 120 can analyze the seating table to determine whether seats that meet the user's seat preferences are available, and then those seats can be automatically selected. In this way, the user himself or herself does not need to select them. Similar techniques can be used to automatically select seats for airplanes, trains, sporting events, etc.

[0106] Figure 5 Depiction Figure 1 An example of how the various components of can be configured to automatically browse all or part of an interactive website, each including one or more interactive web pages. Although these techniques can be used in some embodiments to allow the automated assistant 120 to automatically browse interactive websites without a corresponding third-party agent (150), this is not intended to be limiting. The techniques described herein can be used to automatically browse all or part of an interactive website in other contexts. For example, the web browser 111 may be equipped with a plug-in that is configured to perform selected aspects of the present disclosure to facilitate automatic website navigation. The user may operate the plug-in while using the web browser 111, for example, by clicking on an icon displayed when the web browser is navigated to an interactive website that has been previously verified for automatic navigation using the techniques described herein. Clicking on the icon can automatically fill in and / or submit as much information as possible from the website, and then the user can be allowed to interact with individual interactive elements to provide missing information.

[0107] exist Figure 5 In step 1 of , a user provides natural language input to automated assistant 120. In this example, the user asks: "Order me a thin crust pepperoni pizza from hypotheticalpizza.com". Assume that the interactive website at hypotheticalpizza.com does not have a corresponding third-party agent (150). Automated assistant 120 can analyze the user's request to determine that the user's intent is to order a pizza from hypotheticalpizza.com, and that automated assistant 120 should use the parameters "toasted pepperoni" and "thin crust" to fulfill the order.

[0108] Based on the analysis, at step 2, the automated assistant 120 and / or the website navigation engine 128 can retrieve, for example, from the web page database 127, the open interactive web pages of the website at the requested web identifier (hypotheticalpizza.com). At step 3, a current user state can be generated based at least in part on the retrieved open web pages and / or based on other information such as the user's preferences, calendar, etc. As previously noted, this current user state can take various forms, such as feature vectors, semantically rich embeddings, etc. In some cases, the web page content can be analyzed to generate one or more universal identifier matrices based on one or more interactive elements of the web page. Data indicating these universal identifier matrices can be included in the user state.

[0109] At step 4, the user state may be compared with one or more candidate past user states 560 contained in the past user state database 129. 1-M Match (M is a positive integer). This matching can be based on various criteria, such as a similarity measure between the current user state and each of the candidate past user states 560. For example, in some embodiments, only those candidate past user states 550 that have a similarity measure that satisfies a threshold with the current user can be selected, or the top most similar candidate past user states can be selected. Alternatively, the corresponding universal identifier matrices of the candidate past user states and the current user state can be matched.

[0110] It is not necessary that the candidate past user states have been generated from the same website as the current user state. In many cases, it may be sufficient to generate the candidate past user states from functionally similar web pages, so that similar actions will be successful whether performed on the functionally similar web page or the web page used to generate the current user state. In this way, in some embodiments the website navigation engine 128 can automatically navigate portions of heretofore unknown or unencountered websites based on their similarity to known websites for which scripts have been previously validated.

[0111] like Figure 5 As depicted in , each candidate past user state 560 can be associated with one or more actions. For example, the first candidate past user state 5601 is associated with action 1,1, action 1,2, ..., and action 1,N (N is a positive integer). The second candidate past user state 5602 is associated with action 2,1, action 2,2, ..., and action 2,N, etc. Each action can correspond to, for example, an interactive web page element. In some embodiments, each action can be represented by a universal identifier matrix generated from specific interactive elements encountered on a web page in the past. Thus, an action may include, for example, entering a parameter into a text field, selecting a parameter from a drop-down list, checking a radio button or check box, toggling a toggle switch, activating a "Submit" or "Confirm" or "Next" button, etc.

[0112] At step 5, one or more actions may be selected and performed based on various criteria. If successful, the next web page in the script may be loaded through the interactive website at step 6, and the process may be repeated. If the selected action is not successful, another action may be selected until an action is successful. In some embodiments, when an action is successful, the current user state may be added to the past user state database 127, for example, in association with at least one successful action. In this way, the system "learns" how to automatically navigate the interactive website and / or is able to verify / re-verify a known script through the interactive website.

[0113] exist Figure 5 In the example of , each candidate past user state 560 has N potential actions. This may be because, for example, all possible actions become available to each candidate past user state 560. In various embodiments, one or more strategies may specify which actions to attempt, or in what order to attempt actions. In some such embodiments, all actions 1-N may be ranked, for example, according to one or more strategies, based on various criteria such as the similarity between their corresponding universal identifier matrices (which, as noted above, may be associated with past user states and / or their related actions). However, this is not intended to be limiting. In some embodiments, each past user state 560 may be associated only with those actions that match the interactive elements (and therefore, the universal identifier matrix) found in the underlying web pages used to generate those past user states. This actually reduces the search space for potential actions for a given candidate past user state.

[0114] Other strategies are contemplated herein. For example, in some embodiments, the state / action may be governed by a strategy that evolves over time, e.g., similar to reinforcement learning. For example, a strategy may take the form of a neural network or other statistical model that is trained to receive the current user state as input and provide data indicating which action should be performed first, second, etc. as output. For example, the output may be the probability that each potential action will succeed. In some embodiments, the actions most likely to succeed may be performed, for example, in order of their probability of success.

[0115] In some embodiments, whenever an interactive website is successfully navigated using the techniques described herein, whether by humans or automatically, a strategy can be continuously trained. For example, each web page in a sequence of web pages operated to browse an interactive website can be used to generate a state. The interaction with the website can be divided into "rounds". During each round, actions can be selected and performed based on the strategy and the current user state to interact with the interactive elements of the new web page. If a series of rounds through the web pages of the interactive website are successful, a cumulative reward may be generated and used to train a strategy for each current user state / action pair of each round. For example, a cumulative reward may be used to perform techniques such as gradient descent and / or back propagation for a strategy for each user state / action performed. Over time, a strategy may become better at selecting actions based on the current user state in order to successfully and automatically browse an interactive website.

[0116] Additionally or alternatively, in some embodiments, for example, if the selected action is unsuccessful, a separate per-round reward (or penalty) may be triggered. For example, if the action performed includes attempting to fill in a credit card number field with a date, the underlying website may issue a warning (e.g., using JavaScript) that the input is invalid. This may constitute a failure of the action, which may cause a per-round penalty to be triggered and used to train the policy, e.g., as a negative training example. Similarly, a positive per-round reward may be triggered when the action is successful.

[0117] Fig. 6A -B shows two examples of how a universal identifier matrix may be generated in accordance with various embodiments. Fig. 6A -B, depicts a user interface for a fake railroad. This interface can be operated by a user to book train tickets. For example, the user can type or otherwise enter the source of his or her desired ticket in input box 670 ("Von" translates to "From" in English). The user can type or otherwise enter the destination of his or her desired ticket in input box 671 ("Nach"). And so on. Once the user has provided input for the various interactive elements, the user can then search for available tickets by pressing button 668. Notably, Fig. 6A The text in the interface of -B is in German. However, even non-German speakers are likely to understand what each of the interactive elements represents based on the fact that those interactive elements have a visual and / or geometric context that is likely familiar from other similar websites, such as a website for purchasing airline tickets.

[0118] Thus, in various embodiments, the described techniques may be employed to generate a universal identifier matrix 672A, 672B for each interactive element of an interface (i.e., a web page) that includes data indicating both the corresponding interactive elements and other visible content of the web page / interface that immediately surrounds the corresponding interactive elements when the web page is rendered. More generally, the universal identifier matrix acts as a selector for HTML / XML nodes based on their geometric context and type, so that they can be easily retrieved from a web page even if the web page structure and / or DOM tree changes significantly. In some embodiments, nodes retrieved using the universal selector matrix of a recently modified web page may be of the same type (e.g., radio button, text input, etc.) and may be able to receive the same actions as the original nodes.

[0119] exist Figure 6A-6B A "grid-based" approach for generating a universal identifier matrix is ​​described in . Fig. 6A , a first universal identifier matrix 672A is generated for button 668. The first universal identifier matrix 672A includes nine cells in a three-by-three arrangement. Each cell may have a predetermined width and / or height that may be changed as needed / as desired. Other numbers, sizes, and / or arrangements of cells are contemplated herein, such as five-by-five, nine-by-nine, etc. The first universal selector 672A is centered on the active portion of button 668 (and therefore captures the active portion of button 668). As shown in the enlarged version of the first universal identifier matrix 672A at the bottom, the left top cell captures the text "AN" from the interface, the left middle cell captures the text "SUCHEN" from the interface, and the bottom middle cell captures the text "SUCHE" from the interface. In some embodiments, only entire words completely captured by a cell may be included in the universal identifier matrix, but this is not required.

[0120] exist Figure 6B 6, the second universal identifier matrix 672B is centered on the text input 670 ("VON"). As shown in the enlarged version of the second universal identifier matrix 672B at the bottom, the left top cell captures the text "VON" from the interface, the right top cell captures the text "FAKE" from the interface, and the left bottom cell captures the text "NACH" from the interface.

[0121] These universal identifier matrices 672A-B and Fig. 6A -B may be stored, for example, in a database 127 of past user states. For example, each past user state may include data indicating these universal identifier matrices. Additionally or alternatively, these universal identifier matrices may be used to define actions that can be performed in those past user states. As previously described, when encountering a website associated with a fake railroad during automatic navigation, Fig. 6A -B interface, parameters provided by the user (either beforehand as part of the natural language input or later) can be used to attempt to interact with the interactive elements that form the basis of these universal identifier matrices.

[0122] In some embodiments, a particular DOM node (or more generally, an interactive element) may be searched for in a given web page (or webview) as follows. A reference universal matrix generated from the searched post node may be retrieved. A first universal matrix may be generated that captures the entire web page (i.e., covers the entire web page). Then, for each cell of this first universal matrix, another smaller universal identifier matrix may be generated that is centered in the cell and fits the entire cell. Then, the process may loop through each of the smaller universal identifier matrices to find the closest smaller universal identifier matrix to the reference universal identifier matrix. Then, the process may loop through all DOM nodes found within the closest smaller universal identifier matrix to identify DOM nodes of the same type as the reference universal identifier matrix.

[0123] In some implementations of this "grid-based" approach, the similarity between universal identifier matrices can be determined based on the distance between them. For example, the "distance" between two grid-based universal identifier matrices can be calculated based on a set of words by counting the total number of differences across the grid cells. Figure 7 An example pseudocode for determining the distance between two universal identifier matrices under a "grid-based" approach is depicted in FIG. Note that the distance can be calculated between universal identifier matrices of different grid sizes. In some embodiments, Figure 7 The following line of code in "distance += SetDiff(Ny.GetTerms(i), Nx.GetTerms(j)).size;" can be replaced with the following code:

[0124]

[0125] This change can make the distance measure more robust to term variations by using the inverse document frequency (idf). In the above replacement code, idf is a mapping from a string to log(|D| / |D(term)|), where |D| is the number of common identifier matrices and |D(term)| is the number of common identifier matrices that have the term as text in the cells surrounding the central cell.

[0126] like Fig. 6A One benefit of the grid-based approach to generating the universal identifier matrix, as demonstrated in Figure 1-B, is that it scales linearly with the number of cells in the grid. However, other approaches for generating the universal identifier matrix are contemplated herein. For example, in some embodiments, the universal identifier matrix may be defined as a matrix of cells associated with a given interactive element.<location,text> A collection of pairs. <location>It can be the orientation relative to the center of a bounding rectangle (or other shape) centered on the interactive element of interest. Each element surrounding the interactive element of interest can be classified by its position relative to the interactive element. For example, <(-10,10):"first","name"> can refer to the text "firstname" located ten pixels (or other distance unit) to the left and ten pixels (or other distance unit) above the interactive element of interest. When comparing two universal identifier matrices constructed in this manner, the distances between the individual pairs can be calculated to determine the overall similarity between the two universal identifier matrices. Although the computational complexity is increased, this method can provide a more accurate representation of the geometric / visual context of the interactive element. Example pseudocode for such an approach is depicted in Figure 8 middle.

[0127] Fig. 9 900 is a flowchart illustrating an example method 900 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of the automated assistant 120 and / or the website navigation engine 128. Furthermore, although the operations of the method 900 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0128] At block 902, the system may receive natural language input from a user at an input component of a computing device (e.g., client device 106, 206). In various implementations, the user may direct the natural language input to an automated assistant 120 that operates at least in part (e.g., with the aid of automated assistant client 108) on the computing device. At block 904, the system, e.g., with the aid of cloud-based automated assistant component 119, may analyze the natural language input to identify the user's intent and one or more parameters associated with executing the user's intent.

[0129] At block 906, the system may obtain a web page including one or more interactive elements. In some implementations, the web page may be identified based on one or more of a natural language input, a user's intent, or one or more parameters. For example, the web page may be operable to execute the user's intent. The one or more interactive elements may be operable to input one or more parameters.

[0130] At block 908, the system may determine a current user state associated with the user. As previously described, the current user state (and generally any user state) may be represented by various types of data structures, such as a feature vector, an embedding, one or more universal identifier matrices, etc. In various implementations, the current user state may be determined based at least in part on the content of the web page (including its constituent interactive elements).

[0131] At block 910, the system may identify one or more candidate past user states (e.g., 560) from the database of past user states 127 based on, for example, the current user state determined (e.g., generated) at block 908. 1-M ). In some embodiments, each of the one or more candidate past user states may include, for example, one or more candidate actions for interacting with one or more interactive elements of a web page. In some embodiments, the one or more candidate actions may have been previously verified as being able to perform at least a portion of the intent.

[0132] At block 912, the system may select a particular candidate past user state from one or more candidate past user states based at least in part on a first criterion. Various first criteria may be employed, such as a similarity metric (e.g., distance) between two states, a similarity metric (e.g., distance) between universal identifier matrices thereof, and the like. At block 914, the system may select a particular candidate action from one or more candidate actions associated with the particular candidate past user state based at least in part on a second criterion. For example, the system may select an action for a past user state represented by a universal identifier matrix that is most similar to a universal identifier matrix generated from an interactive element of a web page used to generate the current user state.

[0133] At block 916, the system may use one or more parameters to perform the selected candidate action to interact with one or more interactive elements of the web page. For example, one parameter may be used to select from a drop-down menu, another parameter may be used to fill in a text input field, etc. At block 918, the system may provide a natural language output indicating that at least a portion of the user's intent has been performed to the user as audio or visual output, for example, with the aid of the automated assistant 120.

[0134] Fig.10 An example method 1000 for practicing selected aspects of the present disclosure is depicted in accordance with various embodiments. In particular, the method 100 may be performed to revalidate previously validated web pages / websites, such as where those web pages have changed. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, including an automated assistant 120 and / or a website navigation engine 128. Furthermore, although the operations of the method 1000 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0135] At block 1002, the system may determine an identifier of a web page, such as a URL, domain, IP address, etc. In various implementations, the identifier may have been previously verified to indicate that the web page was last known to be operable to implement the user's intent. In various implementations, the web page may be last known to include a first set of one or more interactive elements that are known to be operable to input one or more parameters associated with the intent.

[0136] At block 1004, the system may retrieve (e.g., download) an updated version of the web page based on the identifier. At block 1006, the system may analyze the updated version of the web page retrieved at block 1004 to identify a second set of one or more interactive elements. At block 1008, the system may generate an updated set of corresponding universal identifier matrices based on the second set of one or more interactive elements identified from the updated version of the web page retrieved at block 1004. Each updated set may include data indicating both the corresponding interactive element of the second set and other visual content of the web page that immediately surrounds the corresponding interactive element when the web page is rendered.

[0137] At block 1010, the system can determine a mapping between the updated set of universal identifier matrices and a known set of universal identifier matrices generated based on a first set of one or more interactive elements last known to be included with the web page. In other words, the system attempts to make the interactive elements of the updated web page consistent with the interactive elements of the previous version of the web page, for example, using past user state and other information available in past user state database 129.

[0138] At block 1012, the system can attempt actions previously performed on the first set of one or more interactive elements on one or more corresponding interactive elements of the second set based on the mapping. In other words, the system attempts to use the same actions that were previously verified to see if they are still valid and to determine if the mapping is reasonable.

[0139] Based on one or more results of the attempt, at block 1014, the system can selectively classify the identifier of the web page as verified or invalid. For example, if the same actions succeed on an updated version of the website, those actions (e.g., forming part of a script through an interactive website) can be considered verified. Thus, the system can make the newly verified identifier available as part of a plurality of verified web page identifiers (corresponding to verified past user states) that can be used to facilitate natural language-based automatic navigation through websites that include web pages associated with the verified web page identifiers.

[0140] Fig.11 An example method 1100 for practicing selected aspects of the present disclosure is depicted in accordance with various embodiments. In particular, Fig.11 The operations of method 1100 may be used to analyze heretofore unknown interactive websites in order to facilitate automatic browsing of those websites. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, including automated assistant 120 and / or website navigation engine 128. Furthermore, although the operations of method 1100 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0141] At block 1102, the system can obtain an unknown web page including one or more interactive elements, for example based on a website identifier provided by a user. The unknown web page can be operable to perform at least a portion of a purchase (e.g., purchase of goods or services), and the one or more interactive elements can be operable to input one or more parameters to perform at least a portion of the purchase.

[0142] At block 1104, the system can generate a corresponding first set of one or more universal identifier matrices, for example, based on one or more interactive elements of the unknown web page. As previously described, each universal identifier matrix of the first set can include data indicating both a corresponding interactive element of the unknown web page and other visual content of the unknown web page that immediately surrounds the corresponding interactive element when the unknown web page is rendered.

[0143] At block 1106, the system may identify one or more candidate past user states from the database 127 of past user states, for example, based at least in part on the first set of universal identifier matrices. These may again be identified using various criteria, such as overall (e.g., entire web page) similarity / distance or similarity / distance between corresponding interactive elements (or between universal identifier matrices generated therefrom). As in previously described implementations, each respective candidate past user state of the one or more candidate past user states may include one or more candidate actions for interacting with one or more interactive elements of the known web page used to generate the respective candidate past user state. And in various embodiments, a given candidate past user state of the one or more candidate past user states may be identified based on a comparison of the first set of universal identifier matrices and the second set of one or more universal identifier matrices associated with the given candidate past user state.

[0144] At box 1108, the system may attempt one or more candidate actions associated with one or more candidate past user states on one or more interactive elements of the unknown web page. If method 1100 is automatically performed to facilitate automatic navigation of unknown websites in the future, in some embodiments, the constituent web pages of the unknown website can be downloaded into a test system (e.g., a sandbox) and dummy values ​​can be used as parameters for testing purposes. At box 1110, the system may selectively classify the unknown web page as verified or invalid based on one or more results of the attempt. As before, verified web pages may be used to facilitate natural language-based automatic pairing navigation through websites that include verified web pages.

[0145] Certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, user location, user time, user biometric information, and user activities and demographic information, relationships between users, etc.), and provide the user with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user to do so.

[0146] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to that user to provide permission or authorization as to whether information is to be collected and which portions of the information are to be collected. For example, one or more such control options may be provided to a user over a communications network. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be summarized as a larger area so that the user's specific location cannot be determined.

[0147] Fig.12 12 is a block diagram of an example computing device 1210 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, website navigation engine 128, automated assistant 120, and / or other components can include one or more components of the example computing device 1210.

[0148] The computing device 1210 typically includes at least one processor 1214 that communicates with a number of peripheral devices via a bus subsystem 1212. These peripheral devices may include a storage subsystem 1224 (including, for example, a memory subsystem 1225 and a file storage subsystem 1226), a user interface output device 1220, a user interface input device 1222, and a network interface subsystem 1216. The input and output devices allow a user to interact with the computing device 1210. The network interface subsystem 1216 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0149] The user interface input devices 1222 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 1210 or onto a communication network.

[0150] User interface output device 1220 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computing device 1210 to a user or to another machine or computing device.

[0151] Storage subsystem 1224 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 1224 may include an executable Figure 7-9 Selected aspects of the method and implementation Figure 1 The logic of the various components is depicted in .

[0152] These software modules are typically executed by processor 1214 alone or in combination with other processors. The memory 1225 used in storage subsystem 1224 can include many memories, including a main random access memory (RAM) 1230 for storing instructions and data during program execution and a read-only memory (ROM) 1232 in which fixed instructions are stored. File storage subsystem 1226 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functionality of certain implementations can be stored in storage subsystem 1224 by file storage subsystem 1226, or in other machines accessible to processor 1214.

[0153] The bus subsystem 1212 provides a mechanism for the various components and subsystems of the computing device 1210 to communicate with each other as intended. Although the bus subsystem 1212 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0154] The computing device 1210 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Fig.12 The description of the computing device 1210 depicted in FIG. 1 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 1210 may have more Fig.12 The computing devices depicted may have greater or fewer components.

[0155] Although several implementations have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications of the teachings used. Those skilled in the art will recognize or be able to use more than one routine experiment to ascertain many equivalents of the specific implementations described herein. Therefore, it should be understood that the foregoing implementations are presented only by way of example, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementations disclosed herein are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.< / location> < / topping> < / topping> < / artist> < / artist> < / artist>

Claims

1. A method implemented using one or more processors, comprising: determining an identifier of a web page, wherein the identifier was previously verified to indicate that the web page was last known to be operable to perform an intent of a user, and wherein the web page was last known to include a first set of one or more interactive elements known to be operable to input one or more parameters associated with the intent; retrieving a latest version of the webpage using the identifier; analyzing the updated version of the web page to identify a second set of one or more interactive elements; Generating a first set of universal identifier matrices, wherein the generating comprises: for each interactive element in the identified second set, generating a corresponding universal identifier matrix, the corresponding universal identifier matrix comprising a two-dimensional grid of cells that semantically captures the interactive element and its surrounding visual context, wherein each cell of the two-dimensional grid has a predetermined width and height, and the generating comprises: positioning the two-dimensional grid over the interactive element to spatially align the underlying interactive element with a central cell of the two-dimensional grid such that other content segments surrounding the interactive element are covered by one or more other cells of the two-dimensional grid that are spatially adjacent to the central cell, and including spatially captured content within each cell of the two-dimensional grid of cells in the universal identifier matrix; determining a reference set of a matrix of universal identifiers, the reference set of universal identifiers being generated based on the first set of one or more interactive elements previously known to be included with a prior version of the web page associated with the identifier; calculating a similarity measure between the universal identifier matrix of the first set and each universal identifier matrix of the universal identifier matrix of the known set; matching one or more universal identifier matrices of the first set with one or more universal identifier matrices of the known set based on the similarity measure; Based on the matching, performing an action previously performed on the first set of one or more interactive elements on one or more corresponding interactive elements in the second set; and selectively classifying the identifier of the webpage as validated or invalid based on one or more results of the executing; wherein verified web page identifiers are marked as usable for facilitating natural language-based automatic navigation through a website including the web page associated with the verified web page identifier, and invalid web page identifiers are marked as unusable for natural language-based automatic navigation or are discarded.

2. The method according to claim 1, wherein: The identifier of the web page includes a uniform resource locator ("URL").

3. The method according to claim 1, wherein: The verified web page identifier may be used with an automated assistant to facilitate the natural language-based automated navigation through the website including the web page associated with the verified web page identifier.

4. The method according to claim 3, wherein: The automated assistant facilitates the natural language-based automated navigation via a spoken conversation with a user using a microphone and a speaker.

5. The method according to claim 4, wherein: The natural language based automated navigation is facilitated without rendering content on a display.

6. A method implemented using one or more processors, comprising: obtaining an unknown webpage including one or more interactive elements, wherein the unknown webpage is operable to perform at least a portion of a purchase and the one or more interactive elements are operable to input one or more parameters for performing the at least a portion of the purchase; Generating a first set of universal identifier matrices, wherein the generating comprises, for each interactive element of the one or more interactive elements, generating a corresponding universal identifier matrix, the corresponding universal identifier matrix comprising a two-dimensional grid of cells that semantically captures the interactive element and its surrounding visual context, wherein each cell of the two-dimensional grid has a predetermined width and height, and the generating comprises: positioning the two-dimensional grid over the interactive element to spatially align the underlying interactive element with a central cell of the two-dimensional grid such that other content segments surrounding the interactive element are covered by one or more other cells of the two-dimensional grid that are spatially adjacent to the central cell, and including spatially captured content within each cell of the two-dimensional grid of cells in the universal identifier matrix; identifying one or more candidate past user states from a database of past user states based at least in part on one or more universal identifier matrices in the first set of universal identifier matrices, wherein each corresponding candidate past user state of the one or more candidate past user states comprises one or more candidate actions for interacting with one or more interactive elements of a known web page used to generate the corresponding candidate past user state, and wherein a given candidate past user state of the one or more candidate past user states is identified based on matching one or more universal identifier matrices of the first set of universal identifier matrices with one or more reference universal identifier matrices associated with the given candidate past user state, wherein the matching is based on a similarity measure between one or more universal identifier matrices of the universal identifier matrices and one or more reference universal identifier matrices of the reference universal identifier matrices; attempting one or more candidate actions associated with one or more of the candidate past user states on one or more of the interactive elements of the unknown web page; and selectively classifying the unknown webpage as verified or invalid based on one or more results of the attempts; Therein, the verified web pages are marked as being useful for facilitating natural language-based automatic navigation through a website including the verified web pages.

7. The method according to claim 6, wherein: The validated web page may be used with an automated assistant to facilitate the natural language-based automated navigation through the website including the validated web page.

8. The method according to claim 7, wherein: The automated assistant facilitates the natural language-based automated navigation via a spoken conversation with a user using a microphone and a speaker.

9. The method according to claim 8, wherein: The natural language based automated navigation is facilitated without rendering content on a display.

10. The method of claim 6, further comprising generating a new user status in response to classifying the unknown webpage as verified.

11. The method according to claim 8, wherein: The new user state is generated based on the unknown web page and added to the database of past user states.

12. A system comprising one or more processors and a memory storing instructions, the instructions, in response to being executed by the one or more processors, causing the one or more processors to: Determine the identifier of the web page, where the identifier was previously verified to indicate that the webpage was last known to be operable to perform an intent of a user, and wherein the webpage was last known to include a first set of one or more interactive elements known to be operable to input one or more parameters associated with the intent; retrieving a latest version of the webpage using the identifier; analyzing the updated version of the web page to identify a second set of one or more interactive elements; Generating a first set of universal identifier matrices, wherein the instructions for generating include instructions for: for each interactive element in the identified second set, generating a corresponding universal identifier matrix, the corresponding universal identifier matrix comprising a two-dimensional grid of cells that semantically capture the interactive element and its surrounding visual context, wherein each cell of the two-dimensional grid has a predetermined width and height, and the generating comprises: positioning the two-dimensional grid over the interactive element to spatially align the underlying interactive element with a central cell of the two-dimensional grid such that other content segments surrounding the interactive element are covered by one or more other cells of the two-dimensional grid that are spatially adjacent to the central cell, and including spatially captured content within each cell of the two-dimensional grid of cells in the universal identifier matrix; determining a reference set of a matrix of universal identifiers, the reference set of universal identifiers being generated based on the first set of one or more interactive elements previously known to be included with a previous version of the web page associated with the identifier; calculating a similarity measure between the universal identifier matrix of the first set and each universal identifier matrix of the universal identifier matrix of the known set; matching one or more universal identifier matrices of the first set with one or more universal identifier matrices of the known set based on the similarity measure; performing, based on the matched universal identifier matrix, an action previously performed on the first set of one or more interactive elements on one or more corresponding interactive elements in the second set; and selectively classifying the identifier of the webpage as validated or invalid based on one or more results of performing an action; wherein verified web page identifiers are marked as usable for facilitating natural language-based automatic navigation through a website including the web page associated with the verified web page identifier, and invalid web page identifiers are marked as unusable for natural language-based automatic navigation or are discarded.

13. The system according to claim 12, wherein: The identifier of the web page includes a uniform resource locator ("URL").

14. The system according to claim 12, wherein: The verified web page identifier may be used with an automated assistant to facilitate the natural language-based automated navigation through the website including the web page associated with the verified web page identifier.

15. The system of claim 14, wherein: The automated assistant facilitates the natural language-based automated navigation via a spoken conversation with a user using a microphone and a speaker.

16. The system of claim 15, wherein: The natural language based automated navigation is facilitated without rendering content on a display.

17. A system comprising one or more processors and a memory storing instructions, the instructions, in response to being executed by the one or more processors, causing the one or more processors to: An unknown web page including one or more interactive elements is obtained, wherein: The unknown web page is operable to perform at least a portion of a purchase, and the one or more interactive elements are operable to input one or more parameters for performing the at least a portion of the purchase; generating a first set of universal identifier matrices, wherein the generating comprises, for each interactive element of the one or more interactive elements, wherein the instructions for generating comprise instructions for: generating a universal identifier matrix, the respective universal identifier matrix comprising a two-dimensional grid of cells that semantically captures the interactive element and its surrounding visual context, wherein each cell of the two-dimensional grid has a predetermined width and height, and the generating comprises: positioning the two-dimensional grid over the interactive element to spatially align the underlying interactive element with a central cell of the two-dimensional grid such that other content segments surrounding the interactive element are covered by one or more other cells of the two-dimensional grid that are spatially adjacent to the central cell, and including spatially captured content within each cell of the two-dimensional grid of cells in the universal identifier matrix; identifying one or more candidate past user states from a database of past user states based at least in part on one or more universal identifier matrices in the first set of universal identifier matrices, wherein each corresponding candidate past user state of the one or more candidate past user states comprises one or more candidate actions for interacting with one or more interactive elements of a known web page used to generate the corresponding candidate past user state, and wherein a given candidate past user state of the one or more candidate past user states is identified based on matching one or more universal identifier matrices of the universal identifier matrices in the first set with one or more reference universal identifier matrices associated with the given candidate past user state, wherein the matching is based on a similarity measure between one or more universal identifier matrices of the universal identifier matrices and one or more reference universal identifier matrices of the reference universal identifier matrices; attempting one or more candidate actions associated with one or more of the candidate past user states on one or more of the interactive elements of the unknown web page; and selectively classifying the unknown webpage as verified or invalid based on one or more results of the attempted one or more candidate actions; Therein, the verified web pages are marked as being useful for facilitating natural language-based automatic navigation through a website including the verified web pages.

18. The system of claim 17, wherein: The validated web page may be used with an automated assistant to facilitate the natural language-based automated navigation through the website including the validated web page.

19. The system of claim 17, further comprising instructions for generating a new user status in response to classifying the unknown webpage as verified.

20. The system of claim 19, wherein: The new user state is generated based on the unknown web page and added to the database of past user states.

21. A method implemented using one or more processors, comprising: receiving natural language input from a user at an input component of a computing device, wherein the natural language input is directed by the user to an automated assistant operating at least in part on the computing device and identifies an intent of the user; retrieving a web page including one or more interactive elements, wherein the web page is operable to execute the identified intent of the user, and the one or more interactive elements are operable to input one or more parameters associated with executing the intent of the user; generating one or more embeddings representing one or more aspects of the natural language input and at least some of the interactive elements of the retrieved web page as the user's current state; processing the one or more embeddings using one or more neural networks to generate an output indicative of a plurality of candidate actions for interacting with one or more of the interactive elements of the webpage; selecting a given candidate action from the plurality of actions based on one or more criteria; performing the selected candidate action to interact with one or more of the interactive elements of the web page using one or more of the parameters; and Output is provided by the automated assistant to the user as audio or visual output indicating a result of performing the selected candidate action.

22. The method according to claim 21, wherein: The one or more neural networks include strategies for automatic navigation of web pages.

23. The method according to claim 21, wherein: The one or more criteria include a ranking of the plurality of candidate actions.

24. The method according to claim 21, wherein: The output indicative of the plurality of candidate actions includes data indicative of a sequence in which the plurality of candidate actions should be performed.

25. The method of claim 21, further comprising: Determine whether execution of the selected candidate action was successful.

26. The method according to claim 25, wherein: The output indicative of the result of performing the selected candidate action includes an indication of whether the selected candidate action was successful.

27. The method of claim 25, further comprising: determining a reward or penalty based on whether the execution of the selected candidate action was successful; as well as One or more of the neural networks are trained based on the determined reward or penalty.

28. The method of claim 21, wherein: The output indicating the result of performing the selected candidate action includes navigating from the web page to a new web page.

29. A system comprising one or more processors and a memory storing instructions that, in response to being executed by the one or more processors, cause the one or more processors to: receiving natural language input from a user at an input component, wherein: The natural language input is directed by the user to an automated assistant operating at least in part on the computing device and identifies the user's intent; retrieving a web page including one or more interactive elements, wherein the web page is operable to execute the identified intent of the user, and the one or more interactive elements are operable to input one or more parameters associated with executing the intent of the user; generating one or more embeddings representing one or more aspects of the natural language input and at least some of the interactive elements of the retrieved web page as the user's current state; processing the one or more embeddings using one or more neural networks to generate an output indicative of a plurality of candidate actions for interacting with one or more of the interactive elements of the webpage; selecting a given candidate action from the plurality of actions based on one or more criteria; performing the selected candidate action to interact with one or more of the interactive elements of the web page using one or more of the parameters; and Output is provided by the automated assistant to the user as audio or visual output indicating a result of performing the selected candidate action.

30. The system of claim 29, wherein: The one or more neural networks include strategies for automatic navigation of web pages.

31. The system of claim 29, wherein: The one or more criteria include a ranking of the plurality of candidate actions.

32. The system of claim 29, wherein: The output indicative of the plurality of candidate actions includes data indicative of a sequence in which the plurality of candidate actions should be performed.

33. The system of claim 29, further comprising instructions for determining whether performance of the selected candidate action was successful.

34. The system of claim 33, wherein: The output indicative of the result of performing the selected candidate action includes an indication of whether the selected candidate action was successful.

35. The system of claim 33, further comprising instructions for: determining a reward or penalty based on whether the execution of the selected candidate action was successful; and One or more of the neural networks are trained based on the determined reward or penalty.

36. The system of claim 29, wherein: The output indicating the result of performing the selected candidate action includes navigating from the web page to a new web page.

37. At least one non-transitory computer-readable medium comprising instructions that, in response to being executed by one or more processors, cause the one or more processors to: receiving natural language input from a user at an input component, wherein: The natural language input is directed by the user to an automated assistant operating at least in part on the computing device and identifies the user's intent; retrieving a web page including one or more interactive elements, wherein the web page is operable to execute the identified intent of the user, and the one or more interactive elements are operable to input one or more parameters associated with executing the intent of the user; generating one or more embeddings representing one or more aspects of the natural language input and at least some of the interactive elements of the retrieved web page as the user's current state; processing the one or more embeddings using one or more neural networks to generate an output indicative of a plurality of candidate actions for interacting with one or more of the interactive elements of the webpage; selecting a given candidate action from the plurality of actions based on one or more criteria; performing the selected candidate action to interact with one or more of the interactive elements of the web page using one or more of the parameters; and Output is provided by the automated assistant to the user as audio or visual output indicating a result of performing the selected candidate action.

38. The at least one non-transitory computer-readable medium of claim 37, wherein: The one or more neural networks include strategies for automatic navigation of web pages.

39. The at least one non-transitory computer-readable medium of claim 37, wherein: The one or more criteria include a ranking of the plurality of candidate actions.

40. The at least one non-transitory computer-readable medium of claim 37, wherein: The output indicative of the plurality of candidate actions includes data indicative of a sequence in which the plurality of candidate actions should be performed.

Images