Secure Digital Assistant Integration in Web Pages

Abstract

Provides secure digital assistant integration with web pages. The system receives an intent manifest data structure that maps actions of the digital assistant to link templates of electronic resources developed by third-party developer devices. The system validates the electronic resources based on the intent manifest data structure. The system receives an identifier of the client computing device from a data exchange component of an inline frame of the electronic resource loaded by the client computing device. The system receives a foreground state of the electronic resource from a live state sharing API. The system selects data values of parameters based on the foreground state and the intent manifest data structure. The system provides the data values. An authorization component generates an authorization prompt, receives input, and provides the data values to the live intent execution API of the electronic resource to perform an action.

Description

Background Art

[0001] An application can be installed on a computing device. The computing device can execute the application. The application can present digital content. Summary of the Invention

[0002] At least one aspect relates to a system for secure digital assistant integration with a web page. The system can include a data processing system having one or more processors and a memory. The data processing system can receive an intent manifest data structure from a third-party developer device, the intent manifest data structure containing a mapping between a plurality of actions of a digital assistant and a plurality of link templates of an electronic resource developed by the third-party developer device. The data processing system can verify the electronic resource based on the intent manifest data structure via a verification policy. The data processing system can receive an identifier of a client computing device that executes the electronic resource from a data exchange component of an inline frame of the electronic resource loaded by the client computing device. The data processing system can query a live status sharing application programming interface of the electronic resource. The data processing system can receive a foreground state of the electronic resource from the live status sharing application programming interface in response to the query. The data processing system determines parameters based on the foreground state and the intent manifest data structure. The data processing system selects a data value of the parameters from a data repository based on the identifier of the client computing device. The data processing system can provide the data value to an authorization component of an inline frame of the electronic resource loaded on the client computing device. The data processing system can provide the data value to enable the authorization component to perform one or more functions. The authorization component can generate an authorization prompt. The authorization component can receive input from the client computing device in response to the authorization prompt. The authorization component can, in response to authorization of the data value, send the data value to a live intent execution application programming interface of the electronic resource. The live intent execution application programming interface can cause the electronic resource to perform one of a plurality of actions using the data value.

[0003] The data exchange component can restrict the electronic resource in a parent frame from accessing the identifier of the client computing device. The third-party developer device that develops the electronic resource can be prohibited from accessing the identifier of the client computing device.

[0004] The data processing system can authorize the data exchange component to be loaded in the inline frame of the electronic resource in response to verification of the electronic resource via the verification policy.

[0005] The data processing system can verify the electronic resource based on a trusted site list.

[0006] The data processing system can receive a request from the data exchange component executed by the client computing device. The data processing system can query the live status sharing application programming interface of the electronic resource in response to the request.

[0007] A data processing system can receive data packets from a voice navigator and a response component executed by a client computing device, the data packets carrying input audio signals detected by sensors of the client computing device. The data processing system can identify a request for a candidate data value from the data packets. The data processing system can provide a data value as the candidate data value in response to the request.

[0008] The data processing system can provide the data value to a live intent execution application programming interface to enable the live intent execution application programming interface to input the data value into an input text box of an electronic resource.

[0009] The data processing system can determine multiple parameters to be used for executing an action provided by an electronic resource based on a foreground state. The data processing system can select multiple data values corresponding to the multiple parameters based on an identifier of the client computing device. The data processing system can provide the multiple data values to an authorization component for the authorization component to provide to the live intent execution application programming interface. The live intent execution application programming interface can be configured to use the multiple data values to bypass one or more states used by the electronic resource to execute the action.

[0010] The data processing system can determine one or more subsequent states of an electronic resource based on the foreground state and an intent manifest data structure. The data processing system can determine one or more parameters based on the one or more subsequent states. The data processing system can select one or more data values of the one or more parameters based on the identifier before the electronic resource enters the one or more subsequent states.

[0011] The data processing system can provide a data value for authorization by an authorization component and input into a live intent execution application programming interface before the electronic resource requests the data value.

[0012] The data processing system can provide the data value to the client computing device to enable the client computing device to construct a deep link with the data value and load the deep link in a web browser executed by the client computing device. The electronic resource can be or include a web page.

[0013] The data processing system can construct a link with the data value based on a link template mapped to the action among multiple link templates. The data processing system can provide the link to the live intent execution application programming interface via a data exchange component.

[0014] At least one aspect relates to a method for secure digital assistant integration with a web page. The method may be performed by a data processing system having at least one processor. The method may include: the data processing system receiving an intent manifest data structure from a third-party developer device, the intent manifest data structure containing a mapping between a plurality of actions of a digital assistant and a plurality of link templates of an electronic resource developed by the third-party developer device. The method may include: the data processing system validating the electronic resource based on the intent manifest data structure via a validation policy. The method may include: the data processing system receiving an identifier of the client computing device on which the electronic resource is executed from a data exchange component of an inline frame of the electronic resource loaded by the client computing device. The method may include: the data processing system querying a live state sharing application programming interface of the electronic resource. The method may include: the data processing system receiving a foreground state of the electronic resource from the live state sharing application programming interface in response to the query. The method may include: the data processing system determining a parameter based on the foreground state and the intent manifest data structure. The method may include: the data processing system selecting a data value of the parameter from a data repository based on the identifier of the client computing device. The method may include: the data processing system providing the data value to an authorization component of an inline frame of the electronic resource loaded on the client computing device, such that the authorization component: generates an authorization prompt; receives an input from the client computing device in response to the authorization prompt; and in response to authorization of the data value, sends the data value to a live intent execution application programming interface of the electronic resource, such that the electronic resource performs one of the plurality of actions using the data value.

[0015] At least one aspect relates to a computer program product configured to cause a data processing system to perform a method for securely integrating a digital assistant with a web page when implemented on the data processing system.

[0016] Any aspect of the present disclosure or the individual features and / or combinations of features defined below with respect to any particular embodiment of the present disclosure may be used separately or individually, independently or in combination with any other defined feature, in any other aspect or embodiment of the present disclosure.

[0017] Furthermore, the present disclosure is intended to cover a device configured to perform any feature described herein with respect to a method and / or a method of using or producing, using or manufacturing any device feature described herein.

[0018] These and other aspects and embodiments will be discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and embodiments and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and embodiments. The drawings provide an illustration and further understanding of the various aspects and embodiments and are incorporated into and form a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings are not necessarily to scale. In the various figures, like reference symbols and designations indicate like elements. For simplicity, not every component may be labeled in each figure. In the figures:

[0020] Figure 1 is a diagram of an example system for integrating a secure digital assistant with a web page;

[0021] Figure 2 is a diagram of an example operation of a system for integrating a secure digital assistant with a web page;

[0022] Figure 3 is a diagram of an example method for integrating a secure digital assistant with a web page;

[0023] Figure 4 is a block diagram of the overall architecture of a computer system that illustrates elements that may be used to implement the systems and methods described and illustrated herein (including, for example, Figure 1 and Figure 2 the system depicted in Figure 3 and the method depicted in DETAILED DESCRIPTION

[0024] The following is a more detailed description of various concepts and their implementations related to methods, apparatuses, and systems for integrating a secure digital assistant with a web page. The various concepts introduced above and discussed in more detail below may be implemented in any of many ways.

[0025] This disclosure generally relates to integrating a secure digital assistant with a web page, an electronic document, or other electronic resources. A client computing device may render or load a web page. The web page may include input fields or provide prompts for input. The input may be provided by a user of the web page. The input may include information associated with the user, such as a username, password, account information, electronic transaction information, or preference information. However, the user may not have access to data to be provided for input into a mobile device. Additionally, the client computing device may have a limited user interface or input capabilities to receive input from the user. The web page may operate in a sandbox or restricted computing environment where the web page is blocked from accessing portions of the memory on the client computing device or a server containing account information. As more and more web pages are accessed or rendered on client computing devices and third - party developers increasingly request input data values to perform actions or execute services, providing such input for web pages or integrating with a digital assistant while maintaining a secure communication channel may pose challenges due to limited input interfaces on mobile devices, inefficiencies resulting from providing input through limited input interfaces, or inability to easily access input information.

[0026] The technical solution of the present disclosure relates to securely integrating a digital assistant with electronic resources such as electronic documents or web pages. This technical solution can allow data transmission between the electronic resources and the server, and can enable an input to be provided to the electronic resources, thereby providing improved user input. The data transmission can provide capabilities such as identification, electronic transaction processing, customization, or context information to a third-party electronic resource to improve the processing flow while maintaining the security of the entire system.

[0027] To securely integrate a digital assistant with a third-party electronic resource, the systems and methods of this technical solution include one or more application programming interfaces ("APIs") for third-party electronic resource developers to integrate with the digital assistant, a JavaScript library that securely hosts digital assistant functionality on the third-party electronic resource, a context suggestion system integrated with the API and the JavaScript library, and a voice navigation and response system integrated with the API and the JavaScript library.

[0028] The API interface of this technical solution can include or be associated with an intent list, an in-situ intent execution API, and an in-situ status sharing API. The intent list can refer to or include a data file provided by a third-party web developer that clarifies the mapping between digital assistant intents and uniform resource locator ("URL") templates of the electronic resources. An intent can refer to or include a messaging object that describes how the digital assistant or other systems perform actions. The intent can refer to, include, or define an action object. The intent can be mapped to a link (e.g., a URL) that implements the action.

[0029] The in - situ intent - execution API may include JavaScript callbacks implemented by third - party developers for electronic resources to handle digital assistant intents triggered by a data - processing system or a client computing device. This technical solution can use JavaScript to execute intents because executing intents via a link or URL may be technically challenging or unavailable. For example, if the link is not elucidated in the intent manifest, it may be difficult or impossible to execute via the link or URL. Due to errors, vulnerabilities, or glitches in the intent manifest, the link may not be published in the intent manifest. The link may not be published in the intent manifest because the third - party developer chooses not to publish it. If the links are not global links or globally accessible links, they may not be published in the intent manifest. Executing via a link may degrade the user experience or result in a sub - optimal user experience because it may cause the web browser or other applications to reload the web page. Executing via a link may consume more computing resources, such as network - bandwidth usage, processor usage, or memory usage, because this may cause a complete web - page request to the server via the network and then reload the entire web page. The JavaScript execution of an intent or action can be more efficient compared to executing via a URL link because it does not reload the page and allows execution without a globally accessible or published link.

[0030] The in - situ status - sharing API may include or provide JavaScript callbacks. A callback may refer to or include a function that is executed after another function has completed execution. The in - situ status - sharing API can be implemented by third - party developers so that, when requested by the data - processing system, the site can publish the foreground status. The data - processing system can query or request the foreground status from the in - situ status - sharing API. In response to the request, the in - situ status - sharing API can provide the foreground status. The foreground status may refer to the current semantic state of an electronic resource, such as the content being displayed on a web page or the functions or actions being executed or available. The status may include one or more entities that represent real - world or physical concepts as structured data in the foreground of the electronic resource. An entity may refer to a person, a place, or a thing. An entity may have a unique identifier. An entity may include attributes, types, and descriptions. An entity may include relationships with one or more other entities. Entities can provide structure to the data. The status may include one or more digital assistant intents that are transiently available in the current context of the electronic resource.

[0031] The JavaScript library of the present technical solution can securely and reliably host digital assistant functionality as an overlay rendered on a third-party electronic resource, and can provide interaction and authenticated callbacks to the data processing system in a manner opaque to the third-party electronic resource. The JavaScript library of the present technical solution can provide secure communication because access by the third-party electronic resource to data associated with the JavaScript library or communication with the data processing system can be prohibited or blocked before authorization. The secure provision of such data values can reduce the processor, memory, or battery consumption of the computing device by reducing the amount of latency caused by entering data values or launching other applications on the computing device to obtain data values.

[0032] The data processing system of the present technical solution can include a data value predictor component (or context autofill suggestion system) that accepts the foreground state as an input. The data processing system can receive the foreground state from the JavaScript library, and the JavaScript library receives the foreground state from the live state sharing API of the third-party electronic resource. The foreground state can indicate or identify the current intent associated with the electronic resource. Using the foreground state information, the data processing system can search a data repository or database linked to the client computing device (or its account) that renders the electronic resource. The data processing system can search the database to select or predict the data value of the current intent parameter. If the data processing system identifies a selection or an acceptable prediction of the parameter, the data processing system can provide the value to the JavaScript library. The JavaScript library can render the data value for authorization. If the data value is authorized, the data value can be provided or passed to the electronic resource. The JavaScript library can provide the authorized data value to the third-party electronic resource through a link (e.g., a URL deep link) or a JavaScript intent execution API.

[0033] For example, the electronic resource can include a car rental website. The data processing system can identify the current foreground state, which indicates the intent of book_car_rental (to_location, from_location, start_time, end_time). The data processing system can search and identify data related to upcoming flight reservations stored in a database associated with the account of the client computing device corresponding to rendering the third-party electronic resource. The data processing system can predict the data value of the intent parameter based on the data in the database. The data processing system can send the predicted data value of the parameter to the client computing device. The data processing system can perform an action corresponding to the intent on the third-party electronic website in response to authorization.

[0034] The present technical solution may include a voice navigation and response system (e.g., a voice navigator and a response component or a digital assistant component). When the data processing system provides structured intent parsing through a natural language processing component, the data processing system may invoke the voice navigator and response component, and the structured intent parsing may be processed by a third-party electronic resource integrated with a digital assistant interface and a JavaScript library. The present technology may convert user intent parsing into a URL link that can be used to navigate the electronic resource or a JavaScript digital assistant intent execution call. After the JavaScript library executes the intent on the third-party electronic resource, the JavaScript library may request the foreground state from the JavaScript callback of the electronic resource. The voice navigator and response component or the data processing system may match the foreground state data with a voice response (text-to-speech) template that has been pre-associated with the matched user intent. The voice navigator and response component may render a text-to-speech response to the user by passing the state data into the template. The present technology may allow the user to perform voice navigation across the entire website and hear a text-to-speech (“TTS”) answer after each voice navigation, thereby providing a mechanism to enable user input.

[0035] Figure 1 FIG. is an illustration of an example system 100 for secure digital assistant integration with a web page. System 100 may include a content selection infrastructure. System 100 may include an application delivery infrastructure. System 100 may include an online application store or marketplace. System 100 may include a data processing system 102. The data processing system 102 may communicate via a network 101 with one or more of a third-party (“3P”) developer device 162 (or application developer device) or a client computing device 128 (or client device or computing device). System 100 may also communicate with other devices, such as third-party devices, content provider devices, or digital surface devices.

[0036] The network 101 may include a computer network such as the Internet, a local area network, a wide area network, a metropolitan area network, or other local area networks, an intranet, a satellite network, and other communication networks such as a voice or data mobile telephone network. The network 101 may be used to access information resources such as web pages, websites, domain names, or uniform resource locators, which may be presented, output, rendered, or displayed on at least one client computing device 128 (such as a laptop computer, a desktop computer, a tablet computer, a digital assistant device, a smartphone, a wearable device, a portable computer, or a speaker). For example, a user of the client computing device 128 may access information or data provided by the data processing system 102 or the 3P developer device 162 via the network 101.

[0037] Network 101 may include or constitute a display network, e.g., a subset of information resources available on the Internet, associated with a content delivery or search engine results system, or eligible to include third-party digital components as part of a digital component delivery campaign. Data processing system 102 may use Network 101 to access information resources, such as web pages, websites, domain names, or uniform resource locators that may be presented, output, rendered, or displayed by client computing device 128. For example, a user of client computing device 128 may access information or data provided by data processing system 102 or 3P developer device 162 via Network 101.

[0038] Network 101 may be any type or form of network and may include any of the following: a peer-to-peer network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, an SDH (Synchronous Digital Hierarchy) network, a wireless network, and a wired network. Network 101 may include wireless links, such as infrared channels or satellite bands. The topology of Network 101 may include a bus, star, or ring network topology. Network 110 may include a mobile phone network using any one or more protocols for communication between mobile devices, including: Advanced Mobile Phone Protocol (“AMPS”), Time Division Multiple Access (“TDMA”), Code Division Multiple Access (“CDMA”), Global System for Mobile Communications (“GSM”), General Packet Radio Service (“GPRS”), or Universal Mobile Telecommunications System (“UMTS”). Different types of data may be transmitted via different protocols, or the same type of data may be transmitted via different protocols.

[0039] System 100 may include at least one data processing system 102. Data processing system 102 may include at least one logic device, such as a computing device having a processor to communicate via Network 101 with, e.g., client computing device 128 or 3P developer device 162 or other networked devices or third-party devices. Data processing system 102 may include at least one computing resource, server, processor, or memory. For example, data processing system 102 may include multiple computing resources or servers located in at least one data center. Data processing system 102 may include multiple logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm, or machine farm. The servers may also be geographically dispersed. The data center or machine farm may be managed as a single entity, or the machine farm may include multiple machine farms. The servers within each computer farm may be heterogeneous: one or more servers or machines may operate according to one or more types of operating system platforms.

[0040] Servers in the machine farm can be stored in a high-density rack system together with an associated storage system and can be located in an enterprise data center. For example, integrating servers in this way can improve system manageability, data security, physical security of the system, and system performance by locating the servers and high-performance storage systems on a localized high-performance network. Centralizing all or some of the components of the data processing system 102, including servers and storage systems, and coupling them with advanced system management tools can more effectively utilize server resources, thus saving power and processing requirements and reducing bandwidth usage.

[0041] System 100 can include, access, or interact with at least one 3P developer device 162. The 3P developer device 162 can include at least one logical device, such as a computing device having a processor to communicate with a client computing device 128 or a data processing system 102 via a network 101, for example. The 3P developer device 162 can include at least one computing resource, server, processor, or memory. For example, the 3P developer device 162 can include multiple computing resources or servers located in at least one data center.

[0042] The 3P developer device 162 can provide audio-based digital components for the client computing device 128 to present or display as audio output digital components. The digital components can include the offering of goods or services, such as a message based on the statement "Can I call a taxi for you?". For example, the 3P developer device 162 can include a memory for storing a series of audio digital components that can be provided in response to a voice-based query. The 3P developer device 162 can also provide audio-based digital components (or other digital components) to the data processing system 102, where they are stored in the data repository 114. The data processing system 102 can select the audio digital components and provide the audio digital components to the client computing device 128 (or instruct the 3P developer device 162 to provide them to the client computing device 126). The audio-based digital components can specifically be audio or can be combined with text, image, or video data.

[0043] The 3P developer device 162 may include a data processing system 102 and interface with or communicate with it. The 3P developer device 162 may include a client computing device 128 and interface with or communicate with it. The 3P developer device 162 may include a client computing device 128 and interface with or communicate with it, and the client computing device 128 may be a mobile computing device. The 3P developer device 162 may include a 3P developer device 162 and interface with or communicate with it. For example, the 3P developer device 162 may provide digital components to the client computing device 128 for execution by the client computing device 128. The 3P developer device 162 may provide digital components to the data processing system 102 for storage by the data processing system 102. The 3P developer device 162 may provide rules or parameters related to the digital components to the data processing system 102.

[0044] The client computing device 128 may download electronic resources, electronic documents, or applications developed by the 3P developer device 162. The client computing device 128 may download the application or electronic resource from the data processing system 102 via the network 101. The client computing device 128 may load an electronic document or resource. The client computing device 128 may execute the application. The client computing device 128 may execute, launch, trigger, access, or use the application in response to a user input or a trigger event or condition. The application may include a front-end component and a back-end component. The client computing device 128 may execute or provide the front-end component of the application, while the data processing system 102 or the 3P developer device 162 provides the back-end component of the application.

[0045] The client computing device 128 may include at least one sensor 152, transducer 154, audio driver 156, or preprocessor 158, interface-connected or in communication therewith. The client computing device 128 may include a display device 160, such as a light indicator, light-emitting diode (“LED”), organic light-emitting diode (“OLED”), or other visual indicator configured to provide a visual or optical output. The sensor 152 may include, for example, an ambient light sensor, proximity sensor, temperature sensor, accelerometer, gyroscope, motion detector, GPS sensor, position sensor, microphone, or touch sensor. The transducer 154 may include a speaker or a microphone. The audio driver 156 may provide a software interface to the hardware transducer 154. The audio driver may execute an audio file or other instructions provided by the data processing system 102 to control the transducer 154 to generate a corresponding acoustic or sound wave. The preprocessor 158 may include a processing unit having hardware configured to detect keywords and perform actions based on the keywords. Before transmitting an item to the data processing system 102 for further processing, the preprocessor 158 may filter out one or more items or modify an item. The preprocessor 158 may convert an analog audio signal detected by a microphone into a digital audio signal and transmit one or more data packets carrying the digital audio signal to the data processing system 102 via the network 101. In some cases, the preprocessor 158 may transmit data packets carrying some or all of the input audio signal in response to detecting an instruction to perform such a transmission. The instruction may include, for example, a trigger keyword or other keyword or approval to transmit a data packet including the input audio signal to the data processing system 102.

[0046] The client computing device 128 may be associated with an end user who inputs a voice query as audio input (via the sensor 152) into the client computing device 128 and receives an audio output provided in the form of computer-generated speech, which may be provided to the client computing device 128 from the data processing system 102 (or 3P developer device 162) and output from the transducer 154 (e.g., a speaker). The computer-generated speech may include a recording from a human or computer-generated language.

[0047] The client computing device 128 (or computing device, or client device, or digital device) may or may not include a display. For example, the computing device may include a limited type of user interface, such as a microphone and a speaker. In some cases, the primary user interface of the client computing device 128 may be a microphone and a speaker, or a voice interface. For example, the primary user interface of the client computing device 128 may include a voice-based or audio-based user interface. The client computing device 128 may include a display and have a voice-based or audio-based primary user interface. The primary user interface of the client computing device 128 may be conversational. A conversational user interface may refer to a user interface that is at least partially driven or facilitated by the natural language processor component 106 of the data processing system 102.

[0048] The data processing system 102 may include a content delivery system having at least one computing resource or server. The data processing system 102 may include at least one interface 104 to interface with or communicate with. The data processing system 102 may include at least one natural language processor component 106 to interface with or communicate with. The data processing system 102 may include at least one direct action application programming interface ("API") 108 to interface with or communicate with. The interface 104, the natural language processing component 106, and the direct action API 108 may provide a conversational API or digital assistant functionality. The conversational API or digital assistant may communicate with or interface with one or more voice-based interfaces or various digital assistant devices or surfaces to provide data or receive data or perform other functionality.

[0049] The data processing system 102 may include at least one verification component 110 to interface with or communicate with. The data processing system 102 may include at least one data value predictor component 112 to interface with or communicate with. The data processing system 102 may include at least one data repository 114 to interface with or communicate with.

[0050] The interface 104, the natural language processor component 106, the direct action API 108, the verification component 110, and the data value predictor component 112 may each include at least one processing unit or other logic device, such as a programmable logic array engine, or a module configured to communicate with the data repository 114 or database. The interface 104, the natural language processor component 106, the direct action API 108, the verification component 110, the data value predictor component 112, and the data repository 114 may be separate components, a single component, or part of the data processing system 102. The system 100 and its components, such as the data processing system 102, may include hardware elements, such as one or more processors, logic devices, or circuits.

[0051] The data processing system 102 may obtain anonymized computer network activity information associated with a plurality of client computing devices 128 (or computing devices or digital assistant devices). A user of the client computing device 128 or mobile computing device may affirmatively authorize the data processing system 102 to obtain network activity information corresponding to the client computing device 128 or mobile computing device. For example, the data processing system 102 may prompt a user of the client computing device 128 to consent to obtaining one or more types of network activity information. The client computing device 128 may include a mobile computing device such as a smart phone, a tablet computer, a smart watch, or a wearable device. The identity of the user of the client computing device 128 may remain anonymous, and the client computing device 128 may be associated with a unique identifier (e.g., a unique identifier of the computing device or the user of the client computing device 128 provided by the user or the data processing system 102). The data processing system 102 may associate each observation with the corresponding unique identifier.

[0052] The data processing system 102 can interface with the 3P developer device 162. The 3P developer device 162 can include or refer to the device of a content provider. The content provider can establish an electronic content campaign. The electronic content campaign can be stored as content data in the data repository 114. The electronic content campaign can refer to one or more content groups corresponding to a common theme. The content campaign can include a hierarchical data structure that includes content groups, digital component data objects, and content selection criteria. To create a content campaign, the content provider can specify values for the campaign-level parameters of the content campaign. The campaign-level parameters can include, for example, a campaign name, a preferred content network for delivering digital component objects, a value of the resources used for the content campaign, start and end dates of the content campaign, a duration of the content campaign, a schedule for delivering digital component objects, a language, a geographical location, a type of computing device on which the digital component objects are to be provided. In some cases, an impression can refer to the time when a digital component object is retrieved from its source (e.g., the data processing system 102 or the content provider), and impressions are countable. In some cases, since click fraud may occur, bot activities can be filtered out and excluded as impressions. Thus, in some cases, an impression can refer to a metric of the response of a web server to a page request from a browser, which is filtered out from bot activities and error codes and recorded at a point as close as possible to the opportunity to render the digital component object for display on the client computing device 128. In some cases, an impression can refer to a visible or audible impression; for example, making the digital component object at least partially visible (e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70% or more) on the display device 160 of the client computing device 128, or audible through the speaker of the client computing device 128. A click or selection can refer to a user's interaction with a digital component object, such as a voice response to an audible impression, a mouse click, a touch interaction, a gesture, a shake, an audio interaction, or a keyboard click. A conversion can refer to a user taking an expected action with respect to a digital component object; for example, purchasing a product or service, completing a survey, visiting a physical store corresponding to the digital component, or completing an electronic transaction.

[0053] The content provider can further establish one or more content groups for the content campaign. The content groups include one or more digital component objects and corresponding content selection criteria, such as keywords, words, items, phrases, geographical locations, types of computing devices, moments, interests, topics, or vertical criteria. The content groups under the same content campaign can share the same campaign-level parameters, but can have customized specifications for content-group-level parameters, such as keywords, negative keywords (e.g., blocking the delivery of digital components in the case of the presence of negative keywords on the main content), quotes of keywords, or parameters associated with the quotes or the content campaign.

[0054] To create new content groups, content providers can provide values for content group-level parameters for the content groups. Content group-level parameters include, for example, a content group name or content group theme, and bids for different content placement opportunities (e.g., automatic placement or hosted placement) or outcomes (e.g., clicks, impressions, or conversions). The content group name or content group theme can be one or more terms that the content provider can use to capture the topic or theme of the digital component objects for which content groups are to be selected for display. For example, an auto dealer can create different content groups for each brand of vehicle it represents and can further create different content groups for each model of vehicle it represents. Examples of content group themes that an auto dealer can use include, for example, "Model A sports car", "Model B sports car", "Model C sedan", "Model C truck", "Model C hybrid", or "Model D hybrid". For example, an example content campaign theme can be "Hybrid vehicles" and include content groups for "Model C hybrid" and "Model D hybrid".

[0055] Content providers can provide one or more keywords and digital component objects to each content group. The keywords can include items related to products or services associated with or identified by the digital component objects. The keywords can include one or more items or phrases. For example, an auto dealer can include "sports car", "V6 engine", "four-wheel drive", "fuel efficiency" as keywords for a content group or content campaign. In some cases, negative keywords can be specified by the content provider to avoid, prevent, block, or deactivate the content placement of certain items or keywords. The content provider can specify the match type for selecting digital component objects, such as exact match, phrase match, or broad match.

[0056] Content providers can provide one or more keywords for the data processing system 102 to use to select digital component objects provided by the content provider. The content provider can identify one or more keywords to bid on and can further provide the bid amounts for the various keywords. The content provider can provide additional content selection criteria for the data processing system 102 to use to select digital component objects. Multiple content providers can bid on the same or different keywords, and the data processing system 102 can run a content selection process or an advertising auction in response to receiving an indication of the keywords in an electronic message.

[0057] Content providers may provide one or more digital component objects for selection by the data processing system 102. The data processing system 102 may select a digital component object that matches the resource allocation, content scheduling, maximum bid, keywords, and other selection criteria specified for a content group when a content placement opportunity becomes available. Different types of digital component objects may be included in a content group, such as voice digital components, audio digital components, text digital components, image digital components, video digital components, multimedia digital components, or digital component links. A digital component object (or digital component) may include, for example, a content item, an online document, audio, an image, a video, multimedia content, or sponsored content. After selecting a digital component, the data processing system 102 may transmit the digital component object for rendering on the computing device 128 or display device 160 of the client computing device 128. Rendering may include displaying the digital component on the display device or playing the digital component via the speaker of the client computing device 128. The data processing system 102 may provide instructions for rendering the digital component object to the computing device 128. The data processing system 102 may instruct the client computing device 128 or the audio driver 156 of the client computing device 128 to generate an audio signal or sound wave.

[0058] The data repository 114 may include one or more local or distributed databases and may include a database management system. The data repository 114 may include computer data storage or memory and may store one or more of the authentication policy 116, intent list 118, actions 120, link templates 122, account information 124, and data values 126, as well as other data. The data repository 114 may store one or more of the authentication policy 116, intent list 118, actions 120, link templates 122, account information 124, and data values 126 in one or more data structures, databases, data files, indexes, or other types of data storage.

[0059] The data repository 114 may store the intent list 118. The intent list 118 may be provided by the 3P developer device 162. The intent list 118 may be configured for an electronic resource. The intent list 118 may be specific to an electronic resource, such as a website, a web page, or other electronic document. The intent list 118 may include a data file or data structure. The intent list 118 may include actions 120 and link templates 122. The intent list 118 may map actions 120 to link templates 122. The intent list 118 may link, bind, associate, or relate actions 120 to link templates 122. The intent data structure 118 may be in a format such as JavaScript object format, which has JavaScript object properties such as name / value pairs.

[0060] Action 120 may refer to or include an intent. Action 120 may refer to or include a function to be performed on or via an electronic resource. Action 120 may be a messaging object that describes how a system performs a task or function. Action 120 may be used to facilitate a system or 3P developer device 162 to perform an action or request an action to be performed. Action 120 may be defined in an action package that includes the name or intent of Action 120 and an indication of a user query that matches the intent. The user query may correspond to link template 122.

[0061] Link template 122 may include a template with data values for parameters having placeholders. Data processing system 102 may use link template 122 to construct a link. The link may refer to a URL or other reference or pointer to an electronic resource. Link template 122 may be referred to as urlTemplate. An example link template 122 may be: https: / / m_taxiapp_com / ?action=setPickup{&pickup[latitude],pickup[longitude],pickup[nickname],pickup[formatted_address],dropoff[latitude],dropoff[longitude],dropoff[nickname],dropoff[formatted_address]}. In this example, the link template includes placeholders for parameter values indicated using square brackets "[]". The link may include one or more parameters. The link may include the domain of the electronic resource, the action to be performed, and the parameters for performing the action. Thus, intent list 118 may map an action to link template 122 that can be used to complete the action.

[0062] Data repository 114 may store validation policy 116 as a data file, data structure, or other storage format. Validation policy 116 may include one or more rules, policies, logics, thresholds, comparisons, or functions used by at least validation component 110 to validate intent list 118 of an electronic resource provided by 3P developer device 162. After validation component 110 uses validation policy 116 to validate the intent list, data processing system 102 may store intent list 118 in data repository 114. An example of validation policy 116 may include determining whether intent list 118 includes a determined action or format of a link template, and if the format matches a predetermined format indicated in validation policy 116, permitting the intent list.

[0063] The data repository 114 may store one or more accounts 124. The accounts 124 may include account information. The accounts 124 may be associated with or for a user of the computing device 128. The accounts 124 may include, store, or otherwise indicate or provide information or data values 126 associated with the user of the client computing device 128. A user of the client computing device 126 may establish an account 124 with the data processing system 102. The accounts 124 may include any electronic or digital account. The accounts 124 may include profile information, historical information, or other data values 126 associated with the user of the client computing device 128. The accounts 124 may include information previously provided to the data processing system 102 by the client computing device 128. The data values 126 may include, for example, information such as electronic account information, identifiers, address information, or preferences. The data values 126 may include information associated with the user that may be used to facilitate a transaction process on the 3P electronic resource 134, or information that may be input into an input form or text box in the 3P electronic resource 134.

[0064] The data processing system 102 may include an interface 104 (or interface component) that is designed, configured, constructed, or operated to receive and send information using, for example, data packets. The interface 104 may receive and send information using one or more protocols, such as network protocols. The interface 104 may include a hardware interface, a software interface, a wired interface, or a wireless interface. The interface 104 may facilitate converting or formatting data from one format to another. For example, the interface 104 may include an application programming interface that includes definitions for communicating between various components, such as software components. The interface 104 may communicate via the network 101 with one or more of the client computing device 128 or the 3P developer device 162.

[0065] The data processing system 102 may interact with an application, script, or program installed on the client computing device 128, such as an app that is used to transmit an input audio signal to the interface 104 of the data processing system 102 and drive a component of the local client computing device to render an output audio signal. The data processing system 102 may receive a data packet or other signal that includes or identifies an audio input signal. The interface 104 may interface with or communicate with one or more components of the client computing device 128. The interface 104 may communicate with, for example, a Web browser 130, a JavaScript library 148, a live status sharing API, or a data exchange component 140 of the client computing device 128, or an authentication component 142 of the client computing device 128.

[0066] The data processing system 102 may include a natural language processor (“NLP”) component 106. For example, the data processing system 102 may execute or run the NLP component 106 to receive or obtain an audio signal and parse the audio signal. For example, the NLP component 106 may provide an interaction between a human and a computer. The NLP component 106 may be configured with techniques for understanding natural language and allowing the data processing system 102 to derive meaning from human or natural language input. The NLP component 106 may include or be configured with machine learning-based techniques, such as statistical machine learning. The NLP component 106 may utilize decision trees, statistical models, or probabilistic models to parse the input audio signal. For example, the NLP component 106 may perform functions such as named entity recognition (e.g., given a text stream, determining which items in the text map to proper names such as people or places, and the type of each such name, such as person, location, or organization), natural language generation (e.g., converting information or semantic intent in a computer database into understandable human language), natural language understanding (e.g., converting text into a more formal representation, such as a first-order logic structure that a computer module can manipulate), machine translation (e.g., automatically translating text from one human language to another), morpheme segmentation (e.g., separating words into individual morphemes and identifying the category of the morphemes, which can be challenging given the complexity of the morphemes or structure of the words in the language), question answering (e.g., determining an answer to a human language question, which can be specific or open-ended), semantic processing (e.g., processing that can occur after identifying words and encoding their meanings to relate the identified words to other words with similar meanings).

[0067] The NLP component 106 may convert an audio input signal into recognized text by comparing the input signal with a stored set of representative audio waveforms (e.g., in the data repository 114) and selecting the closest match. The set of audio waveforms may be stored in the data repository 114 or other databases accessible by the data processing system 102. The representative waveforms are generated across a large number of users and may then be enhanced with samples of the users' utterances. After the audio signal is converted into recognized text, the NLP component 106 matches the text with words that are associated with actions that the data processing system 102 can service, for example, via user training across the board or by manual specification. Aspects or functionality of the NLP component 106 may be executed by the data processing system 102 or the client computing device 128. For example, a local NLP component may be executed on the client computing device 128 to perform aspects of converting an input audio signal into text and sending the text to the data processing system 102 via data packets for further natural language processing.

[0068] An audio input signal can be detected by a sensor 152 or a transducer 154 (e.g., a microphone) of the client computing device 128. Via the transducer 154, the audio driver 156, or other components, the client computing device 128 can provide the audio input signal to the data processing system 102 (e.g., via the network 101), where the audio input signal can be received (e.g., via the interface 104) and provided to the NLP component 106 or stored in the data repository 114. The audio input signal detected by the sensor 152 can include an initial keyword, a hot word, or a trigger word, which indicates to the client computing device 128 that the input audio signal will be sent to the data processing system 102.

[0069] The client computing device 128 can include an audio driver 156, a transducer 154, a sensor 152, and a preprocessor component 158. The sensor 152 can receive or detect an input audio signal (e.g., a voice input). The preprocessor component 158 can be coupled to the audio driver, the transducer, and the sensor. The preprocessor component 158 can identify an initial keyword, a hot word, a trigger keyword, or other symbols in the input audio signal, which indicate that the input audio signal will be sent to the data processing system 102 for processing by the NLP component 106. The preprocessor component 158 can filter the input audio signal to create a filtered input audio signal (e.g., by removing certain frequencies or suppressing noise, or removing the initial keyword or hot word). The preprocessor component 158 can convert the filtered input audio signal into a data packet (e.g., using a software or hardware digital-to-analog converter). In some cases, the preprocessor component 158 can convert the unfiltered input audio signal into a data packet and send the data packet to the data processing system 102. The preprocessor component 158 can send the data packet to the data processing system 102, which includes one or more processors and memories that execute a natural language processor component, an interface, a speaker recognition component, and a direct action application programming interface.

[0070] The data processing system 102 can receive a data packet from the preprocessor component 158 via the interface, and the data packet includes the filtered (or unfiltered) input audio signal detected by the sensor. The data processing system 102 can identify an acoustic signature from the input audio signal. The data processing system 102 can identify an electronic account 124 corresponding to the acoustic signature based on a lookup in the data repository (e.g., querying a database). In response to the identification of the electronic account 124, the data processing system 102 can establish a session and an account for the session. The account 124 can include a profile with one or more policies. The data processing system 102 can parse the input audio signal to identify a request and a trigger keyword corresponding to the request.

[0071] The data processing system 102 can provide a status to a pre-processor component 158 of the client computing device 128. The client computing device 128 can receive an indication of the status. The audio driver can receive an indication of the status of the profile and generate an output signal based on the indication. The audio driver can convert the indication into an output signal, such as a sound signal or an acoustic output signal. The audio driver can drive a transducer 154 (e.g., a speaker) to generate sound based on the output signal generated by the audio driver.

[0072] In some cases, the client computing device 128 can include a display device 160. The display device 160 can include one or more LEDs, lights, displays, or other components or devices configured to provide an optical or visual output. The pre-processor component 158 can cause a light source to provide a visual indication corresponding to the status. For example, the visual indication can be an on status indicator light, a change in the color of light, a light pattern having one or more colors, or a visual display of text or an image.

[0073] The NLP component 106 can obtain an input audio signal. Based on the input audio signal, the NLP component 106 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate the intent or topic of the input audio signal. The trigger keyword can indicate the type of action that may be taken. The trigger keyword can be a wake-up signal or a hot word that instructs the client computing device 128 to convert subsequent audio input into text and send the text to the data processing system 102 for further processing. For example, the NLP component 106 can parse the input audio signal to identify at least one request to leave home for dinner and a movie at night. The trigger keyword can include at least one word, phrase, root, or part of a word, or a derivative that indicates the action to be taken. For example, the trigger keyword "go" or "going to" from the input audio signal can indicate the need for transportation. In this example, the input audio signal (or the identified request) does not directly express the intent for transportation, but the trigger keyword indicates that transportation is an auxiliary action for at least one other action indicated by the request.

[0074] The NLP component 106 can parse the input audio signal to identify, determine, retrieve, or obtain requests and trigger keywords. For example, the NLP component 106 can apply semantic processing techniques to the input audio signal to identify trigger keywords or requests. The NLP component 106 can apply semantic processing techniques to the input audio signal to identify trigger phrases that include one or more trigger keywords (such as a first trigger keyword and a second trigger keyword). For example, the input audio signal can include the sentence "I want to take a ride to the airport". The NLP component 106 can apply semantic processing techniques or other natural language processing techniques to the data packet including this sentence to identify the request or trigger phrases "want to take a ride" and "airport". The NLP component 106 can further identify multiple trigger keywords, such as "want" and "take a ride". For example, the NLP component 106 can determine that the trigger phrase includes trigger keywords and a second trigger keyword.

[0075] The NLP component 106 can filter the input audio signal to identify trigger keywords. For example, the data packet carrying the input audio signal can include "It would be great if I could get someone that could help me go to the airport", in which case, the NLP component 106 can filter out one or more items as follows: "it", "would", "be", "great", "if", "I", "could", "get", "someone", "that", "would", or "help". By filtering out these items, the NLP component 106 can more accurately and reliably identify trigger keywords, such as "go to the airport", and determine that this is a request for a taxi or ridesharing service.

[0076] In some cases, the NLP component 106 can determine that the data packet carrying the input audio signal includes one or more requests. For example, the input audio signal can include the sentence "I want to buy audiobooks and subscribe to movies monthly". The NLP component 106 can determine that this is a request for audiobook and streaming multimedia services. The NLP component 106 can determine whether this is a single request or multiple requests. The NLP component 106 can determine that these are two requests: a first request to a service provider that provides audiobooks, and a second request to a service provider that provides movie streaming. In some cases, the NLP component 106 can merge the determined multiple requests into one request and then send this one request to the 3P developer device 162. In some cases, the NLP component 106 can send each request to another service provider device, or send the two requests separately to the same 3P developer device 162.

[0077] The data processing system 102 may include a direct action API 108 that is designed and constructed to generate an action data structure in response to a request based on a trigger keyword. The direct action API 108 may generate the action data structure to cause an application to perform a corresponding action. The direct action API 108 may send the action data structure to an application installed on the client computing device 128 to cause the client computing device 128 to perform the corresponding action or initiate an action. The action data structure generated by the direct action API 108 may include a deep link for an application installed on the client computing device 128. Then, the application installed on the client computing device 128 may perform the action or communicate with the 3P developer device 162 or the 3P developer device 162 to perform the operation.

[0078] The processor of the data processing system 102 may invoke the direct action API 108 to execute a script that generates a data structure to be provided to an application installed on the client computing device 128, the 3P developer device 162, or other service providers to obtain digital components, content, reservation services, or products (such as reserving a car from a car-sharing service or reserving an audiobook). The direct action API 108 may obtain data from the data repository 114 and data received from the client computing device 128 with the end-user's consent to determine location, time, user account, logistics, or other information to enable the 3P developer device 162 to perform an operation, such as reserving a car from a car-sharing service. Using the direct action API 108, the data processing system 102 may also communicate with the 3P developer device 162 to complete the operation (in this example, by making a car-sharing reservation).

[0079] The direct action API 108 may perform a specified action to meet the end-user's intent determined by the data processing system 102. Based on the action specified in its input and the parameters or rules in the data repository 114, the direct action API 108 may execute code or a dialogue script that identifies the parameters required to satisfy the user's request. The direct action API 108 may execute an application that meets or realizes the end-user's intent. Such code may, for example, look up additional information such as the name of a home automation service or a third-party service in the data repository 114, or it may provide an audio output to be rendered at the client computing device 128 to ask the end-user questions, such as the intended destination of the requested taxi. The direct action API 108 may determine the parameters and may package the information into an action data structure, which can then be sent to another component of the data processing system 102 for implementation.

[0080] The direct action API 108 can receive instructions or commands from the NLP component 106 or other components of the data processing system 102 to generate or construct an action data structure. The direct action API 108 can determine the type of action in order to select a template stored in the data repository 114. The action can be implemented by an application provided by the data processing system 102 and submitted by the 3P developer device 162. The application can perform the action or facilitate the execution of the action. Example types of actions can include, for example, a viewing action, a listening action, a reading action, a navigation action, or a weather action. The types of actions can include or be configured to provide, for example, services, products, reservations, tickets, multimedia content, audiobooks, manage subscriptions, adjust subscriptions, transfer digital currency, make purchases, or music. The types of actions can further include the types of services or products. For example, the types of services can include car sharing services, food delivery services, laundry services, cleaning services, repair services, home services, device automation services, or media streaming services. The types of products can include, for example, clothes, shoes, toys, electronics, computers, books, or jewelry. The types of reservations can include, for example, dinner reservations or hair salon appointments. The types of tickets can include, for example, movie tickets, stadium tickets, or airplane tickets. In some cases, the types of services, products, reservations, or tickets can be classified according to price, location, type of transportation, availability, or other attributes.

[0081] The NLP component 106 can parse the input audio signal to identify the request and the trigger keyword corresponding to the request, and provide the request and the trigger keyword to the direct action API 108 so that the direct action API generates a first action data structure in response to the request based on the trigger keyword. When identifying the type of the request, the direct action API 108 can access the corresponding template from a template repository (e.g., the data repository 114). The template can include fields in a structured data set, which can be filled by the direct action API 108 to further perform the operation requested by the input audio detected by the client computing device 128 (such as the operation of sending a taxi to pick up the end user at the pick-up location and sending the end user to the destination location). The direct action API 108 or the client computing device 128 can start or trigger an application to satisfy the request in the input audio. For example, a car-sharing service application can include one or more of the following fields: device identifier, pick-up location, destination location, number of passengers, or service type. The direct action API 108 can fill the fields with values. To fill the fields with values, the direct action API 108 can check, poll, or obtain information from one or more sensors 152 of the client computing device 128 or the user interface of the client computing device 128. For example, the direct action API 108 can use a location sensor such as a GPS sensor to detect the source location. The direct action API 108 can obtain further information by submitting a survey, prompt, or query to the user side of the client computing device 128. The direct action API 108 can submit a survey, prompt, or query via the interface 104 of the data processing system 102 and the user interface of the client computing device 128 (e.g., an audio interface, a voice-based user interface, a display, or a touch screen). Therefore, the direct action API 108 can select a template for the action data structure based on the trigger keyword or the request, and use the information detected by the data value predictor component 112 from one or more sensors 152 or obtained through the user interface to fill one or more fields in the template, and generate, create, or construct an action data structure to facilitate the 3P developer device 162 to perform operations.

[0082] The system 100 can include a third party (“3P”) developer device 162 or communicate with a third party (“3P”) developer device 162. The 3P developer device 162 can include Figure 4One or more systems depicted or components of system 400. The 3P developer device 162 may include or be associated with one or more computing devices or servers. The 3P developer device 162 may generate, construct, or develop electronic resources or electronic documents. The electronic document may refer to or include web pages, HTML documents, digital media files, images, text, or web-based applications. The electronic document may include input form fields, buttons, graphical user interface elements, or widgets. The electronic document may be presented via the computing device 128 and may be configured to receive input from a user via an interface of the computing device 128. The electronic document may generate prompts or other requests for input from the user. The electronic document may present visual output or audio output. The 3P developer device 162 may generate, construct, or develop one or more portions of the electronic document. The electronic document may be referred to as a 3P document (or 3P electronic resource 134) as it may be provided by the 3P developer device 162. The 3P developer device 162 may provide the 3P electronic resource 134 (e.g., electronic document) to the client computing device 128, or to a caching server that provides the 3P electronic resource to the client computing device 128.

[0083] For example, the 3P developer device 162 may include an online retailer. The online retailer may generate an electronic document that is a web page for a product sold by the online retailer. The electronic document may request input from the user to complete a transaction, such as a financial account number. In another example, the 3P developer device 162 may include a package delivery provider, and the electronic document may provide tracking information. The electronic document may request a tracking number from the user in order to perform a lookup and determine the tracking status. The user may enter the tracking number via an interface of the computing device 128.

[0084] However, due to limited input capabilities on some computing devices 128 (e.g., small touchscreens or keyboards, voice input only), it may be challenging to enter the requested information into the electronic document. Additionally, the requested input may not be easily accessible and may result in other remote procedure calls or lookups to external sources or external accounts to obtain the requested input information. For example, the user may log in to an account or data repository different from the electronic document to obtain the information requested by the electronic document. On some computing devices 128 with limited capabilities, accessing these external accounts to obtain the information requested by the electronic document may be challenging, inefficient, or impossible. Therefore, the 3P developer device 162 may provide the electronic document to the data processing system 102 of the present technical solution.

[0085] The data processing system 102 may include a verification component 110, interface with or access the verification component 110, and the verification component 110 is designed, constructed, or operated to receive an intent manifest 118 from a third-party developer device 162. The verification component 110 may verify the intent manifest based on a verification policy 116. The verification component 110 may store the intent manifest in a data repository 114 of the data processing system 102 in response to the verification of the intent manifest.

[0086] The data processing system 102 may receive an intent manifest 118 from a third-party developer device 162. The intent manifest 118 (or intent manifest data structure) may include a mapping between actions of a digital assistant and link templates of electronic resources developed by the third-party developer device 162. The intent manifest 118 may be specific to or configured for an electronic resource. The intent manifest 118 may facilitate the integration of a digital assistant (e.g., via a voice navigator and response component 150) with a web page (e.g., a 3P electronic resource 134).

[0087] The intent manifest 118 data structure may include a definition for an action having one or more fields. The action may include an intent name and an implementation. The implementation may refer to a technique or process for performing the action. The implementation may include, for example, a URL link template, or a technique for invoking a JavaScript intent action API. The implementation may include one or more parameters integrated with the URL link template. The values of the parameters may be predicted, selected, generated, or identified by a data value predictor component 112.

[0088] The intent list data structure 118 can have name / value pairs. For example, the name can be "intentName", and the value can be "actions.intent.example_action". The name / value pairs can be separated by ":". The intent list 118 data structure can be: {"action": [{"intentName": "action.intent.NAME", "fulfillment": [{"urlTemplate": exampledomain_com / ?action=exampleaction1{parameter1[parameter1_value],parameter2[parameter2_value],parameter3[parameter3_value]}", "parameter": [{"intentParameter": "exampleaction.parameter1", "isRequired": true, "urlParameter": "parameter1[parameter1_value]"}, {"intentParameter": "exampleaction.parameter2", "isRequired": true, "urlParameter": "parameter2[parameter2_value]"}, {"intentParameter": "exampleaction.parameter3", "isRequired": true, "urlParameter": "parameter3[parameter3_value]"}.

[0089] For example, the intent list 118 data structure of the 3P electronic resource 134 that provides a ridesharing or ride reservation function can include:

[0090]

[0091]

[0092]

[0093] The 3P developer device 162 may construct, generate, or develop a list of intents for an electronic resource. The 3P developer device 162 may construct, generate, or develop an electronic resource 134. The verification component 110 may receive the list of intents 118 submitted by the 3P developer device 162. The verification component 110 may use one or more verification policies 116 stored in the data repository 114 to verify the list of intents 118. The verification component 110 may retrieve the verification policy 116 from the data repository 114 for application to the list of intents. To verify the list of intents, the verification component 110 may parse the list of intents. The verification component 110 may parse the list of intents in response to receiving the list of intents from the 3P developer device 162. The verification component 110 may verify the list of intents in response to a request to verify the list of intents. The verification component 110 may receive a request to verify the list of intents from the 3P developer device 162 or from a component of the data processing system 102.

[0094] The verification component 110 may use the verification policy 116 to verify the list of intents. The verification policy 116 may indicate the types of content, format, script, function, or components that are permitted or prohibited from the list of intents. The verification component 110 may parse the list of intents or extract data from the list of intents. The verification component 110 may compare the output of parsing the list of intents or the result of extracting from the list of intents with the verification policy 116 to determine whether one or more items or components in the list of intents are prohibited. If the list of intents passes the verification policy 116 (e.g., the verification component 110 does not detect any prohibited items indicated by the verification policy 116), the verification component 110 may indicate that the list of intents is valid. However, if the verification component 110 detects one or more prohibited items indicated by the verification policy 116 in the list of intents, the verification component 110 may determine that the list of intents is invalid.

[0095] For example, the verification policy 116 may indicate that a list of intents missing one or more pieces of information such as action definitions, parameters, data values, or links is invalid or incorrect. By way of example, if the list of intents does not include one or more of an intent name, a link template (or URL template), a parameter, a parameter value, or an indication of whether the parameter is required, the verification list 110 using the verification policy 116 may determine that the list of intents is invalid. If a name is not paired with a value, or a value is not paired with a name, the verification component 110 using the verification policy 116 may determine that the list of intents is missing information.

[0096] The verification policy 116 can indicate a valid format for the intent list 118. The verification component 110 can use the verification policy 116 to determine whether the intent list is valid or invalid based on the format of the permitted intent list. For example, a valid format can be a JavaScript Object Notation ("JSON") file. A JSON file can refer to a lightweight format for storing and transmitting data. A JSON file can include an array of records. The array of records can include information about actions, implementations, link templates, or parameters of link templates. The JSON file can be constructed using syntax rules. The syntax rules can include: for example, data in name / value pairs, data separated by commas, braces holding objects, and square brackets holding arrays. The verification policy 116 can include these syntax rules as the permitted format for the intent list data structure 118. The verification component 110 can use this verification policy 116 together with the syntax rules to determine whether the format of the intent list data structure 118 is valid and to determine whether to verify or invalidate the intent list data structure 118 received by the verification 3P developer device 162.

[0097] The verification policy 116 can include testing a link constructed using a link template provided in the intent list 118. For example, the verification component 110 can use the intent list 118 to construct a test link based on the verification policy 116. The verification component 110 can input data values of parameters in the link template and then execute the constructed link to determine whether the link works and can perform an action, or to determine whether the link is broken or causes another failure. Thus, the verification component 110 can determine whether the intent list 118 defines an action, parameters, and a link template in a way that results in the construction of a working link for performing an action. The verification component 110 can use the intent list 118 to generate a link or initiate an action to determine whether the 3P developer device 162 receives a request and the data values of the parameters for performing the action. The verification component 110 can receive a status indication from the 3P developer device 162 that indicates whether the execution of the action is successful or failed.

[0098] The verification policy 116 can include determining whether the intent list 118 includes any malicious code or is vulnerable to intrusion or security vulnerabilities. The verification policy 116 can include a list of trusted links or a list of links that are not trusted or unauthorized. The verification component 110 can use the verification policy 116 to determine whether the links included in the intent list 118 are authorized or unauthorized based on a predetermined list in order to verify or invalidate the intent list 118. For example, a website can be maliciously configured to circumvent the restrictions established by the same-origin policy of a web browser. The web browser 130 can use the same-origin policy to prevent different domains associated with different inline frames from accessing each other's data. The data processing system 102 can determine that a website is invalid or malicious by identifying the website in the link template 122 in the intent list 118 and determining whether the link is valid. This can be based on a predetermined list of trusted websites or a list of predetermined untrusted or malicious websites. Thus, using the intent list 118, the data processing system 102 can verify the 3P electronic resource 134.

[0099] The verification component 110 can apply or execute the verification policy 116 to determine whether to block, deny, prevent, or remove the intent list 118 from the intent list 118. Thus, the verification component 110 can use the verification policy 116 to determine whether to verify or not verify the intent list 118 based on whether the intent list 118 has no missing information, is in the correct format, or is available for constructing a working link. For example, the verification component 110 can verify the intent list 118 to store it in the data repository 114 of the data processing system 102 in response to determining that the format is correct and there is no missing information from the intent list 118. For example, if the verification component 110 detects an incorrect or unauthorized format or missing information in the intent list 118, the verification component 110 can determine not to verify the intent list 118 in response to detecting the format error or missing information in the intent list 118 and remove the intent list 118 from storage in the data repository 114.

[0100] The verification component 110 may verify the intent manifest 118 before storing the intent manifest 118 in the memory of the data processing system 102 or in the data repository 114. The data processing system 102 may store the verified intent manifest 118. The data processing system 102 may determine not to store an invalid intent manifest 118. The data processing system 102 may determine to remove the invalid intent manifest 118 that fails the verification process performed by the verification component 110. By determining not to store the invalid intent manifest 118, the data processing system 102 may reduce the memory or storage utilization in the data repository 114. The data processing system 102 may prevent or mitigate error activities occurring on the client computing device 128 by not forwarding the invalid document to the client computing device 128, thereby preventing the client computing device 128 from executing or rendering the invalid intent manifest 118 that may contain errors or unauthorized functionality. The data processing system 102 may prevent or mitigate security failures by determining not to use the invalid intent manifest 118 to implement actions or intents from the client computing device 128. Thus, the verification component 110 may reduce the computing resource utilization (e.g., memory utilization) of the data processing system 102, reduce or prevent errors or crashes on the client computing device 128, and avoid security failures on the client computing device 128. Security failures may occur because the intent manifest 118 containing the link template may lead to an intrusion or vulnerability that can be exploited by a malicious third party. The data processing system 102 may use the verification policy 116 to determine not to store such intent manifests 118 in the data repository 114 and not to use such intent manifests 118 to implement intents or actions from the client computing device 128.

[0101] The data processing system 102 may provide a prompt indicating the verification status to the 3P developer device 162. The data processing system 102 may indicate verification success or verification failure or unsuccessful. If the data processing system 102 determines that the intent manifest 118 is invalid or fails verification, the data processing system 102 may automatically resolve, modify, or fix the errors detected in the intent manifest 118 so that the intent manifest 118 can be verified, or the data processing system 102 may send a request to the 3P developer device 162 to resolve the errors detected in the intent manifest 118.

[0102] The data processing system 102 can automatically resolve, debug, or fix the intent list 118 in response to detecting an error or the intent list 118 being invalid. The data processing system 102 can automatically debug or resolve the intent list 118 by removing or erasing the error or invalid code. For example, the data processing system 102 can remove references to parameters that are unavailable or not used to perform an action. The data processing system 102 can remove references to actions that the digital assistant system cannot perform. The data processing system 102 can automatically resolve the intent list 118 that contains code with an invalid format by converting or reformatting the code into a valid or permitted format. For example, the data processing system 102 can detect that the syntax of the intent list 118 is not in the JSON format and automatically convert the intent list 118 into a permitted syntax or format, such as JSON or some other permitted format. Thus, the validation component 110 can use the validation policy 116 to determine whether the intent list 118 is valid, determine whether to store the intent list 118, reject the intent list 118, or resolve the intent list 118 before storing it in the data repository 114. The data repository 114 can save or store the intent list 118 that has been verified.

[0103] The client computing device 128 can include or execute a web browser 130. The web browser 130 can include an application that is designed, constructed, or operated to render or present electronic content. The web browser 130 can include or can be, for example, an application. The web browser 130 can be a native application, a web application, or other component for transmitting a request for a 3P electronic resource 134, receiving the 3P electronic resource 134, and rendering the 3P electronic resource 134. The web browser 130 can be configured to send a request for the 3P electronic resource 134 to the data processing system 102 or the 3P developer device 162 or some other server, such as a cache server. In some cases, the data processing system 102 can include a cache server that can intercept requests to access the 3P electronic resource 134. Intercepting the request can mean that the cache server receives the request for the 3P electronic resource 134 instead of the 3P developer device 162. The cache server can intercept the request by configuring the web browser 130 with the IP address of the cache server so that requests for electronic documents from the 3P developer device 162 are sent to the cache server instead of the 3P developer device 162 or a server associated with the 3P developer device 162. By configuring the web browser 130 to send requests to the cache server instead of the 3P developer device 162, the system 100 can reduce the latency or delay associated with the response to requests for electronic documents.

[0104] The Web browser 130 can load the 3P electronic resource 134 in the Web browser 130. The Web browser 130 can receive the 3P electronic resource 134 from the data processing system 102 or the 3P developer device 162 or other servers. The Web browser 130 can parse or process the 3P electronic resource 134 (e.g., an electronic document or a web page) to render or present the 3P electronic resource 134 in the Web browser 130. The Web browser 130 can parse the 3P electronic resource 134 to determine whether to retrieve, download, or obtain or utilize other resources of the 3P electronic resource 134.

[0105] The Web browser 130 can send one or more requests to one or more servers to download one or more additional files or resources associated with the 3P electronic resource 134. The additional files or resources can include, for example, cascading style sheets ("css" files) or images. The css file can be a text file for formatting the content on the electronic document and can include information such as fonts, sizes, colors, spacing, borders, or the position of HTML information on the electronic document. When downloading one or more files or resources associated with the electronic document, the Web browser 130 can build the electronic document. The Web browser 130 can build the display with the electronic document by combining the information found in the retrieved electronic document (e.g., the original HTML file) and other information found in the resources. The Web browser 130 can build a document object model ("DOM"), which can include a model of the positions where things are displayed on the page according to HTML. The DOM can map the page in a relational way. The Web browser 130 can build a CSS object model ("SSOM"), which can map according to the CSS using styles which styles should be applied to different parts of the electronic document. The Web browser 130 can build a render tree, which can include combining the DOM and the CSSOM together to create a model of how the layout and painting of the electronic document should be.

[0106] The Web browser 130 can render or paint the electronic document in the parent frame 132. As opposed to loading into an inline frame 138, the parent frame 132 can refer to loading the electronic document into the Web browser 130 itself. In some cases, the Web browser 130 can load the electronic document into an inline frame 138. For example, the Web browser 130 or the electronic document can establish one or more inline frames 138 and load the content of the 3P electronic resource into the inline frame.

[0107] The 3P electronic resource 134 may include HTML content, JavaScript content, XML content, or other types of content. The 3P electronic resource 134 may include a JavaScript (“JS”) library 148. The JS library 148 may be embedded or included in the 3P electronic resource 134. The 3P developer device 162 may provide or establish the 3P electronic resource 134 for the JS library 148. The 3P developer device 162 may download the JS library 148 from the data processing system 102 (e.g., from the data repository 114), and then install, link, include, or otherwise provide the 3P electronic resource 134 to the JS library 148 such that when the client computing device 128 downloads the 3P electronic resource 134, the JS library 148 will be included with the 3P electronic resource 134.

[0108] The JS library 148 may securely and reliably host digital assistant functionality as an overlay rendered on the 3P electronic resource 134 and be capable of providing interactive and authenticated callbacks to the data processing system 102 in a manner opaque to the 3P electronic resource 134. The JavaScript library 148 of the present technical solution may provide secure communication because access by the 3P electronic resource 134 to data associated with the JavaScript library 148 or communication with the data processing system 102 may be prohibited or blocked before authorization. The secure provision of such data values may reduce the processor, memory, or battery consumption of the computing device by reducing the amount of latency caused by entering data values or launching other applications on the client computing device 128 to obtain data values.

[0109] The JS library 148 may be hosted within an iframe 138. The JS library 148 may provide or execute a data exchange component 140 and an authorization component 142. The JS library 148 may include code, programs, scripts, rules, or logic to provide digital assistant functionality for the 3P electronic resource 134. The digital assistant functionality may include, for example, a voice interface for NLP processing by the NLP component 106, voice-based navigation of the 3P electronic resource 134, and predicted data values for performing actions on the 3P electronic resource 134.

[0110] The JS library 148 may load or establish the iframe 138 to communicate with the data processing system 102. The iframe 138 may be linked to the data processing system 102 or a network domain associated with the data processing system 102. The JS library 148 or one or more components hosted within the iframe 138 may communicate with one or more other iframes or the parent frame 132 of the 3P electronic resource 134 using the publish message API.

[0111] The JS library 148 hosted in the inline frame 138 can access the data stored in the data repository 114. The JS library 148 can access the data repository 114, while the 3P electronic resource 134 can be prohibited from accessing the data repository 114. The JS library 148 can be configured with an identifier, token, or other credential that allows it to communicate with the data processing system 102 and its data repository 114. Since the inline frame 138 hosts other network domains, the web browser 130 can prohibit or block the 3P electronic resource 134 of the 3P developer device 162 from accessing certain data in different network domains associated with the data processing system 102. The web browser 130 can use the same-origin policy for different domains to prevent interaction with each other and limit access to another domain.

[0112] After the web browser 130 constructs an electronic document, the JS library 148 can establish an inline frame 138 of the web browser 130. The web browser 130 can include the inline frame 138. The inline frame 138 can refer to an inline frame. The inline frame 138 can be an HTML document embedded inside another HTML document within the web browser 130. The web browser 130 can use the inline frame 138 element as an overlay where digital assistant functionality can be provided. The inline frame 138 can be embedded into the web browser 130. The web browser 130 can load the data exchange component 140 and the authorization component 142 in the inline frame 138. The data processing system 102 can authorize the data exchange component 140 to be loaded into the inline frame 138 of the 3P electronic resource 134 in response to the verification of the 3P electronic resource 134 by the verification policy 116. The data processing system 102 can verify the 3P electronic resource 134 based on the intent manifest 118. The 3P electronic resource 134 can be verified by the intent manifest for the 3P electronic resource 134 verified by the verification component 110. If the verification component 110 uses one or more verification policies 116 to verify the intent manifest 118, the data processing system 102 can determine that the 3P electronic resource 134 is authorized to load the JS library 148 into the inline frame 138, and can enable the data exchange component 140 to access the identifier of the client computing device 128 and allow communication between the data exchange component 140 and the data processing system 102. However, if the data processing system 102 deems the intent manifest 118 to be invalid, the data processing system 102 can prevent the data exchange component 140 from being established, which can refer to or include denying the data exchange component 140 access to the data processing system 102 or account 124 information. The web browser 130 can restrict components from accessing certain parts of the web browser 130 or accessing certain memory or functionality of the client computing device 128. Thus, the web browser 130 can establish security constraints or other controls for the inline frame 138 or the parent frame 132 to limit the type of access or functionality provided by the inline frame 138 or the parent frame 132.

[0113] The web browser 130 may include or execute a data exchange component 140. The data exchange component 140 may include one or more rules, scripts, or programs. The data exchange component 140 loaded into the inline frame 138 via the JS library 148 may determine an identifier of the client computing device 128. The identifier may be associated with an account 124 linked or corresponding to the client computing device 128. The identifier may be an account identifier of the client computing device 128. The identifier may be an alphanumeric identifier, a token, a key, a numeric identifier, or other identifier. The identifier may be stored in a memory or other storage on the client computing device 128. However, the 3P electronic resource 134 may be restricted from accessing the memory storing the identifier of the client computing device 128. The web browser 130 may prevent unauthorized components from accessing the identifier. The data exchange component 140 of the JS library 148 loaded into the inline frame 138 may access the memory because the data exchange component 140 is associated with the same origin or source (such as the network domain of the data processing system 102) as the identifier. Thus, the data exchange component 140 may obtain the identifier of the account 124 from the memory of the client computing device 128. The data exchange component 140 may restrict access by the 3P electronic resource 134 in the parent frame 132 to the identifier of the client computing device 128 through the web browser 130's same-origin policy or other configuration. The 3P developer device 162 developing the 3P electronic resource 134 may be prohibited from accessing the identifier of the client computing device 128.

[0114] The data exchange component 140 may include or be configured with one or more protocols to communicate with the web browser 130, the parent frame 132, the data processing system 102, or the client computing device 128. The data exchange component 140 may communicate with one or more components of the web browser 130 by sending messages. The data exchange component 140 may send messages to, receive messages from, or send messages between components of the inline frame 138, the parent frame 132, or the 3P e-resource 134. For example, the web browser 130 (e.g., via the data exchange component 140) may send a message to the inline frame 138 (or the data exchange component 140) using, for example, "iframeE1.contentWindow.postMessage". Via the data exchange component 140, the web browser 130 or the parent frame may use, for example, "window.addEventListener('message')" to receive messages. The inline frame 138 (or the data exchange component 140) may send a message to the web browser 130 using, for example, "window.parent.postMessage". The inline frame 138 (e.g., the data exchange component 140) may use, for example, "window.addEventListener('message')" to receive messages. The postMessage() technique may accept parameters such as message and targetOrigin. The message parameter may include a string or object to be sent to the receiving window. The targetOrigin parameter may include the Uniform Resource Locator ("URL") of the window to which the message is being sent. The protocol, port, and hostname of the target window may be set to match this parameter of the message being sent. Using a wildcard, such as "*", may match any URL.

[0115] The data exchange component 140 and other components or resources loaded in the web browser 130 can communicate with each other. For example, the data exchange component 140 can correspond to an inline frame 138, and the 3P electronic resource 134 can execute in the inline frame 138 that is a child frame of the parent frame 132. In another example, the data exchange component 140 can be loaded into a separate inline frame 138, in which case the data exchange component 140 and the 3P electronic resource 134 (e.g., the in-situ intent execution API 144) can communicate with each other using the parent frame 132 of the web browser 130 as a relay. For example, the parent frame 132 (e.g., the first frame) can have two child inline frames (e.g., the second inline frame and the third inline frame). The second inline frame can communicate with the parent frame 132, which can relay the communication to the third inline frame 138. The third inline frame 138 can reply to the communication by sending a message back to the parent frame 132, and the parent frame can relay the message to the second inline frame 138.

[0116] The data exchange component 140 can send or provide the identifier of the client computing device 128 to the data processing system 102. The data processing system 102 can receive the identifier of the client computing device 128 that executes the 3P electronic resource 134 from the data exchange component 140 of the inline frame 138 of the 3P electronic resource 134 loaded from the client computing device 128. The data processing system 102 can query the in-situ status sharing API 146 for information about the status of the 3P electronic resource 134. The data processing system 102 can query the in-situ status sharing API 146 in response to receiving the identifier of the client computing device 128 or other requests.

[0117] The in-situ status sharing application programming interface (“API”) 146 can be used to configure or construct the 3P electronic resource 134. The 3P developer device 162 can use the in-situ status sharing API 146 to develop or construct the 3P electronic resource 134. The 3P developer device 162 can develop or construct the 3P electronic resource 134 to interface with the in-situ status sharing API 146. The in-situ status sharing API 146 can be designed, constructed, or operated to determine the semantic foreground state of the 3P electronic resource 134 and send the semantic foreground state information to the data processing system 102.

[0118] The on-site status sharing API 146 may include one or more rules, logics, codes, scripts, or programs configured to identify, detect, or determine the semantic state of the 3P electronic resource 134. The on-site status sharing API 146 may include a schema definition or a repository including entities such as people, places, or things and the relationships between the entities. The on-site status sharing API 146 may include a monitor or tracker component to identify the current state of the 3P electronic resource 134. For example, the on-site status sharing API 146 may parse the foreground of the 3P electronic resource 134 to identify the content being displayed, or any tags or markup languages that may indicate the semantic foreground state. The on-site status sharing API 146 may detect text, metadata, input fields, buttons, or other graphical user interface widgets. The on-site status sharing API 146 may use semantic analysis or processing techniques to convert the detected information into structured data corresponding to the schema.

[0119] The on-site status sharing API 146 may include or provide JavaScript callbacks. A callback may refer to or include a function that is executed after another function has completed execution. The on-site status sharing API 146 may be implemented by the 3P developer device 162 so that the site can publish the semantic foreground state when requested by the data processing system 102. The data processing system 102 may query or request the semantic foreground state from the on-site status sharing API 146. In response to the request, the on-site status sharing API 146 may provide the semantic foreground state. The semantic foreground state may refer to the current semantic state of the 3P electronic resource 134, such as the content being displayed on a web page or the functions or actions being executed or available. The semantic state information may be encoded or communicated using a schema that provides a structure for the semantic state. The semantic state may include one or more entities that represent real-world or physical concepts as structured data in the foreground of the electronic resource. An entity may refer to a person, a place, or a thing. An entity may have a unique identifier. An entity may include attributes, types, and descriptions. An entity may include relationships with one or more other entities. An entity may provide a structure for the data. The semantic state may include one or more digital assistant intents that are instantaneously available in the current context of the electronic resource.

[0120] The on-site status sharing API 146 may be configured by the 3P developer device 162 using semantic state information for one or more states of the 3P electronic resource. For example, the semantic state information for a ride-sharing electronic resource may include actions such as "ride" or "book a ride" or "ride request". Additional semantic foreground information may include the location, destination, vehicle type, or pick-up time of the client computing device 128. In another example, the electronic resource may correspond to tickets for a concert. The semantic foreground information may include "ticket", "purchase", "price", or "quantity".

[0121] The data processing system 102 may receive the semantic foreground state of an electronic resource from the on-site status sharing API 146. The data processing system 102 may receive semantic foreground state information from the Web browser 130 or from the 3P developer device 162 via the client computing device 128. For example, the 3P developer device 162 may receive semantic foreground state information from the on-site status sharing API 146. The data processing system 102 may query the 3P developer device 162 for semantic foreground information using a unique identifier associated with the 3P electronic resource 134 rendered on the client computing device 128. The data processing system 102 may receive on-site status sharing information in response to querying the 3P developer device 162.

[0122] The data processing system 102 may receive semantic foreground state information via the data exchange component 140. The data processing system 102 may receive the semantic foreground state from the JS library 148, which receives the semantic foreground state from the on-site status sharing API 146 of the third-party electronic resource 134. The data exchange component 140 may interface or communicate with the on-site status sharing API 146 using the messaging protocol of the Web browser 130. The data processing system 102 may query the data exchange component 140 for status information. The data exchange component 140 may query the on-site status sharing API 146 for the current semantic foreground state of the 3P electronic resource 134. The on-site status sharing API 146 may provide the semantic foreground state to the data exchange component 140, which may forward the semantic foreground state to the data processing system 102.

[0123] The data processing system 102 may receive the semantic foreground state of the electronic resource 134 from the on-site status sharing API 146 of the electronic resource 134 via the data exchange component 140. The data processing system 102 may receive information in response to a query. The data processing system 102 may include a data value predictor component 112 that is designed, constructed, or operated to determine a parameter based on the semantic foreground state and the intent inventory data structure 118, and to select a data value for the parameter based on the identifier of the client computing device 128.

[0124] The data value predictor component 112 can accept a semantic foreground state as input. The semantic foreground state can indicate or identify the current intent associated with the electronic resource. The data value predictor component 112 can use the semantic foreground state information to search a data repository or database linked to the client computing device 128 (or its account 124) that renders the electronic resource 134. The data value predictor component 112 can search the database to predict the data values of the parameters of the current intent. If the data value predictor component 112 identifies an acceptable prediction, the data value predictor component 112 can provide the predicted value to the JS library 148 (or its components). The JS library 148 (e.g., via the authorization component 142) can present the predicted data value for authorization. If the predicted data value is authorized, the data value can be provided or passed to the electronic resource. The JS library 148 can provide the predicted and authorized data value to the third-party electronic resource 134 via a link (e.g., a URL deep link) or a JavaScript intent execution API 144.

[0125] For example, the electronic resource can include a car rental website. The data processing system can identify the current semantic foreground state that indicates the intent of book_car_rental (to_location, from_location, start_time, end_time). The data processing system can search for and identify data regarding upcoming flight reservations stored in the database that is associated with the account of the client computing device corresponding to rendering the third-party electronic resource. The data processing system can predict the data values of the intent parameters based on the data in the database. The data processing system can send the predicted data values of the parameters to the client computing device. The data processing system can perform an action corresponding to the intent on the third-party electronic website in response to authorization.

[0126] The data value predictor component 112 can identify data values in response to semantic foreground state information provided by the in-field state sharing API 146 using one or more selection techniques. Using the semantic foreground state information, the data value predictor component 112 can identify actions in the intent list 118 for the 3P electronic resource 134. The data value predictor component 112 can perform a lookup in the data repository 114 to identify or select an intent list data structure 118 that corresponds to or matches the semantic foreground state. The data value predictor component 112 can use semantic selection techniques or other selection or matching techniques to identify the intent list 118 of the 3P electronic resource 134. For example, the data value predictor component 112 can determine the domain of the 3P electronic resource 134 and then identify one or more intent lists 118 that have link templates 122 that match the domain of the 3P electronic resource 134. Thereafter, the data value predictor component 112 can select the intent list 118 of the 3P electronic resource 134 that contains an action corresponding to the semantic foreground state. For example, if the semantic foreground state indicates "book a ride", the data value predictor component 112 can select the intent list 118 that has the action "book a ride".

[0127] The in-field state sharing API 146 can provide semantic foreground state information corresponding to the actions in the intent list 118 so that the data value predictor component 112 can identify the match. The data value predictor component 112 can use various matching or selection techniques to predict the match. The data value predictor component 112 can determine a match score between each intent list 118 or action 120 and the semantic foreground information to determine the highest scoring match or the most relevant match.

[0128] After identifying the matching intent list 118 or action 120, the data value predictor component 112 can determine the parameters of the link template 122. The intent list 118 maps the action 120 to the link template 122. The data value predictor component 112 can identify the link template of the action 122 that corresponds to the action 120 corresponding to the semantic foreground state. The data value predictor component 112 can identify the parameters of the link template 122. The link template 122 can include one or more parameters. The parameters can include parameter names. The parameters in the link template 122 can serve as placeholders for parameter data values.

[0129] The data value predictor component 112 can identify data values for parameters of the link template 122. The data value predictor component 112 can access the data repository 114 to identify account information 124 storing the data values 126. The data value predictor component 112 can perform a lookup in the account 124 data structure to determine data values 126 for parameters in response to the intent manifest 118, and facilitate the 3P electronic resource 134 or the 3P developer device 162 to perform services, actions, or functions. The data value predictor component 112 can select or identify values for the action data structure that can be used by the direct action API 108 to generate and can be sent to the 3P developer device 162 to execute or implement the requested action.

[0130] The data value predictor component 112 can use semantic processing techniques, selection criteria, machine learning, or other techniques to select or identify candidate data values for parameters of the link template 122 in the intent manifest 118. The data value predictor component 112 can access one or more sources to determine data values. For example, the data value predictor component 112 can access the account data structure 124 containing data values 126 associated with the client computing device 128 or its user. The data processing system 102 can be configured to respond to an authorization query from the client computing device 128 with an external data source associated with the client computing device 128.

[0131] The data value predictor component 112 can identify one or more data values in response to semantic foreground state information received from the data exchange component 140. The data value predictor component 112 can identify multiple data values. The data value predictor component 112 can determine to send one or more data values identified by the data value predictor component 112 to the data exchange component 140 or the web browser 130. In some embodiments, the data value predictor component 112 may not be able to identify specific data values directly in response to the context information, and can determine to send a subset of the identified data values based on ranking or filtering techniques. For example, each data value can be associated with a confidence score, a ranking score, or a correlation score. The data value predictor component 112 can determine to send the data values with the highest rankings because those data values may be most likely to respond to the semantic foreground information of the 3P electronic resource 134. In some cases, the data value predictor component 112 can send the top three data values, the top five data values, the top ten data values, or other quantities of data values.

[0132] For example, the semantic foreground state information can indicate the intent list 118 with action 120, which has a geographical address parameter requested by the 3P electronic resource 134 to perform a service or action. The data value predictor component 112 can perform a lookup in the data value data structure 126 of the account 124 corresponding to the client computing device 128 to identify the address. The data value 126 can include one or more addresses. The data value predictor component 112 can send one or more addresses retrieved from the data value data structure 126 in response to the request. In another example, the link template 122 can indicate that the 3P electronic resource 134 requests financial account information to perform an action or service. The data value predictor component 112 can perform a lookup in the data value 126 data structure to identify one or more account identifiers, and send the one or more account identifiers to the Web browser 130 via the network 101. Therefore, the data value predictor component 112 can generate data values in response to the intent list 118.

[0133] The data processing system 102 can determine multiple parameters for performing the action provided by the electronic resource 134 based on the semantic foreground state. For example, the intent list 118 with action 120 that matches or corresponds to the received semantic foreground state can include multiple parameters. The data processing system 102 can select multiple data values corresponding to the multiple parameters based on the identifier of the client computing device 128. For example, for the "book a ride" action, the parameters can include "boarding longitude", "boarding latitude", "account identifier", "financial account information", "boarding time", "destination longitude", or "destination latitude". The data processing system 102 can provide the data values to the authorization component 142, so that the authorization component 142 can provide the data values to the in-situ intent execution API 144. Then, the in-situ intent execution API 144 can use the data values to bypass one or more states used by the electronic resource 134 to perform the action.

[0134] 3P electronic resources 134 may have used multiple states, pages, processes, prompts, or requests to obtain data values entered for parameters used to perform an action. For example, a transaction process for booking a ride may include: a first page where the user initiates a request, a second page where the user enters a pick-up location, a third page where the user enters a destination, a fourth page where the user selects payment information, and a fifth page where the user sends the request. However, since the data value predictor component 112 can use the intent manifest 118 to identify multiple parameters required to perform the action 120, the JS library 148 can provide all data values of the parameters of the action 120 in a single communication or transmission, or a series of data packets (which are part of a single transmission). When multiple parameters and data values are received, the in-situ intent execution API 144 can bypass one or more pages in the ride booking transaction process and directly proceed to perform the action of requesting a ride, or request confirmation of the execution of the ride. For example, the in-situ intent execution API 144 can skip the second page, the third page, or the fourth page. Thus, the data processing system 102, via the JS library 148, the in-situ status sharing API 146, and the in-situ intent execution API 144, can facilitate input and can reduce computational resource consumption and remote procedure calls by bypassing one or more pages, requests, or prompts to perform the action.

[0135] The data processing system 102 may determine one or more subsequent states of the electronic resource 134 based on the semantic foreground state and the intent list data structure 118. For example, the intent list structure 118 may include multiple parameters of the link template 122. The multiple parameters may indicate subsequent requests for input data values. The data processing system 102 may determine that the 3P electronic resource 134 is configured to request data value information from the user in one or more subsequent states. The subsequent states may include different web pages, drop-down menus, buttons, prompts, or other graphical user interface elements for the input data value information. The data processing system 102 may determine the multiple subsequent states based on the semantic foreground information. The data processing system 102 may determine the multiple subsequent states based on the historical state information of the 3P electronic resource 134 or based on the historical information associated with the semantic foreground state information. The semantic foreground state may be associated with a set of predetermined subsequent states or may historically be followed by one or more states. For example, the semantic foreground state of purchasing sports shoes may typically be followed by requests for shoe size, address, billing information, and shipping method. The data processing system 102 may select one or more data values for one or more parameters based on the identifier of the client computing device 128 or the account 124 identifier before the electronic resource enters one or more subsequent states. For example, before the 3P electronic resource 134 requests sports shoe size, billing information, or other information, the data processing system 102 may select the data values via the data value predictor component 112. The data processing system 102 may provide the data values before the 3P electronic resource 134 enters the subsequent states, thereby allowing the 3P electronic resource to bypass those states or making those states more efficient by making the input data values immediately available after entering the states.

[0136] The data processing system 102 may provide the data values of the parameters to the JS library 148. The data processing system 102 may provide the data values of the parameters to the data exchange component 140 or the authorization component 142. For example, the data processing system 102 may provide the selected candidate data values to the authorization component 142 of the JS library 148 in the inline frame 138 to determine whether the 3P electronic resource 134 is authorized to receive the data value. The authorization component 142 may generate an authorization prompt, receive input from the client computing device 128 in response to the authorization prompt, and send the data value to the in-situ intent execution API 144 of the electronic resource 134 in response to the authorization of the data value to enable the electronic resource 134 to perform an action using the data value.

[0137] The authorization component 142 may include one or more rules, policies, codes, programs, or scripts. The authorization component 142 may be established or hosted by the JS library 148 in the embedded framework 138. In the case of no authorization, the authorization component 142 may securely receive data values without sharing or consenting to its access to the data with the 3P electronic resource 134 or the 3P developer device 162. The web browser 130 may block the 3P electronic resource 134 from accessing the data values received by the authorization component 142.

[0138] The authorization component 142 may be constructed or operated to generate a prompt including one or more data values received from the data processing system 102. The authorization component 142 may generate a graphical user interface, window, button, or other notification including one or more data values. The authorization component 142 may generate a prompt containing the data values before consenting to the 3P electronic resource 134 to access the data values, so as to maintain a secure communication channel. The authorization component 142 may generate a pop-up window or other user interface elements having one or more buttons or controls. The authorization component 142 may determine to overlay the window on the 3P electronic resource 134. For example, the authorization component 142 may generate a suggested drop-down menu or an autofill drop-down menu, or make suggestions at the position corresponding to the input form field or input text box on the 3P electronic resource 134. The authorization component 142 may render the data values on the 3P electronic resource 134 in a separate embedded framework that is secure and inaccessible to the 3P electronic resource 134.

[0139] The authorization component 142 may be configured to prohibit the 3P electronic resource 134 from accessing the data values before authorizing the data values. For example, the authorization component 142 may have accessed the data values to generate a prompt, but the in-situ intent execution API 144 and the 3P electronic resource 134 may not be able to access the data values unless it is authorized by the client computing device 128. In addition, the 3P electronic resource 134 or the in-situ intent execution API 144 may not be able to access all candidate data values sent from the data processing system 102 to the web browser 130 and provided by the authorization component 142 in the prompt. Instead, the 3P electronic resource 134 may be consented to access the data values authorized by the client computing device 128, but not the other candidate data values displayed by the authorization component 142 in the prompt but not selected by the client computing device 128 to be provided to the 3P electronic resource 134. Therefore, the JS library 148 may be configured to send only the authorized data values to the in-situ intent execution API 144 for input into the 3P electronic resource 134.

[0140] The authorization component 142 can provide data values for display and includes input buttons to allow the client computing device 128 to select a data value or authorize a data value to be sent to the 3P electronic resource 134. For example, the authorization component 142 can receive three different addresses from the data value predictor component 112. The authorization component 142 can provide an indication of these three candidate addresses in a secure manner via the web browser 130. The authorization component 142 can include buttons or other input mechanisms to allow the client computing device 128 to select one of the three candidate addresses. The web browser 130 can receive the selection of the data value from the client computing device 128 or transmit or provide authorization of the data value to the 3P electronic resource 134. The web browser 130 can receive input from the client computing device 128 for the authorized data value in response to a prompt.

[0141] In response to the authorization of the data value by the authorization component 142, the JS library 148 can provide the data value to the 3P electronic resource 134 or execute a link constructed using the link template 122 and the authorized data value. The data processing system 102 can use the link template 122 to construct a link and provide the link to the JS library 148 (e.g., via the data exchange component 140). The data exchange component 140 can provide the link to the in-situ intent execution API 144 for execution. The in-situ intent execution API 144 can execute or initiate the constructed link to initiate the execution of an action.

[0142] In some cases, the JS library 148 can provide the data value to the in-situ intent execution API 144. The in-situ intent execution API 144 can obtain the data value and initiate the execution of an action without redirecting the web browser 130 to a different web page via a link. The in-situ intent execution API 144 can input the data value into the 3P electronic resource 134 and cause the 3P electronic resource 134 to use the data value to execute an action. For example, the in-situ intent execution API 144 can input an address into an input form field in the 3P electronic resource 134 and then select a link or other trigger to initiate the processing of the address to execute a function. The in-situ intent execution API 144 can input one or more authorized data values into one or more input fields in the 3P electronic resource 134. The data processing system 102 can provide the data value to the in-situ intent execution API 144 to cause the in-situ intent execution API 144 to input the data value into an input text box of the electronic resource 134.

[0143] The data processing system 102 can provide a data value to an authorized component 142 for authorization and input into the in-situ intent execution API 144 before the electronic resource 134 requests the data value. The data processing system 102 can predict or determine the data value for input based on one or more parameters in the intent manifest 118 of the electronic resource 134 and provide the data value to the client computing device 128 before the electronic resource 134 has to request the data value.

[0144] The data processing system 102 can provide the data value to the client computing device so that the client computing device 128 can use the data value to create a deep link and load the deep link into the web browser 130 executed by the client computing device 128. For example, the data processing system 102 provides the link template 122 to the client computing device 128. After the data value is authorized, the JS library 148 or the in-situ intent execution API 144 can use the data value to construct a link. In some cases, the in-situ intent action API 144 can have a built-in link template or other intent execution techniques. The in-situ intent execution API 144 can generate, construct, or build a link or other command with the data value. The in-situ intent execution API 144 can determine whether to construct a deep link with the data value with parameters or generate another type of command to send to the 3P developer device 162 to perform the action.

[0145] For example, to book a flight, the in-situ intent execution API 144 can construct a deep link with the data value and launch the deep link in the web browser 130. The in-situ intent execution API 144 can cause the web browser 130 to load the deep link with the data value to display available flights and prices and allow the user to select a flight. In another example, such as to book a ride, the in-situ intent execution API 144 can determine to generate a command with the data value to perform the action of booking a ride without causing the web browser 130 to load a new web page. Instead, the in-situ intent execution API 144 can determine that it may be more effective to display a prompt requesting permission or authorization to book a ride (or perform the action 120) with the predicted data value. Once confirmed, the in-situ intent execution API 144 can send the command to the 3P developer device 162 to cause the 3P developer device 162 to complete the action. Therefore, from the perspective of computing device processing, it may be more effective to avoid redirecting the web browser 130 to a new web page and loading the web page and instead send a command to the 3P developer device 162 to perform the action.

[0146] The in - field intent execution API 144 can determine, based on a policy, whether to generate and load a deep link or send a command without loading a deep link. For example, the policy can be whether additional data value input is required to execute an action. If an action can be executed based on all the predicted data values, it may be more efficient to send a command without loading a deep link. However, if additional input is required, such as selecting a flight from multiple options, the in - field intent execution API 144 can load a deep link. The data processing system 102 can construct a link with data values based on the link template 122 mapped to the action 120 and provide the link to the in - field intent execution API 144 via the data exchange component 140. The in - field intent execution API 144 can determine whether to load the link or send the link, or a link - based command or information, to the 3P developer device 162 (e.g., a server associated with the 3P developer device 162 to complete the action 120) to execute the action.

[0147] The in - field intent execution API 144 can determine whether a data value or link is valid. For example, if the electronic resource 134 is related to tracking shipping information and the input data value is a tracking number, the in - field intent execution API 144 can determine whether the format of the data value corresponds to a predetermined format of the tracking number used by the 3P developer device 162 (e.g., alphanumeric, number of digits, order of numbers and letters). After determining that the data value is valid, the in - field intent execution API 144 can construct a deep link with the data value.

[0148] The in - field intent execution API 144 can include a JavaScript callback implemented by the 3P developer device 162 to enable the electronic resource 134 to handle digital assistant intents triggered by the data processing system 102 or the client computing device 128. The digital assistant intent can refer to the action 120 in the intent list 118.

[0149] The client computing device 128 can include a voice navigator and response component 150. The voice navigator and response component 150 can be interface - connected to one or more of the sensor 152, transducer 154, audio driver 156, pre - processor 158, or display device 160. The voice navigator and response component 150 can include one or more components or functionalities of the data processing system 102, such as the NLP component 106 or the direct action API 108.

[0150] The voice navigator and response component 150 may be referred to as a digital assistant component or client or local digital assistant component. The data processing system 102 may be referred to as a server digital assistant component. When the data processing system 102 provides structured intent parsing (e.g., actions 120 or links constructed based on link templates 122) via the natural language processor component 106 that can be processed by a third-party electronic resource 134 integrated with the voice navigator and response component 150 and the JS library 148, the data processing system 102 may invoke the voice navigator and response component 150. This technology can transform user intent parsing into URL links or JavaScript intent execution calls, which can be used to navigate the electronic resource 134 via the in-situ intent execution API 144. After the JS library 148 executes an intent on the third-party electronic resource 134 via the in-situ intent execution API 144, the JS library 148 may request the foreground semantic state from the JavaScript callback of the electronic resource 134. The voice navigator and response component 150 or the data processing system 102 may match the foreground state data with a voice response (text-to-speech) template that has been pre-associated with the matching user intent. The voice navigator and response component 150 may render a text-to-speech response to the user by passing the state data to the template. This technology enables users to perform voice navigation across the entire website and hear text-to-speech ("TTS") answers after each voice navigation.

[0151] The voice navigator and response component 150 may allow voice-based navigation on the electronic resource 134. The voice navigator and response component 150 may receive audio input from a sensor 152 (e.g., a microphone). The voice navigator and response component 150 may send (or send pre-processed audio input via data grouping) the audio input to the data processing system 102. The data processing system 102 may determine the intent in the audio input via the NLP component 106. The data processing system 102 may determine that the intent is to perform an action on the electronic resource. The data processing system 102 may query the in-situ state sharing API 146 to determine the semantic foreground state of the electronic resource 134. Thus, the voice input detected by the voice navigator and response component 150 may include a request to perform an action received from the user. In response to the voice-based request, the data processing system 102 may query the in-situ state sharing API 146 to determine the semantic foreground state of the electronic resource 134, select the intent manifest 118, and predict or select data values based on the actions 120 or link templates 122 of the intent manifest.

[0152] The data processing system 102 can provide data values to the authorization component 142. The authorization component 142 can interface with the voice navigator and response component 150 to present the data values via visual output or audio output. The authorization component 142 can interface with the voice navigator and response component 150 to obtain authorization or input via voice audio input. The authorization component 142 can pass the data value to the in-field intent execution API 144 in response to a voice input of an authorized data value. Thus, the data processing system 102 can receive a data packet from the voice navigator and response component 150 executed by the client computing device, the data packet carrying an input audio signal detected by the sensor 152 of the client computing device 128. The data processing system 102 can identify a request for a candidate data value from the data packet and provide the data value as the candidate data value in response to the request.

[0153] The in-field intent execution API 144 can perform an action on the 3P electronic resource 134 in response to receiving the data value. In some cases, if the user interface is a voice-based user interface provided by the voice navigator and response component 150, the in-field intent execution API 144 can determine not to load a deep link, thereby reducing computing resource utilization by avoiding having to draw or load a web page on the display device 160.

[0154] Figure 2 is an illustration of the operation of a system 200 for secure communication in a web page. The system 200 can include Figure 1 the depicted system 100 or Figure 4 one or more components of the depicted system 400. The system 200 can include a data processing system 102. The data processing system 102 can communicate with, interface with, or interact with the 3P developer device 162. In action 202, the data processing system can receive an intent manifest from the 3P developer device 162. The 3P developer device 162 can provide or upload the intent manifest to the data processing system 102. In action 204, the data processing system 102 can determine whether the intent manifest is valid. The data processing system 102 can use a validation policy to determine whether the intent manifest is valid. The validation policy can consider whether the code type, syntax, format, or links in the intent manifest are trusted. For example, if an electronic document does not contain name / value pairs, the data processing system 102 can determine that the electronic document is invalid.

[0155] If the data processing system 102 determines that the intent list is invalid, the data processing system 102 may apply security constraints and notify the 3P developer device 162 in action 206. The data processing system 102 may generate a prompt or notification indicating that the intent list verification has failed or is invalid. The data processing system 102 may further indicate the reason for the invalidity of the intent list and provide suggestions on how to resolve, fix, or modify the intent list to make it valid. If in action 204 the data processing system 102 determines that the intent list is valid, the data processing system 102 may continue to store the intent list in the data repository in action 208.

[0156] In action 210, the client computing device 128 may load an electronic resource in the web browser 130. For example, the electronic resource may include a web page. In action 212, the client computing device 128 may receive voice input. The voice navigator and response component 150 may detect the voice input via the microphone or sensor of the client computing device 128. In action 214, the client computing device 128 may send a data packet including audio input corresponding to the detected voice input to the data processing system 102. In action 214, the data processing system 102 may use natural language processing (e.g., via the NLP component 106) to process the audio input to determine the intent.

[0157] In action 216, the data processing system 102 may determine whether to request status information from the electronic resource. The data processing system 102 may determine whether to request status information based on the intent. If the intent corresponds to an action on the electronic resource, the data processing system 102 may determine to request status information in decision box 216. However, if the intent is not related to the electronic resource (e.g., a request to lower the volume or other requests not related to the electronic resource), the data processing system 102 may determine to exit in action 218.

[0158] If the data processing system 102 determines that the intent is related to an action to be performed via the electronic resource, the data processing system may enter action 220 and request status information. The data processing system 102 may query the live status sharing API 146 to obtain semantic foreground status information. In action 222, the data processing system 102 may receive semantic foreground status information from the live status sharing API 146.

[0159] In operation 224, data processing system 102 may receive status information and determine parameters. The data processing system 102 may access an intent list data structure to select an intent list for an electronic resource corresponding to the status information. The intent list may include actions corresponding to or in response to the intent determined in operation 214 based on the voice input 212. The data processing system 102 may select an intent list that maps actions to link templates to identify parameters associated with the actions and link templates.

[0160] In operation 226, data processing system 102 may receive an identifier of client computing device 128. The data processing system 102 may receive the identifier at any point in the process. For example, the data processing system 102 may receive the identifier in response to loading the resource in 210, loading the input voice in 212, when requesting the status information in 220. In operation 228, data processing system 102 may determine a data value using the identifier and the parameters. The data processing system 102 may access one or more data sources linked to the client computing device 128 or the identifier to determine, predict, select, or identify candidate data values.

[0161] In operation 230, data processing system 102 may provide the selected or candidate data value to the authorization component 142 of client computing device 128. The authorization component 142 may execute in an overlay in the JS library 148 that prevents the electronic resource from accessing the data value until authorized. The authorization component 142 may receive an input (e.g., voice, keyboard, mouse, gesture, or other input) indicating whether the data value is authorized. In some cases, multiple candidate data values may be provided and the user may select one or more data values for input.

[0162] In decision block 232, the authorization component 142 may determine whether to provide the data value to the electronic resource. If the authorization component 142 determines that the data value is authorized, the authorization component may provide the data value to the in-situ intent execution API 234 via the JS library 148 to perform the action. The JS library may cause the in-situ intent execution API 234 to complete the action via the electronic resource or the 3P developer device 162.

[0163] However, if the authorization component 142 determines that the data value is not authorized, the authorization component 142 can provide an indication to the data processing system 102. In decision block 236, the data processing system 102 can determine whether to update the data value in response to the authorization component 142 not authorizing the data value. If the number of updates is less than a threshold number (e.g., 2, 3, 4, 5, or more), or based on the type of intention or preference of the third-party developer device 162 indicated in the electronic resource or intention list, if there are other candidate values available, the data processing system 102 can determine to update the value. If the data processing system 102 determines to update the data value, the data processing system 102 can return to action 228 to select another data value. If the data processing system 102 determines not to update the data value, the data processing system 102 can enter action 238 and terminate the communication.

[0164] Figure 3 is an illustration of an example method for secure communication in a mobile page. Method 300 can be performed by Figure 1 system 100 depicted by Figure 2 system 200 depicted by, or Figure 4 one or more components, systems, or elements of system 400 depicted by. For example, method 300 can be performed by a data processing system. In action 302, the data processing system can receive an intention list. The data processing system can receive the intention list from a 3P developer device. The intention list can map actions to link templates and indicate parameters for performing the actions.

[0165] In action 304, the data processing system can validate the intention list. The data processing system can use a validation policy to validate the intention list. Validating the intention list can include: for example, determining whether the intention list includes certain types of content, code, links, or formats. If the intention list does not include prohibited content, code, or formats, the data processing system can validate the intention list. If the intention list contains prohibited content, code, links, or formats, the data processing system can invalidate the intention list. By invalidating certain intention lists, the data processing system can reduce security risks, errors, vulnerabilities, crashes, and wasted computing resource utilization on the client computing device.

[0166] If, in operation 306, the data processing system determines that the intent list is invalid, the data processing system may proceed to operation 308 to determine whether to automatically modify the intent list. The data processing system may determine whether to automatically modify the intent list based on one or more factors or policies. If the 3P developer device authorizes or instructs the data processing system to automatically modify the invalid intent list, the data processing system may determine to automatically modify the intent list. If the reason for determining that the intent list is invalid corresponds to a problem that the data processing system is configured to remedy, the data processing system may determine to automatically modify the intent list. For example, if the intent list is invalid due to formatting or the format of a markup language, and the data processing system is configured to reformat the intent list into a permitted format (e.g., JSON), the data processing system may proceed to reformat the intent list. If the modification involves removing references to unauthorized actions or parameters, the data processing system may determine to automatically modify the intent list. If aspects that may cause further errors or vulnerabilities in the intent list need to be removed (e.g., removing domains in a link template), the data processing system may determine not to make modifications. Thus, the data processing system may determine to modify the intent list based on the number or type of validation failures.

[0167] If the data processing system determines not to automatically modify the intent list, the data processing system may proceed to operation 310 and notify the 3P developer that the intent list is invalid, the reason for the invalidity of the intent list, and request the 3P developer to resolve the issues in the intent list.

[0168] If, in operation 308, the data processing system determines to automatically modify the intent list, the data processing system may proceed to operation 312 and modify the intent list. The data processing system may proceed to operation 312 to modify the intent list by reformatting the intent list based on the validation policy.

[0169] The data processing system may proceed to operation 314 to store the intent list in the data repository of the data processing system. If, in operation 306, the data processing system determines that the intent list is valid based on the validation policy, the data processing system may proceed to 314 to store the intent list in the data repository. In response to the validation of the intent list or the modification of the intent list, the data processing system may store the intent list in the data repository of the data processing system.

[0170] In operation 316, the data processing system may receive an identifier of a client computing device. The identifier may correspond to an account linked to the client computing device. The account may include information or data values associated with the client computing device. The account may include data values based on the historical network utilization of the client computing device. The account may include information stored by the client computing device. The account may be stored on the data processing system or one or more external sources. The account may include information from one or more external sources or servers associated with the client computing device.

[0171] In operation 318, the data processing system may receive semantic foreground state information. The data processing system may query the presence state sharing API for the state information. The data processing system may query the state sharing API in response to a request to perform an action. The data processing system may receive the state information in response to the query.

[0172] In operation 320, the data processing system may determine parameters. The data processing system may select an intent manifest and identify actions and link templates. The actions and link templates may indicate the parameters. In operation 322, the data processing system may select data values for the parameters in the intent manifest based on the account information associated with the identifier of the client computing device.

[0173] In operation 324, the data processing system may provide the data values to an authorization component of the client computing device. The authorization component may execute within an overlay on the electronic resource such that the data values are not accessible to the electronic resource until the data values are authorized to be provided to the electronic resource. The authorization component may present the data values through an overlay, prompt, notification, pop-up window, inline frame, or audio output. The authorization component may receive an input to authorize or reject the data values. If the data values are authorized, the authorization component may pass the data values to the electronic resource through a JS library and an intent execution API to cause the electronic resource to perform an action based on the data values.

[0174] Figure 4is a block diagram of an example computer system 400. The computer system or computing device 400 may include or be used to implement system 100, or its components, such as data processing system 102. The data processing system 102 may include a smart personal assistant or a voice-based digital assistant. The computing system 400 includes a bus 405 or other communication components for transferring information, and a processor 410 or processing circuitry coupled to the bus 405 for processing information. The computing system 400 may also include one or more processors 410 or processing circuitry coupled to the bus for processing information. The computing system 400 also includes a main memory 410 coupled to the bus 405 for storing information and instructions to be executed by the processor 415, such as random access memory (RAM) or other dynamic storage devices. The main memory 415 may be or may include data repository 145. The main memory 415 may also be used to store location information, temporary variables, or other intermediate information during the execution of instructions by the processor 410. The computing system 400 may further include a read only memory (ROM) 420 or other static storage devices coupled to the bus 405 for storing static information and instructions for the processor 410. A storage device 425 (such as, a solid state device, a magnetic disk, or an optical disk) may be coupled to the bus 405 to persistently store information and instructions. The storage device 425 may include data repository 145 or be a part of data repository 145.

[0175] The computing system 400 may be coupled via the bus 405 to a display 435 (such as, a liquid crystal display, or an active matrix display) to display information to a user. An input device 430, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 405 for transmitting information and command selections to the processor 410. The input device 430 may include a touch screen display 435. The input device 430 may also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for transmitting direction information and command selections to the processor 410 and for controlling cursor movement on the display 435. For example, the display 435 may be a part of the data processing system 102, the client computing device 128, or Figure 1 other components in.

[0176] The processes, systems, and methods described herein can be implemented by a computing system 400 in response to execution of an arrangement of instructions contained in main memory 415 by a processor 410. Such instructions can be read into main memory 415 from another computer-readable medium, such as a storage device 425. Execution of the arrangement of instructions contained in main memory 415 causes the computing system 500 to perform the illustrative processes described herein. One or more processors in a multiprocessing arrangement can also be employed to execute the instructions contained in main memory 415. In conjunction with the systems and methods described herein, hardwired circuitry can be used in place of or in combination with software instructions. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0177] Although example computing systems have been described in Figure 4 , the subject matter (including the operations described in this specification) can be implemented using other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them.

[0178] Where a system discussed in this document collects personal information about a user or may make use of personal information, the user may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions, or activities; a user's preferences; or a user's location), or to control whether or how content from a content server or other data processing system may be received that may be more relevant to the user. Additionally, before storing or using certain data, the specific data can be anonymized in one or more ways so that personally identifiable information is removed when generating parameters. For example, a user's identity can be anonymized so that the user's personal identity information cannot be determined, or a user's geographic location from which location information can be derived (such as a city, postal code, or state level) can be generalized so that the user's specific location cannot be determined. Accordingly, the user can control the manner in which a content server collects and / or uses information about the user.

[0179] The subject matter and operations described in this specification can be implemented in a digital electronic circuit system, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more circuits of computer program instructions encoded on one or more computer storage media for execution by, or to control the operation of, a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus. A computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included in them. When the computer storage medium is not a propagated signal, the computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer can also be one or more separate components or media (e.g., multiple CDs, disks, or other storage devices), or included in them. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0180] The terms "data processing system", "computing device", "component", or "data processing apparatus" encompass all kinds of devices, apparatus, and machines for processing data, including, for example, programmable processors, computers, system-on-chips, or multiple or combinations of the foregoing. The apparatus can include dedicated logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer programs under discussion, e.g., code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or a combination of one or more of them. The apparatus and the execution environment can implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures. For example, the direct action API 108 or the NLP component 106 and other components of the data processing system 102 can include or share one or more data processing apparatuses, systems, computing devices, or processors.

[0181] A computer program (also referred to as a program, software, software application, application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including deployment as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored as part of a file that holds other programs or data (e.g., one or more scripts in a markup language document), or in a single file dedicated to the program under consideration, or in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0182] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs (e.g., of data processing system 102 ) to perform actions by operating on input data and generating output. The processes and logical flows can also be performed by, and the apparatus can also be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)). Apparatus suitable for storing computer program instructions and data includes all forms of non-volatile memory, media, and memory devices, including: for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0183] The subject matter described herein can be implemented in a computing system that includes a back-end component (e.g., as a data processor), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification), or a combination of such back-end, middleware, or front-end components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the World Wide Web), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0184] A computing system such as system 100 or system 400 can include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network (e.g., network 101). The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data to the client device (e.g., data packets representing digital components) (e.g., to display data to a user interacting with the client device and to receive user input from the user). Data generated at the client device (e.g., the result of a user interaction) can be received at the server (e.g., by data processing system 102 from client computing device 128 or 3P developer device 162).

[0185] Although the operations are depicted in the drawings in a particular order, such operations need not be performed in the particular order shown or in sequential order, and all illustrated operations need not be performed. The acts described herein can be performed in a different order.

[0186] The separation of the various system components is not required in all embodiments, and the described program components can be included in a single hardware or software product. For example, the NLP component 106 and the direct action API 108 can be a single component, application, or program, or a logical device having one or more processing circuits, or part of one or more servers of the data processing system 102.

[0187] Now, after having described some illustrative embodiments, it is apparent that the foregoing description is illustrative only and not restrictive, presented by way of example. Specifically, although many of the examples presented herein relate to specific combinations of method acts or system elements, these acts and these elements may be combined in other ways to achieve the same purpose. The acts, elements, and features discussed in connection with one embodiment are not intended to be excluded from other embodiments or from similar roles in other embodiments.

[0188] The terminology and phraseology used herein is for descriptive purposes and should not be regarded as limiting. The use herein of the terms including, comprising, having, containing, involving, and their variants is intended to cover the items listed thereafter, their equivalents, and other items, as well as alternative embodiments consisting exclusively of the items listed thereafter. In one embodiment, the systems and methods described herein are composed of each combination, or all, of one or more of the described elements, acts, or components.

[0189] Any reference herein to an embodiment or an element or an act of a system or method in the singular form may also embrace embodiments including a plurality of these elements, and any reference herein to any embodiment or element or act in the plural form may also embrace embodiments including only one element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to a single or multiple configurations. Any reference to an act or an element based on any information, act, or element may include embodiments in which the act or element is based, at least in part, on any information, act, or element.

[0190] Any embodiment disclosed herein may be combined with any other embodiment or example, and references to "an embodiment", "certain embodiments", "one embodiment", etc. are not necessarily mutually exclusive and are intended to indicate that the particular features, structures, or characteristics described in connection with that embodiment may be included in at least one embodiment or example. As used herein, such terms do not necessarily all refer to the same embodiment. Any embodiment may be combined inclusively or exclusively with any other embodiment in any manner consistent with the aspects and embodiments disclosed herein.

[0191] References to "or" may be construed as inclusive, such that any terms described using "or" may indicate any one of a single, more than one, and all of the described terms. References to "at least one of 'A' and 'B'" may include only 'A', only 'B', and both 'A' and 'B'. Such references used in conjunction with "comprising" or other open-ended terms may include additional items.

[0192] Reference numerals are included in the drawings, the detailed description, or any claims after a technical feature to increase the intelligibility of the drawings, the detailed description, and the claims. Accordingly, reference numerals, or their absence, have no limiting effect on the scope of any claim element.

[0193] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. The foregoing embodiments are illustrative and not restrictive of the systems and methods described. Accordingly, the scope of the systems and methods described herein is indicated by the appended claims rather than the foregoing description, and changes within the meaning and range of equivalents of the claims are included therein.

Claims

1. A system for secure digital assistant integration with a web page, comprising: A data processing system including at least one processor, configured to: Receive an intent manifest data structure from a third-party developer device, the intent manifest data structure containing a mapping between multiple actions of a digital assistant and multiple link templates of an electronic resource developed by the third-party developer device; Verify the electronic resource based on the intent manifest data structure via a verification policy; Receive an identifier of the client computing device that executes the electronic resource from a data exchange component of an inline frame of the electronic resource loaded by the client computing device; Receive a foreground state of the electronic resource from a live state sharing application programming interface; Determine parameters based on the foreground state and the intent manifest data structure; Select a data value of the parameters from a data repository based on the identifier of the client computing device; Provide the data value to an authorization component of the inline frame of the electronic resource loaded on the client computing device, so that the authorization component: Generate an authorization prompt; Receive an input from the client computing device in response to the authorization prompt; And In response to authorization of the data value, send the data value to a live intent execution application programming interface of the electronic resource, so that the electronic resource uses the data value to execute one of the multiple actions.

2. The system according to claim 1, wherein, The data exchange component executed in the inline frame of the electronic resource restricts the electronic resource in the parent frame from accessing the identifier of the client computing device, and the third-party developer device that developed the electronic resource is prohibited from accessing the identifier of the client computing device.

3. The system according to claim 1, wherein: The data processing system is configured to authorize the data exchange component to be loaded in the inline frame of the electronic resource in response to verification of the electronic resource via the verification policy.

4. The system according to claim 1, wherein: The data processing system is configured to verify the electronic resource based on a trusted site list.

5. The system according to claim 1, wherein the data processing system is configured to: Receive a request from the data exchange component executed by the client computing device; and Query the live state sharing application programming interface of the electronic resource in response to the request.

6. The system according to claim 1, wherein, The data processing system is configured to: Receive a data packet from a voice navigator and a response component executed by the client computing device, the data packet carrying an input audio signal detected by a sensor of the client computing device; Identify a request for a candidate data value from the data packet; And In response to the request, provide the data value as the candidate data value.

7. The system according to claim 1, wherein the data processing system is configured to: Provide the data value to the live intent execution application programming interface, so that the live intent execution application programming interface inputs the data value into an input text box of the electronic resource.

8. The system according to claim 1, wherein the data processing system is configured to: Determine a plurality of parameters to be used for performing the action provided by the electronic resource based on the foreground state; Select a plurality of data values corresponding to the plurality of parameters based on the identifier of the client computing device; And Provide the plurality of data values to the authorization component so that the authorization component provides the plurality of data values to the in-situ intent execution application programming interface, wherein the in-situ intent execution application programming interface is configured to use the plurality of data values to bypass one or more states used by the electronic resource to perform the action.

9. The system according to claim 1, wherein the data processing system is configured to: Determine one or more subsequent states of the electronic resource based on the foreground state and the intent list data structure; Determine one or more parameters based on the one or more subsequent states; And Select one or more data values of the one or more parameters before the electronic resource enters the one or more subsequent states based on the identifier.

10. The system according to claim 1, wherein the data processing system is configured to: Provide the data value for authorization by the authorization component and input it into the in-situ intent execution application programming interface before the electronic resource requests the data value.

11. The system according to claim 1, wherein: The data processing system is configured to provide the data value to the client computing device so that the client computing device establishes a deep link with the data value and loads the deep link in a web browser executed by the client computing device.

12. The system according to claim 1, wherein The electronic resource includes a web page.

13. The system according to any one of claims 1-12, wherein the data processing system is configured to: Establish a link with the data value based on the link template among the plurality of link templates that maps to the action among the plurality of actions; and Provide the link to the in-situ intent execution application programming interface via the data exchange component.

14. A method for integrating a secure digital assistant with a web page, comprising: Receiving, by a data processing system including at least one processor, an intent list data structure from a third-party developer device, the intent list data structure containing a mapping between a plurality of actions of a digital assistant and a plurality of link templates of an electronic resource developed by the third-party developer device; Verifying, by the data processing system, the electronic resource based on the intent list data structure via a verification policy; Receiving, by the data processing system, an identifier of the client computing device that executes the electronic resource from a data exchange component of an inline frame of the electronic resource loaded by the client computing device; Querying, by the data processing system, a live state sharing application programming interface of the electronic resource; Receiving, by the data processing system, in response to the query, a foreground state of the electronic resource from the live state sharing application programming interface; Determine parameters based on the foreground state and the intent list data structure through the data processing system; Select a data value of the parameter from a data repository based on the identifier of the client computing device through the data processing system; Provide, through the data processing system, the data value to an authorization component of an inline frame of the electronic resource loaded on the client computing device, so that the authorization component: Generate an authorization prompt; Receive an input from the client computing device in response to the authorization prompt; And In response to authorization of the data value, send the data value to a live intent execution application programming interface of the electronic resource, so that the electronic resource uses the data value to execute one of the multiple actions.

15. The method according to claim 14, wherein A data exchange component executed in the inline frame of the electronic resource restricts the electronic resource in a parent frame from accessing the identifier of the client computing device, and a third-party developer device that develops the electronic resource is prohibited from accessing the identifier of the client computing device.

16. The method according to claim 14, comprising: Authorize the data exchange component to be loaded in the inline frame of the electronic resource in response to verification of the electronic resource via the verification policy.

17. The method according to claim 14, comprising: Verify the electronic resource based on a list of trusted sites.

18. The method according to claim 14, comprising: Receive a request from the data exchange component executed by the client computing device; And Query a live state sharing application programming interface of the electronic resource in response to the request.

19. The method according to claim 14, comprising: Receive a data packet from a voice navigator and a response component executed by the client computing device, the data packet carrying an input audio signal detected by a sensor of the client computing device; Identify a request for a candidate data value from the data packet; And In response to the request, provide the data value as the candidate data value.

20. The method according to any one of claims 14 to 19, comprising: Provide the data value to the live intent execution application programming interface, so that the live intent execution application programming interface inputs the data value into an input text box of the electronic resource.

Images