Authorization of voice activation of access to additional functionality using device

Abstract

Techniques are described for granting access to an additional function requested by a user input received by a device to a device. A system may receive, from a device, a first user input requesting content of a content type requiring access by an additional function in order to send the content to the device. The system may exchange data with an access control system to determine that the device is capable of performing additional functionality. The system may prompt a user whether the additional function access should be granted. In response to receiving the determined user input, the system may cause the access control system to grant the additional function access to the device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 346,438, filed on May 27, 2022, entitled “VOICE-ACTIVATED DEVICE AUTHORIZATION TO ACCESS ADDITIONAL FUNCTIONALITY,” and U.S. Patent Application No. 17 / 852,829, filed on June 29, 2022, entitled “VOICE-ACTIVATED DEVICE AUTHORIZATION TO ACCESS ADDITIONAL FUNCTIONALITY USING ADEVICE,” the contents of each of which are expressly incorporated herein by reference in their entirety. Background Art

[0003] Natural language processing systems have been developed to the extent that humans can use their voice and natural language text input to interact with computing devices. Such systems use techniques to recognize words spoken and written by human users based on received input data of various qualities. Speech recognition combined with natural language understanding processing techniques enables voice-based user control of computing devices to perform tasks based on the user's spoken input. Speech recognition and natural language understanding processing techniques may be collectively or separately referred to herein as spoken language understanding (SLU) processing. Computers, handheld devices, telephone computer systems, kiosks, and various other devices may use SLU processing to improve human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0005] Figure 1 is a conceptual diagram illustrating a system for granting a device access to additional functionality requested by a user input received by the device according to an embodiment of the present disclosure.

[0006] Figure 2 is a conceptual diagram illustrating an example process that may be performed to grant access to additional functionality to a device according to an embodiment of the present disclosure.

[0007] Figure 3 is a conceptual diagram illustrating yet another example process that may be performed for granting access to additional functionality to a device according to an embodiment of the present disclosure.

[0008] Figure 4is a conceptual diagram illustrating yet another example process that may be performed for granting access to additional functionality to a device according to an embodiment of the present disclosure.

[0009] Figure 5 is a conceptual diagram illustrating example system components that may be used to process user input according to an embodiment of the present disclosure.

[0010] Figure 6 is a diagram of an illustrative architecture in which sensor data is combined to identify one or more users in accordance with an embodiment of the present disclosure.

[0011] Figure 7 is a system flow chart illustrating an example of a speech-based user recognition process according to an embodiment of the present disclosure.

[0012] Figure 8 is a conceptual diagram of components of a device according to an embodiment of the present disclosure.

[0013] Fig. 9 is a block diagram conceptually illustrating example components of a device according to an embodiment of the present disclosure.

[0014] Fig.10 is a block diagram conceptually illustrating example components of a system according to an embodiment of the present disclosure.

[0015] Fig.11 An example of a computer network used for the entire system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Automatic speech recognition (ASR) processing involves converting audio data including speech into tokens or other textual representations of the speech. Similarly, natural language understanding (NLU) processing involves enabling a computer to derive meaning from natural language user input (such as spoken input). ASR processing and NLU processing are often used together as part of a spoken language processing component of a system. Text-to-speech (TTS) processing involves converting text and / or other data into audio data that is synthesized to resemble human speech. Natural language generation (NLG) processing involves automatically converting data into natural language (e.g., English) content.

[0017] The system can also be configured to perform actions in response to spoken natural language user input (i.e., utterances). For example, for a spoken natural language user input of "play music by [artist name]," the system can output music performed by the indicated artist. For another example, for a spoken natural language user input of "roll down driver's window," the system can roll down the driver's window of the vehicle that captured the spoken natural language user input. In another example, for a spoken natural language user input of "what's the weather like today," the system can output synthesized speech of weather information based on the user's location.

[0018] The vehicle, and more specifically the head unit of the vehicle, may be configured to communicate with the aforementioned systems in order to process spoken natural language user input. The "head unit" of a vehicle (also sometimes referred to as an infotainment system) is a computing component of the vehicle that provides a unified hardware interface including screens, buttons, and controls for the various integrated information and entertainment functions of the vehicle.

[0019] In response to spoken natural language user input received by the vehicle, the system may need to send data to the vehicle for presentation to the user. For example, if the spoken natural language user input requests the output of music, the system may need to send audio data corresponding to the requested music and optionally image data corresponding to album cover, etc. to the vehicle for presentation. For another example, if the spoken natural language user input requests weather information, the system may need to send audio data including synthesized speech corresponding to the requested weather information and optionally image data corresponding to the requested weather information to the vehicle for presentation. In another example, if the spoken natural language user input requests traffic information, the system may need to send audio data including synthesized speech corresponding to the current traffic information of the vehicle's location and optionally image data corresponding to the traffic information to the vehicle for presentation. For another example, if the spoken natural language user input requests the output of a podcast, the system may need to send audio data corresponding to the requested podcast and optionally image data corresponding to the requested podcast to the vehicle for presentation.

[0020] Devices and systems may exchange data via an access control system. As used herein, an "access control system" refers to a system configured to receive data (e.g., audio data, text data, video data, data that can be used to update a machine learning model running on a device, data for performing over-the-air updates, diagnostic / monitoring / control data, etc.) from a source device and route the data to a target destination, and vice versa. For example, where the device is a vehicle, the access control system may be a cellular network system configured to control the exchange of data with the vehicle via cellular data transmission and the exchange of data with the system via Internet data transmission.

[0021] Before the access control system allows the additional functionality to be provided to the device via the access control system (e.g., one or more types of data to be transmitted between the device and the system via the access control system), the access control system may require that the device be authorized to access the additional functionality (e.g., to receive one or more types of data). For example, the access control system may be configured to receive certain audio data including synthesized speech from the system and send the audio data to the device without requiring authorization to access the additional functionality, but such authorization may be required if long-form audio data (e.g., corresponding to music, podcasts, videos, etc.) is to be sent to the device on behalf of the system. In some embodiments, authorization to access the additional functionality may need to be purchased for the device. Authorization to access the additional functionality from the access control system may also be obtained by the device based on a device identifier, user profile, purchase, IP address, and / or MAC address associated with the device.

[0022] Among other things, the present disclosure provides a technology for granting access to additional functions requested by a user of the device to a device (e.g., a vehicle) by an access control system. Although the examples provided herein use examples of vehicles, the techniques described herein can be applied to a variety of different device types, depending on the system configuration. As can be appreciated, a device can have the ability to obtain authorization for one or more specific functions from one or more specific access control systems, and the techniques described herein can be applied to one or many different specific functions. For example, the access control system can be a mandatory portal in a commercial residence (e.g., a hotel), where the mandatory portal can authorize access to additional functions (e.g., Internet access) of a device (e.g., a smart phone, a laptop, a smart TV, etc.) based on a device identifier, a user profile, an IP address, a MAC address, and / or a purchase associated with the device. For another example, the access control system can be a home network configured with parental control, where the home network can authorize access to additional functions (e.g., Internet access to a specific content type) of a device (e.g., a smart phone, a laptop, a smart TV, etc.) based on a device identifier, a user profile, an IP address, and / or a MAC address associated with the device. As another example, the access control system may be a tiered content provider (e.g., Amazon Prime Video, Hulu, Netflix, etc.), where the tiered content provider may authorize a device (e.g., a smart phone, laptop, smart TV, etc.) to access additional functionality (e.g., access to specific content) based on a device identifier, user profile, IP address, MAC address, and / or purchases associated with the device.

[0023] According to the present disclosure, a system may determine that a first spoken natural language user input received from a device requests an output content type. The system may determine that additional functionality access is required from an access control system in order to permit the device to output the content type. For example, the system may determine that additional functionality access is required based at least in part on a device corresponding to a particular device type (e.g., a vehicle). For another example, the system may determine that additional functionality access is required based on the type of functionality requested. The system may determine that the device currently does not have authorization from the access control system to access the additional functionality. For example, the system may determine that status data associated with the device indicates that the device has not yet received authorization from the access control system.

[0024] After determining that the device has not received authorization to access the requested additional functionality from the access control system, the system can query the access control system for information related to receiving access to the additional functionality. For example, the system can generate and send an access token corresponding to the profile data of the device (and / or the instant user) to the access control system.

[0025] The access control system may use the access token to receive profile data, and may use the profile data to determine that the device is capable of performing functions that require additional access. In response to the determination, the access control system may send data indicating that the device is capable of accessing the additional functions, and additional function access information (e.g., pricing information) for accessing the additional functions to the system. The system may cause the device to output synthesized speech and / or display one or more images indicating that additional function access is required to output the content type, the additional function access information, and a request for authorization to obtain the additional function access on behalf of the user.

[0026] After causing the aforementioned output, the system may receive a second spoken natural language user input from the device requesting that additional functionality access be granted. In response to receiving the second spoken natural language user input, the system may request additional functionality access from the access control system. Thereafter, the system may receive an indication from the access control system that additional functionality access has been granted, and may cause the device to output synthesized speech to the user and / or display one or more images to indicate that additional functionality access has been granted.

[0027] In some embodiments, the system can determine a user identifier associated with the second spoken natural language user input, determine that the user identifier is authorized to obtain additional functionality access (e.g., based on a user identifier associated with or represented in the device's profile data), and in response, can obtain the additional functionality access from the access control system.

[0028] In some embodiments, after determining that the state data indicates that the device has not been granted additional functionality access, the system can determine the number of user inputs (e.g., spoken natural language user inputs) received from the device and unable to be executed based on the device not being granted additional functionality access. In response to determining that the number of user inputs meets a condition (e.g., a threshold number of spoken natural language user inputs), the system can generate an access token that enables the access control system to access profile data of the device (and / or the instant user). Such a determination can help prevent the user from being so queried and degrading the user experience.

[0029] In some embodiments, the system may receive spoken natural language user input from a device specifically requesting that additional functionality access be granted. The system may use state data associated with the device to determine that additional functionality access has not yet been granted to the device. The system may query an access control system for additional functionality access information for the device. For example, the system may generate and send an access token corresponding to profile data for the device (and / or user) to the access control system. The system may receive additional functionality access information and an indication that the device is able to access additional functionality from the access control system. The system may cause the device to output synthesized speech and / or display one or more images to indicate additional functionality access information and a request for authorization to obtain additional functionality access on behalf of the user.

[0030] After causing the aforementioned output, the system may receive a second spoken natural language user input from the device authorizing additional functionality access to be obtained on behalf of the user. In response to receiving the second spoken natural language user input, the system may request additional functionality access from the access control system. Thereafter, the system may receive an indication from the access control system that additional functionality access has been granted, and may cause the device to output synthesized speech to the user and / or display one or more images to indicate that additional functionality access has been granted.

[0031] Among other things, the teachings of the present disclosure provide an improved user experience. For example, the present disclosure improves the user experience by permitting a user to interact with the system and authorize the system to obtain access to additional functionality for a user device (e.g., a vehicle) on behalf of the user (from an access control system). Aspects of the present disclosure minimize the number of user / system interactions required for an access control system to grant access to additional functionality to a device.

[0032] Systems according to the present disclosure may be configured to include user permissions and may only perform activities disclosed herein with user approval. Thus, the systems, devices, components, and techniques described herein are generally configured to limit processing where appropriate and to process user data only in a manner that ensures compliance with all appropriate laws, regulations, standards, and the like. Systems and techniques may be implemented on a geographic basis to ensure compliance with laws in various jurisdictions and entities in which components and / or users of the systems are located.

[0033] Figure 1 A system 100 is shown for granting access to a device for additional functionality requested by spoken natural language user input received by the device. The system 100 may include a device 110 (local to a user 105), a system 120, and an access control system 130 that communicate via a network 199 (at least a portion of which is provided by an access control system 130). Although the device 110 is shown as a vehicle, it should be understood that the present disclosure is applicable to any device that is configured to receive spoken natural language user input and for which the system 120 may require additional functionality access (from the access control system 130) to access a particular functionality. The network may include the Internet and / or any wide area network or local area network, and may include wired network hardware, wireless network hardware, satellite network hardware, and / or cellular network hardware.

[0034] System 100 may include various components. Figure 1 , system 100 may include a dispatch component 135, an orchestrator component 140, an automatic speech recognition (ASR) component 145, a natural language understanding (NLU) component 150, a state management component 155, a skill component 160, an access control skill component 165, a user identification component 170, a verification skill component 175, an event component 180, an access control component 185, a consent component 190, an identity component 195, and a verification system 197 (in communication with the verification skill component 175). Although the accompanying drawings show components in a particular arrangement, those skilled in the art will appreciate that different combinations and / or arrangements of components are possible, depending on the configuration of the system, without departing from the present disclosure. In addition, in some embodiments, a subset or all of the above components may be included in system 120.

[0035] refer to Figure 1 and Figure 2 (Notice Figure 2 Steps 1 to 9b in correspond to Figure 1 In steps 1 to 9b), device 110 (via one or more microphones or via a connection to another device having one or more microphones, such as another device wirelessly connected to the vehicle) can receive input audio of spoken natural language user input by user 105. Device 110 can generate input audio data corresponding to the input audio. Alternatively, device 110 can receive user input in the form of selecting a graphical user interface (GUI) button, a gesture, typed natural language text, etc.

[0036] Device 110 may generate input data 205 to include input audio data (or data representing other types of user input received) and one or more other data instances, such as, but not limited to, a device identifier corresponding to device 110 and / or time data corresponding to the time when spoken natural language user input was received and / or the input data was generated.

[0037] Device 110 may send input data 205 to dispatch component 135 (via network 199 and at step 1). Dispatch component 135 may be configured to send input data 205 to orchestrator component 140 (at step 2).

[0038] In the event that input data 205 includes input audio data of spoken natural language user input, orchestrator component 140 may identify input audio data 210 (included in input data 205 ) and send (at step 3 ) input audio data 210 to ASR component 145 .

[0039] The ASR component 145 is configured to process the input audio data 210 to generate ASR output data 215 corresponding to the spoken natural language user input included in the input audio data 210. Figure 5 The processing of ASR component 145 is described in detail. ASR output data 215 may include text or some other (e.g., tokenized) representation of spoken natural language user input. For example, ASR output data 215 may represent a transcription of input audio data 210. ASR component 145 may send (at step 4) ASR output data 215 to orchestrator component 140, which may send (at step 5) ASR output data 215 to NLU component 150.

[0040] Where input data 205 includes input text data corresponding to typed natural language user input, orchestrator component 140 may identify the input text data (in input data 205 ) and send the input text data to NLU component 150 .

[0041] The NLU component 150 is configured to process the ASR output data 215 (or input text data) and generate NLU output data 220. Figure 5The processing of the NLU component 150 is described in detail. The NLU output data 220 may include one or more NLU hypotheses, each representing a respective semantic interpretation of the ASR output data 215. For example, each NLU hypothesis may include an intent determined by the NLU component 150 to represent the spoken natural language user input, as represented in the ASR output data 215. Each NLU hypothesis may also include one or more entity types and corresponding entity values ​​corresponding to entities determined by the NLU component 150 to be mentioned in the spoken natural language user input, as represented in the ASR output data 215. The NLU component 150 may send (at step 6) the NLU output data 220 to the orchestrator component 140.

[0042] Alternatively, the orchestrator component 140 may send the input audio data to a spoken language understanding (SLU) component (of the system 100 ) that is configured to generate NLU output data from the input audio data without generating ASR output data as an intermediary.

[0043] Where input data 205 includes data representing selection of a GUI button, orchestrator component 140 may send such data to a GUI user input component that is configured to process the data and determine NLU output data (or other data) representing the user input and usable by downstream skill components. Where input data 205 includes data representing a gesture, orchestrator component 140 may send such data to a gesture detection component that is configured to process the data and determine NLU output data (or other data) representing the user input and usable by downstream skill components.

[0044] As previously mentioned, the system 100 may include a state management component 155. The state management component 155 is configured to maintain a record of the current state of the device 110. The device 110 may send state data 225 (e.g., via the network 199) to the dispatch component 135, which may send the state data 225 (in Figure 1 7 in step 150) is sent to the state management component 155. The state data 225 at least represents the additional function access state of the device 110.

[0045] Whenever there is a state change associated with device 110, device 110 can send state data to dispatch component 135. For example, when device 110 is powered on (e.g., the vehicle housing device 110 is ignited), a state change may occur. In this example, when device 110 is powered on, device 110 can send state data indicating that device 110 is in a powered-on state and indicating the functional access level (e.g., access to standard functions or access to additional functions) that device 110 has to dispatch component 135. As another example, a state change may occur when the functional access level of device 110 changes. In this example, the state data can indicate a new functional access status of the device. In some embodiments, device 110 can be configured to send state data (at least representing the current functional access status of device 110) as part of input data 205.

[0046] The state management component 155 may be configured to receive state data 225 (in Figure 1 7) and updating the device identifier of device 110 to be associated with the current function access state. For example, the function access state may be <connectedwithnoadditionalaccess>or <connectedwithadditionalaccess>.

[0047] Prior to or at least partially in parallel with at least one of ASR component 145 and NLU component 150 processing, orchestrator component 140 can identify a device identifier represented in input data 205 and can query state management component 155 (at step 8) for state data 225 representing at least a current function access state of device 110.

[0048] Orchestrator component 140 (or another component of system 100, such as Figure 5 9a) may determine whether NLU output data 220 includes an intent to gain access to additional functionality (or the highest scoring intent in the event that NLU output data 220 includes more than one NLU hypothesis). If orchestrator component 140 (or skill selection component 585) determines that NLU output data 220 does not include an intent to gain access to additional functionality, orchestrator component 140 may send (at step 9a) device data 230 representing device 110 (e.g., including a device identifier such as a serial number, data representing a device type of device 110 (such as a vehicle), etc.), NLU output information 220, and state information 225 to skill component 160. Alternatively, if orchestrator component 140 (or skill selection component 585) determines that NLU output data 220 includes an intent to obtain access to additional functionality (or the highest scoring intent if NLU output information 220 includes more than one NLU hypothesis), orchestrator component 140 can send (at step 9b) device data 230, NLU output signal 220, and state data 225 to access control skill component 165.

[0049] For example, if the NLU output data 220 includes an intent to play music, the skill component 160 may be a music skill component. For another example, if the NLU output data 220 includes an intent to output a podcast, the skill component 160 may be a podcast skill component. As another example, if the NLU output data 220 includes an intent to output weather information, the skill component 160 may be a weather skill. For another example, if the NLU output data 220 includes an intent to output traffic information, the skill component 160 may be a traffic skill component. As another example, if the NLU output data 220 includes an intent to make a restaurant reservation, the skill component 160 may be a restaurant reservation skill. Those skilled in the art will appreciate that the foregoing use cases are not limiting, and that other types of intents and corresponding skill components are also within the scope of the present disclosure.

[0050] As described above, the orchestrator component 140 may send the state data 225 to the skill component 160 (at step 9a) or the access control skill component 165 (at step 9b). In some embodiments, the orchestrator component 140 may send the state data 225 and the NLU output data 220 (and optionally the device data 230 and / or other data) to a policy component (not shown), which is configured to implement one or more policies for determining whether authorization to access additional functionality is required in order to respond to user input. For example, when the NLU output data 220 (or the highest scoring NLU hypothesis therein) includes a specific intent, the policy component may implement a policy indicating that the aforementioned additional functionality access is required. For another example, when the NLU output data 220 (or the highest scoring NLU hypothesis therein) includes a specific intent, the policy component may implement a policy indicating that the aforementioned additional functionality access is not required. As another example, when the NLU output data 220 (or the highest scoring NLU hypothesis therein) includes a specific entity (or entity type), the policy component may implement a policy indicating that additional functionality access is required. As another example, when the NLU output data 220 (or the highest scoring NLU hypothesis therein) includes a particular entity (or entity type), the policy component can implement a policy indicating that additional functional access is not required.

[0051] If the policy component determines that additional functionality access is required based on the NLU output data 220, the policy component can determine whether the status data 225 indicates that the additional functionality access has been granted (e.g., determine whether the status data 225 indicates that the additional functionality access has been granted). <connectedwithnoadditionalaccess>still <connectedwithadditionalaccess>The current device status.

[0052] The policy component may generate and send policy evaluation data indicating whether additional functionality access is required to orchestrator component 140. In this case, orchestrator component 140 may send the policy evaluation data (instead of state data 225) to skill component 160 (at step 9a) or access control skill component 165 (at step 9b).

[0053] Reference now Figure 1 and Figure 3 (Notice, Figure 3 Steps 10 to 22 in correspond to Figure 1 In steps 10 to 22), after the skill component 160 receives the device data 230, the NLU output data 220, and the state data 225, the skill component 160 can determine whether the NLU output information 220 includes an intent that its execution requires additional functional access.

[0054] The skill component 160 may determine whether the execution of the intent requires additional function access in various ways. In some embodiments, the skill component 160 may make this determination based at least in part on the device data 230. For example, the skill component 160 may determine that the execution of the intent requires additional function access based at least in part on the device type included in the device data 230. As a specific example, when the device data 230 indicates a vehicle device type, the skill component 160 may determine that additional function access is required. In some embodiments, when the execution of the intent involves a specific type of function (e.g., outputting a type of data), the skill component 160 may determine that additional function access is required. In some embodiments, the skill component 160 may use an authorized function list to determine whether the execution of the intent requires additional function access. For example, the authorized function list may indicate which intents the skill component 160 may execute when the additional function access of the device 110 is not active. In some embodiments, the authorized function list may indicate which intents the skill component 160 may not execute when the additional function access of the device 110 is not active. For example, when execution of the intent involves an intent to output long-form audio data (e.g., corresponding to music, podcasts, videos, etc.), skill component 160 can determine that additional functionality access is required. In some embodiments, skill component 160 can determine that additional functionality access is required based on a combination of device type and functionality / intent type (e.g., based on an intent to output long-form audio data and the device type being a vehicle).

[0055] If the skill component 160 determines, based on the device's access state, intent, and authorized function list, that the device 110 has been granted authorization to access the functionality required by the intent, the skill component 160 can execute the intent included in the NLU output data 220 (e.g., causing the long-form audio data to be sent to the device 110 for output). Alternatively, if the skill component 160 determines, based on the device's access state, intent, and authorized function list, that the device 110 has not been granted the necessary authorization to access the functionality required by the intent, the skill component 160 can send (at step 10b) the device identifier 305 (represented in the device data 230 received by the skill component 160) to the access control component 185. In some embodiments, the skill component 160 can use an application program interface (API) to call the access control component 185 to determine whether the user 105 should be queried to obtain (e.g., purchase) additional functionality access for the device 110.

[0056] In the skill component 160, the above Figure 2 In the discussed policy determination data embodiment, the skills component 160 may simply determine whether the policy determination data indicates that additional functionality access is required.

[0057] In response to receiving the device identifier 305, the access control component 185 can determine whether to prompt the user 105 to obtain additional functionality access to the device 110. In some embodiments, the state management component 155 can send (at step 10a) the state data 225 to the event component, which in turn can send (at step 11a) the state data 225 to the access control component 185. The access control component 185 can implement one or more guardrail strategies for determining whether the user 105 should be queried to obtain (i.e., purchase) additional functionality access to the device 110. This configuration of the access control component 185 can help prevent the user 105 from being queried in this way and reduce the user experience. For example, the access control component 185 can implement the following strategy: for every n user inputs requesting the execution of an intent that has not yet been granted functional authorization for the device 110, the device 110 should be caused to output a prompt to obtain additional functionality access to the device 110 no more than once. At runtime, to implement the policy, access control component 185 may determine the number of prior user inputs: (i) prior user inputs received from device 110 corresponding to device identifier 305 since the last time device 110 was caused to output a prompt to obtain additional functionality access, and (ii) prior user inputs corresponding to one or more intents that require additional functionality access that has not yet been granted to device 110. Access control component 185 may also determine whether the number of prior user inputs satisfies a condition (e.g., is equal to or greater than n user inputs). If access control component 185 determines that the number of prior user inputs satisfies the condition, access control component 185 may send (at step 11b) prompt decision data 310 to skill component 160 that will cause device 110 to output a prompt to query user 105 whether additional functionality access should be obtained. Alternatively, if the access control component 185 determines that the number of previous user inputs does not satisfy the condition, the access control component 185 may send (at step 11b) prompt decision data 310 to the skill component 160 indicating that the device 110 will not output a prompt to query the user 105 whether to obtain additional functionality access (and optionally, instead output a prompt indicating that additional functionality access is required to perform the function requested by the immediate user input).

[0058] In the event that the prompt decision data 310 indicates that the device 110 should not output a prompt asking whether additional functionality access should be obtained, processing of the immediate user input can cease when the skill component 160 receives the prompt decision data 310 (and the skill component 160 can optionally cause the device 110 to present synthesized speech and / or display one or more images to indicate that additional functionality access is required to respond to the immediate user input). In the event that the prompt decision data 310 indicates that the device 110 should output a prompt asking whether additional functionality access should be obtained, the skill component 160 can send (at step 12) the NLU output data 220, the prompt decision data 310, and the device data 230 to the access control skill component 165.

[0059] As mentioned above about Figure 2 As discussed, orchestrator component 140 may send device data 230, NLU output database 220, and state data 225 (or policy determination data) to access control skill component 165 at step 9b, instead of to skill component 160 at step 9a, depending on the intent (or highest scoring intent) in NLU output data 220. If orchestrator component 140 sends device data 230, NLU output data 220, and state data 225 (or policy determination data) to access control skill component 165 at step 9b, access control skill component 165 may determine whether state data 225 indicates that additional functionality access has been granted to device 110 (using the manner described above with respect to skill component 160). Regardless of whether access control skill component 165 receives data at step 9b or at step 12, access control skill component 165 may query access control component 185 (at step 13) for additional functionality access information associated with device identifier 305. Such a query may include the access control skills component 165 sending the device identifier 305 (of the device 110 ) to the access control component 185 .

[0060] The access control component 185 can be configured to receive the device identifier 305 and determine additional functionality access information (e.g., information corresponding to the access control system 130, pricing information, and / or other information relevant to the user 105 making an informed decision about whether the user 105 wants to authorize the access control skills component 165 to obtain additional functionality access for the device 110). In some embodiments, the additional functionality access information can be provided from the access control system 130 to the access control component 185.

[0061] In some embodiments, the access control component 185 may be configured to determine whether user input has been received from the device 110 accepting a clickwrap agreement of the access control system 130. When the access control skill component 165 receives such user input, the consent component 190 may associate a device identifier (corresponding to the device from which the user input was received) with a flag or other indicator / data representing said acceptance of the clickwrap agreement. The access control component 185 may query the consent component 190 to determine whether the device identifier 305 is associated with a flag or other indicator / data representing that the clickwrap agreement has been accepted via user input received from the device 110. The consent component 190 may determine whether the device identifier 305 is associated with such a flag or other indicator / data, and send (at step 14) consent data 315 representing the determination of the consent component 190 to the access control component 185.

[0062] If the access control component 185 determines that the consent data 315 indicates that the user has accepted the click-through consent (or other) agreement (or where such consent data is not required), the access control component 185 can send (step 15) request data 320 to the identity component 195 to generate profile token data for device profile data (of the device 110) and / or user profile data (of the user 105). The identity component 195 can be configured to retrieve the profile data from the profile storage device (described below in detail). Figure 5 The profile data may include user profile data corresponding to the user 105 and / or device profile data corresponding to the device 110.

[0063] The identity component 195 can be configured to generate archive token data 325 for accessing the archive data in a secure manner. For example, the identity component 195 can generate a JavaScript Object Notation (JSON) Web Token (JWT) including at least the archive data, and encrypt the JWT using an encryption method known in the art / industry. In some embodiments, the encryption method can be an asymmetric encryption method (e.g., RSA 256). The identity component 195 can send (at step 16) the archive token data 325 to the access control component 185.

[0064] The access control component 185 may send (at step 17) the profile token data 325 to the access control system 130. In some embodiments, the access control component 185 may also send the consent data 315 to the access control system 130. In other embodiments, the access control component 185 may send a representation of the consent data 315 to the access control system 130 (e.g., data that simply indicates whether the click-through licensing agreement has been accepted without indicating a particular device 110 or user 105).

[0065] After receiving the profile token data 325 (at step 18), the access control system 130 may send (at step 18) the profile token data 325 to the identity component 195. Thereafter, the identity component 195 may decrypt the profile token data 325 using a field / industry known decryption method to determine the corresponding profile data for the device 110 and / or user 105. The identity component 195 may send (at step 19) the profile data 327 to the access control system 130.

[0066] Access control system 130 may use profile data 327 to determine that device 110 is configured to perform a function that requires additional functionality access. For example, access control system 130 may determine that profile data 327 indicates that device 110 includes at least one hardware component capable of exchanging data with access control system 130 via cellular data transmission, such as a transmitter and receiver configured to communicate with a cellular tower. For another example, access control system 130 may use consent data 315 to determine that consent has been given to a click-through licensing agreement of access control system 130, and based thereon determine that additional functionality access may be provided to device 110.

[0067] The access control system 130 may determine additional functionality access data 330 for the device 110. The access functionality access data 330 may include various data. For example, the additional functionality access data 330 may include a monthly price, an amount of data permitted to be sent to the device 110 in a single month, and the like. As another example, the additional functionality access data 330 may indicate the additional functionality permitted by obtaining additional functionality access. The access control system 130 may send (at step 20) the additional functionality access data 330 to the access control component 185. The access control component 185 may then send (at step 21) the additional functionality access data 330 to the access control skill component 165.

[0068] The access control skill component 165 may be configured to cause generation of output audio data (including synthesized speech) representing the additional functionality access data 330. For example, and while Figure 3 1, but the access control skill component 165 can send the additional functionality access data 330 along with the name of the access control system 130 to a natural language generation (NLG) component of the system 100. The natural language generation component can generate a natural language output based on the additional functionality access data 330 and the name of the access control system 130. The text-to-speech (TTS) component of the system 100 (in Figure 5 , and described in detail below) can process the natural language output (of the natural language generation component) to generate the above-mentioned output audio data including synthesized speech. The access control skill component 165 can also cause the generation of output image data, wherein the output image data indicates the additional function access data 330 and the name of the access control system 130. As an example, the synthesized speech can be "We're sorry, but this action requires additional access rights, and your vehicle can receive [amount of data] data per month at [subscription price] per month. Do you want me to purchase this for you?" or the like. Alternatively, if the access control system 130 determines that the device 110 cannot perform the additional function, the synthesized speech can be "We're sorry, but your vehicle cannot support the requested function" or the like. Alternatively, for another example, where access control system 130 determines that device 110 is capable of supporting additional functionality access but device 110 has not yet been used to accept the required click-through licensing agreement of access control system 130, the synthesized speech may be "We're sorry, but this action requires additional access and your vehicle is eligible to receive [amount of data] data per month at [subscription price] per month. If you would like me to purchase this for you, please indicate your acceptance of the terms and conditions of access control system name." or the like.

[0069] The access control skill component 165 can cause the output audio data (and optionally the output image data) to be sent to the device 110 for presentation to the user 105. For example, the access control skill component 165 can send the output audio data (and optionally the output image data) to the dispatch component 135, which in turn can send the output audio data (and optionally the output image data) to the device 110 via the network 199.

[0070] In response to receiving the output audio data (and optionally the output image data), the device 110 may output audio corresponding to the output audio data (and optionally display one or more images corresponding to the output image data). In response thereto, the device 110 may receive audio of spoken natural language user input (or some other type of user input, such as selecting a graphical user interface (GUI) button, a gesture, etc.) from the user 105 and generate input audio data corresponding to the audio (or other type of input data corresponding to the user input).

[0071] Device 110 sends input audio data (or other input data) to dispatch component 135 for processing. For example, device 110 may send input audio data (or other input data) to dispatch component 135 via network 199.

[0072] Dispatching component 135 can send the input audio data (or other input data) to orchestrator component 140. In the event that orchestrator component 140 receives the input audio data, orchestrator component 140 can send the input audio data to ASR component 145. ASR component 145 can process (as described below with respect to Figure 5 Detailed description) input audio data to generate ASR output data corresponding to the spoken natural language user input. ASR component 145 can send the ASR output data to the arranger component 140.

[0073] The orchestrator component 140 can send the ASR output data to the NLU component 150. The NLU component 150 can process (as described below with respect to Figure 5 The NLU component 150 may transmit the NLU output data to the orchestrator component 140.

[0074] Alternatively, the orchestrator component 140 may send the input audio data to a spoken language understanding (SLU) component (of the system 100 ) that is configured to generate NLU output data from the input audio data 405 without generating ASR output data as an intermediary.

[0075] Orchestrator component 140 may send the NLU output data (or other data representing user input) to access control skills component 165 .

[0076] refer to Figure 1 and Figure 4 (Notice, Figure 4 Steps 22 to 30 in correspond to Figure 1 22 to 30 in ), access control skill component 165 may determine whether the NLU output data (or other data representing user input) indicates that user 105 has provided authorization to obtain additional functionality access to device 110. If access control skill component 165 determines that the NLU output data indicates that authorization has not been provided, access control skill component 165 may stop the process of obtaining additional functionality access. Conversely, if access control skill component 165 determines that the NLU output data indicates that authorization has been provided, access control skill component 165 may (at step 22) cause input audio data 405 (as input by the immediate user) to be sent to user identification component 170 to determine the identity of user 105. For example, access control skill component 165 may send a request for user identity data to orchestrator component 140, and orchestrator component 140 may then send input audio data 405 to user identification component 170.

[0077] The user recognition component 170 is configured to receive the input audio data 405 and determine the identity of the user 105. In general, the user recognition component 170 can process the input audio data 405 to determine the speech characteristics represented therein, and can (in a storage device of voice profiles, each voice profile corresponding to the speech characteristics of a different user) determine a user identifier 410 corresponding to a stored voice profile corresponding to a speech characteristic that is similar or identical to the speech characteristic determined in the input audio data. Figure 6 and Figure 7 To describe additional details of how the user identification component 170 can process.

[0078] User identification component 170 may (at step 23) cause user identifier 410 to be sent to access control skills component 165. For example, user identification component 170 may send user identifier 410 to orchestrator component 140, and orchestrator component 140 may send user identifier 410 to access control skills component 165.

[0079] The access control skills component 165 receives the user identifier 410 and may determine (using the user identifier 410) whether the user 105 is permitted to authorize the access control skills component 165 to cause the device 110 to obtain additional functionality access. For example, the access control skills component 165 may determine whether the user identifier 410 is associated with (or represented in) the device profile data of the device 110 (in other words, determine whether the device 110 is a registered device of the user 105). If the access control skills component 165 determines that the user identifier 410 is not associated with (or represented in) the device profile data of the device 110 (in other words, determine that the user 105 is not permitted to authorize the access control skills component 165 to cause the device 110 to obtain additional functionality access), the access control skills component 165 may stop processing regarding obtaining additional functionality access. Conversely, if the access control skill component 165 determines that the user identifier 410 is associated with (or represented in) the device profile data of the device 110 (in other words, determines that the user 105 is permitted to authorize the access control skill component 165 to enable the device 110 to obtain additional functionality access), the access control skill component 165 may determine additional functionality access request data 415 for a payment transaction for the additional functionality access. In some embodiments, the additional functionality access request data 415 may include payment information (e.g., credit card information, address, etc.) from the user profile data corresponding to the user identifier 410, as well as pricing information represented in the additional functionality access data 330 previously received by the access control skill component 165 at step 21. In some embodiments, the additional functionality access request data 415 may additionally include the user identifier 410. The access control skill component 165 may send (at step 24) the additional functionality access request data 415 to the verification skill component 175. In some embodiments, access control skill component 165 may send additional functionality access request data 415 to orchestrator component 140 , and orchestrator component 140 may send additional functionality access request data 415 to verification skill component 175 .

[0080] The verification skill component 175 is configured to receive the additional functionality access request data 415 and use the additional functionality access request data 415 to perform a payment transaction. For example, the verification skill component 175 may use the payment information (represented in the additional functionality access request data 415) to perform a payment transaction (for the price represented in the additional functionality access request data 415) with a computing system of a financial institution corresponding to the payment information (e.g., corresponding to the credit card information represented in the additional functionality access request data 415). After performing the aforementioned payment transaction (equivalent to purchasing additional functionality access), the verification skill component 175 may generate a verification confirmation access token 420 for accessing the payment confirmation data from the verification skill component 175. The verification skill component 175 may send (at step 25) the verification confirmation access token 420 to the access control skill component 165.

[0081] The access control skill component 165 may send (at step 26) the verification confirmation access token 420 to the access control component 185, which in turn may send (at step 27) the verification confirmation access token 420 to the access control system 130. The access control system 130 may send (at step 28) the verification confirmation access token 420 to the verification system 197.

[0082] In response to receiving the verification confirmation access token 420, the verification system 197 is configured to conduct a payment transaction with the access control system 130 (or a computing system of a financial institution associated with the access control system 130), wherein the verification system 197 transfers the fee of the previous payment transaction (resulting in the generation of the verification confirmation access token 420) to a payment account associated with the access control system 130. This payment transfer is equivalent to the user 105 purchasing the additional functionality access from the access control system 130. In some embodiments, the verification system 197 can communicate with the verification skills component 175. For example, the verification system 197 can send the verification confirmation access token 420 to the verification skills component 175 and receive the transfer of the fee of the previous payment transaction from the verification skills component 175. The verification system 197 can generate verification confirmation data 425 representing the aforementioned transfer of funds to the payment account of the access control system 130. The verification system 197 can send (at step 29) the verification confirmation data 425 to the access control system 130.

[0083] As a result of receiving the verification confirmation data 425, the access control system 130 may deem it permissible to send one or more types of data (e.g., long-form audio data) to the device 110. For example, the access control system 130 may deem it permissible to transfer "higher bandwidth" data between the device 110 and the system 120. As another example, the access control system 130 may deem it permissible to transfer specific types of content between the device 110 and the skills component 160 and / or different components of the system 120.

[0084] In some embodiments, access control system 130 may generate authorization data indicating that additional functionality access has been granted to device 110, and may send the authorization data to access control component 185, which may send the authorization data to access control skills component 165. In some embodiments, in response to receiving the authorization data, access control skills component 165 may cause device 110 to output audio and / or display text and / or one or more images indicating that additional functionality access has been granted.

[0085] In some embodiments, after sending the verification confirmation access token 420 to the access control component 185 (and optionally after receiving the authorization data described above), the access control skill component 165 can send (at step 30) authorization data 430 to the skill component 160, wherein the authorization data 430 indicates that additional functionality access has been granted to the device 110. In response to receiving the authorization data 430, the skill component 160 can cause output data (corresponding to the spoken natural language user input received at step 1) to be sent to the device 110 for presentation to the user 105.

[0086] In some embodiments, the access control skill component 165 may not send the authorization data 430 to the skill component 160. Instead, at some point after the access control system 130 receives the verification confirmation data 425 at step 29, the access control system 130 may cause the state data of the device 110 (in the state management component 155) to indicate that the additional functionality access of the device 110 is active, and may cause the device 110 to output synthesized speech (generated using NLG and TTS processing) and / or display one or more images to indicate that the additional functionality access has been granted. Thus, if the device 110 thereafter again receives the spoken natural language user input previously received at step 1, the skill component 160 (or the policy component described herein) will determine that the additional functionality access of the device 110 is active and cause the output data to be presented.

[0087] Although the access control skill component 165 has been described as causing additional functionality access to be granted after a single prompt is output to the user 105, in some embodiments, the access control skill component 165 may output multiple prompts before causing additional functionality access to be granted. For example, in response to the access control skill component 165 receiving the prompt decision data 310 at step 12, the access control skill component 165 may cause the device 110 to output audio (including synthesized speech) and / or display one or more images to indicate that additional functionality access is desired in response to the spoken natural language user input received at step 1, and query the user 105 whether the user 105 wants to receive information about the additional functionality access. If the user responds affirmatively (e.g., via spoken natural language user input), the access control skill component 165 may then process as described above to cause the additional functionality access data 330 to be granted by the access control system 130. The access control skill component 165 may then cause the device 110 to output audio (including synthesized speech) and / or display one or more images to indicate the additional functionality access data 330 and query the user 105 whether the user 105 wants to grant the additional functionality access to the device 110. Thereafter, the system 100 may be configured as described above with respect to Figure 4 Process as described.

[0088] In some cases, the user 105 may no longer own the device 110 (e.g., because the device 110 is sold, lost, etc.), and may want to cancel the above-mentioned additional function access granted to the device 110. In such a case, the user 105 may provide a user input indicating that the additional function access should be terminated (e.g., a touch input via a touch screen, a spoken natural language input, etc.). The system 120 may process the user input (using ASR processing, NLU processing, SLU processing, touch input processing, etc.) to determine that the user input requests to terminate the additional function access. Therefore, the system 120 may send the NLU output data (or other data representing the user input) to the access control skill component 165. In response to receiving the NLU output data (or other data representing the user input), the access control skill component 165 may send a command to terminate the additional function access of the device 110 to the access control component 185. Such a command may include a device identifier of the device 110. In response to receiving the command, the access control component 185 may send a command to terminate the additional function access of the device 110 to the access control system 130. Such a command may include a device identifier of the device 110. In response to receiving a command from the access control component 185, the access control system 130 may no longer consider one or more types of data (e.g., long-form audio data) to be permissible to be sent to the device 110, may no longer consider "higher bandwidth" data transfer between the device 110 and the system 120 to be permissible, may no longer consider transfer of certain types of content between the device 110 and the skill component 160 and / or different components of the system 120 to be permissible, etc. Additionally, in response to receiving user input requesting termination of additional functionality access, the access control skill component 165 and / or the access control component 185 may cause the state management component 155 to update the state data of the device 110 to indicate that additional functionality access for the device 110 is no longer active.

[0089] Reference now Figure 5 , the following describes example components of a system 100 that can be used to process user input. A user 105 can speak input, and a device 110 can receive audio 111 representing the spoken user input. For example, the user 105 can say "Alexa, what's the weather like" or "Alexa, book me a flight to Seattle." In other examples, the user 105 can provide another type of input (e.g., selecting a button, selecting one or more displayed graphical interface elements, performing a gesture, etc.). The device 110 can send the input data to the system 120 for processing. In examples where the user input is spoken user input, the input data can be audio data 511. In other examples, the input data can be text data or image data.

[0090] In the example of spoken user input, a microphone or microphone array (of or associated with the device 110) can continuously capture audio 11, and the device 110 can continuously process the audio data representing audio 11 when continuously capturing audio, to determine whether voice is detected. Equipment 110 can use various techniques to determine whether audio data includes voice. In some examples, equipment 110 can apply voice activity detection (VAD) technology. Such technology can determine whether there is voice in audio data based on various quantitative aspects of audio data, such as the spectrum slope between one or more frames of audio data, the energy level of audio data in one or more spectral bands, the signal-to-noise ratio of audio data in one or more spectral bands or other quantitative aspects. In other examples, equipment 110 can implement a classifier configured to distinguish voice from background noise. The classifier can be implemented by techniques such as linear classifiers, support vector machines and decision trees. In other examples, device 110 may apply Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques to compare the audio data with one or more acoustic models in memory, which may include models corresponding to speech, noise (e.g., ambient noise or background noise), or silence. Other techniques may also be used to determine whether speech is present in the audio data.

[0091] Once speech is detected in the audio data representing audio 11, device 110 can determine whether the speech is directed to device 110. In some embodiments, a wake-up word detection component can be used to make such a determination. The wake-up word detection component can be configured to detect various wake-up words. In at least some examples, each wake-up word can correspond to the name of a different digital assistant. An example wake-up word / digital assistant name is "Alexa".

[0092] Wake-up word detection is typically performed without language analysis, text analysis, or semantic analysis. Instead, audio data representing audio 11 is analyzed to determine whether specific characteristics of the audio data match a pre-configured acoustic waveform, audio signature, or other data corresponding to a wake-up word.

[0093] Therefore, the wake-up word detection component can compare the audio data with the stored data to detect the wake-up word. One method of wake-up word detection applies a general large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal while performing a wake-up word search in the resulting lattice or confusion network. Another method of wake-up word detection is to establish HMMs for each wake-up word and non-wake-up word speech signal respectively. Non-wake-up word speech includes other spoken words, background noise, etc. One or more HMMs can be established to simulate non-wake-up word speech characteristics, which are called filling models. Viterbi decoding is used to search for the best path in the decoding graph, and the decoded output is further processed to determine the presence of the wake-up word. By combining a hybrid DNN-HMM decoding framework, this method can be extended to include identification information. In another example, the wake-up word detection component 540 can be directly built on a deep neural network (DNN) / recursive neural network (RNN) structure without involving HMM. This architecture can estimate the posterior value of the wake-up word using context data by stacking frames within the context window of the DNN or using RNN. Subsequent posterior threshold adjustment or smoothing is applied to decision making. Other techniques for wake word detection, such as those known in the art, may also be used.

[0094] Once the wake word detection component detects the wake word, the device 110 may “wake up” and send input audio data 511 representing spoken user input to the system 120 .

[0095] System 120 may include an orchestrator component 140 configured to, among other things, coordinate data transfers between components of system 120. Orchestrator component 140 may receive audio data 511 from device 110 and send audio data 511 to ASR component 145.

[0096] The ASR component 145 transcribes the audio data 511 into ASR output data that includes one or more ASR hypotheses. The ASR hypotheses may be configured as textual interpretations of the speech in the audio data 511, or may be configured in another manner, such as one or more tokens. Each ASR hypothesis may represent a different possible interpretation of the speech in the audio data 511. Each ASR hypothesis may be associated with a score (e.g., a confidence score, a probability score, etc.) that indicates that the associated ASR hypothesis correctly represents the speech in the audio data 511.

[0097] The ASR component 145 interprets the speech in the audio data 511 based on the similarity between the audio data 511 and the pre-established language model. For example, the ASR component 145 can compare the audio data 511 with the sound model (e.g., sub-word units such as phonemes, etc.) and the sound sequence to identify words that match the sound sequence of the speech represented in the audio data 511.

[0098] In at least some cases, instead of device 110 receiving spoken natural language user input, device 110 may receive textual (e.g., typed) natural language user input. Device 110 may determine textual data representing the textual natural language user input, and may send the textual data to system 120, where the textual data is received by orchestrator component 140. Orchestrator component 140 may send the textual data or ASR output data to NLU component 150 depending on the type of natural language user input received.

[0099] The NLU component 150 processes the ASR output data or text data to determine one or more NLU hypotheses embodied in the NLU output data. The NLU component 150 may perform intent classification (IC) processing on the ASR output data or text data to determine the intent of the natural language user input. The intent corresponds to an action to be performed in response to the natural language user input. In order to perform IC processing, the NLU component 150 may communicate with a database of words linked to the intent. For example, a music intent database may link words and phrases such as "quiet", "volume off", and "mute" to the <mute> intent. The NLU component 150 identifies intent by comparing words and phrases in the ASR output data or text data with words or phrases in the intent database. In some embodiments, the NLU component 150 may communicate with multiple intent databases, each of which corresponds to one or more intents associated with a particular skill.

[0100] For example, IC processing of a natural language user input "play my workout playlist" may determine the intent of <play music>. For another example, IC processing of a natural language user input "call mom" may determine the intent of <call>. In another example, IC processing of a natural language user input "call mom using video" may determine the intent of <video call>. In yet another example, IC processing of a natural language user input "how's the weather today" may determine the intent of <output weather>.

[0101] The NLU component 150 can also perform named entity recognition (NER) processing on the ASR output data or text data to determine one or more portions, sometimes referred to as slots, of the natural language user input that may be required for NLU post-processing (e.g., processing performed by a skill). For example, NER processing of the natural language user input "play [song name]" can determine an entity type of "song name" and an entity value corresponding to the indicated song name. For another example, NER processing of the natural language user input "call mom" can determine an entity type of "recipient" and an entity value corresponding to "mom". In another example, NER processing of the natural language user input "what's the weather like today" can determine an entity type of "date" and an entity value of "today".

[0102] In at least some embodiments, the intents recognizable by the NLU component 150 can be linked to one or more grammatical frameworks, in which entity types are populated with entity values. Each entity type of the grammatical framework corresponds to a portion of the ASR output data or text data that the NLU component 150 deems to correspond to an entity value. For example, a grammatical framework corresponding to the <play music> intent can correspond to a sentence structure such as "play {artist name}", "play {album name}", "play {song name}", "play {song name} by {artist name}", etc.

[0103] For example, the NLU component 150 may perform NER processing to identify words in the ASR output data or text data as subjects, objects, verbs, prepositions, etc. based on grammatical rules and / or models. The NLU component 150 may then perform IC processing using the identified verbs to identify intent. Thereafter, the NLU component 150 may perform NER processing again to determine a grammatical model associated with the identified intent. For example, the grammatical model for the <play music> intent may specify a list of entity types such as {artist name}, {album name}, {song name}, etc. that are applicable to playing the identified "object" and any object modifiers (e.g., prepositional phrases). The NER processing may then involve searching for corresponding fields in the dictionary, attempting to match the words and phrases in the ASR output data that the NER processing previously marked as grammatical objects or object modifiers with those words and phrases identified in the dictionary.

[0104] NER processing may include semantic tagging, i.e., tagging words or word combinations according to their type / semantics. NER processing may include parsing ASR output data or text data using heuristic grammar rules, or may build models using techniques such as hidden Markov models, maximum entropy models, log-linear models, and conditional random fields (CRFs). For example, NER processing for music skills may include parsing and tagging ASR output data or text data corresponding to "play Mother's Little Helper sung by the Rolling Stones" as {verb}: "sing", {object}: "mother's little helper", {object preposition}: "by", and {object modifier}: "rolling stones". NER processing may identify "play" as a verb based on a database of words associated with music skills, and the IC component determines that it corresponds to the intent of <play music>.

[0105] The NLU component 150 can generate NLU output data including one or more NLU hypotheses, wherein each NLU hypothesis includes an intent and optionally one or more entity types and corresponding entity values. In some embodiments, the NLU component 150 can perform IC processing and NER processing for different skills. A skill can support the same or different intents as another skill. Therefore, the NLU output data may include multiple NLU hypotheses, wherein each NLU hypothesis corresponds to IC processing and NER processing performed on ASR output or text data for different skills.

[0106] Skill primaries component 565 is configured to determine a subset of skill components implemented by or in communication with system 120 that can perform an action in response to a (spoken) user input. Without skill primaries component 565, NLU component 150 may process ASR output data input thereto for each skill component of system 120 or in communication with the system. By implementing skill primaries component 565, NLU component 150 may process ASR output data only for skill components that skill primaries module 565 determines are likely to perform in response to user input. This reduces the overall computing power and latency due to NLU processing.

[0107] The skill preselection component 565 may include one or more ML models. The ML models may be trained to recognize various forms of user input that may be received by the system 120. For example, during a training period, a skill component developer may provide training data representing sample user input that may be provided by a user to invoke a skill component. For example, for a ride-sharing skill component, the skill component developer may provide training data corresponding to "Call me a taxi to [location]", "Get me a ride to [location]", "Book me a taxi to [location]", "Book me a car to [location]", etc.

[0108] System 120 can use sample user input provided by skill component developers to determine other potentially relevant user input structures that a user can try to invoke a particular skill component. These potentially relevant user input structures can also be used to train ML models. During training, skill component developers can be queried as to whether the determined other user input structures are permissible for invoking a skill component from the perspective of the skill component developer. Potentially relevant user input structures can be derived by one or more ML models and can be based on user input schemas provided by different skill component developers.

[0109] Skill component developers can also provide training data that indicates grammar and annotations.

[0110] Each ML model of the skill preselection component 565 can be trained for a different skill component. Alternatively, the skill preselection component 565 can implement one ML model per domain, such as one ML model for skill components associated with the weather domain, one ML model for skill components associated with the ride-sharing domain, and the like.

[0111] The sample user inputs provided by the skill component developer and the potentially relevant sample user inputs determined by the system 120 can be used as binary examples to train the ML model associated with the skill component. For example, some sample user inputs can be positive examples (e.g., user inputs that can be used to invoke the skill component). Other sample user inputs can be negative examples (e.g., user inputs that can not be used to invoke the skill component).

[0112] As described above, the skill preliminary selection component 565 may include a different ML model for each skill component, a different ML model for each domain, or some other combination of ML models. In some embodiments, the skill preliminary selection component 565 may alternatively include a single ML model. The ML model may include a portion trained on features (e.g., semantic features) shared by all skill components. The ML model may also include a skill component specific portion, where each skill component specific portion is trained with respect to a specific skill component. Implementing a single ML model with a skill component specific portion can result in less waiting time than implementing a different ML model for each skill component, because a single ML model with a skill component specific portion limits the number of features processed at each skill component level.

[0113] The parts trained for characteristics common to more than one skill components may be clustered based on domains. For example, the first part of the parts trained for multiple skill components may be trained for weather domain skill components; the second part of the parts trained for multiple skill components may be trained for music domain skill components; the third part of the parts trained for multiple skill components may be trained for tourism domain skill components, etc.

[0114] The skill priming component 565 can make a binary (e.g., yes or no) determination as to which skill components are relevant to the ASR output data. The skill priming component 565 can use one or more ML models described above to make such a determination. If the skill priming component 565 implements a different ML model for each skill component, the skill priming component 565 can run the ML model associated with the enabled skill component, as indicated in the user profile associated with the device 110 and / or the user 105.

[0115] The skill preliminary selection component 565 can generate an n-best list of skill components that can be executed with respect to the user input represented in the ASR output data. The size of the n-best list of skill components is configurable. In an example, the n-best list of skill components can indicate each skill component of the system 120 or in communication with the system, and for each skill component, contains an indication indicating whether the skill component is likely to execute the user input represented in the ASR output data. In another example, instead of indicating each skill component, the n-best list of skill components can indicate only the skill components that are likely to execute the user input represented in the ASR output data. In yet another example, the skill preliminary selection component 565 can implement thresholding so that the n-best list of skill components can indicate no more than a maximum number of skill components. In another example, the skill components included in the n-best list of skill components can be limited by a threshold score, wherein only skill components above a certain score associated with the likelihood of handling the user input are included in the n-best list of skill components.

[0116] The ASR output data may correspond to more than one ASR hypothesis. When this occurs, the skill preselection component 565 may output a different n-best list of skill components for each ASR hypothesis. Alternatively, the skill preselection component 565 may output a single n-best list of skill components that represents the skill components associated with the multiple ASR hypotheses represented in the ASR output data.

[0117] As indicated above, the skill preselection component 565 can implement thresholding such that the n-best list of skill components output therefrom can include no more than a threshold number of entries. If the ASR output data includes more than one ASR hypothesis, the n-best list of skill components can include no more than a threshold number of entries, regardless of the number of ASR hypotheses output by the ASR component 145. Additionally or alternatively, the n-best list of skill components can include no more than a threshold number of entries per ASR hypothesis (e.g., no more than five entries for the first ASR hypothesis, no more than five entries for the second ASR hypothesis, etc.).

[0118] In addition or in lieu of making a binary determination as to whether a skill component is potentially relevant to the ASR output data, the skill preselection component 565 can also generate a confidence score representing the likelihood that the skill component is relevant to the ASR output data. The skill preselection component 565 can perform matrix-vector modifications to obtain confidence scores for all skill components in a single instance of processing the ASR output data.

[0119] The n-best list of skill components including confidence scores that may be output by the skill shortlist component 565 may be represented, for example:

[0120] Story Skill Component, 0.67

[0121] Recipe Skill Component, 0.62

[0122] Information Skill Component, 0.57

[0123] Shopping Skill Component, 0.42

[0124] As indicated, the confidence score output by the skill shortlisting component 565 can be a numeric value. Alternatively, the confidence score output by the skill shortlisting component 565 can be a binned value (e.g., high, medium, low).

[0125] The n-best list of skill components may include only entries for skill components whose confidence scores meet (e.g., equal or exceed) a minimum threshold confidence score. Alternatively, the skill shortlist component 565 may include entries for all skill components associated with the current user's enabled skill components, even if one or more of the skill components are associated with a confidence score that does not meet the minimum threshold confidence score.

[0126] In determining which skill components may be associated with the user input represented in the ASR output data and the corresponding confidence scores, the skill preliminary selection component XAA65 may consider other data. The other data may include usage history data, data indicating skill components enabled relative to the device 110 and / or the user 105, data indicating the device type of the device 110, data indicating the speed of the device 110, the location of the device 110, data indicating the skill component being used to output content via the device 110 when the device 110 receives the immediate user input, etc.

[0127] The thresholding implemented for the n-best list of skill components generated by the skill shortlist component 565, as well as the different types of other data considered by the skill shortlist component 565, is configurable.

[0128] As described above, system 120 can perform speech processing using two different components (e.g., ASR component 145 and NLU component 150). In at least some embodiments, system 120 can implement a spoken language understanding (SLU) component 547 configured to process audio data 511 to determine NLU output data.

[0129] The SLU component 547 can be equivalent to a combination of the ASR component 145 and the NLU component 150. However, the SLU component 547 can process the audio data 511 and directly determine the NLU output data without the intermediate step of generating the ASR output data. Therefore, the SLU component 547 can obtain the audio data 511 representing the spoken natural language user input and attempt to interpret the spoken natural language user input semantically. That is, the SLU component 547 can determine the meaning associated with the spoken natural language user input and then implement the meaning. For example, the SLU component 547 can interpret the audio data 511 representing the spoken natural language user input to derive the desired action. The SLU component 547 can output the most likely NLU hypothesis, or multiple NLU hypotheses associated with corresponding confidence or other scores (such as probability scores, etc.).

[0130] System 120 may include a gesture detection component ( Figure 5 ). System 120 may receive image data representing a gesture, and a gesture detection component may process the image data to determine the gesture represented therein. The gesture detection component may implement a gesture detection process known in the field / industry.

[0131] In embodiments where system 120 receives non-image data (eg, textual data) representing a gesture, orchestrator component 140 may be configured to determine what downstream processing to perform in response to the gesture.

[0132] The system may include a skill selection component 585 configured to determine a skill component or an n-best list of skill components each associated with a confidence score / value to execute in response to a user input. The skill selection component 585 may include a skill component suggestion component, a skill component pre-response component, and a skill component ranking component.

[0133] The skill component suggestion component is configured to determine a skill component that can be processed in response to the user input. In addition to receiving the NLU output data, the skill component suggestion component can also receive contextual data corresponding to the user input. For example, the contextual data can indicate a skill component that causes the device 110 to output content (e.g., music, video, synthesized speech, etc.) when the device 110 captures the user input, one or more skill components indicated as enabled in a profile associated with the user 105 (such as stored in the profile storage device 570), the output capabilities of the device 110, the geographic location of the device 110, and / or other contextual data corresponding to the user input.

[0134] The skill component suggestion component can implement skill component suggestion rules. A skill component developer can provide one or more rules via a skill component developer device that indicate when a skill component should be invoked to respond to user input. In some embodiments, such rules can be specific to intents. In such embodiments, if a skill component is configured to execute for multiple intents, a skill component can be associated with more than one rule (e.g., each rule corresponds to a different intent that the skill component can handle). In addition to being specific to intents, rules can also indicate one or more entity identifiers regarding which skill components should be invoked. For another example, a rule can indicate the output capabilities, geographic location, and / or other conditions of a device.

[0135] Each skill component can be associated with each rule that corresponds to the skill component. For example, a rule may indicate that a video skill component can execute when the user input corresponds to a "play video" intent and the device includes a display or is otherwise associated with a display. As another example, a rule may indicate that a music skill component can execute when the user input corresponds to a "play music" intent and the device is outputting music when the device captures the user input. It should be understood that other examples are possible. In a system where multiple skill components are configured to execute for the same intent, the aforementioned rules enable skill components to be suggested differently at runtime based on various conditions.

[0136] The skill component suggestion component uses the NLU output data, the received context data, and the aforementioned skill component suggestion rules to determine the skill component that is configured to process in response to the user input. Thus, in some embodiments, the skill component suggestion component can be implemented as a rule engine. In some embodiments, the skill component suggestion component can make a binary (e.g., yes / no, true / false, etc.) determination as to whether the skill component is configured to process in response to the user input. For example, if the skill component is associated with a rule corresponding to the intent represented in the NLU output data and the context data, the skill component suggestion component can determine that the skill component is configured to process in response to the user input.

[0137] In some embodiments, the skill component suggestion component can make such a binary determination with respect to all skill components. In some embodiments, the skill component suggestion component can make a binary determination with respect to only some skill components (e.g., only skill components indicated as enabled in the user profile of user 105).

[0138] After the skill component suggestion component completes processing, the skill component pre-response component may be called for execution. The skill component pre-response component is configured to query the skill component determined by the skill component suggestion component as configured to process the user input whether the skill component is actually capable of responding to the user input. The skill component pre-response component may take as input NLU output data including one or more NLU hypotheses, wherein each of the one or more NLU hypotheses is associated with a specific skill component determined by the skill component suggestion component as configured to respond to the user input.

[0139] The skill component pre-response component sends a pre-response query to each skill component determined by the skill component suggestion component. The pre-response query may include an NLU hypothesis associated with the skill component and optionally other contextual data corresponding to the user input.

[0140] The skill component can determine whether the skill component can respond to the user input based on the received pre-response query and optionally other data available to the skill component. For example, the skill component can generate a pre-response indicating that the skill component can respond to the user input, indicating that the skill component needs more data to determine whether the skill component can respond to the user output, or indicating that the skill component cannot respond to the user input.

[0141] Where the pre-response of a skill component indicates that the skill component can respond to the user input or indicates that the skill component requires more information, the pre-response of the skill component may also include various other data indicating the strength of the potential response of the skill component to the user input. Such other data may positively influence the ranking of the skill component by the skill component ranking component of the skill selection component 585. For example, such other data may indicate: the capabilities of the device to be used to output the response of the skill component (e.g., output capabilities or components, such as a connected screen, speaker, etc.); pricing data corresponding to the product or service for which the user input is requesting to purchase or requesting information; the availability of the product for which the user input is requesting to purchase; whether there is a shipping fee for the product for which the user input is requesting to purchase; whether the user 105 already has a profile and / or subscription to the skill component; the user 105 does not subscribe to the skill component, but the skill component offers a free trial / free tier; with respect to the taxi skill component, the cost of the trip based on the origin and destination locations, how long the user 105 needs to wait to be picked up, etc.; and / or other data available to the skill component related to the processing of the user input by the skill component. In some embodiments, the pre-response of the skill component may include an indicator (e.g., a flag indicating the strength of the skill component's ability to personalize its response to the user input).

[0142] In some embodiments, the pre-responses of skill components can be configured to predefined patterns. By requiring pre-responses to conform to a particular pattern (e.g., by requiring skill components to only be able to provide certain types of data in pre-responses), new skill components can be added to the skill component selection function without having to reconfigure the skill selection component 585 each time a new skill component is added. In addition, requiring pre-responses to conform to a pattern limits the amount of value required to train and implement an ML model to rank skill components.

[0143] In some embodiments, the skill component's pre-response may indicate whether the skill component requests exclusive display access (ie, whether the skill component requests that its visual data be presented on the entire display).

[0144] After the skill component pre-response component queries the skill component for pre-response, the skill component ranking component may be called for execution. The skill component ranking component may be configured to select a single skill component from the skill components determined by the skill component suggestion component to respond to the user input. In some embodiments, the skill component ranking component may implement an ML model. In some embodiments, the ML model may be a deep neural network (DNN).

[0145] The skill component ranking component may take as input the NLU output data, skill component pre-responses, one or more skill component preferences of the user 105 (e.g., as represented in a user profile or group profile stored in the profile storage 570), an NLU confidence score for the NLU output data, a device type of the device 110, data indicating whether the device 110 was outputting content when the user input was received, and / or other contextual data available to the skill component ranking component.

[0146] The skill component ranking component ranks the skill components using the ML model. Things that can improve the ranking of the skill component include, for example, the skill component is associated with a pre-response indicating that the skill component can generate a personalized response for the user 105, the NLU hypothesis corresponding to the skill component is associated with an NLU confidence score (e.g., a threshold NLU confidence score) that satisfies the condition that the skill component is outputting content via the device 110 when the device 110 receives user input, etc. Things that can reduce the ranking of the skill component include, for example, the skill component is associated with a pre-response indicating that the skill component cannot generate a personalized response for the user 105, and the NLU hypothesis corresponding to the skill component is associated with an NLU confidence score (e.g., a threshold NLU confidence score, etc.) that fails to meet the condition.

[0147] The skill component ranking component can generate a score for each skill component determined by the skill component suggestion component, wherein the score represents the strength with which the skill component ranking component recommends executing the associated skill component in response to the user input. Such a confidence score can be a numerical score (e.g., between 0 and 1) or a binned score (e.g., low, medium, high).

[0148] The system 120 may include or otherwise communicate with one or more skill components 160. The skill component 160 may process the NLU output data and perform one or more actions in response thereto. For example, for NLU output data including the <play music> intent, the "artist" entity type, and the artist's name as an entity value, the music skill component may output music sung by the indicated artist. For another example, for NLU output data including the <turn on> intent, the "device" entity type, and the entity value of "light", the smart home skill component may cause one or more "smart" lights to operate in the "on" state. In another example, for NLU output data including the <output weather> intent, the "location" entity type, and the entity value corresponding to the geographic location of the device 110, the weather skill component may output weather information for the geographic location. For another example, for NLU output data including the <book a ride> intent, the taxi skill component may book the requested ride. In another example, for NLU output data including the <buy pizza> intent, the restaurant skill component may order a pizza. In another example, for NLU output data including an <output story> intent and an entity type of “title” and an entity value corresponding to the title, the story skill component may output a story corresponding to the title.

[0149] The skill component can cooperate between the device 110 / system 120 and other devices (such as a restaurant electronic ordering system, a taxi electronic reservation system, etc.) to complete certain functions. The input to the skill component can come from voice processing interaction or through other interaction or input sources.

[0150] Skill components can be associated with domains, a non-limiting list of which includes a smart home domain, a music domain, a video domain, a weather domain, a communications domain, a flash briefing domain, a shopping domain, and a custom domain.

[0151] The skill component 160 may process to determine output data in response to spoken user input (eg, based on intent and entity data represented in NLU output data received by the skill component 160 ).

[0152] The system 120 may include a TTS component 580 that generates audio data including synthesized speech. The TTS component 580 is configured to generate output audio data including synthesized speech. The TTS component 580 may use one or more different methods to perform speech synthesis. In a synthesis method called unit selection, the TTS component 580 matches a database of recorded speech with data input to the TTS component 580. The TTS component 580 matches the input data with spoken audio units in the database. Select matching units and connect them together to form speech output. Each unit includes an audio waveform corresponding to a phonetic unit (such as a short .wav file of a specific sound) and a description of various acoustic features associated with the .wav file (such as its pitch, energy, etc.) and other information (such as the position of occurrence of the phonetic unit in a word, sentence or phrase, adjacent phonetic units, etc.). Using all the information in the unit database, the TTS component 580 can match the unit with the input data to create a natural sounding waveform. The unit database may include multiple examples of phonetic units to provide the TTS component 580 with many different options for connecting units into speech. One benefit of unit selection is that, depending on the size of the database, natural-sounding speech output can be generated. The larger the unit database, the more likely the TTS component 580 is to construct natural-sounding speech.

[0153] Unit selection speech synthesis can be performed as follows. Unit selection includes a two-step process. First, the TTS component 580 determines what speech units to use, and then it combines these speech units so that the specific combined unit matches the desired phonemes and acoustic features to create the desired speech output. Units can be selected based on a cost function, which represents the degree of matching of a specific unit with the speech segment to be synthesized. The cost function can represent a combination of different costs, and different costs represent different aspects of the working effect of a specific speech unit for a specific speech segment. For example, the target cost indicates the degree of matching of a given speech unit with the characteristics of the desired speech output (e.g., tone, prosody, etc.). The connection cost represents the degree of matching of a speech unit with a continuous speech unit so that the speech units are connected together in the final synthesized speech. The total cost function is a combination of the target cost, the connection cost, and other costs that can be determined by the TTS component 580.

[0154] In another synthesis method known as parametric synthesis, parameters such as frequency, volume, noise, etc. are varied by the TTS component 580 to create an artificial speech waveform output. Parametric synthesis can use acoustic models and various statistical techniques to match the data input to the TTS component 580 with the desired output speech parameters. Parametric synthesis can include the ability to ensure accuracy at high processing speeds, and the ability to process speech without a large database associated with unit selection, but also typically produces an output speech quality that may not match the output speech quality of the unit selection. Unit selection and parameter techniques can be performed separately or combined together and / or combined with other synthesis techniques to produce speech audio output.

[0155] Parametric speech synthesis can be performed as follows. The TTS component 580 can include an acoustic model or other model that can convert data input to the TTS component 580 into a synthesized acoustic waveform based on audio signal manipulation. The acoustic model includes rules that can be used to assign specific audio waveform parameters to input phoneme units and / or prosodic annotations. These rules can be used to calculate a score representing the likelihood that a specific audio output parameter (such as frequency, volume, etc.) corresponds to a portion of the input data.

[0156] The TTS component 580 can use a variety of techniques to match the speech to be synthesized with the input phoneme units and / or prosodic annotations. A common technique is to use a hidden Markov model (HMM). HMM can be used to determine the probability that the audio output should match the text input. HMM can be used to convert parameters from the language and acoustic space into parameters used by a vocoder (i.e., a digital speech encoder) to artificially synthesize the desired speech. Using HMM, multiple states are presented, where these states together represent one or more potential acoustic parameters to be output to the vocoder, and each state is associated with a model such as a Gaussian mixture model. The transition between states can also have an associated probability, indicating the possibility of reaching the current state from the previous state. The sound to be output can be represented as a path between HMM states, and multiple paths can represent multiple possible audio matches of the same input text. Each part of the text can be represented by multiple potential states corresponding to different known pronunciations of phonemes and their parts (such as phoneme identity, stress, accent, position, etc.). The initial determination of the probability of a potential pronunciation can be associated with a state. When the TTS component 580 processes new text, the state can change or remain unchanged based on the processing of the new text. For example, the pronunciation of previously processed words may be changed based on the words processed subsequently. The Viterbi algorithm can be used to find the most likely state sequence based on the processed text. The HMM can generate speech in parameterized form, including parameters such as fundamental frequency (f0), noise envelope, spectrum envelope, etc., which are translated into audio clips by the vocoder. Output parameters can be configured for a specific vocoder, such as a STRAIGHT vocoder, a TANDEM-STRAIGHT vocoder, a vocoder based on HNM (harmonics plus noise), a CELP (code excited linear prediction) vocoder, a GlottHMM vocoder, an HSM (harmonic / stochastic model) vocoder, or other vocoders.

[0157] In addition to calculating the potential state of an audio waveform as a potential match for a phoneme unit, the TTS component 580 can also calculate the potential states of other potential audio outputs (such as various pronunciations of the phoneme / E / ) as potential acoustic matches for the phoneme unit. In this way, multiple states and state transition probabilities can be calculated.

[0158] The possible states and possible state transitions calculated by the TTS component 580 may generate a plurality of potential audio output sequences. Based on the acoustic model and other potential models, the potential audio output sequences may be scored according to a confidence level of the TTS component 580. The highest scoring audio output sequence including the parameter stream to be synthesized may be selected, and digital signal processing may be performed by a vocoder or similar component to create an audio output including a synthesized speech waveform corresponding to the parameters of the highest scoring audio output sequence (and, if the correct sequence was selected, to the input data).

[0159] System 120 may include user recognition component 170. User recognition component 170 may use various data to recognize one or more users. User recognition component 170 may use audio data 511 as input. User recognition component 170 may perform user recognition by comparing voice characteristics in audio data 511 with stored user voice characteristics. User recognition component 170 may additionally or alternatively perform user recognition by comparing biometric data (e.g., fingerprint data, iris data, retina data, etc.) received by system 120 and associated with natural language user input with stored user biometric data. User recognition component 170 may additionally or alternatively perform user recognition by comparing image data (e.g., including representations of at least one feature of a user) received by system 120 and stored image data including representations of features of different users. User recognition component 170 may perform other or additional user recognition processes, including those known in the art. For a particular natural language user input, user recognition component 170 may perform processing on stored user data associated with device 110 that receives the natural language user input.

[0160] The user identification component 170 determines whether the natural language user input originated from a particular user. For example, the user identification component 170 can determine a first value representing the likelihood that the natural language user input originated from a first user, a second value representing the likelihood that the natural language input originated from a second user, etc. The user identification component 170 can also determine an overall confidence level regarding the accuracy of the user identification process.

[0161] User identification component 170 may output a single user identifier corresponding to the most likely user to initiate the natural language user input. Alternatively, user identification component 170 may output multiple user identifiers (e.g., in the form of an N-best list) with respective values ​​representing the likelihood that the respective users initiated the natural language user input. The output of user identification component 170 may be used to inform NLU processing, processing performed by skill component 160, processing performed by access control skill component 165, and processing performed by other components of system 120 and / or other systems.

[0162] The system 120 may include a profile storage 570. The profile storage 570 may include various data related to individual users, user groups, devices, etc. As used herein, a "profile" refers to a set of data associated with a user, user group, device, etc. The data for a profile may include: preferences specific to a user, user group, device, etc.; input and output capabilities of one or more devices; Internet connection data; user bibliographic data; subscription data; skill component enablement data; and / or other data.

[0163] The profile storage device 570 may include one or more user profiles. Each user profile may be associated with a different user identifier. Each user profile may include various user identification data (e.g., name, gender, address, language, etc.). Each user profile may also include the user's preferences. Each user profile may include one or more device identifiers, each device identifier representing a corresponding device registered to the user. Each user profile may include component identifiers of skill components that the user has enabled. When the user enables a skill component, the user provides permission to allow the skill component to execute in response to the user's input. If the user does not enable the skill component, the skill component may be prevented from processing the user's input.

[0164] The profile storage device 570 may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a home device profile associated with a user profile, and the user profile is associated with multiple users of a single home device. The group profile may include preferences shared by all user profiles associated with it. Each user profile associated with a group profile may additionally include preferences specific to the user associated with it. That is, a user profile may include preferences different from one or more other user profiles associated with the same group profile. A user profile may be an independent profile, or may be associated with a group profile. A group profile may be associated with (or include) one or more device profiles, which correspond to one or more devices associated with the group profile.

[0165] Profile storage 570 may include one or more device profiles. Each device profile may be associated with a different device identifier. A device profile may include various device identification data, input / output characteristics, networking characteristics, etc. A device profile may also include one or more user identifiers corresponding to one or more user profiles associated with the device profile. For example, a profile for a home device may include a user identifier for a user of the home device.

[0166] Apart from Figure 5 In addition to the components shown, the system 120 can also include one or more of a dispatch component 135 , a state management component 1555 , an event component 180 , an access control skill component 165 , an authentication skill component 175 , a consent component 190 , and an identity component 195 .

[0167] In some embodiments, device 110 may include the voice processing components described above with respect to system 120, and may be configured to perform the processing described above with respect to system 120. In such embodiments, various operations discussed herein may be performed by device 110 without necessarily involving 120, such as in processing and executing voice commands. In other embodiments, device 110 may perform certain operations and system 120 may perform other operations in a joint effort to process and execute voice commands.

[0168] like Figure 6 As shown, the user recognition component 170 may include one or more subcomponents, including a visual component 608, an audio component 610, a biometric component 612, a radio frequency (RF) component 614, a machine learning (ML) component 616, and a recognition confidence component 618. In some cases, the user recognition component 170 may monitor data and determinations from one or more subcomponents to determine the identity of one or more users associated with data input to the device 110 and / or system 120. The user recognition component 170 may output user recognition data 410, which may include a user identifier associated with the user determined by the user recognition component 170 to have initiated the data input to the device 110 and / or system 120. The user recognition data 410 may be used to inform processes performed by various components of the device 110 and / or system 120.

[0169] The visual component 608 can receive data from one or more sensors (e.g., cameras) that can provide images or sensors (e.g., motion sensors) that indicate motion. The visual component 608 can perform facial recognition or image analysis to determine the identity of the user and associate the identity with a user profile associated with the user. In some cases, when the user faces the camera, the visual component 608 can perform facial recognition and identify the user with high confidence. In other cases, the visual component 608 may have a low confidence in the identity of the user, and the user recognition component 170 can determine the identity of the user using a determination from an additional component. The visual component 608 can be used in conjunction with other components to determine the identity of the user. For example, the user recognition component 170 can use data from the visual component 608 and data from the audio component 610 to identify which user's face appears to be speaking while the device 110 facing the user captures audio, so as to identify the user who speaks input to the device 110 and / or the system 120.

[0170] The overall system of the present disclosure may include a biometric sensor that transmits data to a biometric component 612. For example, the biometric component 612 may receive data corresponding to a fingerprint, an iris or retina scan, a thermal scan, a user's weight, a user's body shape, pressure (e.g., within a floor sensor), etc., and may determine a biometric profile corresponding to the user. For example, the biometric component 612 may distinguish between a user and sound from a television. Thus, the biometric component 612 may incorporate biometric information into a confidence level for determining the identity of the user. The biometric information output by the biometric component 612 may be associated with specific user profile data such that the biometric information uniquely identifies the user's user profile.

[0171] Radio frequency (RF) component 614 can use RF positioning to track devices that a user may be carrying or wearing. For example, a user (and a user profile associated with the user) can be associated with a device. The device can transmit RF signals (e.g., Wi-Fi, The device may detect the signal and indicate the strength of the signal to RF component 614 (e.g., as a received signal strength indication (RSSI)). RF component 614 may use the RSSI to determine the identity of the user (with an associated confidence level). In some cases, RF component 614 may determine that the received RF signal is associated with a mobile device that is associated with a particular user identifier.

[0172] In some cases, a personal device (such as a phone, tablet, wearable, or other device) may include some RF or other detection processing capabilities so that a user speaking an input can scan, tap, or otherwise identify his / her personal device to device 110. In this way, a user can "register" with system 100 so that system 100 can determine who spoke a particular input. This registration can occur before, during, or after the input is spoken.

[0173] The ML component 616 can track various user behaviors as a factor in determining a confidence level of a user's identity. For example, a user may follow a regular schedule that places the user in a first location during the day (e.g., at work or school). In this example, the ML component 616 will consider past behaviors and / or trends when determining the identity of a user providing input to the device 110 and / or system 120. Thus, the ML component 616 can use historical data and / or usage patterns over time to increase or decrease the confidence level of a user's identity.

[0174] In at least some cases, the recognition confidence component 618 receives the determinations from the various components 608, 610, 612, 614, and 616 and can determine a final confidence level associated with the identity of the user. In some cases, the confidence level can determine whether to perform an action in response to the user input. For example, if the user input includes a request to unlock a door, the confidence level may need to be above a threshold, which may be higher than a threshold confidence level required to perform a user request associated with playing a playlist or sending a message. The confidence level or other score data may be included in the user recognition data 410.

[0175] The audio component 610 can receive data from one or more sensors (e.g., one or more microphones) that can provide audio signals to facilitate identification of the user. The audio component 610 can perform audio recognition on the audio signal to determine the identity of the user and the associated user identifier. In some cases, various aspects of the device 110 and / or system 120 can be configured at a computing device (e.g., a local server). Therefore, in some cases, the audio component 610 operating on the computing device can analyze all sounds to facilitate identification of the user. In some cases, the audio component 610 can perform voice recognition to determine the identity of the user.

[0176] The audio component 610 may also perform user recognition based on the audio data 511 input into the device 110 and / or system 120 for speech processing. The audio component 610 may determine a score indicating whether speech in the audio data originates from a particular user. For example, a first score may indicate the likelihood that speech in the audio data 511 originates from a first user associated with a first user identifier, a second score may indicate the likelihood that speech in the speech data 511 originates from a second user associated with a second user identifier, and so on. The audio component 610 may perform user recognition by comparing speech characteristics represented in the audio data 511 with stored user speech characteristics (e.g., a stored voice archive associated with the device 110 that captures spoken user input).

[0177] Figure 7 A user identification process is shown which may be performed by the user identification component 170. The ASR component 145 performs ASR processing on the ASR feature vector data 750. The ASR confidence data 707 may be passed to the user identification component 170.

[0178] The user identification component 170 performs user identification using various data, including user identification feature vector data 740, feature vectors 705 representing speech profiles of users of the system 100, ASR confidence data 707, and other data 709. The user identification component 170 can output user identification data 410 that reflects a certain confidence that the user input is spoken by one or more specific users. The user identification data 410 can include one or more user identifiers (e.g., corresponding to one or more speech profiles). Each user identifier in the user identification data 410 can be associated with a corresponding confidence value that represents the likelihood that the user input corresponds to the user identifier. The confidence value can be a numeric value or a binned value.

[0179] The feature vector 705 input to the user recognition component 170 may correspond to one or more voice archives. The user recognition component 170 may use the feature vector 705 to compare with a user recognition feature vector 740 representing the current user input to determine whether the user recognition feature vector 740 corresponds to one or more of the feature vectors 705 of the voice archives. Each feature vector 705 may be the same size as the user recognition feature vector 740.

[0180] In order to perform user recognition, the user recognition component 170 can determine the device 110 from which the audio data 511 originates. For example, the audio data 511 can be associated with metadata including a device identifier representing the device 110. The device 110 or the system 120 can generate the metadata. The system 100 can determine the group profile identifier associated with the device identifier, can determine the user identifier associated with the group profile identifier, and can include the group profile identifier and / or the user identifier in the metadata. The system 100 can associate the metadata with the user recognition feature vector 740 generated from the audio data 511. The user recognition component 170 can send a signal to the voice archive storage device 785, wherein the signal only requests the audio data and / or feature vector 705 associated with the device identifier, group profile identifier, and / or user identifier represented in the metadata (depending on whether the audio data and / or the corresponding feature vector are stored). This limits the range of possible feature vectors 705 considered by the user recognition component 170 at runtime, and thus reduces the amount of time to perform the user recognition process by reducing the amount of feature vectors 705 that need to be processed. Alternatively, the user recognition component 170 may access all of the audio data and / or feature vectors 705 (or some other subset thereof) available to the user recognition component 170. However, based on the amount of audio data and / or feature vectors 705 to be processed, accessing all of the audio data 511 and / or feature vectors 705 will likely increase the amount of time required to perform the user recognition process.

[0181] If the user recognition component 170 receives audio data from the voice archive storage device 785, the user recognition component 170 can generate one or more feature vectors 705 corresponding to the received audio data.

[0182] The user identification component 170 may attempt to identify the user who spoke the speech represented in the audio data 511 by comparing the user identification feature vector 740 to the feature vector 705. The user identification component 170 may include a scoring component 722 that determines a corresponding score indicating whether one or more specific users (represented by the feature vector 705) spoke the user input (represented by the user identification feature vector 740). The user identification component 170 may also include a confidence component 724 that determines the overall accuracy of the user identification process (such as the accuracy of the scoring component 722) and / or the individual confidence values ​​that may be identified by the scoring component 722 for each user. The output from the scoring component 722 may include different confidence values ​​for each received feature vector 705. For example, the output may include a first confidence value for a first feature vector 705a (representing a first voice profile), a second confidence value for a second feature vector 705b (representing a second voice profile), and so on. Although shown as two separate components, the score component 722 and the confidence component 724 can be combined into a single component, or can be separated into more than two components.

[0183] Scoring component 722 and confidence component 724 can implement one or more trained machine learning models known in the art (such as neural networks, classifiers, etc.). For example, scoring component 722 can use probabilistic linear discriminant analysis (PLDA) technology. PLDA scoring determines the possibility that user identification feature vector 740 corresponds to a specific feature vector 705. PLDA scoring can generate a confidence value for each feature vector 705 considered, and can output a list of confidence values ​​associated with the corresponding user identifier. Scoring component 722 can also use other techniques (such as GMM, generating Bayesian models, etc.) to determine the confidence value.

[0184] Confidence component 724 can input various data, including information about ASR confidence, speech length (e.g., number of frames or other measured length input by the user), audio condition / quality data (such as signal interference data or other metric data), fingerprint data, image data or other factors, to consider the confidence of user recognition component 170 about linking the user to the confidence value input by the user. Confidence component 724 can also consider the confidence value and associated identifier output by scoring component 722. For example, confidence component 724 can determine that lower ASR confidence or poor audio quality or other factors may lead to lower confidence of user recognition component 170. And higher ASR confidence or better audio quality or other factors may lead to higher confidence of user recognition component 170. The precise determination of confidence can depend on the configuration and training of the confidence component 724 and the model implemented by it. Confidence component 724 can operate using a variety of different machine learning models / techniques (such as GMM, neural network, etc.). For example, confidence component 724 can be a classifier configured to map the score output by scoring component 722 to a confidence value.

[0185] The user identification component 170 can output user identification data 410 specific to one or more user identifiers. For example, the user identification component 170 can output user identification data 410 for each received feature vector 705. The user identification data 410 can include a numerical confidence value (e.g., 0.0 to 1.0, 0 to 1000, or any range that the system is configured to operate). Thus, the user identification data 410 can output an n-best list of potential users with numerical confidence values ​​(e.g., user identifier 123 - 0.2, user identifier 234 - 0.8). Alternatively or in addition, the user identification data 410 can include binned confidence values. For example, a first range of calculated identification scores (e.g., 0.0 to 0.33) can be output as "low", a second range of calculated identification scores (e.g., 0.34 to 0.66) can be output as "medium", and a third range of calculated identification scores (e.g., 0.67 to 1.0) can be output as "high". The user identification component 170 may output an n-best list of user identifiers with binned confidence values ​​(e.g., user identifier 123 - low, user identifier 234 - high). A combined binning and numerical confidence value output is also possible. The user identification data 410 may include only information related to the highest scoring identifiers determined by the user identification component 170, rather than a list of identifiers and their corresponding confidence values. The user identification component 170 may also output an overall confidence value that the individual confidence values ​​are correct, wherein the overall confidence value indicates the confidence of the user identification component 170 in the output result. The confidence component 724 may determine the overall confidence value.

[0186] Confidence component 724 can determine differences between confidence values ​​when determining user identification data 410. For example, if the difference between the first confidence value and the second confidence value is large, and the first confidence value is above the threshold confidence value, user identification component 170 can identify the first user (associated with feature vector 705 associated with the first confidence value) as the user who spoke the user input with a higher confidence than if the difference between the confidence values ​​is small.

[0187] The user identification component 170 may perform threshold processing to avoid outputting incorrect user identification data 410. For example, the user identification component 170 may compare the confidence value output by the confidence component 724 with a threshold confidence value. If the confidence value does not meet (e.g., does not equal or exceed) the threshold confidence value, the user identification component 170 may not output the user identification data 410, or may only include an indicator of the user who cannot identify the user input in the data 410. In addition, the user identification component 170 may not output the user identification data 410 until sufficient user identification feature vector data 740 has been accumulated and processed to verify the user above the threshold confidence value. Therefore, the user identification component 170 may wait until a sufficient threshold amount of audio data of the user input has been processed, and then output the user identification confidence data 410. The confidence component 724 may also take into account the amount of audio data received.

[0188] The user identification component 170 may default to outputting binned (e.g., low, medium, high) user identification confidence values. However, in some cases, this may be problematic. For example, if the user identification component 170 computes a single binned confidence value for multiple feature vectors 705, the system may not be able to determine which particular user initiated the user input. In this case, the user identification component 170 may override its default setting and output a numerical confidence value. This enables the system to determine the user who initiated the user input and is associated with the highest numerical confidence value.

[0189] The user identification component 170 can use other data 709 to inform the user identification process. When performing the user identification process, the trained model or other components of the user identification component 170 can be trained to use other data 709 as input features. Depending on the system configuration, the other data 709 can include a variety of data types and can be obtained from other sensors, devices, or storage devices. The other data 709 can include the time of day when the audio data 511 is generated by the device 110 or received from the device 110, the day of the week when the audio data 511 is generated by the device 110 or received from the device 110, etc.

[0190] Other data 709 may include image data or video data. For example, facial recognition may be performed on image data or video data received from device 110 (or another device) from which audio data 511 is received. Facial recognition may be performed by user recognition component 170. The output of the facial recognition process may be used by user recognition component 170. That is, the facial recognition output data may be used in conjunction with a comparison of the user recognition feature vector 740 and one or more feature vectors 705 to perform a more accurate user recognition process.

[0191] Other data 709 may include location data of device 110. The location data may be specific to the building in which device 110 is located. For example, if device 110 is located in user A's bedroom, this location may increase the user identification confidence value associated with user A and / or decrease the user identification confidence value associated with user B.

[0192] Other data 709 may also include data indicating the type of device 110. Different types of devices may include, for example, smart watches, smart phones, tablets, and vehicles. The type of device 110 may be indicated in a profile associated with the device 110. For example, if the device 110 from which the audio data 511 is received is a smart watch or a vehicle belonging to user A, the fact that the device 110 belongs to user A may increase the user identification confidence value associated with user A and / or decrease the user identification confidence value associated with user B.

[0193] Other data 709 may include geographic coordinate data associated with device 110. For example, a group profile associated with a vehicle may indicate multiple users (e.g., user A and user B). The vehicle may include a global positioning system (GPS) that indicates the latitude and longitude coordinates of the vehicle when the vehicle generated the audio data 511. Therefore, if the vehicle is located at coordinates corresponding to user A's workplace / building, this may increase the user identification confidence value associated with user A and / or reduce the user identification confidence values ​​of all other users indicated in the group profile associated with the vehicle. The profile associated with device 110 may indicate global coordinates and associated locations (e.g., work, home, etc.). One or more user profiles may also or alternatively indicate global coordinates.

[0194] Other data 709 may include data representing the activities of a particular user, which may be useful for performing user identification processing. For example, a user may have recently entered a code to disable a home security alarm. Devices 110 represented in a group profile associated with a home may have generated audio data 511. Other data 709 may reflect signals from a home security alarm regarding disabled users, disabled times, and the like. If a mobile device (such as a smart phone, Tile, dongle, or other device) known to be associated with a particular user is detected to be close to device 110 (e.g., physically close, connected to the same Wi-Fi network, or otherwise nearby), this may be reflected in other data 709 and considered by user identification component 170.

[0195] Depending on the system configuration, the other data 709 can be configured to be included in the user identification feature vector data 740, so that all data related to the user data to be processed by the scoring component 722 can be included in a single feature vector. Alternatively, the other data 709 can be reflected in one or more different data structures to be processed by the scoring component 722.

[0196] Illustrative components and processes of device 110 are described below. Figure 8 As shown, in at least some embodiments, system 120 can receive audio data 511 from device 110 to recognize speech in the received audio data 511, and perform functions in response to the recognized speech. In at least some embodiments, these functions involve sending instructions (e.g., commands) from system 120 to device 110 to cause device 110 to perform actions, such as outputting synthesized speech via a speaker, and / or controlling one or more auxiliary devices by sending control commands to the one or more auxiliary devices.

[0197] Thus, when device 110 is capable of communicating with system 120 via network 199, some or all of the functions that can be performed by system 120 can be performed by sending one or more instructions to device 110 over network 199, which can then process the instructions and perform one or more corresponding actions. For example, system 120 can use instructions included in response data (e.g., response) to instruct device 110 to output synthesized speech via a speaker of device 110 (or otherwise associated with device 110), output content (e.g., music) via a speaker of device 110 (or otherwise associated with device 110), display content on a display of device 110 (or otherwise associated with device 110), and / or send instructions to an auxiliary device (e.g., instructions to turn on a smart light). It should be understood that system 120 may be configured to provide other functionality in addition to the functionality discussed herein, such as, but not limited to, providing step-by-step instructions for navigating from a source location to a destination location, conducting e-commerce transactions on behalf of user 105 as part of a shopping function, establishing a communication session (e.g., an audio or video call) between user 105 and another user, and the like.

[0198] As previously described, the device 110 may include a wake word detection component 540 configured to detect a wake word (e.g., "Alexa") that indicates to the device 110 that the audio data 511 will be processed to determine the NLU output data. In at least some embodiments, the hybrid selector 824 of the device 110 may send the audio data 511 to the wake word detection component 540. If the wake word detection component 540 detects the wake word in the audio data 511, the wake word detection component 540 may send an indication of such detection to the hybrid selector 824. In response to receiving the indication, the hybrid selector 824 may send the audio data 511 to the system 120 and / or the ASR component 845 implemented by the device 110. The wake word detection component 540 may also send an indication to the hybrid selector 824 that the wake word was not detected. In response to receiving such an indication, the hybrid selector 824 may avoid sending the audio data 511 to the system 120 and may prevent the ASR component 845 from processing the audio data 511. In this case, the audio data 511 may be discarded.

[0199] Device 110 may perform its own speech processing using on-device language processing components such as SLU component 847, ASR component 845, and / or NLU component 850, similar to the manner discussed above with respect to the SLU component 547, ASR component 145, and NLU component 150 of the system implementation. The device 110 may also internally include or otherwise access other components, such as a TTS component 880 (configured to process in a manner similar to the TTS component 580 implemented by the system 120), an archive storage device 870 (configured to store archive data similar to the archive storage device 570 implemented by the system 120), a skill selection component 885 (configured to process in a manner similar to the skill selection component 585 implemented by the system 120), a skill preliminary selection component 865 (configured to process in a manner similar to the skill preliminary selection component 565 implemented by the system 120), one or more skills 860 (configured to process in a manner similar to one or more skill components 160 implemented by the system 120 and / or communicating with the system), a user identification component 875 (configured to process in a manner similar to the user identification component 170 implemented by the system 120), a verification skill component 175, an access control skill component 165, a consent component 190, an identity component 195 and / or other components. In at least some embodiments, profile storage 870 may store only profile data for a user or group of users specifically associated with device 110 .

[0200] In at least some embodiments, a language processing component on the device may not have the same capabilities as a language processing component implemented by system 120. For example, a language processing component on the device may be configured to process only a subset of user inputs that can be handled by a language processing component implemented by the system. For example, such a subset of user inputs may correspond to local types of user inputs, such as those that control components of device 110. In such cases, the language processing component on the device may be able to interpret and respond to local types of user inputs more quickly, for example, than processing involving system 120. If device 110 attempts to process user inputs for which the language processing component on the device is not necessarily best suited, the NLU output data determined by the component on the device may have low confidence or other metrics indicating that the processing by the language processing component on the device may not be as accurate as the processing done by system 120.

[0201] The hybrid selector 824 of the device 110 may include a hybrid proxy (HP) 826 configured to proxy traffic to / from the system 120. For example, the HP 826 may be configured to send messages to / from a hybrid execution controller (HEC) 827 of the hybrid selector 824. For example, the HP 826 may be used to send command / instruction data received from the system 120 to the HEC 827. The HP 826 may also be configured to allow the audio data 511 to pass to the system 120 while also receiving (e.g., intercepting) the audio data 511 and sending the audio data 511 to the HEC 827.

[0202] In at least some embodiments, the hybrid selector 824 may also include a local request orchestrator (LRO) 828 that is configured to notify the ASR component 845 of the availability of the audio data 511 and otherwise initiate operation of the language processing on the device when the audio data 511 becomes available. In general, the hybrid selector 824 may control the execution of the language processing on the device, such as by sending "execute" and "terminate" events. An "execute" event may instruct a component to continue any suspended execution (e.g., by instructing the component to execute according to a previously determined intent in order to determine an instruction). Meanwhile, a "terminate" event may instruct a component to terminate further execution, such as when the device 110 receives instruction data from the system 120 and chooses to use remotely determined instruction data.

[0203] Thus, when receiving the audio data 511, the HP 826 may allow the audio data 511 to be passed to the system 120, and the HP 826 may also input the audio data 511 to the ASR component 845 by routing the audio data 511 via the HEC 827 of the hybrid selector 824, whereby the LRO 828 notifies the ASR component 845 of the audio data 511. At this point, the hybrid selector 824 may wait for response data from one or both of the system 120 and / or the language processing component on the device. However, the present disclosure is not limited thereto, and in some examples, without departing from the present disclosure, the hybrid selector 824 may only send the audio data 511 to the ASR component 845. For example, the device 110 may process the audio data 511 on the device without sending the audio data 511 to the system 120.

[0204] The ASR component 845 is configured to receive the audio data 511 from the hybrid selector 824 and recognize the speech in the audio data 511, and the NLU component 850 is configured to determine the intent based on the recognized speech (and optionally one or more named entities) and determine how to act on the intent by generating instruction data (e.g., instructing the component to perform an action). In some cases, the instruction may include a description of the intent (e.g., the intent to turn off {device A}). In some cases, the instruction may include (e.g., encoding) an identifier of a second device (such as a kitchen light), and an operation to be performed at the second device. The instruction data may be formatted using Java, such as JavaScript syntax or a JavaScript-based syntax. This may include formatting the instruction using JSON. In at least some embodiments, the device-determined instruction may be serialized, much like how the remote-determined instruction may be serialized for transmission in a data packet over the network 199. In at least some embodiments, the device-determined instruction may be formatted as a program application programming interface (API) call having the same logical operation as the remote-determined instruction. In other words, the device-determined instructions may mimic the remotely-determined instructions by using the same or similar format as the remotely-determined instructions.

[0205] NLU output data (output by NLU component 850) may be selected as available for responding to user input, and local response data may be sent to hybrid selector 824, such as a "ready to execute" response. Hybrid selector 824 may then determine whether to respond to the user input using instruction data from a component on the device, using instruction data received from system 120 (assuming even a remote response is received (e.g., when device 110 is able to access system 120 via network 199)), or determining output data to request additional information from user 105.

[0206] Device 110 and / or system 120 may associate a unique identifier with each user input. Device 110 may include the unique identifier when sending audio data 511 to system 120, and response data from system 120 may include a unique identifier to identify which user input the response data corresponds to.

[0207] Fig. 9 is a block diagram conceptually illustrating device 110 . Fig.10 1 is a block diagram conceptually illustrating example components of a remote device, such as system 120 or access control system 130 (which may assist in ASR processing, NLU processing, etc.) and a skills component. The system (120 / 130) may include one or more servers. As used herein, "server" may refer to a traditional server as understood in a server / client computing structure, but may also refer to a plurality of different computing components that may assist in the operations discussed herein. For example, a server may include one or more physical computing components (such as a rack-mounted server) that are physically and / or network-connected to other devices / components and are capable of performing computing operations. The server may also include one or more virtual machines that simulate a computer system and run on one or more devices. The server may also include other combinations of hardware, software, or firmware, etc. to perform the operations discussed herein. The system (120 / 130) may be configured to operate using one or more of a client-server model, a computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, a peer-to-peer model, sandbox technology, or other computing technologies.

[0208] The system 100 of the present disclosure may include multiple systems (120 / 130), such as one or more systems 120 for performing ASR processing, one or more systems 120 for performing NLU processing and one or more skill components, one or more access control systems 130, etc. In operation, each of these systems may include computer-readable and computer-executable instructions resident on the corresponding system (120 / 130), which will be further discussed below.

[0209] Each of these devices (110 / 120 / 130) may include one or more controllers / processors (904 / 1004), each of which may include a central processing unit (CPU) for processing data and computer-readable instructions and a memory (906 / 1006) for storing data and instructions for the corresponding device. The memory (906 / 1006) may include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory, respectively. Each device (110 / 120 / 130) may also include a data storage device component (908 / 1008) for storing data and controller / processor executable instructions. Each data storage device component (908 / 1008) may include one or more non-volatile storage device types, such as magnetic storage devices, optical storage devices, solid-state storage devices, etc. Each device (110 / 120 / 130) may also be connected to a removable or external non-volatile memory and / or storage device (such as a removable memory card, storage key drive, network storage device, etc.) through a respective input / output device interface (902 / 1002).

[0210] Computer instructions for operating each device (110 / 120 / 130) and its various components may be executed by the controller / processor (904 / 1004) of the respective device, using the memory (906 / 1006) as temporary "working" storage during runtime. The computer instructions for the device may be stored in a non-transitory manner in non-volatile memory (906 / 1006), storage (908 / 1008), or an external device. Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or in lieu of software.

[0211] Each device (110 / 120 / 130) includes an input / output device interface (902 / 1002). Various components can be connected via the input / output device interface (902 / 1002), which will be discussed further below. Additionally, each device (110 / 120 / 130) can include an address / data bus (924 / 1024) for transmitting data between components of the corresponding device. In addition to (or instead of) being connected to other components via the bus (924 / 1024), each component within the device (110 / 120 / 130) can also be directly connected to other components.

[0212] refer to Fig. 9 , the device 110 may include an input / output device interface 902, which is connected to various components, such as an audio output component (such as a speaker 912, a wired headset or a wireless headset (not shown)) or other components capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 920 or a microphone array, a wired headset or a wireless headset (not shown), etc. If a microphone array is included, the approximate distance to the origin of the sound can be determined by acoustic positioning based on the time and amplitude differences between the sounds captured by different microphones in the array. The device 110 may additionally include a display 916 for displaying content. The device 110 may also include a camera 918.

[0213] Via antenna 914, the input / output device interface 902 can be connected to one or more networks 199 via a wireless local area network (WLAN) (such as Wi-Fi) radio, Bluetooth and / or a wireless network radio (such as a radio capable of communicating with a wireless communication network (such as a Long Term Evolution (LTE) network, a WiMAX network, a 3G network, a 4G network, a 5G network, etc.)). Wired connections such as Ethernet may also be supported. Through the network 199, the system can be distributed in a network environment. The I / O device interface (902 / 1002) may also include a communication component that allows data to be exchanged between devices (such as different physical servers in a collection of servers or other components).

[0214] Components of the device 110, system 120, access control system 130, and / or skill component may include their own dedicated processors, memories, and / or storage. Alternatively, one or more components of the device 110, system 120, access control system 130, and / or skill component may utilize the I / O interface (902 / 1002), processor (904 / 1004), memory (906 / 1006), and / or storage (908 / 1008) of the device 110, system 120, access control system 130, or skill component, respectively.

[0215] As described above, multiple devices may be employed in a single system. In this multi-device system, each of the devices may include different components for performing different aspects of system processing. Multiple devices may include overlapping components. As described herein, the components of device 110, system 120, access control system 130, and skill components are illustrative and may be located as stand-alone devices or may be included in whole or in part as components of a larger device or system.

[0216] like Fig.11 As shown, multiple devices may include components of system 100, and the devices may be connected via a network 199. Network 199 may include a local or private network, or may include a wide area network such as the Internet. Devices may be connected to network 199 via a wired connection or a wireless connection. For example, voice-controllable device 110a, smart phone 110b, smart watch 110c, tablet computer 110d, vehicle 110e, voice-controlled device with display 110f, television 110g, washer / dryer 110h, refrigerator 110i, and microwave oven 110j may be connected to network 199 via a wireless service provider, via Wi-Fi or cellular network connection, etc. Other devices are included as network-connected support devices, such as system 120, access control system 130, and / or other devices. Support devices may be connected to network 199 via a wired connection or a wireless connection. Networked devices may capture audio using one or more built-in or connected microphones or other audio capture devices, where processing is performed by an ASR component, an NLU component, or other components of the same device or another device connected via network 199.

[0217] The concepts disclosed herein can be applied in a variety of different devices and computer systems, including, for example, general purpose computing systems, speech processing systems, and distributed computing environments.

[0218] The material herein may also be understood in light of the following terms.

[0219] 1. A computing system comprising: at least one processor; and at least one memory, the at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to: receive first input audio data corresponding to a first spoken natural language user input from a first device associated with first archival data; determine using the first input audio data that the first spoken natural language user input requests output of a first content type; determine that the first device corresponds to a device type; determine, at least in part based on the first device corresponding to the device type, that sending content of the first content type to the first device requires additional functional access; determine that the additional functional access has not been granted to the first device; generate a first access token corresponding to the first archival data; send the first access token to an access control system; after sending the first access token to the access control system, receive additional functional access information and first data indicating that the first device is capable of performing the additional function from the access control system; generate The following first output audio data: indicates that the additional function access is required to output the first content type, indicates the additional function access information, and requests authorization for the first device to obtain the additional function access; causes the first device to present the first output audio data; after causing the first device to present the first output audio data, receives second input audio data corresponding to a second spoken natural language user input from the first device; uses the second input audio data to determine that the second spoken natural language user input requests to grant the additional function access to the first device; grants the additional function access to the first device based at least in part on the second spoken natural language user input request, causing the access control system to grant the additional function access to the first device; receives second data from the access control system indicating that the additional function access has been granted to the first device; generates second output audio data indicating that the additional function access has been granted to the first device; and causes the first device to present the second output audio data.

[0220] 2. A computing system according to clause 1, wherein the at least one memory also includes instructions that, when executed by the at least one processor, also cause the computing system to: determine a number of previously spoken natural language user inputs received from the first device and requiring the additional functionality access after determining that the additional functionality access has not yet been granted to the first device; determine that the number of previously spoken natural language user inputs satisfies a condition; and generate the first access token corresponding to the first archival data based at least in part on determining that the number of previously spoken natural language user inputs satisfies the condition, wherein the access control system is configured to receive the first archival data using the first access token, and the access control system is also configured to generate at least one of the additional functionality access information and the first data based at least in part on the first archival data.

[0221] 3. A computing system according to clause 1 or 2, wherein the first content type corresponds to long-form audio, and wherein the at least one memory also includes instructions that, when executed by the at least one processor, further cause the computing system to: determine that sending the content of the first content type to the first device requires the additional functionality access based at least in part on determining that the first device corresponds to the device type and that the first content type corresponds to the long-form audio.

[0222] 4. A computing system comprising: at least one processor; and at least one memory, the at least one memory comprising instructions, which when executed by the at least one processor cause the system to: receive first input data corresponding to a first user input from a first device; determine that the first user input requests content of a first content type; determine that sending the content of the first content type to the first device requires additional functional access; send device profile data corresponding to the first device to an access control system; after sending the device profile data, receive first data from the access control system indicating that the first device is capable of performing additional functions; generate first output data based at least in part on receiving the first data, the first output data requesting authorization to grant the additional functional access to the first device; cause the first device to present the first output data; receive second input data corresponding to a second user input; determine that the second user input authorizes granting the additional functional access to the first device; and cause the access control system to grant the additional functional access to the first device.

[0223] 5. A computing system according to clause 4, wherein the at least one memory also includes instructions that, when executed by the at least one processor, also cause the computing system to: determine a number of previous user inputs received from the first device and requiring access to the additional functionality; determine that the number of previous user inputs satisfies a condition; and generate the first output data based at least in part on the number of previous user inputs satisfying the condition.

[0224] 6. A computing system according to clause 4 or 5, wherein the at least one memory also includes instructions that, when executed by the at least one processor, cause the computing system to: generate an access token corresponding to the device profile data; send the access token to the access control system; and after sending the access token, receive second data from the access control system, wherein the second data indicates at least one requirement for granting access to the additional functionality to the first device.

[0225] 7. A computing system according to clauses 4, 5 or 6, wherein the at least one memory also includes instructions that, when executed by the at least one processor, further cause the computing system to: determine a user identifier associated with the second user input; determine that the user identifier is authorized to be used to grant the additional functionality access to the first device; and cause the access control system to grant the additional functionality access to the first device based at least in part on determining that the user identifier is authorized to be used to grant the additional functionality access.

[0226] 8. A computing system according to clauses 4, 5, 6 or 7, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing system to: determine status data corresponding to the first device; determine that the status data indicates that access to the additional functionality has not yet been granted to the first device; and send the device profile data to the access control system based at least in part on determining that the status data indicates that access to the additional functionality has not yet been granted to the first device.

[0227] 9. A computing system according to clauses 4, 5, 6, 7 or 8, wherein the at least one memory also includes instructions that, when executed by the at least one processor, further cause the computing system to: determine that the first device corresponds to a device type; and determine that sending the content of the first content type to the first device requires the additional functionality access based at least in part on the first device corresponding to the first device type.

[0228] 10. A computing system according to clauses 4, 5, 6, 7, 8 or 9, wherein the first content type corresponds to long-form audio, and wherein the at least one memory also includes instructions that, when executed by the at least one processor, further cause the computing system to: determine that sending the content of the first content type to the first device requires the additional functionality access based at least in part on the first content type corresponding to the long-form audio.

[0229] 11. A computing system according to clauses 4, 5, 6, 7, 8, 9 or 10, wherein the at least one memory further includes instructions that, when executed by the at least one processor, cause the computing system to: determine an intent representing the first user input; and determine, based at least in part on the intent, that sending the content of the first content type to the first device requires access to the additional functionality.

[0230] 12. A computing system according to clauses 4, 5, 6, 7, 8, 9, 10 or 11, wherein the at least one memory also includes instructions that, when executed by the at least one processor, cause the computing system to: receive second data from the access control system indicating that access to the additional functionality has been granted to the first device; and after receiving the second data, cause the first device to output the content of the first content type.

[0231] 13. A computer-implemented method, comprising: receiving first input data corresponding to a first user input from a first device; determining that the first user input requests content of a first content type; determining that sending the content of the first content type to the first device requires additional functional access; sending device profile data corresponding to the first device to an access control system; after sending the device profile data, receiving first data from the access control system indicating that the first device is capable of performing additional functions; generating first output data based at least in part on receiving the first data, the first output data requesting authorization to grant the additional functional access to the first device; causing the first device to present the first output data; receiving second input data corresponding to a second user input; determining that the second user input authorizes granting the additional functional access to the first device; and causing the access control system to grant the additional functional access to the first device.

[0232] 14. A computer-implemented method according to clause 13, further comprising: determining a number of previous user inputs received from the first device and requiring access to the additional functionality; determining that the number of previous user inputs satisfies a condition; and generating the first output data based at least in part on the number of previous user inputs satisfying the condition.

[0233] 15. A computer-implemented method according to clause 13 or 14, further comprising: generating an access token corresponding to the device profile data; sending the access token to the access control system; and after sending the access token, receiving second data from the access control system, wherein the second data indicates at least one requirement for granting access to the additional functionality to the first device.

[0234] 16. A computer-implemented method according to clauses 13, 14 or 15, further comprising: determining a user identifier associated with the second user input; determining that the user identifier is authorized to be used to grant the additional functionality access to the first device; and causing the access control system to grant the additional functionality access to the first device based at least in part on determining that the user identifier is authorized to be used to grant the additional functionality access.

[0235] 17. A computer-implemented method according to clauses 13, 14, 15 or 16, further comprising: determining status data corresponding to the first device; determining that the status data indicates that access to the additional functionality has not yet been granted to the first device; and sending the device profile data to the access control system based at least in part on determining that the status data indicates that access to the additional functionality has not yet been granted to the first device.

[0236] 18. A computer-implemented method according to clauses 13, 14, 15, 16 or 17, further comprising: determining that the first device corresponds to a device type; and determining that sending the content of the first content type to the first device requires the additional functionality access based at least in part on the fact that the first device corresponds to the first device type.

[0237] 19. A computer-implemented method according to clauses 13, 14, 15, 16, 17 or 18, wherein the first content type corresponds to long-form audio, and wherein the computer-implemented method further comprises: determining that sending the content of the first content type to the first device requires the additional functionality access based at least in part on the first content type corresponding to the long-form audio.

[0238] 20. A computer-implemented method according to clauses 13, 14, 15, 16, 17, 18 or 19, further comprising: determining an intent representing the first user input; and determining, based at least in part on the intent, that sending the content of the first content type to the first device requires the additional functionality access.

[0239] The above aspects of the present disclosure are intended to be illustrative. They are selected to explain the principles and applications of the present disclosure, and are not intended to be exhaustive or limit the present disclosure. Many modifications and variations of the disclosed aspects may be obvious to those skilled in the art. Personnel with ordinary skills in the computer and speech processing fields should recognize that the components and process steps described herein can be interchangeable with other components or steps or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. In addition, it is obvious to those skilled in the art that the present disclosure can be implemented without some or all of the specific details and steps disclosed herein.

[0240] Aspects of the disclosed system may be implemented as a computer method or article of manufacture, such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be computer-readable and may include instructions for causing a computer or other device to perform the processes described in the present disclosure. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, a hard drive, a solid-state memory, a flash drive, a removable disk, and / or other media. In addition, components of the system may be implemented as firmware or hardware, such as an acoustic front end (AFE), which includes, among other things, analog and / or digital filters (e.g., filters configured as digital signal processor (DSP) firmware).

[0241] Conditional language used herein, such as, inter alia, "may," "can," "likely," "for example," and the like, unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and / or states. Thus, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included or to be performed in any particular embodiment with or without other input or prompting. The terms "comprise," "include," and "have," and the like are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, and the like. In addition, the term "or" is used in an inclusive sense (rather than an exclusive sense), such that when used, for example, to connect a list of elements, the term "or" represents one, some, or all of the elements in the list.

[0242] Unless otherwise specifically stated, disjunctive language such as the phrase "at least one of X, Y, Z" should generally be understood in context to indicate that an item, term, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one X, at least one Y, or at least one Z each be present.

[0243] As used in this disclosure, the term "a" or "an" may include one or more items, unless explicitly stated otherwise. In addition, the phrase "based on" is intended to mean "based at least in part on," unless explicitly stated otherwise.< / connectedwithadditionalaccess> < / connectedwithnoadditionalaccess> < / connectedwithadditionalaccess> < / connectedwithnoadditionalaccess>

Claims

1. A computing system comprising: at least one processor; as well as at least one memory, the at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to: receiving first input data corresponding to a first user input from a first device; determining that the first user input requests content of a first content type; determining that sending the content of the first content type to the first device requires additional functionality access; sending device profile data corresponding to the first device to the system; After sending the device profile data, receiving first data from the system indicating that the first device is capable of performing an additional function; generating first output data based at least in part on receiving the first data, the first output data requesting authorization to grant the additional functionality access to the first device; causing the first device to present the first output data; receiving second input data corresponding to a second user input; determining that the second user input authorizes granting the additional functionality access to the first device; as well as The system is caused to grant the additional functionality access to the first device.

2. The computing system of claim 1 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: determining a number of previous user inputs received from the first device that required access to the additional functionality; Determining that the number of previous user inputs satisfies a condition; and The first output data is also generated based at least in part on a number of the previous user inputs satisfying the condition.

3. The computing system of claim 1 or 2, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: generating an access token corresponding to the device profile data; sending the access token to the system; as well as After sending the access token, second data is received from the system, wherein the second data indicates at least one requirement for granting the additional functionality access to the first device.

4. The computing system of claim 1 , 2 or 3, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: determining a user identifier associated with the second user input; determining that the user identifier is authorized for use in granting the additional functionality access to the first device; as well as The system is caused to grant the additional functionality access to the first device based at least in part on determining that the user identifier is authorized for use in granting the additional functionality access.

5. The computing system of claim 1, 2, 3 or 4, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: determining status data corresponding to the first device; determining that the status data indicates that the additional functionality access has not been granted to the first device; as well as The device profile data is sent to the system based at least in part on determining that the status data indicates that the additional functionality access has not been granted to the first device.

6. The computing system of claim 1, 2, 3, 4, or 5, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: determining that the first device corresponds to a device type; and The additional functionality access is determined to be required to send the content of the first content type to the first device based at least in part on the first device corresponding to a first device type.

7. The computing system of claim 1, 2, 3, 4, 5, or 6, wherein the first content type corresponds to long form audio, and wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: The additional functionality access is determined to be required to send the content of the first content type to the first device based at least in part on the first content type corresponding to the long form audio.

8. The computing system of claim 1, 2, 3, 4, 5, 6, or 7, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: determining an intent representing the first user input; and A determination is made based at least in part on the intent that the additional functionality access is required to send the content of the first content type to the first device.

9. The computing system of claim 1, 2, 3, 4, 5, 6, 7, or 8, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to: receiving second data from the system indicating that the additional functionality access has been granted to the first device; and After receiving the second data, the first device is caused to output the content of the first content type.

10. A computer-implemented method comprising: receiving first input data corresponding to a first user input from a first device; determining that the first user input requests content of a first content type; determining that sending the content of the first content type to the first device requires additional functionality access; sending device profile data corresponding to the first device to the system; After sending the device profile data, receiving first data from the system indicating that the first device is capable of performing an additional function; generating first output data based at least in part on receiving the first data, the first output data requesting authorization to grant the additional functionality access to the first device; causing the first device to present the first output data; receiving second input data corresponding to a second user input; determining that the second user input authorizes granting the additional functionality access to the first device; as well as causing the system to grant the additional functionality access to the first device; 11. The computer-implemented method of claim 10, further comprising: determining a number of previous user inputs received from the first device that required access to the additional functionality; Determining that the number of previous user inputs satisfies a condition; as well as The first output data is also generated based at least in part on a number of the previous user inputs satisfying the condition.

12. The computer-implemented method of claim 10 or 11, further comprising: generating an access token corresponding to the device profile data; sending the access token to the system; as well as After sending the access token, second data is received from the system, wherein the second data indicates at least one requirement for granting the additional functionality access to the first device.

13. The computer-implemented method of claim 10, 11 or 12, further comprising: determining a user identifier associated with the second user input; determining that the user identifier is authorized for use in granting the additional functionality access to the first device; as well as The system is caused to grant the additional functionality access to the first device based at least in part on determining that the user identifier is authorized for use in granting the additional functionality access.

14. The computer-implemented method of claim 10, 11, 12 or 13, further comprising: determining status data corresponding to the first device; determining that the status data indicates that the additional functionality access has not been granted to the first device; as well as The device profile data is sent to the system based at least in part on determining that the status data indicates that the additional functionality access has not been granted to the first device.

15. The computer-implemented method of claim 10, 11, 12, 13 or 14, further comprising: determining that the first device corresponds to a device type; as well as The additional functionality access is determined to be required to send the content of the first content type to the first device based at least in part on the first device corresponding to a first device type.

Images