Standardizing analytic metrics across multiple devices

Abstract

Implementations relate to generating standardized metrics from device-specific metrics generated during interaction between a user and an automated assistant. The metrics are indicative of events occurring when processing user interactions with the automated assistant, and are specific to a specific configuration of the device with which the user is interacting. The conversion map is determined based on the device characteristics, which can be used to convert the device metrics to standardized metrics. Analytical metrics are generated based on standardized metrics, and cannot be generated from device metrics. Some implementations include visually rendering analytical metrics such that one or more of the analytical metrics are rendered more prominently than other metrics.

Description

Background Art

[0001] Client devices may have different capabilities and may execute applications and / or otherwise perform differently depending on the hardware and / or software present on the device. In some cases, the presence of one or more applications executing on a device may change the behavior of other applications and / or the behavior of hardware included in the device. In order to monitor the behavior of such devices, metrics are often generated that indicate the timing of events so that developers can examine how the device behaves when a user performs one or more actions. For example, metrics may be generated that indicate the timing of events that occur when a user interacts with an automated assistant that is executing on a client device.

[0002] However, due to the numerous software and hardware configurations possible on any number of client devices, comparisons between the behavior of different devices may be difficult. For example, while a developer may be interested in monitoring the behavior of an application executing on different devices, comparing metrics generated from separate devices may be difficult. Therefore, generating aggregate metrics across multiple devices - each with its own capabilities and configurations - may result in complex analysis to compensate for the numerous variations possible for each of the devices. Summary of the invention

[0003] Implementations disclosed herein relate to generating standardized metrics from device-specific metrics initially generated during a user's interaction with an automated assistant. Some implementations include identifying device metrics generated while the user is interacting with the automated assistant, identifying a set of transformation mappings based on one or more characteristics of a client device with which the user is interacting, generating a standardized set of metrics from the device metrics using the transformation mappings, and generating analytical metrics by comparing the standardized set of metrics to other standardized metrics generated from device metrics of one or more other client devices.

[0004] Without conversion to standard metrics, generating analysis metrics will be different depending on the device. For example, without conversion to standard metrics, it may be necessary to create a first analysis program to process device metrics from a first device set, it may be necessary to create a second analysis program to process device metrics from a second device set, it may be necessary to create a third analysis program to process device metrics from a third device set, and so on. This may require executing multiple different analysis programs simultaneously on a remote server, which may consume a large amount of memory and / or other resources of the remote server. In addition, this may require separate maintenance (e.g., updating) of each of the different analysis programs, which may require programmers to make great efforts and make extensive use of client device resources used by programmers. By converting to standard metrics according to the implementation disclosed herein, a single analysis program that operates on standard metrics can be used when generating analysis metrics. Although a single analysis program will not be able to generate analysis data based on device metrics, it can generate analysis data based on standard metrics generated from device metrics. In addition, the efficiency of memory and / or other resources can be achieved using a single analysis program because there is no need to execute and / or maintain multiple analysis programs simultaneously.

[0005] In some implementations, a metric conversion from an initial set of metrics to a standardized set of metrics can be performed on a client device that is at least partially executing an automated assistant. For example, a client device can generate device metrics that indicate the time of one or more events that occur when a user is interacting with an instance of an automated assistant that is being executed on the client device. The client device can additionally execute one or more components that can identify a conversion mapping based on device characteristics, which can be used to standardize the device metrics into standardized metrics. The client device can generate standardized metrics and then transmit the standardized metrics to a remote computing device for further analysis. Moreover, for example, a second client device can generate standardized metrics from the device metrics generated by the second device and transmit these standardized metrics to a remote computing device. In response, the remote computing device can perform a comparison of the standardized metrics and generate analytical metrics, such as analytical metrics indicating aggregated operations and / or behaviors of the device.

[0006] In some implementations, the conversion from the initial set of metrics to the standardized set of metrics can be performed by a different computing device. For example, a client device can provide the device metrics to a remote computing device, which can perform the conversion to the standardized metrics and further transmit the standardized metrics to another remote computing device for further comparison to generate analysis metrics.

[0007] In some implementations, when a user interacts with an automated assistant, a metric indicating the time of one or more events may be generated. For example, a user may utter the words "OK Assistant, send a message to Bob". The automated assistant may continuously process a small amount of audio data to determine whether a hot word (e.g., an invocation phrase such as "OK Assistant" or "Hey Assistant") has been spoken. When a hot word is detected, the automated assistant may further process the audio data following the invocation phrase. Once processed, the automated assistant may determine whether it can fulfill a request (e.g., "what time isit") included in the audio data, or whether the request should be transmitted to one or more applications for further processing. For example, for audio data including a request to "send a message to Bob", the automated assistant may provide the request to a messaging application for further processing.

[0008] For each event that occurs, a timestamp can be generated that indicates when the event starts and / or is completed. For example, the microphone can be continuously active, thereby continuously receiving audio data, and optionally performing certain limited processing on the audio data (e.g., calling phrase detection processing). At a certain moment, the automated assistant can start processing the audio data (or perform additional processing on the audio data), which can be a detectable event (e.g., an "AUDIO_PROCESS_START (audio_processing_start)" event). Processing audio data can include performing speech to text processing, and another event (e.g., a "TRANSCRIPTION_START (transcription_start)" event) can be identified using a corresponding timestamp. At a certain moment, transcription can be completed, and another event (e.g., a "TRANSCRIPTION_END (transcription_end)" event) can be generated. Therefore, in some cases, at least three events can be generated for processing audio data.

[0009] After audio processing, the automated assistant can determine whether to further process the request included in the audio data, or transfer the request to another application for further processing. As with processing audio data, one or more events can be generated to indicate events that occur when further processing the audio data. For example, one or more components of the automated assistant can determine how to fulfill the request included in the audio data. When the automated assistant has determined how to fulfill the request (e.g., by generating a response and / or by transmitting the request to another application), an event (e.g., "FULFILLMENT_DETERMINATION") can be recorded, which may include a timestamp indicating when the automated assistant made the determination. In the case of forwarding a request to another application, an event can be generated, which indicates the time to start sending the request, the time to complete sending the request, the time to start receiving a response from the application, the completion time of receiving a complete response, the delay between receiving a response from the client and rendering the response, and / or other events that may occur during processing the request.

[0010] In some implementations, events can be generated by an operating system and / or other device-specific applications. Because each operating system can log events differently (e.g., different names for events, different events logged), a first device may have a different metric log than a second device—even if users of the two devices each interact with an instance of the same automated assistant. Thus, in some implementations, metrics may not be in a standardized format that would otherwise allow events occurring on different devices to be compared to each other.

[0011] As an example, a first device may include hardware for the receipt and initial processing of audio data. The hardware may include a microphone, and the software may include a digital signal processor, which may be specific to the device and different for a second device. For example, through experimentation, a delay may be determined for the first device that indicates a time delay between the onset of speech and the initial detection of speech by the processor. Similarly, for the second device, a delay may be determined that is different from the delay of the first device. Thus, for each device, a conversion map may be identified that indicates a known time delay between the time when an event is recognized by the device and the time when the event actually occurs.

[0012] In some implementations, once device metrics have been identified, each metric indicating the time of an event that occurs when a user is interacting with an automated assistant, one or more conversion mappings can be identified that can convert device-specific metrics into standardized metrics. For example, for a first device, "TRANSCRIPTION_ START" and "TRANSCRIPTION_END" events can be generated, each with a timestamp indicating when transcription begins and when it is completed by the automated assistant. For a second device, "TRANSCRIPTION_START" and "TRANSCRIPTION_END" events can be recorded, each with a timestamp indicating when automatic speech recognition begins (TRANSCRIPTION_START) and when it is completed by the automated assistant (TRANSCRIPTION_END). However, for each device, there may be time lags, for example, based on the operation of the specific hardware and / or software of the device. Therefore, the timestamps recorded for each event may not reflect the exact time when these events occurred. Based on the identified conversion mappings, each of the timestamps of the events can be adjusted so that the time lag can be taken into account when comparing the resulting standardized events, thereby allowing comparisons between the operations of the devices in a standardized manner.

[0013] In some implementations, one or more metrics may be based on the operation of applications other than the automated assistant executed on the client device. For example, in some cases, a user may speak an invocation phrase, thereby invoking the automated assistant, and the invocation is followed by "call Bob". In response, the automated assistant may provide information to an application configured to make a phone call, which may utilize the information to fulfill the user's request. In some implementations, the response of the application (in this case, the phone application) may be used to generate metrics indicating timed events performed by the application, such as events indicating when the application receives a request, the time the application responds to a request, the time the application provides graphics rendering, and / or other behavior of the application.

[0014] In some implementations, metrics from separate devices, once normalized, can be compared to each other to generate analytical metrics. The analytical metrics can include generalizations of the operation and / or behavior of the automated assistant and / or other application so that the overall performance of the application can be evaluated on different devices that are executing different software and / or have different hardware capabilities. Therefore, analytical metrics cannot be generated from device-specific metrics, but must first be normalized to make the comparison more meaningful to the end user.

[0015] In some implementations, comparisons can be performed between standardized metrics from devices that share one or more characteristics. For example, devices that have common specific settings (e.g., language preferences) can be compared to each other to generate analytical metrics that can be analyzed to better understand the performance of one or more applications being executed on the devices. Also, for example, devices can be compared based on their geographic location, connection information, and / or other characteristics that may affect the performance of the device's hardware and / or software.

[0016] In some implementations, one or more analytical metrics may be rendered visually to provide a user with an interface for inspecting and / or evaluating the metric. For example, once the device metrics are standardized and the standardized metrics are compared to each other (e.g., averaged, aggregated), a graphical interface may be rendered that illustrates a timeline of average times of events on multiple devices. Moreover, for example, rendering may be performed so that one or more baseline values ​​and / or changes compared to baseline values ​​may be identified more quickly. For example, for a given analytical metric, a baseline delay of 5 ms may be an industry standard, or may be determined based on analytical metrics generated based on standardized metrics of a large number of device types. Once multiple standardized metrics have been aggregated, a delay of 10 ms may be identified for the metric. In this case, a timeline may be rendered that indicates the metric in a specific color (e.g., red indicates higher than the baseline), and / or the metric may be displayed more prominently in other ways to indicate to the user who is inspecting the interface that there may be a problem with the operation of the hardware and / or software of the device that generated the original metric.

[0017] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations will be described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 illustrates an environment in which the implementations described herein may be implemented.

[0019] Figure 2 Another environment in which the implementations described herein may be implemented is illustrated.

[0020] Figure 3 A timeline of generated metrics indicative of events that occur when a user interacts with an automated assistant is illustrated.

[0021] Figure 4 A flow chart illustrating a method for processing normalized metrics generated from device-specific metrics.

[0022] Figure 5A flow chart illustrating a method for processing normalized metrics generated from device-specific metrics across multiple devices.

[0023] Figure 6 An example architecture is illustrated that can be used to implement one or more of the methods described herein. DETAILED DESCRIPTION

[0024] First go to Figure 1 , illustrating an example environment in which various implementations may be performed. Figure 1 An assistant device 100 (i.e., a client device executing an automated assistant client and / or via which an automated assistant may be otherwise accessed) is included, which executes an instance of an automated assistant client 120. One or more cloud-based automated assistant components may be implemented on one or more computing systems (collectively, "cloud" computing systems) that are communicatively coupled to assistant device 100 via one or more local area networks and / or wide area networks (e.g., the Internet). An instance of automated assistant client 120, optionally via interaction with one or more of the cloud-based automated assistant components, may form what appears to be a logical instance of an automated assistant from a user's perspective, with which the user may engage in a human-computer dialogue.

[0025] The client device 100 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a wearable device including a computing device (e.g., a watch with a computing device, glasses with a computing device, a virtual or augmented reality computing device).

[0026] Additional and / or alternative assistant devices may be provided, and in some of those implementations, speaker characteristics of a particular user may be shared among assistant devices for which the user is a registered user. In various implementations, assistant device 100 may optionally operate one or more other applications (e.g., application 130) in addition to automated assistant 120, such as a messaging client (e.g., SMS, MMS, online chat), a browser, etc. In some of those various implementations, one or more of the other applications may optionally interface with automated assistant 100 (e.g., via an application programming interface), or include their own instances of automated assistant applications (which may also interface with any cloud-based automated assistant components).

[0027] The automated assistant 120 conducts a human-computer dialogue session with the user via the user interface input and output devices of the client device 100. In order to protect user privacy and / or save resources, in many cases, the user must usually explicitly call the automated assistant 120 before the automated assistant will fully process the spoken utterance. The explicit call of the automated assistant 120 may occur in response to a certain user interface input received at the client device 100. For example, the user interface input that can call the automated assistant 120 via the client device 100 may optionally include the actuation of the hardware and / or virtual buttons of the client device 100. In addition, the automated assistant client may include one or more local engines, such as a call engine that is operable to detect the presence of one or more verbal general call wake-up words. The call engine may call the automated assistant 120 in response to detecting one of the verbal call wake-up words. For example, the call engine may call the automated assistant 120 in response to detecting a verbal call wake-up word such as "HeyAssistant", "OK Assistant" and / or "Assistant". The invocation engine may continuously process (e.g., when not in an "inactive" mode) a stream of audio data frames based on the output of one or more microphones from the client device 100 to monitor for the occurrence of a spoken invocation phrase. While monitoring for the occurrence of a spoken invocation phrase, the invocation engine discards (e.g., after temporarily storing in a buffer) any audio data frames that do not include a spoken invocation phrase. However, when the invocation engine detects the occurrence of a spoken invocation phrase in a processed audio data frame, the invocation engine may invoke the automated assistant 120. As used herein, "invoking" the automated assistant 120 may include causing one or more previously inactive functions of the automated assistant 120 to be activated. For example, invoking the automated assistant 120 may include causing one or more local engines and / or cloud-based automated assistant components to further process the audio data frame based on which the invocation phrase was detected, and / or one or more subsequent audio data frames (where no further processing of the audio data frames occurred prior to the invocation). For example, the local and / or cloud-based components may process the captured audio data using an ASR model in response to the invocation of the automated assistant 120.

[0028] In some implementations, multiple automated assistants can be executed on client device 100, and the spoken invocation phrase can be different for each automated assistant. For example, a first automated assistant can have an invocation phrase "OK Assistant 1", and when a user speaks the phrase "OK Assistant 1", the first automated assistant can be invoked so that the first automated assistant can process additional audio data before and / or after the invocation phrase. Similarly, when the user speaks a second invocation phrase such as "OK Assistant 2", a second automated assistant also executing on client device 100 can be invoked, whereby the second automated assistant can process additional audio data before and / or after the invocation phrase. As further described herein, the audio data can be encrypted using speech features generated from a portion of the audio data that includes the user speaking the invocation phrase. Continuing with the previous example, in some implementations, audio data including a user saying “OK Assistant 1” can be encrypted using utterance features generated from a portion of the audio data including the user saying “OK Assistant 1” such that only a first automated assistant (e.g., an automated assistant invoked with “OK Assistant 1”) that has access to the speaker features generated from the user saying “OKAssistant 1” can decrypt the audio data.

[0029] Figure 1 The automated assistant client 120 in is illustrated as including an automatic speech recognition (ASR) engine 122, a natural language understanding (NLU) engine 124, a text-to-speech (TTS) engine 126, and a fulfillment engine 128. In some implementations, one or more of the illustrated engines may be omitted (e.g., instead implemented solely by the cloud-based automated assistant component 140) and / or additional engines may be provided (e.g., the invocation engine described above).

[0030] The ASR engine 122 may process audio data capturing a spoken utterance to generate a recognition of the spoken utterance. For example, the ASR engine 122 may utilize one or more ASR machine learning models to process the audio data to generate a prediction of the recognized text corresponding to the utterance. In some of those implementations, the ASR engine 122 may generate a corresponding confidence metric for each of the one or more recognized terms indicating a confidence that the predicted term corresponds to the spoken utterance.

[0031] TTS engine 126 can convert text into synthesized speech, and can rely on one or more speech synthesis neural network models to do so. For example, TTS engine 126 can be used to convert a text response into audio data that includes a synthesized version of the text, and the synthesized version can be rendered in an audible manner via a hardware speaker of assistant device 100.

[0032] The NLU engine 124 determines the semantic meaning of the audio and / or the text converted from the audio by the ASR engine, and determines assistant actions corresponding to those semantic meanings. In some implementations, the NLU engine 124 determines the assistant action as the intent and / or parameter determined based on the recognition of the ASR engine 122. In some cases, the NLU engine 124 can parse the intent and / or parameter based on a single utterance of the user, and in other cases, prompts can be generated based on unresolved intent and / or parameters, those prompts are rendered to the user, and the NLU engine 124 uses the user response to those prompts to parse the intent and / or parameters. In those cases, the NLU engine 124 can optionally work in conjunction with a dialog manager engine (not illustrated) that determines unresolved intent and / or parameters and / or generates corresponding prompts. The NLU engine 124 can use one or more NLU machine learning models to determine intent and / or parameters.

[0033] Fulfillment engine 128 may cause execution of an assistant action determined by NLU engine 124. For example, if NLU engine 124 determines that the assistant action is "turning on the kitchen lights," fulfillment engine 128 may cause corresponding data to be transmitted (either directly to the lights or to a remote server associated with the manufacturer of the lights) to cause the "kitchen lights" to be "turned on." As another example, if NLU engine 124 determines that the assistant action is "provide a summary of the user's meetings for today," fulfillment engine 128 may access the user's calendar, summarize the user's meetings for the day, and cause the summary to be rendered at assistant device 100 in a visual and / or audible manner.

[0034] Device metrics generator 140 may generate metrics based on events that occur while a user is interacting with automated assistant 120. Device metrics may be device-specific metrics that indicate the timing of events performed by one or more components of client device 100. For example, a user may utter the words “OK Assistant, send a message to Bob”. A first event may indicate the time at which audio data processing is started by TTS engine 126. Additionally, device metrics generator 140 may generate another event to indicate that an invocation of automated assistant 120 has been detected (e.g., determining an indication of an event that a user invoked the automated assistant by saying “OKAssistant”). Additionally, device metrics generator 140 may generate another metric to indicate that a user uttered a request (e.g., “send a message to Bob”). Thus, for each action performed by one or more components of client device 100, an event may be generated that indicates the time of the event.

[0035] In some implementations, any of the components of automated assistant 120, software executing on client device 100, and / or hardware components of client device 100 can generate metrics that can be included in device metrics. For example, a speech detection component can generate an event when the start of speech is detected, an NLU engine 124 can generate an event when the meaning of the speech has been determined (e.g., a determination that the speech includes a request), a TTS engine 126 can generate an event indicating the start and / or end of speech-to-text conversion, and / or any other component that performs one or more actions. Additionally, device metric generator 140 can generate metrics indicating events performed by one or more other applications. For example, a metric can be generated when automated assistant 120 transmits a request to application 130, another metric can be generated when application 130 responds to the request, another metric can be generated when application 130 renders a graphical element, and so on.

[0036] refer to Figure 3 , provides a timing diagram illustrating metrics that may be generated by device metric generator 140. As illustrated, a user has spoken the utterance "OK Assistant, navigate to the office". The utterance includes an invocation 300 (i.e., "OK Assistant") and a request 305 (i.e., "navigate to the office"). In response, automated assistant 120 may generate text from the audio data, determine that the invocation phrase has been spoken, and begin processing request 305.

[0037] Once the call has been identified, the request 305 is provided to an application, such as application 130. The application may perform processing of the request and further provide a response to the request. As illustrated, the response includes providing a voice 310 indicating the action being taken (e.g., "Navigating to the office"), and a graphical display 315 of a map application. Thus, the user is provided with a response to the request 305 from application 130, causing application 130 to render a visual representation of a map that can be used for navigation on a graphical interface of client device 100 according to the request.

[0038] The device metric generator 130 can generate metrics for each event that occurs during a user's interaction with the automated assistant 120. The device action timeline 320 indicates multiple metrics generated in response to the user's interaction with the automated assistant and the actions performed by the application 130. As illustrated, the first metric "Voice Detected" indicates the time when the device's processor first detected that the user was speaking. Next, the "Voice Processed" metric indicates that the processor of the client device 100 has processed the first portion of the audio data to determine what the user said. Subsequently, once processed, the automated assistant 120 can determine that "Call Detected" and further continue to process subsequent audio data. Finally, at the "Request Sent" metric, the request is provided to the application 130, which can further process the request. In response, the application 130 provides audio 310 indicating "Navigating to the Office" and causes the rendering 315 of a graphical interface.

[0039] Device metrics such as those illustrated on timeline 320 may be provided to server 150 for further processing. Transformation mapping module 190 may determine one or more transformation mappings that may be used to generate standardized metrics from the device metrics. For example, referring again to Figure 3 , the normalized metric timeline 325 includes an “actual speech start” metric that is not included in the device metric timeline 320. The conversion mapping module 190 may determine the length of the delay based on the hardware and / or software of the client device 100. For example, the client device 100 may provide device characteristics such as the hardware type, operating system, and / or other software currently executing on the device to the server 150, and the conversion mapping module 190 may determine the conversion mapping based on the specific configuration of the client device 100.

[0040] As another example, the standardized metric timeline 325 includes a standardized metric of "request received," which may indicate the time when the request 305 is received from the user, as well as any delays that may be caused by processing the request and / or transmitting information between components. In addition, the standardized metric timeline 325 includes an "automated assistant response time," which indicates the time between determining that the spoken speech includes a request and providing the request to the application 130. Finally, an "application processing time" metric is determined based on the time it takes the application 130 to respond to the provided request.

[0041] In some implementations, the normalized metrics can be determined based on events not included in the device metrics. Figure 3 , "speech start" is not included in the device metrics, but is determined based on the conversion map. Therefore, the determined delay between the first recognition of speech and the actual speech start is determined by the conversion map and depends on the characteristics of the device, such as the microphone, the speech processing speed, and / or other factors that must be considered before the delay between the actual speech start and the detected speech start can be determined. In some implementations, one or more standardized metrics can be determined based on multiple device metrics. For example, "application processing time" can be determined based on metrics indicating when a request is provided to the application 130 (e.g., "request sent") and when the application 130 first provides a response (e.g., "application response").

[0042] Analysis engine 170 can determine one or more analysis metrics based on standardized metrics, for example, which developers can use to determine the performance of a device in a standardized manner. In some implementations, metrics from multiple devices can be aggregated to generate metrics that indicate the behavior of automated assistant 120 and / or other applications 130 on different devices. Therefore, standardized metrics can be compared to each other in a standardized form, which is not possible when only device metrics are used. For example, delays introduced based on hardware differences can be removed from application response times, so that only delays in application responses caused by the application can be examined and / or further analyzed.

[0043] refer to Figure 2, an environment illustrating multiple client devices 205 and 210 is provided, each client device including different hardware and / or software components. The first client device 205 includes an operating system 225 and a hardware component 230, while the second client device 210 includes an operating system 235 and a hardware component 240. As previously described, the device metric generator 215 can generate device metrics for the first client device 205, and the device metric generator 220 can generate device metrics for the second client device 210. In some implementations, once each of the device metric generators has generated device metrics, the metrics can be provided to the server 150 for further analysis.

[0044] In some implementations, at least a portion of the conversion of the device metric to the standardized metric may be performed by the client device. For example, in some implementations, at least a portion of the actions performed by the conversion mapping module 160 and / or the standard metric generator 160 may be performed by the client device. In some implementations, the client device may provide device characteristics to the server 150 to indicate the hardware and / or software components present on the device. For example, the first client device 205 may provide the device metric and the indication of the operating system 225 and / or the hardware component 230 to the server 150. In response, the conversion mapping module 160 may determine the conversion mapping based on the indication of the operating system 225 and / or the hardware component 230, and the standard metric generator 160 may determine the standardized metric based on the received device metric and the determined conversion mapping. In some implementations, at least a portion of the actions performed by the standard metric generator 160 and / or the conversion mapping module 160 may be performed by another remote computing device. For example, the first client device 205 and the second client device 210 can provide the device metrics to the second remote computing device, which can determine the conversion mapping and further generate the standardized metrics. The standardized metrics can then be provided to the server 150, which can perform further analysis to generate analytical metrics.

[0045] In some implementations, the first client device 205 and / or the second client device 210 may provide additional device characteristics that may be utilized by the analysis engine 170 when generating analysis metrics. For example, a user of the first client device 205 may set the language setting of the first client device 205 to a first language, and a user of the second client device 210 may set the language preference setting to a second language. In response, the analysis engine 170 may determine whether the standardized metrics from the two devices should be compared. In the case where the language preference settings are the same, the analysis metrics generated for the devices may be more meaningful to the developer than in the case where the language preferences are different. As another example, in some implementations, the first client device 205 and / or the second client device 210 may provide geographic location and / or other device information that may be utilized by the analysis engine 170 when generating analysis metrics for the devices.

[0046] In some implementations, a user can interact with instances of an automated assistant on multiple devices. For example, a user can begin interacting with an instance of an automated assistant executing on a first client device 205, which can be a smart speaker located at a first location. Subsequently, the user can continue the same interaction with a second instance of the automated assistant on a second client device 210, which can be executed on a smart phone located at a second location. Thus, again referring to Figure 3 , at least a portion of the device metrics 320 may be generated by the first client device 205, and another portion of the device metrics 320 may be generated by the second client device 210. In this case, the two client devices 205 and 210 may have different hardware and / or software configurations, and in order to compare the device metrics with each other or with the metrics of another device, the metrics generated by each device may require a separate conversion map. Thus, the first client device 205 may generate metrics that are converted to standardized metrics via a first conversion map, and the second client device 210 may generate metrics that are converted to standardized metrics via a second conversion map.

[0047] Reference again Figure 1 , the interface renderer 180 may generate a graphical interface indicating one or more of the analysis metrics generated by the analysis engine 170. For example, the interface renderer 180 may generate a graphical interface indicating one or more of the analysis metrics generated by the analysis engine 170. Figure 3, to provide a visual aid when analyzing the resulting analytical metrics. In some implementations, one or more of the analytical metrics provided via the graphical interface can be rendered so that the metrics of most interest to the developer are rendered differently than other metrics. For example, for a given metric, a baseline standard can be identified. In the event that a device (or an aggregated metric across multiple devices) does not meet the baseline standard, the metric can be rendered in a different color and / or rendered more prominently than other metrics.

[0048] In some implementations, analysis engine 170 can generate analysis metrics using standardized metrics generated from device metrics of first client device 205 and standardized metrics generated from device metrics of second client device 210 that indicate, for example, the behavior of an instance of an automated assistant executing on each of the devices. For example, the device metrics of first client device 205 can depend at least in part on operating system 225 and / or hardware components 210. Thus, given the number of differences that may exist between different software and / or hardware on the devices, it may not be possible to compare the behavior of an automated assistant to the behavior of an automated assistant being executed on second client device 210. By generating standardized metrics and then comparing the standardized metrics from two devices, developers can be provided with better analysis tools to determine how hardware and / or software affects the performance of an application.

[0049] refer to Figure 4 , a flowchart illustrating a method for processing standardized metrics generated from device metrics is provided. In some implementations, one or more steps of the method may be omitted or combined, and / or one or more additional steps may be included in the method. The method may be Figure 1 One or more components instantiated in an environment are executed.

[0050] At step 405, device metrics are identified. The device metrics are generated by a client device, such as client device 100, via a component that shares one or more characteristics with device metrics generator 140. In some implementations, the metrics are generated during an interaction between a user and an automated assistant executing on the client device. The metrics can indicate the timing of events that occurred during the interaction, such as an indication of a response by the automated assistant, one or more applications, and / or a start time and / or stop time of processing input. For example, the device metrics can include an event to indicate when processing of audio data has begun, and a second device metric can be generated to indicate a time when processing of the audio data has ended.

[0051] At step 410, a plurality of transformation mappings are determined. The plurality of transformation mappings may be determined by a component that shares one or more characteristics with the transformation mapping module 190. For example, once the device metrics generator 140 of the client device 100 generates the device metrics, the device metrics may be provided to the server 150 for further processing. In some implementations, an indication of the transformation mappings and / or information related to the hardware and / or software capabilities and / or configuration of the client device 100 may be provided to the transformation mapping module 190 so that the transformation mapping module 190 may determine the transformation mapping based on the device characteristics.

[0052] At step 415, a standardized metric is generated. The standardized metric is generated by transforming the device metric using the determined transformation map. For example, the transformation map may indicate that events are adjusted and / or new events are generated to be included in the standardized metric based on the processing capabilities of the client device 100. Thus, in some cases, the transformation map may change the time of one or more events included in the device metric. In some cases, for example, the transformation map may indicate the generation of additional metrics indicating events that are not included in the device metric.

[0053] At step 420, the remote computing device processes the standardized metrics to generate analytical metrics. The analytical metrics may include one or more interferences determined based on the standardized metrics. In some implementations, a remote computing device such as server 150 is unable to generate analytical metrics from device metrics. For example, without converting the device metrics to standardized metrics, one or more metrics may be incorrect, inaccurate, and / or not included, making it impossible to otherwise generate analytical metrics. In some implementations, analytical metrics are generated by a component that shares one or more characteristics with analysis engine 170.

[0054] refer to Figure 5 , a flowchart illustrating a method for processing standardized metrics from multiple devices is provided. In some implementations, one or more steps of the method may be omitted or combined, and / or one or more additional steps may be included in the method. The method may be Figure 2 One or more components instantiated in an environment are executed.

[0055] At step 505, a first device metric is received from a first client device. The client device may be associated with Figure 2 The first device metric is generated during the user's interaction with the automated assistant executing on the first client device. As previously described, the metrics can each indicate the time of an event that occurred during the interaction. For example, the first device metric can be associated with Figure 3 The metrics illustrated in and described above share one or more characteristics.

[0056] At step 510, a second device metric is received from a second client device. The second client device may be connected to Figure 2 The first client device and the second client device 210 share one or more characteristics. For example, the first client device and the second client device may have different capabilities, different software, different operating systems, different hardware, and / or have different applications installed and / or executing when the metrics are generated. Therefore, the first device metrics and the second device metrics are not standardized across the devices and cannot be meaningfully compared to generate analytical metrics.

[0057] At step 515, a plurality of first conversion mappings are determined using the one or more first characteristics. The first conversion mappings are based on, for example, hardware and / or software of the first client device and may be unique to the device. At step 520, a plurality of second conversion mappings are determined using the one or more second characteristics. The second conversion mappings are based on, for example, hardware and / or software of the second client device and may be different from the first conversion mappings.

[0058] At step 525, a first normalized metric is generated from the first device metric using the first transformation map. At step 530, a second normalized metric is generated from the second device metric. In both cases, the device-specific metrics are transformed so that the two sets of metrics can be further analyzed in a more meaningful manner than would be possible in performing analysis (e.g., comparison, aggregation) on the device-specific metrics.

[0059] At step 535, at least a portion of the first standardized metric is compared to at least a portion of the second standardized metric. The comparison may be performed by a component that shares one or more characteristics with the analysis engine 170. At step 540, an analysis metric is generated based on the comparison. As previously described, due to differences in device characteristics, it may not be possible to generate a resulting analysis metric from the first device metric and the second device metric. For example, the first device and the second device may have different hardware components, each of which introduces an amount of delay in one or more operations, thereby biasing one or more metrics. By normalizing the device metrics prior to comparison, delays from hardware and / or software components may be compensated for, making the resulting analysis metric more meaningful and accurate to the developer.

[0060] In some implementations, interactions with an automated assistant may occur on multiple devices. For example, in some cases, a user may begin an interaction with an automated assistant via a smart phone, continue the interaction on a smart speaker, and end the interaction on another device integrated into a vehicle. In some implementations, each of these devices may have different hardware and / or software for executing the automated assistant and / or one or more other applications. Thus, for each portion of the interaction, a different device-specific metric may be identified. In some implementations, for each of the device-specific metrics utilized during the interaction, a conversion mapping may be identified for each set of metrics. Thus, for an interaction, multiple conversion mappings may be utilized to convert the metrics of the interaction into standardized metrics.

[0061] Figure 6 6 is a block diagram of an example computing device 610 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 610 generally includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow for user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0062] The user interface input device 622 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 610 or onto a communication network.

[0063] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0064] The storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include a computer program for executing Figure 3 A to Figure 4 Selected aspects of the method of B and / or for implementing Figure 1 The logic of the various components depicted in .

[0065] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 626 in the storage subsystem 624, or in other machines accessible to the processor 614.

[0066] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0067] The computing device 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for illustrating some implementations. Many other configurations of computing device 610 are possible, and these configurations are similar to those of FIG. Figure 6 The computing devices depicted in FIG. 1 may have more or fewer components than those depicted in FIG.

[0068] Implementations disclosed herein include a method comprising: identifying device metrics generated during an interaction between a user and an automated assistant being executed on a client device, wherein the device metrics indicate the timing of events that occurred during the interaction; determining a plurality of transformation mappings based on one or more characteristics of the client device; generating standardized metrics, wherein generating the standardized metrics comprises converting one or more metrics in the device metrics into one or more standard metrics using the transformation mappings; and causing a remote computing device to process the standardized metrics when generating analytical metrics, wherein the remote computing device is unable to generate the analytical metrics from the device metrics.

[0069] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0070] In some implementations, generating the normalized metric is performed by the client device, and causing the remote computing device to process the normalized metric in generating the analytical metric includes transmitting, by the client device, the normalized metric to the remote computing device.

[0071] In some implementations, generating the standardized metrics is performed by a second remote computing device, and the method further includes: transmitting the device metrics by the client device to the second remote computing device, and causing the remote computing device to process the standardized metrics when generating the analytical metrics including transmitting the standardized metrics by the second remote computing device to the remote computing device.

[0072] In some implementations, when generating the analytical metrics, the remote computing device compares the standardized metrics to the second standardized metrics. In some of those implementations, the method further includes: identifying second device metrics generated during an interaction between the additional user and a second automated assistant being executed on a second client device; determining a plurality of second transformation mappings based on one or more characteristics of the second client device; and generating the second standardized metrics, wherein generating the second standardized metrics includes transforming one or more of the metrics in the second device metrics using the second transformation mappings.

[0073] In some implementations, the client device further executes an application, and wherein at least a given one of the device metrics is based on a time of one or more events related to the application.

[0074] In some implementations, the one or more characteristics of the client device include operating system characteristics based on an operating system executing on the client device.

[0075] In some implementations, the one or more characteristics of the client device include hardware characteristics of one or more hardware components of the client device.

[0076] In some implementations, a given standardized metric in the standardized metrics indicates a time of a given event that is not included in the events indicated by the device metrics. In some of those implementations, generating the standardized metric includes inferring the given standardized metric based on one or more device metrics in the device metrics using a given transformation map in the transformation maps. In other implementations, the given transformation map is generated based on a temporal change caused by a component and based on a determination that the client device includes the component. In yet other of those implementations, the given event is a response from an application to a request included in a spoken utterance processed by an automated assistant.

[0077] In another aspect, an implementation disclosed herein includes another method, comprising: receiving, from a first client device, first device metrics generated during a first interaction between a first user and a first instance of an automated assistant being executed on the first client device, wherein the first device metrics indicate the time of events that occurred during the first interaction; receiving, from a second client device, second device metrics generated during a second interaction between a second user and a second instance of an automated assistant being executed on the second client device, wherein the second device metrics indicate the time of events that occurred during the second interaction; determining a plurality of first transformation mappings based on one or more first characteristics of the first client device; determining a plurality of second transformation mappings based on one or more second characteristics of the second client device; generating a first standardized metric, wherein generating the first standardized metric includes converting one or more of the first device metrics into one or more standard metrics using the first transformation mapping; generating a second standardized metric, wherein generating the second standardized metric includes converting one or more of the second device metrics into one or more standard metrics using the second transformation mapping; comparing at least one of the first standardized metrics with at least one of the second standardized metrics; and generating an analytical metric based on the comparison.

[0078] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0079] In some implementations, at least one of the characteristics of the first client device is different than at least one of the characteristics of the second client device.

[0080] In some implementations, the method further includes: receiving additional first device characteristics from the first client device; receiving additional second device characteristics from the second client device; and before comparing at least one of the first device metrics to at least one of the second device metrics: determining that at least one of the additional first device characteristics is similar to at least one of the additional second device characteristics. In some of those implementations, the additional first device characteristics include a geographic location of the first client device. In other of those implementations, the additional first device characteristics include a language preference setting of the first client device.

[0081] In some implementations, the method further includes rendering an interface indicating one or more of the standardized metrics. In some of those implementations, the method further includes identifying a benchmark standard for the analytical metric of interest; and based on whether the analytical metric of interest meets the benchmark standard, visually rendering the analytical metric of interest.

[0082] Various implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described herein. Other implementations may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component) that includes a processor that is operable to execute stored instructions to perform a method such as one or more of the methods described herein. Still other implementations may include a system of one or more servers that include one or more processors that are operable to execute stored instructions to perform a method such as one or more of the methods described herein.

[0083] Where certain implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user to collect, store, and / or use the user's personal information.

[0084] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information associated with that user, providing permission or authorization as to whether to collect information and as to which portions of the information are to be collected. For example, one or more such control options may be provided to a user via a communications network. In addition, specific data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

Claims

1. A method performed by one or more processors, the method comprising: identifying device metrics generated during an interaction between a user and an automated assistant executing on a client device, wherein the device metrics indicate timing of events occurring during the interaction; determining a plurality of conversion mappings based on one or more characteristics of the client device; generating a standardized metric, wherein generating the standardized metric comprises converting one or more of the device metrics into one or more standard metrics using the conversion map; and A remote computing device is caused to process the normalized metrics when generating analytical metrics, wherein the remote computing device is unable to generate the analytical metrics from the device metrics.

2. The method of claim 1, wherein: Generating the normalized metric is performed by the client device, and wherein causing the remote computing device to process the normalized metric when generating the analytical metric comprises: The normalized metric is transmitted by the client device to the remote computing device.

3. The method according to any one of claims 1 and 2, wherein: Generating the normalized metric is performed by a second remote computing device, and the method further comprises: transmitting, by the client device, the device metrics to the second remote computing device; Wherein causing the remote computing device to process the standardized metric when generating the analytical metric comprises: The normalized metric is transmitted by the second remote computing device to the remote computing device.

4. A method as claimed in any preceding claim, wherein: In generating the analytical metric, the remote computing device compares the normalized metric to a second normalized metric.

5. The method of claim 4, further comprising: identifying second device metrics generated during an interaction between an additional user and a second automated assistant executing on a second client device; determining a plurality of second translation mappings based on one or more characteristics of the second client device; as well as The second normalized metric is generated, wherein generating the second normalized metric comprises transforming one or more of the metrics in the second device metric using the second transformation map.

6. A method as claimed in any preceding claim, wherein: The client device is further executing an application, and wherein at least a given one of the device metrics is based on the timing of one or more events related to the application.

7. A method as claimed in any preceding claim, wherein: The one or more characteristics of the client device include operating system characteristics based on an operating system executing on the client device.

8. A method as claimed in any preceding claim, wherein: The one or more characteristics of the client device include hardware characteristics of one or more hardware components of the client device.

9. A method as claimed in any preceding claim, wherein: A given one of the normalized metrics indicates a time of a given event that is not included among the events indicated by the device metrics.

10. The method of claim 9, wherein: Generating the normalized metric includes inferring the given normalized metric based on one or more of the device metrics using a given one of the transformation maps.

11. The method of claim 10, further comprising: The given transformation map is generated based on temporal changes caused by a component and based on a determination that the client device includes the component.

12. A method as claimed in any preceding claim, wherein: The given event is a response from an application to a request included in a spoken utterance processed by the automated assistant.

13. A method performed by one or more processors, the method comprising: receiving, from a first client device, first device metrics generated during a first interaction between a first user and a first instance of an automated assistant executing on the first client device, wherein the first device metrics indicate timing of events occurring during the first interaction; receiving, from a second client device, second device metrics generated during a second interaction between a second user and a second instance of the automated assistant executing on the second client device, wherein the second device metrics indicate times of events occurring during the second interaction; determining a plurality of first conversion mappings based on one or more first characteristics of the first client device; determining a plurality of second conversion mappings based on one or more second characteristics of the second client device; generating a first standardized metric, wherein generating the first standardized metric comprises converting one or more of the first device metrics into one or more standard metrics using the first conversion mapping; generating a second standardized metric, wherein generating the second standardized metric comprises converting one or more of the second device metrics to one or more of the standard metrics using the second conversion mapping; comparing at least one of the first normalized metrics to at least one of the second normalized metrics; and An analytical metric is generated based on the comparison.

14. The method of claim 13, wherein: At least one of the characteristics of the first client device is different from at least one of the characteristics of the second client device.

15. The method of any one of claims 13 and 14, further comprising: receiving additional first device characteristics from the first client device; receiving additional second device characteristics from the second client device; as well as Prior to comparing at least one of the first device metrics to at least one of the second device metrics: At least one of the additional first device characteristics is determined to be similar to at least one of the additional second device characteristics.

16. The method of claim 15, wherein: The additional first device characteristics include a geographic location of the first client device.

17. The method of claim 15, wherein: The additional first device characteristics include a language preference setting for the first client device.

18. The method of any one of claims 13 to 17, further comprising: An interface is rendered that indicates one or more of the normalized metrics.

19. The method of claim 18, further comprising: For the analytical metrics of interest, identify the benchmark standards; as well as The analytical metric of interest is visually rendered based on whether the analytical metric of interest meets the benchmark criterion.

20. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any preceding claim.

21. One or more computing devices configured to perform the method of any one of claims 1 to 19.

Images