Systems and methods for generating remediation recommendations for power and performance issues in semiconductor software and hardware

Abstract

A computer-implemented method is disclosed for generating remediation recommendations for power and performance issues in semiconductor software and hardware. For example, the disclosed system and method can apply rule-based models to telemetry data to generate rule-based root cause outputs and telemetry-based unknown outputs. Furthermore, the disclosed system and method can apply root cause machine learning models to the telemetry-based unknown outputs, analyze severe and complex failure patterns using the telemetry-based unknown outputs, and ultimately generate one or more root cause remediation recommendations specific to the identified failures and the client computing device experiencing the failures.

Description

[Background technology]

[0001] Exemplary automated analysis systems often perform rule-based analysis of telemetric data associated with semiconductor software and / or hardware failures, performance, and power issues.

[0002] The accompanying drawings, which form a part of this specification, illustrate several exemplary embodiments and, together with the following description, illustrate and explain various principles of the present disclosure. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a flow diagram of an example computer-implemented method for generating remedy recommendations for power and performance issues in semiconductor software and hardware, according to one or more embodiments. [Figure 2] FIG. 1 is a schematic sequence diagram of a remedy recommendation system that generates device-specific remedy recommendations that address one or more root causes of a fault, power, or performance issue, according to one or more embodiments. [Figure 3] FIG. 1 is a detailed sequence diagram of a remediation recommendation system for generating device-specific remediation recommendations according to one or more embodiments. [Figure 4] FIG. 2 is a detailed diagram of a remediation recommendation system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] Throughout the drawings, like reference numerals and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are herein described in detail. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0005] The present disclosure is generally directed to systems and methods for generating remediation recommendations for power and performance issues in semiconductor software and hardware. As described above, an exemplary analysis system may use a rules-based approach when analyzing telemetric data from one or more telemetric sources associated with a computing device. Such telemetric data may often be complex and / or meaningless (e.g., "noisy"). An exemplary analysis system generally cannot recognize such complex and noisy data patterns in relation to failures, performance, and power issues associated with semiconductor software and hardware. Furthermore, an exemplary analysis system is often strictly limited to expected telemetric data patterns and progressions. Thus, when unexpected telemetrics are received, an exemplary analysis system generally generates an output that requires additional manual analysis.

[0006] More specifically, exemplary analysis systems often employ rigorous, rule-based analysis of telemetric data when attempting to determine the root causes of various types of system failures. However, such rule-based approaches are effective only when the telemetric data falls into expected formats and fairly simple patterns. Thus, exemplary analysis systems are often limited to high-level problem analysis. Nevertheless, as noted above, the root causes of some system failures and other problems can only be determined upon deep analysis of complex patterns within noisy telemetric data. With this in mind, exemplary analysis systems may fail to improve as much as 50% of reported system failures.

[0007] To remedy these and other problems, the systems and methods described herein efficiently generate accurate, device-specific remediation solutions for faults, performance, and power issues indicated by complex and potentially noisy telemetric data. For example, the systems and methods described herein can receive and transform telemetric data from a variety of telemetric sources based on both rule-based and machine-learning-based models. The systems and methods described herein can further apply these models to the transformed telemetric data to generate accurate predictions that indicate the root causes of problems experienced by semiconductor software and / or hardware. Additionally, the systems and methods described herein can further generate remediation recommendations based on the indicated root causes that end users can apply to their client computing devices to resolve these issues.

[0008] Thus, the systems and methods described herein provide accurate, efficient, flexible, and resilient solutions to problems experienced by exemplary analysis systems. For example, as described above, the exemplary systems inflexibly require that telemetric data be received in expected formats and fairly simple patterns. Therefore, when such exemplary systems receive complex, noisy data, they are unable to generate meaningful, accurate root cause predictions. In contrast, the systems and methods described herein include a resilient data pipeline that can receive complex, noisy, or unexpected telemetric data and transform it into a usable format. By applying both rule-based and machine-learning-based models to such transformed telemetric data in the resilient data pipeline, the systems and methods described herein can analyze severe failure patterns and generate accurate root cause predictions. The systems and methods described herein can further generate accurate remediation recommendations from these predictions that are tailored to end-user computing devices to remedy failures, performance, and power issues experienced by the computing devices.

[0009] As described in more detail below, this disclosure describes various systems and methods for generating accurate, device-specific remediation measures for faults, performance, and power issues. In one embodiment, a method for generating device-specific remediation measures for faults, performance, and power issues can include generating rule-based root cause outputs and telemetry-based unknown outputs from telemetry data of a client computing device, and generating machine-learning-based root cause predictions by applying a root cause machine learning model to the telemetry-based unknown outputs. The method can further include generating one or more root cause recommendations that are specific to the client computing device and based on the rule-based root cause outputs and the machine-learning-based root cause predictions.

[0010] In one or more embodiments, the method may further include receiving telemetry data from the client computing device and parsing the telemetry data based on the rule-based model and the root cause machine learning model prior to generating the rule-based root cause output and the telemetry-based unknown output. In some embodiments, receiving telemetry data from the client computing device may include receiving raw, unstructured data from one or more of an offline log and a real-time telemetry source.

[0011] Also, in some embodiments, parsing the telemetry data based on the rule-based model and the root cause machine learning model can include parsing the telemetry data into structured data based on one or more of offline logs and real-time telemetry sources, parsing the structured data into rule-based model-specific attributes, and parsing the structured data into root cause machine learning model-specific attributes. Furthermore, parsing the telemetry data based on the rule-based model and the root cause machine learning model can further include generating one or more of engineering features, data encoding, customized mappings, and data augmentations based on the rule-based model-specific attributes and the root cause machine learning model-specific attributes.

[0012] In one or more embodiments, the method may also include modifying the rule-based root cause output in light of the machine learning-based root cause prediction, and generating client-specific root cause results from the machine learning-based root cause prediction and the modified rule-based root cause output. In some embodiments, one or more root cause recommendations specific to the client computing device are generated by applying a recommendation engine to the client-specific root cause results, wherein the recommendation engine may include a plurality of trained machine learning-based recommendation units. In some embodiments, the recommendation engine may further include one or more of a lookup table or a rule-based recommendation.

[0013] In at least one embodiment, the method may also include generating filtered remediation recommendations tailored to a user security level associated with the client computing device based on one or more root cause recommendations specific to the client computing device and end-user data extracted from telemetry data from the client computing device. Additionally, in at least one embodiment, the method may further include generating machine learning-based unknowns by applying the root cause machine learning model to the telemetry-based unknown outputs, and retraining the root cause machine learning model based on the machine learning-based unknown outputs.

[0014] In an example embodiment, a system for generating device-specific remedies for faults, performance, and power issues may include at least one processor and a physical memory including computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including generating rule-based root cause predictions and telemetry-based unknown outputs from telemetry data from a client computing device; generating machine-learning-based root cause predictions by applying a root cause machine learning model to the telemetry-based unknown outputs; and generating one or more root cause recommendations that are specific to the client computing device and that are based on the rule-based root cause outputs and the machine-learning-based root cause predictions.

[0015] In some example embodiments, the above-described methods may be encoded as computer-readable instructions on a non-transitory computer-readable storage medium. For example, the computer-readable storage medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to generate rule-based root cause outputs and telemetry-based unknown outputs from telemetry data from a client computing device, generate machine-learning-based root cause predictions by applying a root cause machine learning model to the telemetry-based unknown outputs, and generate one or more root cause recommendations that are specific to the client computing device and that are based on the rule-based root cause outputs and the machine-learning-based root cause predictions.

[0016] The features of any embodiment described herein may be used in combination with each other in accordance with the general principles described herein. These and other embodiments, features, and advantages will be more fully understood from the following detailed description read in conjunction with the accompanying drawings and claims.

[0017]

[0023] Below, with reference to Figures 1-3, a detailed description of an exemplary system and method for generating resilient and device-specific remediation recommendations for fault, performance, and power issues is provided. For example, Figure 1 provides a detailed description of a corresponding computer-implemented method therefor. Also, Figure 2 shows an exemplary schematic sequence diagram of a remediation recommendation system for generating device-specific remediation recommendations that address one or more root causes of fault, power, or performance issues indicated by telemetry data associated with a client computing device. Furthermore, Figure 3 shows an exemplary detailed sequence diagram of the same process. Finally, Figure 4 shows a detailed diagram of the system and method described herein.

[0018] Figure 1 is a flow diagram of an example computer-implemented method 100 for generating resilient and device-specific improvement recommendations that address the root causes of failures, performance, and power issues. The steps illustrated in Figure 1 may be performed by any suitable computer-executable code and / or computer hardware. In one example, each of the steps illustrated in Figure 1 may represent an algorithm whose structure includes and / or is represented by multiple sub-steps, examples of which are provided in more detail below.

[0019] 1 , in step 102, the remediation recommendation system can generate rule-based root cause outputs and telemetry-based unknown outputs from telemetry data of the client computing device. For example, the remediation recommendation system can receive telemetry data from any of a number of telemetric sources (e.g., including the client computing device and other telemetric sources associated with the client computing device), such as offline logs and / or other real-time sources. In one or more embodiments, the remediation recommendation system can parse, process, and otherwise transform the received telemetry data based on one or more models. For example, the remediation recommendation system can parse the telemetry data according to a rule-based model and according to a root cause machine learning-based model.

[0020] In at least one embodiment, the remediation recommendation system may apply a rule-based model to the parsed telemetry data to generate a rule-based root cause output and a telemetry-based unknown output. For example, the rule-based model may include one or more of rules, heuristics, lookup tables, decision trees, etc. Thus, in at least one embodiment, the rule-based model generates the rule-based root cause output by applying the rule-based model to the parsed telemetry data that includes known or expected patterns, values, characteristics, attributes, etc. Thus, in some embodiments, the telemetry-based unknown output may include remaining telemetry data that potentially includes complex patterns and / or noisy data that the rule-based model was unable to process.

[0021] Also, in step 104, the remediation recommendation system may generate a machine learning-based root cause prediction by applying a root cause machine learning model to the unknown telemetry-based output. For example, in at least one embodiment, the remediation recommendation system may apply a root cause machine learning model to the unknown telemetry-based output to generate a machine learning-based root cause prediction. In one or more embodiments, the remediation recommendation system may train the root cause machine learning model to predict the root causes of faults, performance, and power issues based on telemetry data that cannot be analyzed by rule-based models. Thus, the root cause machine learning model can understand severe fault patterns presented by telemetric data that were previously impossible to analyze under rule-based approaches to predict the root causes of various types of faults.

[0022] 1 , in step 106, the remediation recommendation system may generate one or more root cause remediation recommendations that are specific to the client computing device and that are based on the rule-based root cause output and the machine learning-based root cause prediction. For example, the remediation recommendation system may apply a recommendation engine to the root causes indicated by the rule-based root cause output and the machine learning-based root cause prediction to generate at least one root cause remediation recommendation that is specific to the client computing device. In one or more embodiments, the remediation recommendations generated by the recommendation engine may include instructions specific to the client computing device, for example, regarding software updates, disabling problematic system features, enabling various system features, modifying or reconfiguring certain software or hardware features, etc.

[0023] In one or more embodiments, the remediation recommendation system described herein references a number of terms and phrases. For example, the term "root cause" can refer to an action performed by a client computing device that causes a problem. By way of example, the root cause of a software failure problem can include a specific erroneous instruction executed by the client computing device. In some embodiments, the root cause can also include actions performed by the client computing device that do not cause the problem, but that cause other actions to be performed that cause the problem (e.g., a cascading failure).

[0024] As used herein, the term "model" may refer to any group of computer instructions that, when applied to one or more inputs, produces one or more outputs. For example, a "rule-based model" may apply one or more rule-based heuristics (e.g., decision trees, if / then tables, graphs) that, when applied to one or more inputs containing structured data in an expected format, produce one or more outputs. Additionally, a "machine learning-based model" or a "root cause machine learning model" may refer to a layer of weighted nodes trained to generate predictions with an acceptable confidence level when applied to one or more input vectors. A machine learning model may include one or more of a convolutional neural network, a recurrent neural network, a feedforward neural network, or a perceptron neural network. In further embodiments, a machine learning model may include other types of artificial networks.

[0025] As used herein, "telemetry data" may refer to a collection of observational data values ​​associated with the functionality of a client computing device. For example, telemetry data may include register values, stack values, accelerator measurements (e.g., thermal measurements, clock speed measurements), log values ​​(e.g., debug statements, software write output), etc. Telemetry data may be collected at one or more telemetry sources. For example, telemetry data may be collected by one or more components of the client computing device (e.g., installed software) or by a third-party telemetry source (e.g., a service that monitors the client computing device or receives monitoring statements from the client computing device). Telemetry data may be collected, transmitted, and / or received in real time and after the fact (e.g., as with telemetry data from offline logs).

[0026] As used herein, a "remediation recommendation" may refer to a generated output that includes information associated with the root cause of a problem experienced by a client computing device. For example, a remediation recommendation may include a complete description of the problem and the root cause of the problem. The remediation recommendation may further include instructions for resolving the root cause of the problem. For example, a remediation recommendation may include instructions for permanently resolving the problem (e.g., "Update your gaming application to version xyz"), temporarily resolving the problem (e.g., "Increase fan speed by 70% to reduce thermal throttling"), or instructions for working around the problem. In one or more embodiments, the remediation recommendation may be formatted as an email, an SMS message, a pop-up window, or other type of notification. In some embodiments, the remediation recommendation may also include a selectable option for automatically resolving the identified problem.

[0027] As used herein, the term "user security level" may refer to a categorization associated with an end user of a client computing device. For example, the systems and methods described herein may generate remediation recommendations based on the end user's user security level, such that the remediation recommendations do not include information that is overly sensitive or inappropriate for the end user to view. An end user may have a higher user security level, such as an "internal user" or an "administrator." Conversely, an end user may have a lower user security level, such as an "external user."

[0028] As described above, the remediation recommendation system can efficiently and accurately generate remediation recommendations specific to a problem being experienced by a particular client computing device. Figure 2 shows a schematic sequence diagram of a remediation recommendation system 202 (e.g., the remediation recommendation system described above) generating a remediation recommendation 216 for a client computing device 204 based on telemetry data 206 associated with the client computing device 204.

[0029] More particularly, the remediation recommendation system 202 may receive telemetry data 206 in a data block 208. In one or more embodiments, the data block 208 may include one or more parsers, processors, translators, etc. For example, components of the data block 208 may parse the received telemetry data 206 according to an analyzer block 210. In at least one embodiment, the data block 208 may parse the telemetry data 206 according to one or more models in the analyzer block 210.

[0030] 2, the data block 208 can send the parsed telemetry data 206 to an analyzer block 210 for root cause analysis. In at least one embodiment, the analyzer block 210 can include multiple models that process some or all of the parsed telemetry data 206 in parallel or sequentially. Additionally, the analyzer block 210 can utilize the results of one model as some or all of the input to another model. In one or more embodiments, the analyzer block 210 generates client-specific root cause results based on the results, outputs, and / or predictions generated by the one or more models in the analyzer block 210.

[0031] Using the client-specific root cause results of the analyzer block 210, the remediation recommendation system 202 can generate one or more root cause remediation recommendations. For example, the remediation recommendation system 202 can apply a recommendation engine 212 to the client-specific root cause results and a subset of the parsed telemetry data generated by the data block 208. In one or more embodiments, the recommendation engine 212 can include one or more of a machine learning-based unit, a lookup table, rule-based recommendations, and other heuristics. Thus, the remediation recommendation system 202 can apply the recommendation engine 212 to the client-specific root cause results and other telemetry data to generate root cause remediation recommendations that include specific instructions to modify, configure, install, disable, enable, and / or upgrade one or more components of the client computing device 204 to remediate a particular fault, performance, and / or power issue being experienced by the client computing device 204.

[0032] In at least one embodiment, a client computing device 204 may be associated with a particular user security level. By way of example, a client computing device 204 may be associated with a high security level (e.g., similar to an end user with administrative access to the client computing device 204) or a low security level (e.g., similar to a standard non-administrative end user). Accordingly, the remediation recommendation system 202 may generate filtered remediation recommendations 216 by applying a filter engine 214 to the one or more root cause remediation recommendations generated by the recommendation engine 212.

[0033] For example, the filter engine 214 can determine a security level for the client computing device 204 based on the end-user data extracted from the telemetry data 206 by the data block 208. Based on this security level, the filter engine 214 can filter the one or more root cause remediation recommendations to include information and instructions appropriate for that security level. For example, the filter engine 214 can filter the one or more root cause remediation recommendations to include security level appropriate data, potentially including one or more of a detailed description of the failure, a failure pattern, and recommendations specific to the client computing device. The filter engine 214 can then transmit the filtered remediation recommendations 216 to the client computing device 204.

[0034] FIG. 3 illustrates a detailed diagram of the remediation recommendation system 202, which generates one or more remediation recommendations 216. For example, as shown in FIG. 3, the remediation recommendation system 202 can receive telemetry data 206 at an initial parser 302 of a data block 208. In one or more embodiments, the initial parser 302 can receive raw, unstructured data from a variety of sources, including offline logs and real-time telemetry sources. Additionally, the initial parser 302 can separate, convert, process, translate, and otherwise configure the telemetry data 206 based on the one or more sources of the telemetry data 206. Illustratively, the initial parser 302 can parse telemetry data from a particular telemetry source based on knowledge of how the telemetry source patterns and / or structures the telemetry data. Additionally, the initial parser 302 can further extract all information from the received telemetry data 206 without knowledge of the telemetry source's formatting, patterns, and / or structure. For example, the initial parser 302 may extract all information and may attempt to label and / or otherwise categorize the extracted data into a structured format usable by the rule-based parser 304, the machine learning-based parser 306, the analyzer block 210, the recommendation engine 212, and / or the filter engine 214.

[0035] In one or more embodiments, the data block 208 may further include a rule-based parser 304 and a machine learning-based parser 306. In at least one embodiment, the rule-based parser 304 and the machine learning-based parser 306 may further parse the telemetry data 206 based on one or more models in the analyzer block 210. For example, the rule-based parser 304 may further parse the output of the initial parser 302 based on the input structure of the rule-based model 310 in the analyzer block 210. Similarly, the machine learning-based parser 306 may further parse the output of the initial parser 302 based on the input structure of the root cause machine learning model 312. For example, either or both of the rule-based parser 304 and the machine learning-based parser 306 may further parse the telemetry data 206 by extracting certain attributes, features, and values ​​(numeric and categorical) according to the rule-based model 310 and the root cause machine learning model 312, respectively. In some embodiments, the rule-based parser 304 and the machine learning-based parser 306 can parse the output of the initial parser 302 in parallel. In other embodiments, the rule-based parser 304 and the machine learning-based parser 306 can operate in series or sequentially.

[0036] 3, the data block 208 can further include a data processor 308 that can receive output from the rule-based parser 304 and the machine learning-based parser 306. In one or more embodiments, the data processor 308 can generate, enumerate, and encode various data patterns (e.g., sequential patterns with timing information, logical patterns, etc.). For example, the data processor 308 can perform these tasks by generating one or more of engineering features, data encoding (e.g., via one-hot encoding), customized mapping, and / or data augmentation. In at least one embodiment, the data processor 308 can generate any of these outputs based on attributes specific to the rule-based model and / or attributes specific to the root cause machine learning model.

[0037] As described above and shown in FIG. 3 , the improvement recommendation system 202 can generate root cause predictions in the analyzer block 210. More specifically, the rule-based model 310 can receive the transformed telemetry data 206 from the data processor 308 of the data block 208. In one or more embodiments, the rule-based model 310 generates a rule-based root cause output and a telemetry-based unknown output based on the received data. For example, the rule-based model 310 can include one or more of heuristics, lookup tables, decision trees, random forests, if / then statements, etc., associated with potential root causes of various potential problems that the client computing device may experience. Thus, the rule-based model 310 can generate a rule-based root cause output by matching or otherwise correlating the received data with these heuristics, lookup tables, decision trees, random forests, if / then statements, etc. In at least one embodiment, the rule-based model 310 can generate a telemetry-based unknown output from a subset of the received data that cannot be correlated with any of the rule-based approaches embodied by the rule-based model 310.

[0038] In one or more embodiments, as described above, analyzer block 210 can further include root cause machine learning model 312. In at least one embodiment, root cause machine learning model 312 can generate one or more machine learning-based root cause predictions based on the telemetry-based unknown outputs. For example, root cause machine learning model 312 can include a shallow neural network, such as a multilayer perceptron. Furthermore, remediation recommendation system 202 can train root cause machine learning model 312 to learn telemetry-based patterns, which may include complex and / or noisy data. In some embodiments, remediation recommendation system 202 can further apply root cause machine learning model 312 to rule-based root cause outputs from rule-based model 310 in addition to the telemetry-based unknown outputs.

[0039] Additionally, the analyzer block 210 may further include a post-processor 314 that further modifies one or more of the outputs of the rule-based model 310 and the root cause machine learning model 312. For example, in one embodiment, the post-processor 314 may analyze the rule-based root cause output and / or the machine learning-based root cause prediction to check for erroneous outputs, null values, and other undesirable outputs. Additionally, in some embodiments, the post-processor 314 may modify the rule-based root cause output generated by the rule-based model 310 in light of the machine learning-based root cause prediction generated by the root cause machine learning model 312. Illustratively, the post-processor 314 may modify the rule-based root cause output based on a confidence level associated with the machine learning-based root cause prediction. The post-processor 314 may then generate client-specific root cause results from the machine learning-based root cause prediction and the potentially modified rule-based root cause output. In one or more embodiments, the client-specific root cause results indicate the particular components, settings, and / or configurations of the hardware and / or software of the client computing device 204 that caused the failure, power, or performance issue experienced by the client computing device 204.

[0040] The improvement recommendation system 202 can further generate one or more root cause improvement recommendations by applying a recommendation engine 212 to the client-specific root cause results generated by the post processor 314 in the analyzer block 210 based on the rule-based root cause output and the machine learning-based root cause prediction. For example, in one or more embodiments, the recommendation engine 212 can include one or more of a machine learning-based recommendation unit 316, a lookup table 318, and a rule-based recommendation 320. For example, the recommendation engine 212 can apply the machine learning-based recommendation unit 316 to the client-specific root cause results to generate one or more root cause improvement recommendations specific to the client computing device 204. Similarly, the recommendation engine 212 can correlate the client-specific root cause results with the lookup table 318 and / or the rule-based recommendation 320 to further generate one or more root cause improvement recommendations.

[0041] In some embodiments, recommendation engine 212 may utilize lookup table 318 and rule-based recommendations 320 in response to determining that machine learning-based recommendation unit 316 generated a root cause remediation recommendation with a low confidence level. Also, in some embodiments, recommendation engine 212 may combine the remediation recommendations output by machine learning-based recommendation unit 316, lookup table 318, and rule-based recommendations 320 to generate a final root cause remediation recommendation. In additional embodiments, recommendation engine 212 may include fewer or additional components (e.g., decision trees, random forests, etc.).

[0042] In at least one embodiment, recommendation engine 212 can utilize end-user-specific data (e.g., from data block 208) when generating a root cause remediation recommendation. Illustratively, recommendation engine 212 can receive data from data block 208 indicating that an end user of client computing device 204 frequently overclocks the accelerated processor of client computing device 204. In response to determining that overclocking is likely to be the root cause of a problem experienced by client computing device 204, recommendation engine 212 can incorporate this user behavior into the root cause remediation recommendation. For example, recommendation engine 212 can generate a root cause remediation recommendation that includes a recommended accelerated processor clock setting for a user who does not overclock, but can generate a root cause remediation recommendation that includes a maximum accelerated processor clock setting for a user who is likely to overclock. Thus, recommendation engine 212 can generate a root cause remediation recommendation that is not only device-specific, but also specific to the user's observed behavior.

[0043] As described above, the remediation recommendation system 202 can further tailor the remediation recommendation according to the security level associated with the client computing device 204. For example, the remediation recommendation system 202 can apply the end-user filter 322 to the root cause remediation recommendation associated with the end-user data to generate the final remediation recommendation 216. More specifically, the end-user filter 322 can receive the end-user data extracted by the data block 208. Based on this end-user data, the end-user filter 322 can determine a security level associated with the client computing device 204 (e.g., a user of the client computing device 204). The end-user filter 322 can then filter, restructure, expand, or edit the root cause remediation recommendation generated by the recommendation engine 212 according to the determined security level. For example, the end-user filter 322 can add or remove information from the root cause remediation recommendation, including, but not limited to, a detailed description of the failure, a failure pattern, and detailed recommendations for modifying, configuring, or bypassing one or more hardware and / or software components. Additionally, the filter engine 214 can send the filtered root cause remediation recommendations to the client computing device 204 .

[0044] In one or more embodiments, the filter engine 214 may further generate the final remediation recommendation 216 in various ways. For example, the filter engine 214 may generate the final remediation recommendation 216 as part of a report (e.g., either standalone or within an application installed on the client computing device 204). In another embodiment, the filter engine 214 may generate the final remediation recommendation 216 as a notification for the client computing device 204 (e.g., a pop-up window, a banner notification, a system log entry). Additionally, the filter engine 214 may generate the final remediation recommendation 216 as an electronic message to the user of the client computing device 204. For example, the filter engine 214 may generate the final remediation recommendation 216 as an email, a text message, a social media message, etc.

[0045] The remediation recommendation system 202 described herein can be implemented in a variety of ways. Accordingly, FIG. 4 illustrates a block diagram of an example embodiment 400 of the remediation recommendation system 202 as described throughout. For example, as shown in FIG. 4, the embodiment 400 can include the remediation recommendation system 202 running on a server 408. As shown, the remediation recommendation system 202 can receive telemetry data via a network 404 from one or more of a telemetry application 402 and a telemetry service 406 installed on a client computing device 204.

[0046] More specifically, remediation recommendation system 202 may operate as software or firmware and may include a parsing manager 410, a model manager 412, a recommendation manager 414, and a security filter manager 416. While Figure 4 depicts components 410-416 of remediation recommendation system 202 as separate elements, one or more of components 410-416 of remediation recommendation system 202 may be combined in further embodiments. Similarly, in further embodiments, remediation recommendation system 202 may include additional, fewer, or different components.

[0047] In certain embodiments, the improvement recommendation system 202 may represent one or more software applications or programs that, when executed by a processor, cause the processor to perform one or more tasks. For example, as described in more detail below, one or more of the components 410-416 of the improvement recommendation system 202 may represent modules stored and configured to execute on one or more computing devices. One or more of the components 410-416 of the improvement recommendation system 202 shown in FIG. 4 may represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0048] As described above and illustrated in FIG. 4 , the remediation recommendation system 202 may include a parsing manager 410. In one or more embodiments, the parsing manager 410 may perform any of the functions described above in connection with the data block 208. For example, the parsing manager 410 may include and / or direct the initial parser 302, the rule-based parser 304, the machine-learning-based parser 306, and / or the data processor 308. As such, the parsing manager 410 may receive telemetry data from one or more sources (e.g., including the telemetry application 402, the telemetry service 406, and other applications and services). In at least one embodiment, the parsing manager 410 may generate and provide one or more interfaces that enable additional configuration of one or more of the initial parser 302, the rule-based parser 304, the machine-learning-based parser 306, and the data processor 308. By way of example, the parsing manager 410 may enable reconfiguration of one or more of the components of the data block 208. The parsing manager 410 may also allow for the inclusion of one or more additional components within the data block 208, or may allow for the removal of one or more of the components of the data block 208.

[0049] 4, the improvement recommendation system 202 may also include a model manager 412. In one or more embodiments, the model manager 412 may generate, train, retrain, and apply one or more models to various inputs. For example, the model manager 412 may generate the rule-based models 310 in the analyzer block 210, including one or more lookup tables, decision trees, decision graphs, random forests, and additional heuristics. Additionally, the model manager 412 may generate and train the root cause machine learning models 312. For example, the model manager 412 may generate and train a multi-layer perceptron to learn deep, complex patterns in noisy telemetry data. Furthermore, the model manager 412 may generate and train the machine learning-based recommendation unit 316 of the recommendation engine 212. For example, the model manager 412 can train the machine learning-based recommendation unit 316 to generate root cause improvement recommendations based on both the client-specific root cause results generated by the post processor 314 of the analyzer block 210 and the end-user behavior data extracted by the data block 208.

[0050] In at least one embodiment, the model manager 412 can periodically retrain one or more of the models utilized by the improvement recommendation system 202. For example, in one embodiment, the root cause machine learning model 312 can detect faults from among a subset of unknown telemetry-based outputs (e.g., as much as a 25% improvement). In that embodiment, the model manager 412 can retrain the root cause machine learning model 312 using the remainder of the unknown telemetry-based outputs. The model manager 412 can also retrain the root cause machine learning model 312 based on user feedback associated with the final improvement recommendation 216 (e.g., whether the final improvement recommendation 216 was accurate, whether the final improvement recommendation 216 correctly resolved the problem).

[0051] Additionally, as described above, the improvement recommendation system 202 may include a recommendation manager 414. In one or more embodiments, the recommendation manager 414 generates, maintains, and applies the lookup table 318 and the rule-based recommendations 320 within the recommendation engine 212. The recommendation manager 414 may also generate and provide one or more interfaces that enable reconfiguration and additions associated with the lookup table 318 and the rule-based recommendations 320. For example, the recommendation manager 414 may enable reconfiguration of the lookup table 318 and / or the rule-based recommendations 320 to include updated rules and improvement suggestions.

[0052] As also described above, remediation recommendation system 202 may include security filter manager 416. In one or more embodiments, security filter manager 416 may generate, maintain, and apply end-user filters 322 for filter engine 214. For example, security filter manager 416 may generate end-user filters 322 that include correlations between user security levels and permissible remediation recommendation information for each user security level. In at least one embodiment, security filter manager 416 may generate final remediation recommendations 216 in any of a variety of formats (e.g., log entries, notifications, emails).

[0053] 4, the client computing device 204 may include or have installed thereon a telemetry application 402. In one or more embodiments, the telemetry application 402 may be a component-specific monitoring application (e.g., an application that monitors and records telemetry data associated with an acceleration processor). Additionally, the telemetry application 402 may be a third-party application that monitors multiple system components (e.g., hardware and / or software components) to record and transmit telemetry data. In some embodiments, the telemetry application 402 may transmit telemetry data to the remediation advisory system 202 in real time or via one or more logs or other reports.

[0054] 4, the embodiment 400 may include a telemetry service 406. In one or more embodiments, the telemetry service 406 receives, requests, monitors, and / or transmits telemetry data associated with the client computing device 204. In some embodiments, the telemetry service 406 may transmit the telemetry data to the remediation advisory system 202 via the telemetry application 402, or vice versa. In some embodiments, the telemetry service 406 may transmit the telemetry data to the remediation advisory system 202 in real time or via one or more logs or other reports.

[0055] The client computing device 204, the telemetry service 406, and the server 408 may be communicatively coupled to one another through a network 404. The network 404 may represent any type or form of communication network, such as the Internet, and may include one or more physical connections, such as a LAN, and / or wireless connections, such as a WAN.

[0056] 4 illustrates an embodiment 400 that includes the remediation recommendation system 202 on a server 408 (e.g., a cloud-based embodiment), other configurations are possible. For example, the remediation recommendation system 202 may be implemented on a client computing device 204. The remediation recommendation system 202 may also be implemented within a local network (e.g., a WAN or LAN) to receive telemetry data from a group of client computing devices (e.g., via Ethernet or Wi-Fi). Furthermore, in some embodiments, the remediation recommendation system 202 may receive telemetry data only from the telemetry application 402 or the telemetry service 406, but not from both.

[0057] Thus, as described throughout, remediation recommendation system 202 provides a solution to the rigid and often inaccurate remediation recommendations generated by previous systems. For example, by implementing a rule-based approach further augmented with a machine learning-based approach, remediation recommendation system 202 provides a resilient pipeline that can detect faults with a significantly higher proportion of noisy telemetry data. Because remediation recommendation system 202 is more tolerant of noisy data, the remediation recommendations generated by remediation recommendation system 202 are more accurate to the root causes of serious and complex problems. Additionally, because remediation recommendation system 202 provides a fully automatic solution to generating remediation recommendations, these recommendations are generated quickly (on the order of minutes, rather than days or weeks as was common with previous systems).

[0058] The process parameters and order of steps described and / or illustrated herein are provided by way of example only and can be changed as desired. For example, although the steps illustrated and / or described herein may be illustrated or described in a particular order, these steps do not necessarily have to be performed in the order illustrated or described. The various exemplary methods described and / or illustrated herein can also omit one or more of the steps described or illustrated herein or can include additional steps in addition to those disclosed.

[0059] Although various embodiments have been described and / or illustrated herein in the context of a fully functional computing system, one or more of these exemplary embodiments may be distributed as a program product in various forms, regardless of the particular type of computer-readable medium used to actually execute the distribution. The embodiments disclosed herein may also be implemented using modules that perform certain tasks. These modules may include scripts, batch files, or other executable files that may be stored on a computer-readable storage medium or on a computing system. In some embodiments, these modules may configure a computing system to execute one or more of the exemplary embodiments disclosed herein.

[0060] The foregoing description is provided to enable those skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many changes and modifications are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the appended claims and their equivalents.

[0061] Unless otherwise specified, the terms "connected to" and "coupled to" (and their derivatives) as used in this specification and claims should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. Also, the terms "a" or "an" as used in this specification and claims should be interpreted as meaning "at least one of." Finally, for ease of use, the terms "including" and "having" (and their derivatives) as used in this specification and claims are interchangeable with the term "comprising," and have the same meaning.

Claims

1. 1. A computer-implemented method comprising: generating a rule-based root cause output and a telemetry-based unknown output from telemetry data of the client computing device; generating a machine learning based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; and generating one or more root cause remediation recommendations specific to the client computing device and based on the rule-based root cause output and the machine learning-based root cause prediction; A computer-implemented method.

2. prior to generating the rule-based root cause output and the telemetry-based unknown output; receiving the telemetry data from the client computing device; parsing the telemetry data based on a rule-based model and the root cause machine learning model.

10. The computer-implemented method of claim 1.

3. receiving the telemetry data from the client computing device includes receiving raw and unstructured data from one or more of an offline log and a real-time telemetry source; 3. The computer-implemented method of claim 2.

4. Parsing the telemetry data based on the rule-based model and the root cause machine learning model includes: parsing the telemetry data into structured data based on one or more of the offline log and the real-time telemetry source; parsing the structured data into attributes specific to the rule-based model; parsing the structured data into attributes specific to the root cause machine learning model.

4. The computer-implemented method of claim 3.

5. Parsing the telemetry data based on the rule-based model and the root cause machine learning model includes generating one or more of engineering features, data encoding, customized mapping, and data augmentation based on attributes specific to the rule-based model and attributes specific to the root cause machine learning model.

5. The computer-implemented method of claim 4.

6. modifying the rule-based root cause output in light of the machine learning-based root cause prediction; generating client-specific root cause outcomes from the machine learning-based root cause predictions and the modified rule-based root cause outputs; 10. The computer-implemented method of claim 1.

7. applying a recommendation engine to the results of the client-specific root causes to generate remediation recommendations for the one or more root causes specific to the client computing device; 7. The computer-implemented method of claim 6.

8. the recommendation engine comprises a plurality of trained machine learning based recommendation units; 8. The computer-implemented method of claim 7.

9. the recommendation engine comprises one or more of a look-up table or rule-based recommendations; 9. The computer-implemented method of claim 8.

10. generating filtered remediation recommendations tailored to a user security level associated with the client computing device based on the one or more root cause remediation recommendations specific to the client computing device and end user data extracted from the telemetry data of the client computing device; 3. The computer-implemented method of claim 2.

11. generating a machine learning-based unknown output by applying the root cause machine learning model to the telemetry-based unknown output; and retraining the root cause machine learning model based on the unknown machine learning-based output.

10. The computer-implemented method of claim 1.

12. 1. A system comprising: at least one processor; a physical memory containing computer-executable instructions; The computer-executable instructions, when executed by the at least one processor, generating a rule-based root cause output and a telemetry-based unknown output from telemetry data of the client computing device; generating a machine learning based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; and generating one or more root cause remediation recommendations specific to the client computing device and based on the rules-based root cause output and the machine learning-based root cause predictions; causing the at least one processor to perform operations including: system.

13. The computer-executable instructions, when executed by the at least one processor, prior to generating the rule-based root cause output and the telemetry-based unknown output; receiving the telemetry data from the client computing device; parsing the telemetry data based on a rule-based model and the root cause machine learning model; causing the at least one processor to perform operations including: The system of claim 12.

14. receiving the telemetry data from the client computing device includes receiving raw and unstructured data from one or more of an offline log and a real-time telemetry source; The system of claim 13.

15. Parsing the telemetry data based on the rule-based model and the root cause machine learning model includes: parsing the telemetry data into structured data based on one or more of the offline log and the real-time telemetry source; parsing the structured data into attributes specific to the rule-based model; parsing the structured data into attributes specific to the root cause machine learning model.

15. The system of claim 14.

16. Parsing the telemetry data based on the rule-based model and the root cause machine learning model includes generating one or more of engineering features, data encoding, customized mapping, and data augmentation based on attributes specific to the rule-based model and attributes specific to the root cause machine learning model.

16. The system of claim 15.

17. The computer-executable instructions, when executed by the at least one processor, modifying the rule-based root cause output in light of the machine learning-based root cause prediction; generating client-specific root cause outcomes from the machine learning based root cause predictions and the modified rule-based root cause outputs; causing the at least one processor to perform operations including: The system of claim 12.

18. applying a recommendation engine to the results of the client-specific root causes to generate remediation recommendations for the one or more root causes specific to the client computing device; 18. The system of claim 17.

19. the recommendation engine comprises a plurality of trained machine learning based recommendation units; 20. The system of claim 18.

20. A computer-readable storage medium containing one or more computer-executable instructions, The computer-executable instructions, when executed by at least one processor of a computing device, generating a rule-based root cause output and a telemetry-based unknown output from telemetry data of the client computing device; generating a machine learning based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; and generating one or more root cause remediation recommendations specific to the client computing device and based on the rule-based root cause output and the machine learning-based root cause prediction; causing the computing device to perform A computer-readable storage medium.

Images