Feature deployment readiness prediction

Abstract

Embodiments of this disclosure relate to feature deployment readiness prediction. Systems and methods for generating predictive quality metrics are provided. Telemetry data can be received from a first set of devices executing first software. A quality metric for the first software can be generated based on the first telemetry data. Telemetry data can be received from a second set of devices, which are different from the first set of devices. Based on features included in the first and second telemetry data, covariates affecting the quality metric can be identified, and a coarsening-exact matching process can be performed using the identified covariates to generate a predictive quality metric for the first software based on the second set of devices.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2021 / 025689, international application date of April 5, 2021, entered the Chinese national phase on November 29, 2022, Chinese national application number 202180039240.0, and invention title "Feature Deployment Readiness Prediction". Background Technology Evaluating pre-release software candidates (such as beta releases) and inferring software quality based on such evaluations is problematic because the evaluators assessing these candidates often differ from the intended audience (also known as the retail audience). That is, the group of individuals evaluating pre-release software candidates is much smaller than the retail audience, and those who do evaluate them tend to use pre-release software in ways different from retail users. For example, software evaluators are often software enthusiasts and may use pre-release software less, as they are often interested in exploring new features or evaluating pre-release software for adoption purposes. Additionally, software enthusiasts tend to use hardware superior to that used by the retail audience. Therefore, determining how best to infer software quality for a larger retail audience based on pre-release software evaluations from a much smaller group of evaluators is challenging and affects the determination of when new software might be ready for deployment.

[0002] Regarding these and other general considerations, the aspects disclosed herein have been addressed. Furthermore, while relatively specific issues may be discussed, it should be understood that these examples should not be limited to addressing the specific problems identified in the context of this disclosure or elsewhere. Summary of the Invention

[0003] According to examples in this disclosure, a method and system are provided that generate predicted quality metrics for a release version of software based on actual quality metrics from a group or set of software evaluators. That is, the first set of devices may correspond to a limited number of devices used by software evaluators and associated with preview programs, beta programs, or other processes that collect quality metrics from devices executing pre-release software or other software not yet available for retail or public use. The quality metrics of the software associated with the first set of devices can indicate how well the pre-release version or pre-release build of the software performs or functions on the first set of devices. The systems and methods described herein leverage the similarities and differences between the first set of devices and users and a second set of devices and users to predict the quality metrics of the software, where the predicted quality metric is a prediction of how well the pre-release version of the software is expected to function on the second set of devices and users. Therefore, the predicted quality metrics can provide insights into when features, updates, or other features are ready for deployment.

[0004] According to examples of this disclosure, a method for generating a predictive quality metric is provided. The method may include: receiving first telemetry data from a first group of devices executing first software; generating a quality metric for the first software based on the first telemetry data; receiving second telemetry data from a second group of devices, wherein the second group of devices is different from the first group of devices; identifying covariates affecting the quality metric based on features included in the first and second telemetry data; and performing coarsened exact matching using the identified covariates to generate a predictive quality metric for the first software on the second group of devices.

[0005] According to examples of this disclosure, a computer-readable medium is provided. The computer-readable medium may include instructions that, when executed by a processor, cause the processor to: receive first telemetry data from a first set of devices executing first software; generate a quality metric for the first software based on the first telemetry data; receive second telemetry data from a second set of devices, wherein the second set of devices is different from the first set of devices; identify covariates affecting the quality metric based on features included in the first and second telemetry data; perform coarsened exact matching using the identified covariates; identify weights to be assigned to each device in the first set of devices; and generate a predicted quality metric based on the weights assigned to each device in the first set of devices and the identified covariates.

[0006] According to examples of this disclosure, a system for generating a predictive quality metric is provided. The system may include a processor and a memory storing instructions that, when executed by the processor, cause the processor to: receive first telemetry data from each of a first group of devices executing first software; generate a quality metric for the first software based on the first telemetry data; receive second telemetry data from a second group of devices, wherein the second group of devices differs from the first group of devices; identify covariates influencing the quality metric based on features included in the first and second telemetry data; stratify the first group of devices and the second group of devices based on the identified covariates; reweight each device in the first group of devices; and generate a predictive quality metric based on the weights assigned to each device in the first group of devices and the identified covariates.

[0007] Any one of the above aspects combined with any other aspect of the above aspects. Any one of the one or more aspects described herein.

[0008] This disclosure is provided to present the selection of concepts in a simplified form, which will be further described in the detailed description below. This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Further aspects, features, and / or advantages of the embodiments will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0009] Refer to the following figures for non-limiting and non-exhaustive examples.

[0010] Figure 1 The details of the predictive quality metric generation system based on the example of this disclosure are as follows;

[0011] Figure 2 Additional details are described regarding the predictive quality metric generation system based on the example of this disclosure;

[0012] Figure 3 Details of the coarsened precise matching process according to the examples in this disclosure are described;

[0013] Figure 4 This invention describes example software diagnostic feature data used throughout the process of predicting quality metrics according to the present disclosure;

[0014] Figure 5 Examples of balanced and imbalanced features generated from a random forest classifier according to examples in this disclosure are depicted;

[0015] Figure 6A Details of a method, according to an example of this disclosure, for generating a predictive quality metric for a group of devices and users based on existing quality metrics for different groups of devices and users;

[0016] Figure 6B Additional details are described regarding a method for generating predictive quality metrics for groups of devices and users based on existing quality metrics for different groups of devices and users, according to examples of this disclosure;

[0017] Figure 7 Details of a method for determining whether action should be taken based on quality metrics for a first group of devices and users and predicted quality metrics for a second group of devices and users, according to an example of this disclosure, are described.

[0018] Figure 8 The present disclosure describes in detail a method for generating a model that provides predictive quality metrics for a second group of devices and users based on quality metrics associated with a first group of devices and users.

[0019] Figure 9An example of a predictive quality metric generator based on the examples in this disclosure is depicted;

[0020] Figure 10 A block diagram illustrating the physical components (e.g., hardware) of a computing device that can implement various aspects of the present disclosure is depicted.

[0021] Figure 11A The illustration shows a first example of a computing device that can implement various aspects of the present disclosure;

[0022] Figure 11B A second example of a computing device that can implement various aspects of this disclosure is illustrated; and

[0023] Figure 12 The illustration shows at least one aspect of a system architecture for processing data according to an example of this disclosure. Detailed Implementation

[0024] In the following detailed description, reference is made to the accompanying drawings, which form a part thereof, and specific embodiments or examples are illustrated therein. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be practiced as methods, systems, or devices. Therefore, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description should not be considered limiting, but rather the scope of this disclosure is defined by the appended claims and their equivalents.

[0025] The Software Development Lifecycle (SDLC) is an established and standardized framework or methodology for implementing software product development across multiple phases. The ultimate goal behind software product development is to release and deliver the software product to its intended or target users or clients. While software development teams can modify one or more steps of the SDLC, it typically includes requirements analysis, planning, software design, software development, software testing, and deployment and maintenance. Similar to the Software Development Lifecycle, there exists a Software Release Lifecycle to ensure the timely release of a software application from its initial coding stage to its final release. The Software Release Lifecycle extends from the initial development of the software to its final release and includes updates to improve the software or fix software vulnerabilities discovered during development and testing. The fundamental purpose of defining a Software Release Lifecycle is to assess the stability of the software product during development and to further develop the product in the next subsequent phase until final release. Software Release Lifecycles are often described using different terms and phases, but typically include several pre-release phases followed by a release to market or release to the web phase.

[0026] The software release cycle can begin with an initial release of the software to the internal users of the software development organization. This phase is often referred to as the alpha release. The alpha phase can refer to the testing performed on the initial software product after initial development. During the alpha phase, the software can be internally tested by developers and testers within the organization and can be used by various users within the organization. It should be understood that alpha software may be unstable and often requires additional development before mainstream deployment. Alpha software may contain serious bugs, and any resulting instability may lead to crashes or data loss. Alpha software may not include all features planned for the final release.

[0027] After software is released to internal users within the development organization, it can be released to selected user groups for testing in a real-world environment by the intended users. This phase is often referred to as the preview phase, technical preview phase, or beta phase, and the software can be identified as a preview build. Software in the preview phase is typically considered the final testing phase before product launch. Preview builds often have more bugs, more speed or performance issues than the completed software, and may result in crashes or data loss. The focus of testing in the preview phase is to minimize the impact on end users, and typically includes usability testing. The process of delivering the preview build to users is called a preview release or beta release, and it is usually the first time the software is available outside the group or group of organizations that developed it. There can be many preview releases, as new features are added, usability is enhanced, and bugs are found and fixed, and these are also referred to as different preview builds or different preview versions. Such releases can be public or private, depending on whether they are publicly available or only available to a limited audience. Preview builds are often useful for demonstrations and previews within an organization, as well as for anticipated customers. Other terms used in this phase may include, but are not limited to, preview, prototype, technical preview, early access, or release candidate, where a release candidate is generally considered to be the final product to be released unless there are serious problems or defects.

[0028] After the release candidate is released, the software product can be released to clients or otherwise made available to end users. Often referred to as the release-to-retail phase, release-to-manufacturing phase, or release-to-website phase, the latest build (often called the retail build) can be digitally signed, allowing end users to verify the integrity and authenticity of the software purchase. Copies of the retail build can be sent for duplication and mass distribution and / or distribution on the website.

[0029] At each stage of the software release cycle, each build or version of the software may include various features or additions; bugs, crashes, performance issues, and other disruptive factors affecting user satisfaction can be monitored for infrequent failures and failures. Software that doesn't frequently fail is better, and when failures do occur, it recovers easily from them. Metrics can be used to track and evaluate the reliability of the software and any new features that may have been added; these metrics may include, but are not limited to, average failure rate—the average number of failures per deployment unit or per user per cycle, average time between failures / mean time between failures—a measure of the amount of time the software is expected to work correctly until the next major failure, number of crashes, etc. Additionally, software performance can be evaluated so that users understand performance levels and how performance affects their use of the software. Metrics for tracking and monitoring performance may include CPU utilization, error rate, response time, garbage collection, request rate, and overall customer satisfaction. In some examples, quality metrics may be the same as quality measures.

[0030] However, many of the metrics upon which software reliability and performance are tracked and evaluated can be influenced by other factors. For example, the hardware on which the software runs (such as, but not limited to, the CPU, the physical amount of system memory, the amount of available storage, system architecture, etc.) can affect processing speed, the amount of memory available to the software, and the overall user experience. Software running on the latest and greatest hardware can access the hardware resources that allow the most complex calculations to be performed in a short amount of time, while the same software running on an older system with fewer resources may take longer to perform the same calculations and cause the application to run slowly or even fail. In some cases, other software and / or other features added to the software running on the system can lead to reliability and performance issues.

[0031] Furthermore, the metrics upon which software reliability and performance are tracked and evaluated can be influenced by the frequency and manner in which users interact with the software and / or its various features. In other words, users may interact with the software in ways different from other users. For example, one user may extensively use several features of the software, while a second user may freely use all of its features. Therefore, the quality measurements associated with software reliability and performance provided by the first user may differ from those associated with software reliability and performance provided by the second user.

[0032] While software quality metrics obtained from pre-release phases or based on preview builds can be used to infer the quality of software and / or features in a general release or retail build that has not yet been released, actual measurements of software quality from preview builds in the pre-release phase may not match measurements of software quality obtained later in a retail build when the software and / or new features are actually released to a general user group and used by the intended audience. That is, software quality metrics obtained from preview builds in the pre-release phase represent the experience quality of a smaller, selected user group (e.g., a preview audience or group), partly due to differences in hardware and software usage. For example, a measurement of the average time between failure metrics may be acceptable for preview builds during the pre-release phase; however, the same average time between failure metrics may differ for a retail build when the software is actually released. As another example, new features may be added to software that works or functions very well in one environment or for a specific user group, but may fail or reduce user satisfaction for different user groups. Therefore, evaluating a preview build in a preview audience may distort or bias the quality metrics used to infer the quality of a retail build.

[0033] To overcome biases or prejudices added to quality metrics by the preview group during the pre-release phase and to infer software quality for later releases (e.g., retail builds), similarities and differences between two groups or clusters of users and / or devices can be determined. That is, the similarities and differences between a first group or cluster of users and devices performing a preview build (e.g., a preview audience) and a second group or cluster of users and devices performing a retail build (e.g., a retail audience) can be used to better predict or infer software quality metrics or measures at release. By using causal inference matching processes (such as coarsened exact matching), quality metrics associated with each device in the preview group can be reweighted based on the retail group, making the reweighted quality metrics based on the preview group more closely match the actual quality metrics derived from the retail group, thus better reflecting the retail group and predicting quality metrics.

[0034] Therefore, it is essential to first identify the quality metrics associated with aspects of the software and / or features. As an example, software developers, scientists, and engineers can determine that the Mean Time To Failure (MTTF) metric associated with a service, software application, or feature performing on the device can provide a measure of software quality for that specific, identical service, software application, or feature performing on the device. For example, an MTTF measurement for a management service can provide a measure of the software quality of the display management service. Alternatively or additionally, software developers, scientists, and engineers can determine that the Mean Time To Failure (MTTF) metric associated with a service, software application, or feature performing on the device can provide a measure of software quality for different services, software applications, or features performing on the device. For example, an MTTF measurement for a display management service can provide a measure of the software quality of a printer application, where the printer application may rely on the display management service to display content to a user. The selected quality metric is obtained as an independent average (e.g., the average MTTF across all preview devices for the service, software application, or feature).

[0035] When identifying quality metrics, one or more covariates associated with the quality metric can be identified. That is, one or more characteristics or features that influence the identified quality metric can be identified. For example, covariates for the MTTF used to display services may include, but are not limited to, a metric for the device's total physical RAM (total Physical RAM), the amount of time since the last operating system reboot (osTime), the region of the world in which the user / device is located, the device's firmware manufacturer, operating system architecture (e.g., x32 vs x64), processor manufacturer, etc. Such covariates may be a subset of a feature list. The feature list may include those features that describe the device or user and can be used as device and software diagnostic information for two groups or subsets of devices.

[0036] Device and software diagnostic information can be provided by a software telemetry service associated with each device, where software diagnostic information can be provided in real-time or near real-time for devices associated with a preview build. The software and diagnostic information provided by devices in the retail group can be historical in nature and can correspond to software and diagnostic information collected via a release build prior to the preview build. For example, if the preview build corresponds to the second version of the software, the software and diagnostic information provided by devices in the retail group can be associated with the first version of the software.

[0037] Covariates can be identified from subgroups of features on a feature list. For example, software developers, scientists, and engineers might identify thirty features that typically influence the identified quality metric (e.g., subgroups of features from a feature list). Based on the feature list and / or subgroups of features from the feature list, machine learning models can be used to identify covariates.

[0038] Machine learning models that provide an indication of the relative importance or contribution of each feature to the prediction can be used to identify important covariates. For example, a random forest classifier can be used to identify covariates that influence a quality metric. A random forest classifier can be trained with features from a feature list or a subset of features from a subgroup of features identified by software developers, scientists, and engineers as influencing the identified quality metric. Once trained with these features, the random forest classifier can act as a feature selector, providing an indication of the relative importance or contribution of each feature to the prediction. For example, a random forest classifier can provide a relevance score for each feature input during the training of the random classifier model, where the relevance score can be used to select important features and discard the least important ones. That is, the top ten most important features can be selected as covariates; alternatively, or additionally, important features with relevance scores above a threshold can be selected as covariates.

[0039] Once covariates (e.g., a subset of features from a feature list or a subset of features from a subgroup of features) have been identified, each covariate can be evaluated to determine if there is an imbalance or disparity between the preview and retail groups. The data for each identified feature as a covariate can be coarsened or stratified into smaller, more specific groups or buckets based on a predetermined set of criteria, which may correspond to a distribution such as a Gaussian distribution and several strata or buckets. Covariates exhibiting imbalance can be selected and used in the coarsening of the exact match process.

[0040] The coarsening of the exact match process can be used to reweight the quality metrics associated with each device in the preview group based on the retail group. As described below, the weighting for each tier or bucket can be determined based on the number of devices in the preview and retail groups falling within each tier or bucket. The weighting can be applied to the quality metrics of the preview group so that the average mean of the quality metrics better predicts the quality metrics associated with the retail group.

[0041] Figure 1Details of a predictive quality metric generation system 100 according to an example of this disclosure are depicted. The predictive quality metric generation system 100 can utilize existing quality metrics of software associated with a first set of devices 104 and predict quality metrics of software associated with a second set of devices 108, wherein the second set of devices 108 may include more devices than the first set of devices 104. That is, the first set of devices 104 may be a limited number of devices associated with preview programs, beta programs, or other software that executes pre-release software or is not yet available for retail use. Quality metrics of the software associated with the first set of devices 104 can indicate how well a pre-release version or preview of the software performs or functions on devices 112A-112C of the first set of devices 104. In some cases, quality metrics can indicate quality measurements inferred by users, such as measurements associated with user use and / or user feedback. Because the number of devices in the first group of devices 104 can be different from and less than the number of devices in the second group of devices 108 and / or can be used in a different manner than the devices in the second group of devices 108, the predictive quality metric generation system 100 can generate predictive quality metrics that indicate how well a pre-release or preview build or version of the software performs or functions on the number of devices in the second group of devices 108. In other words, the predictive quality metric generation system 100 uses the similarities and differences between the first group of devices 104 and the second group of devices 108 to predict the quality metrics of the software expected to perform at the second group of devices 108.

[0042] like Figure 1As depicted, devices 112A-112F may include, but are not limited to, desktop computers 112A / 112D, tablets 112B / 112E, and / or laptop computers 112C / 112F. Software executing on the first group of devices 104 may include, but is not limited to, operating systems, applications, services, or other instructions executed by the processors of devices within the group of devices 104. Quality metrics may be identified or selected from software developers, engineers, and scientists as the best quality metrics associated with the specific functionality of devices 112A-112C, the software executing on devices 112A-112C, and / or the services executing on the devices. For example, a quality metric may be equal to the Mean Time To Failure (MTTF) of a service such as a print spooler, installation service, or window manager. As a non-limiting example, a quality metric may be equal to the MTTF of a service or an executable associated with graphics rendering. As another non-limiting example, a quality metric may be equal to the number of crashes associated with the printing functionality of a word processing application, where such a quality metric may be used to indicate the quality associated with updates to the operating system or other applications. While MTTF is discussed as a quality metric, other quality metrics can be used, including average failure rate, number of crashes, average disconnection rate, and the number of hardware and / or software events with specific event IDs.

[0043] The similarities and differences between the first group of devices 104 and the second group of devices 108 can be obtained via feature data from telemetry streams 116 and 154. As used herein, “telemetry data” includes, but is not limited to, information about the devices and how they are configured (including hardware attributes such as CPUs, installed memory and storage devices) and quality-related information (such as uptime and sleep details, and the number of crashes or hangs). That is, each of the first group of devices 104, 112A-112C, can provide device and software diagnostic feature data 148 as telemetry stream 116. Device and software diagnostic feature data may include, but is not limited to, the number of crashes associated with hardware / software, total physical RAM, the amount of time since the last operating system restart (osTime), operating system architecture (osArchitecture), disk space, CPU usage (cpuUsage), number of CPU cores (processorCores), CPU manufacturer (processorManufacturer), type of hardware used, installed applications and usage details, reliability information on device drivers, and may be provided as telemetry stream 116 to telemetry storage 124 via network 120. Similarly, each of the second set of devices 108, 112D-112F, can provide device and software diagnostic characteristic data 152 as a telemetry stream 154. This device and software diagnostic characteristic data may include, but is not limited to, hardware / software-related crash counts, total physical RAM, time elapsed since the last operating system reboot (osTime), operating system architecture (osArchitecture), disk space, CPU usage (cpuUsage), number of CPU cores (processorCores), CPU manufacturer (processorManufacturer), type of hardware used, installed applications and usage details, reliability information on device drivers, and may be provided to the telemetry repository 124 via network 120 in the telemetry stream 154. In all respects, the device and software diagnostic characteristic data 152 is associated with software versions prior to those retail-built or executed on the devices in the first set of devices 104. For example, device and software diagnostic feature data 152 can be collected from devices that are performing stable releases or retail builds of the operating system or operating system updates (such as operating system version 4.3), while device and software diagnostic feature data 148 can be collected from devices that are performing preview builds, or from unreleased versions of the operating system or operating system updates (such as operating system version release candidate 4.4).In cases where a previous version does not exist, telemetry data 154 can be obtained from other sources, such as inference from similar software operating systems or software applications.

[0044] Telemetry repository 124 can provide feature data 128 to quality metric generator 132. The quality metric generator can generate quality metrics based on feature data 128, where the quality metric can be the average of feature data 128 obtained from feature data of each device in the first group of devices 104. Quality metrics can be measures that an engineering team tracks and evaluates for performance in a preview build to make a determination about whether a new software product is ready for public release (e.g., a "ship-to-market" determination). As a non-limiting example, the quality metric generator can generate an MTTF quality metric based on the number of times a service associated with graphics rendering exhibits a failure or malfunction, measured in terms of the amount of time since the last operating system reboot. The quality metric can represent the average MTTF across all devices in the first group of devices 104. Quality metric generator 132 can then provide the quality metric to dashboard 136, allowing client device 144 to display the resulting quality metric 140, where quality metric 140 represents a quality metric associated with a specific pre-release version of the software running on the first group of devices 104.

[0045] Telemetry repository 124 can provide feature data 156 to prediction quality metric generator 160. Feature data 156 can be determined by software developers, scientists, and engineers, and selected from features provided in telemetry data stream 154 (from the second set of devices) and telemetry data stream 116 (from the first set of devices). Prediction quality metric generator 160 may include feature data preprocessing module 164, random forest classification model 168, coarse-out exact matching module 172, and quality metric generator 176. Feature data preprocessing module can process feature data 156 and convert it into a usable format to train random forest classification model 168. Random forest classification model 168 can be trained on preprocessed feature data provided from feature data preprocessing module 164 and can identify one or more features as important covariates. For example, random forest classification model 168 can score features representing covariates that influence the quality metric; each covariate feature score can provide an indication of the feature's influence on the quality metric. For example, each covariate feature with a score above a threshold can be selected as a covariate feature influencing the quality metric.

[0046] In some respects, a subset of identified covariates that have a high impact on quality metrics can be selected based on feature imbalance. Feature imbalance, in various respects, refers to the differences present between the first group of devices 104 and the second group of devices 108. As part of the imbalance identification process, covariate features can be coarsened or stratified into smaller, more defined groups or buckets based on a predetermined set of criteria, which may correspond to a distribution such as a Gaussian distribution and several layers or buckets. Covariate features exhibiting imbalance can be selected and utilized during the coarsening-precision-matching process performed by the coarsening-precision-matching module 172.

[0047] Coarse-precise matching is a non-parametric causal inference technique for creating matching datasets to evaluate the impact of treatments on a control population. According to examples in this disclosure, feature data associated with the second group of devices 108 can be considered the control population, while feature data associated with the first group of devices 104 can be considered the treatment population. Therefore, the coarse-precise matching module 172 can be used to assess the impact of preview builds, for example, on devices performing retail builds in the second group of devices 108.

[0048] As will be discussed further below, for example, the output of the coarsening precise matching module 172 may include feature weights that are applied to the quality metric generation process implemented by the quality metric generator 132. Therefore, the quality metric generation process used to create quality metrics at 132 can be provided to the predictive quality metric generator 160 as a preview-built quality metric generation process 138. The quality metric generation process 138 may be an equation or algorithm associated with the generation of the preview-built quality metric. The feature weights determined by the coarsening precise matching module 172 can be applied to the quality metric generation process 138, and a quality metric associated with the second set of devices 108 can be generated. The quality metric associated with the second set of devices 108 can then be provided to the dashboard 136, allowing the client device 144 to display the resulting quality metric 180, where the quality metric 180 represents the quality metric associated with the predictive execution of the preview-built quality metric performed by the second set of devices 108.

[0049] Figure 2Additional details of a predictive quality metric generation system 200 according to an example of this disclosure are described. The predictive quality metric generation system 200 may be the same as or similar to the previously discussed predictive quality metric generation system 100. Thus, at 208, a predictive quality metric generator 204, the same as or similar to the predictive quality metric generator 160, may receive feature data 212 from telemetry stream 116 provided via a first set of devices 104 and feature data 216 from telemetry stream 154 provided via a second set of devices 108. While the telemetry stream data may include, but is not limited to, information about devices and how they are configured (including hardware attributes such as CPU, installed memory, and storage devices) and quality-related information (such as uptime and sleep details, and the number of crashes or hangs), the feature data 212 may be a subset of the feature data provided from the telemetry data 116 and may be based on a specific quality metric. For example, the feature data 212 may include 50 features determined to be associated with a quality metric equal to the printer installation failure rate. For example, such features may include, but are not limited to, total Physical RAM, the amount of time since the last operating system reboot (osTime), operating system architecture (osArchitecture), disk space, CPU usage (cpuUsage), number of CPU cores (processorCores), CPU manufacturer (processorManufacturer), and whether another printer is installed. Similarly, feature data 216 may be a subset of feature data provided from telemetry stream data 154 and may be based on the same specific quality metric. For example, feature data 216 may include the same 50 features identified as being associated with a quality metric equal to the printer installation failure rate. For example, such features may include, but are not limited to, total Physical RAM, the amount of time since the last operating system reboot (osTime), operating system architecture (osArchitecture), disk space, CPU usage (cpuUsage), number of CPU cores (processorCores), CPU manufacturer (processorManufacturer), and whether another printer is installed.

[0050] Feature data 212 and 216 can be preprocessed into a format suitable for training the random forest model 220. A portion of the preprocessed feature data 212 and 216 can be used to train the random forest model 220 and identify features representing covariates that have a high impact on the quality metric. For example, the random forest model 220 can score features representing covariates that influence the quality metric; each feature score can indicate the feature's impact on the quality metric. For example, each feature with a score above a threshold can be selected as a feature with a high impact on the quality metric. Therefore, at 220, a subset of the original 50 features can be selected as features with a high impact on the quality metric of printer installation failure rate.

[0051] Furthermore, at 224, based on feature imbalance, a subset of the original 50 features can be selected as features that have a high impact on the quality metric of printer installation failure rate. As previously discussed, feature imbalance can be assessed in such a way that features with a high impact on the quality metric and representing imbalance between the first group of devices 104 and the second group of devices 108 are utilized during the coarsening precise matching process. In some examples, the distribution of each feature from the first group of devices 104 can be compared with the distribution of each corresponding feature from the second group of devices 104. Features that indicate feature imbalance as assessed by feature distribution can be selected for coarsening precise matching. That is, if the difference between the distribution of features from the first group of devices 104 and the distribution of features from the second group of devices 108 meets or exceeds an imbalance threshold, that feature can be selected as a bin or bucket during the precise coarsening matching process, as further described below. At 228, during the coarsening precise matching process, features that have a high impact on quality metrics and represent the imbalance between the first group of devices 104 and the second group of devices 108 are used to generate weights that will be applied to the coarsening matching features associated with the first group of devices 104.

[0052] For example, go to Figure 3The coarsening exact matching process is described according to examples in this disclosure. As previously stated, feature 308 may represent a feature that has a high impact on quality metrics and indicates an imbalance between the first group of devices 104 and the second group of devices 108. Each device 304 in the first and second groups of devices 104 and 108 may be represented by a feature coarsened to a discrete value using a coarsening or binning strategy (such as the coarsening or binning strategy depicted in 312). Thus, for each device 304, a “BIN signature” 316 can be used for exact matching with other devices 304 having the same “BIN signature” 316. Each matching device 304 must have the exact same BIN signature 316 as its matching device in the other group. For example, a device 304 in the second group of devices 108 with feature data equal to level 1, level 2, level 3, level 4, and level 5 can only match a device in the first group of devices 104 with the exact same value as the feature data of level 1, level 2, level 3, level 4, and level 5. With the selected features and loading strategy, a BIN signature 31 is generated for each device 304 in the second group of devices 108 and matched against devices 304 in the first group of devices 104. Since devices 304 belonging to the second group of devices 108 can have BIN signatures that devices 304 in the first group of devices 104 do not have (and vice versa), a percentage (but not necessarily all) of the second group of devices 108 will be matched. There may be an imbalance between the number of devices 304 in the second group of devices 108 and the number of devices 304 in the first group of devices 104 with the same BIN signature. This distribution variance can be normalized using coarse-grained exact match weights, where the weights can be calculated according to Equation 1. Equation 1

[0053] Since each weight of each BIN signature can be calculated according to Equation 1, the weights can be applied to the first group of devices 104, such as... Figure 3 As depicted in Table 328. Although in Figure 3 The text describes a simplified representation of the quality metric generation process, but it can generate quality metrics corresponding to weighted averages. For example, returning to... Figure 2At 236, a quality metric generation process 232 for the first group of devices 104, identical or similar to the quality metric generation process 138, can be received. In some examples, the quality metric generation process 232 may include device-specific quality metrics for devices in the first group 104. Weights calculated by the coarsening exact matching process can be applied to the quality metrics such that the contribution of each device belonging to the first group of devices 104 is used to predict the quality metric at 244. That is, the predicted quality metric 244 may be a weighted average of weighted quality metrics specific to each device in the first group of devices 104. Then, at 248, the predicted quality metric generated based on the first group of devices 104, the weights provided by the coarsening exact matching process, and the quality metric process used to generate the quality metric for the first group of devices 104 can be stored as the predicted quality metric for the second group of devices 108.

[0054] Figure 4 Example features are depicted throughout the process of predicting quality metrics according to the examples of this disclosure. For example, telemetry flows 116 and 154 may include feature data 148, 152 provided by each of the first group of devices 104 and the second group of devices 108. Example feature data may include, but is not limited to, serviceCrash, applicationTime, osTime, cpuUsage, networkState, #timeOuts, gpuHardwareAttributes, processorManufacturer, totalPhysicalRAM, osArchitecture, diskSize, processorCores, etc. Additional feature data may be included as indicated by ellipses. Feature 402 may represent features identified by software developers, engineers, and scientists as relevant to generating a specific quality metric. For example, feature 402 may represent 50 features identified by software developers, engineers, and scientists as relevant to generating a quality metric equal to the MTTF of a particular process or service (such as a display manager). Covariate features ranked by relative importance or contribution to prediction based on a trained random forest classification model are depicted as ranked covariate features 404. As depicted in covariate feature 404 for ranking, the feature "gpuHardwareAttributes" may not have a significant impact on the quality metric and is therefore ranked at the bottom. Furthermore, covariate features for each ranking, for example, with scores above a threshold, can be selected as features with a high impact on the quality metric and thus can be chosen for coarsening exact matches, as indicated by selected covariate feature 408. In some examples, the imbalance between the feature data of the first group of devices 104 and the feature data of the second group of devices 108 can be utilized to ensure that the features selected for coarsening exact matches are sufficiently dissimilar across the groups of devices.

[0055] For example, go to Figure 5 As depicted in Figure 504, the distribution of devices across each feature level is shown. The features depicted in Figure 504 can be any of the features previously discussed, and feature levels can be associated with binning or bucketing the data associated with the feature into one or more categories (e.g., level 1, level 2, etc.) as shown in Figure 504. As depicted in Figure 504, the distribution of devices at feature levels differs between the first group of devices 104 and the second group of devices 108; therefore, this feature is considered unbalanced. For example, approximately 16% of the devices in group 2 can be classified into level 1 category, while approximately 11% of the devices in group 2 can be classified into level 1 category. The additional levels described in Figure 504 illustrate similar differences between the first and second groups of devices. Therefore, the features represented in Figure 504 can be utilized in the process of coarsening exact matching. Similarly, more specific features, such as region features, can be shown as depicted in Figure 508. Since the difference in the regional distribution between the first group of devices 104 and the second group of devices 108 is shown as unbalanced, the regional features can be utilized during the coarsening precise matching process. However, the distribution of the "processor manufacturer" feature can be depicted in Figure 512. Since the difference in the distribution of "processor manufacturer" between the first group of devices 104 and the second group of devices 108 is not shown as unbalanced, the "processor manufacturer" feature should not be utilized during the coarsening precise matching process.

[0056] Return to Figure 4 Feature 412 satisfies the imbalance requirement and represents four of the original 50 features to be used in the coarsening exact matching process. Data 416 depicts the weights used to generate various BIN signatures using feature 412. For example, a separate device with build, revision, or other software (such as Device_A) can be assigned a weight of 0.3333 from the coarsening exact matching process. As previously discussed... Figure 3 For each device in the first group of devices 104, weights can be applied to existing quality metrics (QMs) 420 at the device level to obtain a weighted quality metric 424 for each device. According to at least one example, the average of all weighted quality metrics (QMs) can be shown, where the weighted average represents the predicted quality metric for the second group of devices 108.

[0057] Figures 6A-6B Details of method 600 according to an example of this disclosure are described. Method 60 is used to generate predicted quality metrics for a group of devices based on existing quality metrics for different groups of devices. The general sequence of the steps of method 600 is as follows: Figures 6A-6BThe following is illustrated. Typically, method 600 begins at 604 and ends at 648. Method 600 may include more or fewer steps, or the order of the steps may differ from those shown in Figure 6. Method 600 may be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 600 may be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SOC), or other hardware device. References will be incorporated herein by reference. Figures 1-5 Method 600 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0058] The method begins at 604, where the process can proceed to 608. At 608, feature data can be received from a first set of devices and a second set of devices. The first set of devices can be a limited number of devices associated with a preview program, a beta program, or other software that executes pre-release software or is not yet available for retail or public use. The second set of devices can be retail devices running release builds of the software. Because the first set of devices may differ from the larger second set of devices and / or may be used in a different manner than the larger second set of devices 108, predictive quality metrics indicative of how well a pre-release version or pre-release build of the software executes or functions across the larger second set of devices can be generated. That is, by leveraging the similarities and differences between the first and second sets of devices, quality metrics of the software expected to execute on the second set of devices can be predicted based on the quality metrics of the software executing on the first set of devices. Therefore, the feature data received at 608 can be dedicated to quality metrics and can be received from both the first and second sets of devices. Quality metrics can be associated with specific device functionality, software executing on the device, and / or services executing on the device. For example, a quality metric could be equal to the Mean Time To Failure (MTTF) of a service such as a print spooler, installation service, or window manager. As a non-limiting example, a quality metric could be the MTTF of a service or an executable associated with graphics rendering. As another non-limiting example, a quality metric could be the number of crashes associated with the printing functionality of a word processing application, where such a quality metric could be used to indicate the quality associated with an update to the operating system. The feature data received at 608 could originate from a first set of devices executing a pre-release version of the software, such as a pre-release version of the operating system or an update to the operating system. The feature data received at 608 could also originate from a second set of devices executing a stable release of the operating system or a stable release of an update to the operating system, where a stable release corresponds to a version of the software released prior to the pre-release version of the software.

[0059] At 612, the feature data received at 608 can be preprocessed to clean the data; in some examples, this cleaning may include removing null values ​​and segmenting and classifying the data. At 616, based on the preprocessed data, training and testing data can be generated for training a random forest classification model or other tree-based classification models. For example, the preprocessed data may be provided in a format specific to the random forest classification model. A portion of the preprocessed data can be used to train the random forest classification model at 620, and once trained, the remainder of the preprocessed data can be used at 624 to predict feature data representing covariates that have a high impact on the quality metric. For example, the random forest classification model can score features representing covariates that influence the quality metric; each feature score can indicate the feature's impact on the quality metric. For example, each feature with a score above a threshold can be selected as a feature with a high impact on the quality metric. At 628, a subset of identified covariates with low impact on the quality metric can be removed based on feature imbalance. Feature imbalance can be assessed so that features that have a high impact on quality metrics and represent the imbalance between the first and second groups of devices are utilized in the coarsening and fine-matching process. In other words, imbalance can represent the level of difference that exists between the first and second groups of devices.

[0060] Go to Figure 6B Features that have a high impact on quality metrics and indicate an imbalance between the first and second groups of devices can be coarsened or stratified into smaller, more defined groups or buckets based on a predetermined set of criteria, which may correspond to a distribution such as a Gaussian distribution and several layers or buckets. Features exhibiting imbalance can be selected and utilized during the coarsening-precision matching process performed by the coarsening-precision matching module. Coarsening-precision matching is a causal inference technique for non-parametrically creating matching datasets to evaluate the impact of a treatment on a control group. According to examples of this disclosure, feature data associated with the larger second group of devices can be considered the control group, while feature data associated with the smaller first group of devices can be considered the treatment group. Therefore, the impact of pre-release software, such as on retail devices in the second group of devices, can be assessed using coarsening-precision matching. As previously discussed... Figures 1-3 As described, at position 636, the coarsening exact matching process can generate weights associated with each coarsening feature, such that the quality metric derived from the first group of devices can be used to predict the quality metric of a second, different group of devices based on the similarity and differences between the two groups of devices. In some examples, the coarsening exact matching process can generate groups that do not have matching members. For example, if a group of devices in the first group has a specific BIN sequence that is not found in the second group, then at position 640, the group of devices in the first group with the specific BIN sequence can be discarded from the weighting process.

[0061] This method can be advanced to 644, where quality metrics can be generated from the first set using weights associated with the coarsening features. For example, as... Figure 3 As described herein, the quality metrics determined for each device can be weighted such that the weighted average of the quality metrics derived from the devices in the first group represents the quality metric associated with the second group. Therefore, at 644, the generated quality metrics can be stored and / or provided to a dashboard for viewing by client devices. Method 600 can then end at 648.

[0062] Figure 7 Details of a method 700 according to an example of this disclosure are depicted, which is used to determine whether an action should be taken based on quality metrics of a first group of devices and predicted quality metrics of a second group of devices. Figure 7 The general sequence of steps in method 700 is shown. Typically, method 700 begins at 704 and ends at 728. Method 700 may include more or fewer steps, or may be combined with… Figure 7 The steps shown are arranged in different orders. Method 700 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 700 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), or other hardware device. References will be incorporated herein by reference. Figure 1 - The system, components, modules, software, data structures, user interface, etc. described in Figure 6 are used to explain method 700.

[0063] The method begins at 704, with the process proceeding to 708. At 708, a quality metric for the current build associated with the first group of devices is received. For example, the quality metric may be the same as or similar to quality metric 140 provided from dashboard 136 and displayed on client device 144. At 712, a predicted quality metric based on the current build of the second group of devices is received. For example, the predicted quality metric may be the same as or similar to predicted quality metric 180 provided from dashboard 136 and displayed on client device 144. At 716, method 700 may determine whether the quality metric for the current build of the first group of devices is related to the predicted quality metric based on the current build of the second group of devices. In some examples, the two quality metrics may be related if the difference between them is less than a threshold. If the two quality metrics are not related, method 700 may proceed to 720, where an alert condition may be triggered. For example, a notification such as a text message, email, or visual indication on a display may provide an indication to the user that the two quality metrics are not related.

[0064] In some examples, and as depicted at 724, if two quality metrics are correlated, the quality metric associated with the first group and / or the predicted quality metric associated with the second group may fall below a threshold, or be out of range due to defects, flaws, or other characteristics of the pre-release software. In this case, an alert condition may be triggered at 720. For example, a notification such as a text message, email, or visual indication on a display may provide the user with an indication that both quality metrics are below the threshold. In some examples, if the quality metric of the current build of the first group of devices falls below the threshold, but the predicted quality metric does not fall below the same or a different threshold, this difference may be partly due to a difference between the first and second group of devices, and a notification such as a text message, email, or visual indication on a display may provide this indication. In some examples, the difference may be partly due to a vulnerability that has a significant impact on the second group of devices but not the first group of devices; alternatively, the difference may be partly due to a defect that has a significant impact on the first group of devices but not the second group of devices. Therefore, in the case where a quality metric or predicted quality metric is out of range, a notification (such as a text message, email, or visual indication on a display) may provide the user with an indication of the same situation. Method 700 can end at 728.

[0065] Figure 8 Details of method 800 according to an example of this disclosure are described. Method 800 is used to generate a model that provides a predicted quality metric for a second group of devices based on a quality metric associated with a first group of devices. The first group of devices may correspond to a first group of devices 104, and the second group of devices may correspond to a second group of devices 108. The general sequence of the steps of method 800 is as follows: Figure 8 The following is shown. Typically, method 800 begins at 804 and ends at 824. Method 800 may include more or fewer steps, or may be combined with... Figure 8 The steps shown are arranged in different orders. Method 800 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 800 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), or other hardware device. References will be incorporated herein by reference. Figures 1-7 Method 800 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0066] The method begins at 804, where the process can proceed to 808. At 808, a quality metric associated with the first group of devices can be received. For example, the quality metric can be determined from empirical data and can be based on multiple features received from telemetry information originating from the first group of devices. That is, the quality metric can be associated with... Figure 1 The quality metric generator 132 depicted generates the same or similar quality metrics. In some examples, quality metrics can be determined by consulting engineers and scientists with knowledge specific to the software being measured. For example, engineers or scientists on a particular software development team can understand that a specific quality metric based on multiple features provides an indication of actual quality and user enjoyment of success. The method can proceed to 812, where a prototype model can be generated based on the quality metrics and features identified and received at 808. For example, a prototype predictive quality metric generator 904 can be assembled based on the quality metrics and identified and received features at 808. At 816, the prototype can be validated by determining whether the features identified and received at 808 provide a quality metric that matches the actual quality metrics of the second set of devices. For example, predictive quality metrics can be generated for a pre-release version of the software that has already been released to the second set of devices. Thus, as part of the validation process, the predictive quality metrics can be compared with the actual quality metrics. At 820, method 800 can proceed to operationalize the prototype, thereby making the prototype operational and providing predictive quality metrics based on quality metrics associated with pre-release software candidates. Method 800 can end at 824.

[0067] Figure 9 A second example of a predictive quality metric generator 900 according to the examples in this disclosure is depicted. The predictive quality metric generator 900 is similar to... Figure 1 The prediction quality metric generator 900 can utilize any tree-based classifier 912 to operate on the preprocessed data provided by the telemetry data preprocessing module 908. That is, the prediction quality metric generator 900 utilizes a random forest classification model 168; however, any tree-based classifier, such as, for example, a boosting tree, can also be used. Furthermore, the prediction quality metric generator 900 utilizes a coarse-fit exact match model as a causal inference technique to create a matching dataset between two groups of devices. The prediction quality metric generator 900 can utilize any causal inference matcher, such as a propensity score matching model. Therefore, the prediction quality metric from the quality metric generator 920 can be based on a tree-based classifier 912 and a causal inference matcher 916.

[0068] Figures 10-12 The associated description provides a discussion of various operating environments in which the aspects of this disclosure can be practiced. However, regarding Figures 10-12The devices and systems illustrated and discussed are for illustrative purposes only and are not intended to limit the wide range of computing device configurations that can be used to practice the aspects of this disclosure described herein.

[0069] Figure 10 This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 1000 that can be used to implement various aspects of the present disclosure. The computing device components described below are applicable to the computing device described above. In a basic configuration, the computing device 1000 may include at least one processing unit 1002 and system memory 1004. Depending on the configuration and type of the computing device, the system memory 1004 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of these memories.

[0070] System memory 1004 may include an operating system 1005 and one or more program modules 1006 adapted to run software applications 1007, such as, but not limited to, a predictive quality metric generator 1023, a quality metric generator 1025, and a dashboard 1026, and / or one or more components supported by the system described herein. As per (but not limited to) at least [the provisions of this disclosure] Figures 1-9 As described, the predictive quality metric generator 1023 may be the same as or similar to the predictive quality metric generator 1060; the quality metric generator 1025 may be the same as or similar to the quality metric generator 132; and the dashboard 1026 may be the same as or similar to the dashboard 136. For example, the operating system 1005 may be adapted to control the operation of the computing device 1000.

[0071] Furthermore, embodiments of this disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other applications, and embodiments of this disclosure are not limited to any particular application or system. This basic configuration is... Figure 10 The components within the dashed line 1008 are illustrated. The computing device 1000 may have additional features or functionalities. For example, the computing device 1000 may also include additional data storage devices (removable and / or non-removable), such as, for example, a hard disk, optical disk, or magnetic tape. Such additional storage... Figure 10 The diagram shows a removable storage device 1009 and a non-removable storage device 1010.

[0072] As stated above, several program modules and data files may be stored in system memory 804. When executed on at least one processing unit 802, program module 806 may perform processes, including but not limited to one or more aspects as described herein. Other program modules that may be used according to various aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, PowerPoint presentation applications, drawing or computer-aided applications, and / or one or more components supported by the system described herein.

[0073] Furthermore, embodiments of this disclosure can be practiced in electronic circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, embodiments of this disclosure can be practiced via a system-on-a-chip (SoC), wherein... Figure 10 Each or more of the components illustrated in the diagram can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functionalities, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the functionality described herein regarding the ability to switch client protocols can be operated via dedicated logic integrated on a single integrated circuit (chip) along with other components of the computing device 1000. Embodiments of this disclosure can also be practiced using other techniques capable of performing logical operations (such as, for example, AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure can be practiced within a general-purpose computer or in any other circuit or system.

[0074] The computing device 1000 may also have one or more input devices 1012, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. It may also include output devices(s) 1014A, such as a monitor, speaker, printer, etc. It may also include an output 1014B corresponding to a virtual display. The foregoing devices are examples and other devices may be used. The computing device 1000 may include one or more communication connections 1016 that allow communication with other computing devices 1050. Examples of suitable communication connections 1016 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial buses (USB), parallel and / or serial ports.

[0075] The term "computer-readable medium" as used herein can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, or program modules. System memory 1004, removable storage device 1009, and non-removable storage device 1010 are examples of computer storage media (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and can be accessed by computing device 1000. Any such computer storage medium may be part of computing device 1000. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0076] Communication media can be implemented by computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and include any information delivery medium. The term "modulated data signal" can describe a signal having one or more characteristics that are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media such as wired networks or direct-line connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0077] Figure 11A and Figure 11B The illustration depicts a computing device or mobile computing device 1100, such as a mobile phone, smartphone, wearable computer (e.g., smartwatch), tablet computer, laptop computer, etc., which can be used to practice various aspects of this disclosure. Reference Figure 11AThe illustration depicts one aspect of a mobile computing device 1100 used to implement these aspects. In a basic configuration, the mobile computing device 1100 is a handheld computer with input and output elements. The mobile computing device 1100 typically includes a display 1105 and one or more input buttons 1110 that allow users to input information into the mobile computing device 1100. The display 1105 of the mobile computing device 1100 can also be used as an input device (e.g., a touchscreen display). If included, optional side input elements 1115 allow for further user input. The side input elements 1115 can be rotary switches, buttons, or any other type of manual input element. In alternative aspects, the mobile computing device 1100 may include more or fewer input elements. For example, in some aspects, the display 1105 may not be a touchscreen. In yet another alternative aspect, the mobile computing device 1100 is a portable telephone system, such as a cellular phone. The mobile computing device 1100 may also include an optional keypad 1135. The optional keypad 1135 can be a physical keyboard or a “soft” keyboard generated on a touchscreen display. In various aspects, the output elements include a display 1105 for displaying a graphical user interface (GUI), a visual indicator 1131 (e.g., a light-emitting diode), and / or an audio transducer 1125 (e.g., a speaker). In some aspects, the mobile computing device 1100 includes a vibration transducer for providing tactile feedback to a user. In another aspect, the mobile computing device 1100 includes input and / or output ports for sending signals to or receiving signals from external sources, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., an HDMI port).

[0078] Figure 11B This is a block diagram illustrating one aspect of the architecture of a computing device, server, or mobile computing device. That is, computing device 1100 can be combined with system (e.g., architecture) 1102 to implement certain aspects. System 902 can be implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 1102 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and cordless phone.

[0079] One or more applications 1166 may be loaded into memory 1162 and run on or associated with operating system 1164. Examples of applications include telephone dialer programs, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, and / or one or more components supported by the system described herein. System 1102 also includes a non-volatile storage area 1168 within memory 1162. The non-volatile storage area 1168 may be used to store persistent information that will not be lost when system 1102 is powered off. Applications 1166 may use information (such as emails or other messages used by email applications) and store it in the non-volatile storage area 1168. A synchronization application (not shown) also resides on system 1102 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 1168 synchronized with the corresponding information stored on the host computer. It should be understood that other applications can be loaded into memory 1162 and run on the mobile computing device 1100 described herein (e.g., prediction quality metric generator 1023, quality metric generator 1025, and dashboard 1026, etc.).

[0080] System 1102 has a power supply 1170, which can be implemented as one or more batteries. The power supply 1170 may also include an external power source (such as an AC adapter) or a power docking bracket for replenishing or recharging the batteries.

[0081] System 1102 may also include a radio interface layer 1172, which performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 1172 facilitates wireless connectivity between system 1102 and the "outside world" via a communication carrier or service provider. Transmissions to and from the radio interface layer 1172 are conducted under the control of the operating system 1164. In other words, communications received by the radio interface layer 1172 can be distributed to application 1166 via the operating system 1164, and vice versa.

[0082] A visual indicator 1120 can be used to provide visual notifications, and / or an audio interface 1174 can be used to generate audible notifications via an audio transducer 1125. In the illustrated configuration, the visual indicator 1120 is a light-emitting diode (LED), and the audio transducer 1125 is a speaker. These devices can be directly coupled to a power supply 1170 so that when activated, they remain on for the duration indicated by the notification mechanism, even if the processor 1160 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 1174 is used to provide and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 1125, the audio interface 1174 can also be coupled to a microphone to receive audible input, such as for facilitating telephone conversations. According to various aspects of this disclosure, as will be described below, the microphone can also be used as an audio sensor to facilitate control of the notification. System 1102 may also include a video interface 1176, which enables the operation of the vehicle-mounted camera to record still images, video streams, etc.

[0083] The mobile computing device 1100 implementing system 1102 may have additional features or functionalities. For example, the mobile computing device 1100 may also include additional data storage devices (removable and / or non-removable) such as disks, optical discs, or magnetic tapes. Such additional storage... Figure 11B The diagram shows a non-volatile storage region 1168.

[0084] As described above, data / information generated or captured by mobile computing device 1100 and stored via system 1102 can be locally stored on mobile computing device 1100, or the data can be stored on several storage media, which can be accessed by the device via radio interface layer 1172 or via a wired connection between mobile computing device 1100 and a separate computing device associated with mobile computing device 1100 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via mobile computing device 1100 via radio interface layer 1172 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use based on well-known data / information transfer and storage components (including email and collaborative data / information sharing systems).

[0085] Figure 12The diagram illustrates one aspect of a system architecture for processing data received from a remote source at a computing system, such as the personal computer 1204, tablet computing device 1206, or mobile computing device 1208 described above. The content displayed at the server device 1202 can be stored in different communication channels or other storage types.

[0086] In some aspects, server device 1202 may employ one or more of predictive quality metric generator 1023, quality metric generator 1025, and dashboard 1026. Server device 1202 may provide data to and from client computing devices such as personal computer 1204, tablet computing device 1206, and / or mobile computing device 1208 (e.g., smartphone) via network 1215. By way of example, the computer system described above may be embodied in personal computer 1204, tablet computing device 1206, and / or mobile computing device 1208 (e.g., smartphone). In addition to receiving graphics data that can be preprocessed at the graphics origin system or post-processed at the receiving computing system, any of these embodiments of the computing device may obtain content from repository 1216.

[0087] Figure 12 An exemplary mobile computing device 1200 is illustrated, capable of performing one or more aspects disclosed herein. Furthermore, the aspects and functionalities described herein can operate on a distributed system (e.g., a cloud-based computing system), wherein application functionalities, memory, data storage and retrieval, and various processing functions can operate remotely to each other on a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed and interacted with on a wall surface, or various types of user interfaces and information can be projected onto the wall surface. Interaction with numerous computing systems that can practice embodiments of the present invention includes keystroke input, touchscreen input, voice or other audio input, gesture input, etc., wherein the associated computing device is equipped with detection functionalities (e.g., a camera) for capturing and interpreting user gestures to control the functionality of the computing device.

[0088] The phrases “at least one,” “one or more,” “or,” and “and / or” are open-ended expressions that are both connected and separate in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0089] The term "a" or "an" entity refers to one or more of the same entity. Therefore, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0090] As used herein, the term "automatic" and its variations refer to any process or operation that is typically continuous or semi-continuous, performed without significant human input. However, a process or operation can be automatic if input is received prior to its execution, even if the execution of the process or operation uses significant or insignificant human input. Such human input is considered significant if it affects the extent to which the process or operation is performed. Human input that permits the execution of a process or operation is not considered "significant."

[0091] Any steps, functions, and operations discussed in this article can be performed continuously and automatically.

[0092] Exemplary systems and methods of this disclosure have been described with respect to computing devices. However, to avoid unnecessarily obscuring this disclosure, many known structures and devices have been omitted from the foregoing description. Such omissions should not be construed as limiting. Specific details are set forth to provide an understanding of this disclosure. However, it should be understood that this disclosure can be practiced in various ways other than those set forth herein.

[0093] Furthermore, while the exemplary aspects illustrated herein show various components of a co-located system, some components of the system may be located remotely, in a remote portion of a distributed network (such as a LAN and / or the Internet), or within a dedicated system. Therefore, it should be understood that system components can be combined into one or more devices, such as servers, communication equipment, or co-located on specific nodes of a distributed network (such as analog and / or digital telecommunications networks, packet-switched networks, or circuit-switched networks). As can be understood from the foregoing description, and for computational efficiency reasons, system components can be arranged anywhere within the distributed network without affecting the operation of the system.

[0094] Furthermore, it should be understood that the various links connecting the elements can be wired or wireless links, or any combination thereof, or any other known or later-developed element capable of providing data to or from the connected element. These wired or wireless links can also be secure links and capable of transmitting encrypted information. For example, the transmission medium used as a link can be any suitable carrier for electrical signals (including coaxial cable, copper wire, and optical fiber), and can take the form of sound waves or light waves (e.g., sound waves or light waves generated during radio wave and infrared data communication).

[0095] Although a flowchart has been drawn for a specific sequence of events, it should be understood that the sequence can be changed, added to, or omitted without substantially affecting the operation of the disclosed configuration and aspects.

[0096] Many variations and modifications of this disclosure may be used. Some features of this disclosure may be provided without providing others.

[0097] In another configuration, the systems and methods of this disclosure may be implemented in combination with a dedicated computer, a programmable microprocessor or microcontroller and (multiple) peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, hardwired electronic or logic circuitry (such as discrete element circuitry), a programmable logic device or gate array (such as a PLD, PLA, FPGA, PAL), a dedicated computer, any comparable component, etc. Generally, any (multiple) devices or components capable of implementing the methods illustrated herein can be used to implement various aspects of this disclosure. Exemplary hardware that may be used in this disclosure includes computers, handheld devices, telephones (e.g., cellular, internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Furthermore, alternative software implementations (including, but not limited to, distributed processing or component / object distributed processing, parallel processing, or virtual machine processing) may be constructed to implement the methods described herein.

[0098] In another configuration, the disclosed method can be readily implemented using software developed in conjunction with an object-oriented or object-based software development environment, which provides portable source code that can be used on a wide variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or entirely in hardware using standard logic circuitry or VLSI design. Whether a system according to this disclosure is implemented using software or hardware depends on the system's speed and / or efficiency requirements, specific functionality, and the particular software or hardware system or microprocessor or microcomputer system being utilized.

[0099] In another configuration, the disclosed method can be implemented in part in software, which can be stored on a storage medium and executed on a general-purpose computer, special-purpose computer, microprocessor, etc., that cooperates with the controller and memory. In these cases, the system and method of this disclosure can be implemented as a program embedded in a personal computer, such as a small application, JAVA®, or CGI script; as a resource residing on a server or computer workstation; or as routines embedded in a dedicated measurement system, system component, etc. The system can also be implemented by physically integrating the system and / or method into a software and / or hardware system.

[0100] This disclosure is not limited to the standards and protocols described herein. Other similar standards and protocols not mentioned herein exist and are considered to be included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically replaced by faster or more efficient equivalents that have essentially the same functionality. Such replacement standards and protocols with the same functionality are considered to be equivalents included in this disclosure.

[0101] According to examples of this disclosure, a method for generating a predicted quality metric is provided. The method may include: receiving first telemetry data from a first group of devices executing first software; generating a quality metric for the first software based on the first telemetry data; receiving second telemetry data from a second group of devices, wherein the second group of devices is different from the first group of devices; identifying covariates affecting the quality metric based on features included in the first and second telemetry data; and performing coarsened exact matching using the identified covariates to generate a predicted quality metric for the first software on the second group of devices.

[0102] At least one aspect of the above method may include: utilizing a tree-based classifier to identify covariates affecting the quality metric based on features included in the first and second telemetry data. At least one aspect of the above method may include: wherein the tree-based classifier is a random forest classifier. At least one aspect of the above method may include: training the tree-based classifier with a subset of data from features in the first and second telemetry data. At least one aspect of the above method may include: performing coarse-fit exact matching using a subset of the identified covariates that are greater than a threshold. At least one aspect of the above method may include: displaying a quality metric for the first software on a display device, the quality metric being close to a predicted quality metric for the first software on a second set of devices. At least one aspect of the above method may include: generating an alert condition when at least one of the quality metric for the first software or the predicted quality metric for the first software is less than a threshold. At least one aspect of the above method may include: wherein the quality metric is the average time of a failure metric for a service performed on the first set of devices. At least one aspect of the above method may include: receiving third telemetry data from a second group of devices executing the second software, generating a second quality metric for the second software based on the third telemetry data, and providing a prediction error based on the difference between the second quality metric for the second software and a predicted quality metric for the first software.

[0103] According to examples of this disclosure, a computer-readable medium is provided. The computer-readable medium may include instructions that, when executed by a processor, cause the processor to: receive first telemetry data from a first group of devices executing first software; generate a quality metric for the first software based on the first telemetry data; receive second telemetry data from a second group of devices, wherein the second group of devices is different from the first group of devices; identify covariates affecting the quality metric based on features included in the first and second telemetry data; perform coarsened exact matching using the identified covariates; identify weights to be assigned to each device in the first group of devices; and generate a predicted quality metric based on the weights assigned to each device in the first group of devices and the identified covariates.

[0104] At least one aspect of the aforementioned computer-readable medium may include instructions that, when executed by a processor, cause the processor to: utilize a tree-based classifier to identify covariates affecting a quality metric based on features included in first and second telemetry data. At least one aspect of the aforementioned computer-readable medium may include: wherein the tree-based classifier is a random forest classifier. At least one aspect of the aforementioned computer-readable medium may include instructions that, when executed by a processor, cause the processor to: train a tree-based classifier using a subset of data from features in the first and second telemetry data. At least one aspect of the aforementioned computer-readable medium may include instructions that, when executed by a processor, cause the processor to: perform coarsening exact matching processing using a subset of identified covariates greater than a threshold. At least one aspect of the aforementioned computer-readable medium may include instructions that, when executed by a processor, cause the processor to: display a quality metric that approximates a predicted quality metric at a display device. At least one aspect of the aforementioned computer-readable medium may include instructions that, when executed by a processor, cause the processor to: generate an alarm condition when at least one of the quality metric or the predicted quality metric is less than a threshold.

[0105] According to examples of this disclosure, a system for generating a predictive quality metric is provided. The system may include a processor and a memory storing instructions that, when executed by the processor, cause the processor to: receive first telemetry data from each of a first group of devices executing first software; generate a quality metric for the first software based on the first telemetry data; receive second telemetry data from a second group of devices, wherein the second group of devices differs from the first group of devices; identify covariates affecting the quality metric based on features included in the first and second telemetry data; stratify the first and second groups of devices based on the identified covariates; reweight each device in the first group of devices; and generate a predictive quality metric based on the weights assigned to each device in the first group of devices and the identified covariates.

[0106] At least one aspect of the aforementioned system may include: wherein the instruction causes the processor to identify covariates affecting a quality metric based on features included in the first and second telemetry data using a tree-based classifier. At least one aspect of the aforementioned system may include: wherein the instruction causes the processor to classify a first group of devices and a second group of devices using a subset of the identified covariates that are greater than a threshold. At least one aspect of the aforementioned system may include: wherein the instruction causes the processor to provide a quality metric to a display device that approximates a predicted quality metric.

[0107] In various configurations and aspects, this disclosure includes components, methods, processes, systems, and / or apparatuses generally as depicted and described herein, including various combinations, sub-combinations, and subsets thereof. Upon understanding this disclosure, those skilled in the art will understand how to make and use the systems and methods disclosed herein. In various configurations and aspects, this disclosure includes providing apparatus and processes in the absence of items not depicted and / or described herein, or in various configurations or aspects thereof (including where such items might have been used in prior apparatus or processes are absent), for example, to improve performance, simplify implementation, and / or reduce implementation costs.

Claims

1. A method for generating a predictive quality metric for software, the method comprising: The computing device receives first telemetry data from a first group of devices executing the first software; The computing device generates a quality metric for the first software based on the first telemetry data; The computing device receives second telemetry data from a second group of devices executing the first software, wherein the second group of devices is different from the first group of devices; The computing device generates a first prediction quality metric for the first software based on the first telemetry data and the second telemetry data, including: The computing device identifies one or more covariates that affect the quality metric for the first software based on features included in the first telemetry data and the second telemetry data; as well as A notification is generated by the computing device when at least one of the quality metric for the first software or the first predicted quality metric for the first software is less than a threshold.

2. The method of claim 1, wherein generating the first predicted quality metric for the first software further comprises: Perform causal inference matching.

3. The method of claim 2, wherein performing the causal inference matching further comprises: Perform propensity score matching.

4. The method of claim 2, wherein performing the causal inference matching further comprises: The identified one or more covariates are used to perform coarse-finite matching to generate the first predicted quality metric for the first software on the second set of devices.

5. The method according to claim 4, further comprising: The coarsened exact match is performed using a subset of the identified one or more covariates that is greater than the threshold.

6. The method according to claim 1, further comprising: A tree-based classifier is used to identify one or more covariates that influence the quality metric for the first software, based on features included in the first and second telemetry data.

7. The method of claim 1, wherein the notification further includes an alarm status, a text message, an email, or a visual indication on a display.

8. The method of claim 1, wherein the quality metric for the first software is a mean time between failures metric for a service performed on the first set of devices.

9. The method according to claim 1, further comprising: Receive third telemetry data from the second group of devices executing the second software; A second prediction quality metric for the second software is generated based on the third telemetry data. as well as The prediction error is provided based on the difference between the second prediction quality metric for the second software and the first prediction quality metric for the first software.

10. A computer storage medium storing instructions that, when executed by a processor, cause the processor to: Receive first telemetry data from the first group of devices executing the first software; Generate a quality metric for the first software based on the first telemetry data; Receive second telemetry data from a second group of devices that execute the first software, wherein the second group of devices is different from the first group of devices; Generating a first prediction quality metric for the first software based on the first telemetry data and the second telemetry data includes: One or more covariates that influence the quality metric for the first software are identified based on features included in the first telemetry data and the second telemetry data. as well as A notification is generated when at least one of the quality metric for the first software or the first predicted quality metric for the first software is less than a threshold.

11. The computer storage medium of claim 10, wherein generating the first predicted quality metric for the first software further comprises: Perform causal inference matching.

12. The computer storage medium of claim 11, wherein performing the causal inference matching further comprises: Perform propensity score matching.

13. The computer storage medium of claim 11, wherein performing the causal inference matching further comprises: The identified one or more covariates are used to perform coarse-finite matching to generate the first predicted quality metric for the first software on the second set of devices.

14. The computer storage medium of claim 13, wherein the instructions further cause the processor to perform the coarsened exact match using a subset of the identified one or more covariates that is greater than a threshold.

15. The computer storage medium of claim 10, wherein the instructions further cause the processor to: use a tree-based classifier to identify the one or more covariates affecting the quality metric for the first software based on features included in the first telemetry data and the second telemetry data.

16. The computer storage medium of claim 10, wherein the notification further includes an alarm status, a text message, an email, or a visual indication on a display.

17. A system for generating predictive quality metrics for software, the system comprising: processor; as well as A memory for storing instructions that, when executed by the processor, cause the processor to: Receive first telemetry data from the first group of devices executing the first software; Generate a quality metric for the first software based on the first telemetry data; Receive second telemetry data from a second group of devices that execute the first software, wherein the second group of devices is different from the first group of devices; One or more covariates that affect the quality metric for the first software are identified based on features included in the first and second telemetry data. A first prediction quality metric for the first software is generated based on the first telemetry data and the second telemetry data. as well as A notification is generated when at least one of the quality metric for the first software or the first predicted quality metric for the first software is less than a threshold.

18. The system of claim 17, wherein generating the first prediction quality metric for the first software further comprises: Perform causal inference matching.

19. The system of claim 18, wherein performing the causal inference matching further comprises: Perform propensity score matching.

20. The system of claim 18, wherein performing the causal inference matching further comprises: The first group of devices and the second group of devices are stratified based on the identified one or more covariates; Based on the first group of devices and the second group of devices in the hierarchy, weights are assigned to each device in the first group of devices; as well as Based on the weights assigned to each device in the first group of devices and the identified one or more covariates, a first predictive quality metric is generated for the first software.

Images