Detect code defects through analysis of telemetry data across internal validation loops

By monitoring operation failure signals in cloud-based systems, identifying operation failure peaks in the development loop, and comparing them with previous versions, the problem of high cost of code defect diagnosis is solved, and automated diagnosis with low error rate is achieved, and processing costs are reduced. .

CN112703486BActive Publication Date: 2024-12-03MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN201980059635.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-09-12
Filing Date
2019-06-25
Publication Date
2024-12-03
Estimated Expiration
2039-06-25

AI Technical Summary

Technical Problem

In cloud-based systems, it is difficult to determine whether operational failures are related to code flaws in new software builds, or whether they are not related to noise, resulting in high diagnostic costs and high error rates.

Method used

By monitoring the operation failure signals of cloud-based application services, use the system monitor to identify operation failure peaks in the outermost development ring and previous development ring, and analyze whether there are corresponding peaks in previous versions. If not, mark them as code Defects and notify IT personnel to debug.

Benefits of technology

It realizes near real-time automated operation failure spike analysis, reduces memory costs for manual code diagnosis, reduces new software build release processing costs, and significantly reduces error rates, reducing the need for code removal and replacement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112703486B_ABST
    Figure CN112703486B_ABST
Patent Text Reader

Abstract

In a non-limiting example of the present disclosure, systems, methods, and devices for detecting code defects associated with cloud-based services are presented. A correlation can be identified between an operation failure spike in the outermost development ring preceding the production environment of a current software build and an operation failure spike in a development ring preceding the outermost development ring of the current software build. It can be determined that there are no operation failure spikes corresponding to the spikes in the outermost development ring in the previous software build. It can be determined that there are no operation failure spikes corresponding to the development ring preceding the outermost development ring of the previous software build. Based on the discovery, the operation associated with the failure spike can be marked as related to the code defect.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] As computing has increasingly moved to the cloud, the systems that support large numbers of users and the cloud-based applications they utilize are constantly being modified. The infrastructure of cloud-based systems requires constant monitoring to maintain, not to mention updating and adding additional features and functionality. As new software builds are added to the infrastructure of cloud-based systems through development cycles and ultimately production environments, it can be difficult to determine whether an operational failure is related to a code defect in the new software build, or whether it simply represents noise that may be completely unrelated to the code defects in those builds.

[0002] Aspects of the present technology disclosed herein have been contemplated relative to this general technology environment.Furthermore, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background. Summary of the invention

[0003] This summary is provided to introduce some concepts in a simplified form, which are further described in the "Detailed Description" section below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description below and will become apparent in part from the description, or may be learned by practicing the disclosure.

[0004] Non-limiting examples of the present disclosure describe systems, methods, and devices for identifying code defects for software builds based on telemetry data received from the current and previous versions of the software build. The telemetry data may include information describing operational failures of cloud-based applications associated with the software build. Operational failure spikes may be identified in the outermost development ring before the production environment ring, and corresponding operational failure spikes in the outermost development ring for the previous version of the software build may be verified as not occurring therein. Operational failure spikes may also be identified in the ring of the current software build before the outermost development ring. The previous ring in which the spike is identified may be an engineering ring or an internal test ring. The corresponding operational failure spike may be identified in the previous version of the software build as not occurring therein. Based on the above identification / determination, code defects for the current software build may be identified. The system monitor may mark the code defect and send a notification to IT personnel for debugging. The notification may include a graphical representation of the operational failure spike in the ring in which the operational failure spike is detected and / or identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting and non-exhaustive examples are described with reference to the following figures:

[0006] Figure 1 is a schematic diagram illustrating an exemplary distributed computing environment for detecting code defects associated with a cloud-based application service.

[0007] Figure 2 Illustrated are rings that may be monitored as part of a software build implemented in a cloud-based application service.

[0008] Figure 3 An exemplary user interface is illustrated for indicating that operational failure spikes in an external development loop and an engineering development loop are correlated with each other and indicate a code defect.

[0009] Figure 4 An exemplary method for detecting code defects associated with a cloud-based application service is provided.

[0010] Figure 5 Another exemplary method for detecting code defects associated with a cloud-based application service.

[0011] Figure 6 and Figure 7 is a simplified diagram of a mobile computing device in which aspects of the present disclosure may be practiced.

[0012] Figure 8 is a block diagram illustrating exemplary physical components of a computing device in which aspects of the invention may be practiced.

[0013] Fig. 9 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0014] Various embodiments will be described in detail with reference to the accompanying drawings, wherein like reference numerals represent like parts and components throughout the several views. Reference to various embodiments does not limit the scope of the appended claims. In addition, any examples set forth in this specification are not intended to be limiting, but merely set forth some of the many possible embodiments for the appended claims.

[0015] The various embodiments and examples described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily appreciate that various modifications and changes may be made without following the exemplary embodiments and applications illustrated and described herein, and without departing from the true spirit and scope of the claims.

[0016] Examples of the present disclosure provide systems, methods, and devices for detecting code defects associated with cloud-based application services. For example, the code defects may be included in new software packages, updates, and / or patches. In some examples, the code defects may be associated with one or more cloud-based applications, such as cloud-based document processing applications, spreadsheet applications, calendar applications, presentation applications, storage applications, video applications, real-time electronic messaging applications, voice messaging applications, video communication applications, and / or email applications. In some examples, a system monitor may monitor signals associated with failures of operations involving one or more cloud-based applications. For example, when an operation for a cloud-based application fails, and / or an operation for a cloud-based application causes an application crash or failure when it is executed, a signal indicating the presence of an operation event or an operation failure may be reported to the system monitor. The system monitor may monitor one or more operations associated with a cloud-based application. For example, the system monitor may monitor a "save" operation, a "send" operation, a "new document" operation, a "copy" operation, a "paste" operation, and any other operation that may be performed by a cloud-based application.

[0017] According to an example, the system monitor may monitor operation failure signals from the outermost development ring (i.e., the development ring before the production environment) for software building. In some examples, when the system monitor has received and / or detected a threshold number of operation failure signals for an operation within a threshold duration, the system monitor may determine whether a similar operation failure spike has occurred in a previous development ring for software building. That is, the system monitor may determine whether a threshold number of operation failure signals for the operation was also received, reported, and / or otherwise identifiable in the previous development ring during the threshold duration for software building. In some examples, due to the larger number of operations being executed by a larger user base in the outer development ring, the number of operation failures required to identify an operation spike in the outermost development ring may be higher than the number of operation failures required to identify an operation spike in the previous development ring. In addition, in some examples, rather than simply using absolute numbers to identify an operation failure spike, a ratio and / or percentage of failed operations to non-failed operations for a given operation may be used when determining whether an operation failure spike has occurred.

[0018] If an operation failure spike for a particular operation in the outermost development ring of a software build has been identified, and a corresponding operation failure spike for the particular operation has also been identified in a previous development ring of the software build, the system monitor can then identify whether the operation spike for the particular operation also exists in a previous version of the software build. That is, the system monitor can identify whether operation failure spikes corresponding to those operation failure spikes identified in the new software build also appear in one or more rings and / or environments in the previous version of the software build. In some examples, the system monitor can analyze past operation failure data for one or more previous versions of the software build to make this determination. According to some examples, if the system monitor identifies a corresponding operation failure spike in a previous version of the software build as not occurring in each corresponding development ring (e.g., the outermost development ring and the previous development ring), the system monitor can mark the operation failure and its corresponding spike in the current software build as being related to a code defect (rather than being related to system noise).

[0019] In other examples, further verification can be performed before marking the operation failure and / or its corresponding spike as being related to a code defect. That is, in some examples, the system monitor can additionally determine whether an operation error of an elevated level for a specific operation has occurred in the production environment for the current software build and / or the previous version of the software build. In such an example, if the system monitor determines that the elevated level of the operation error for the specific operation has not occurred in the production environment for the previous version of the software build, and / or the elevated level of the operation error for the specific operation does exist in the production environment of the current software build, then the system monitor can subsequently mark the operation failure and its corresponding spike in the current software build as being related to a code defect. In some examples, the elevated level must meet a threshold number of operation failures within a threshold duration so that the system monitor marks the operation failure in the current software build as being related to a code defect. In other examples, the elevated level must meet a threshold ratio and / or percentage of failed operations to successful operations so that the system monitor marks the operation failure in the current software build as being related to a code defect.

[0020] The systems, methods and devices described in this article provide technical advantages for detecting code defects associated with cloud-based application services. Since the automation of operation failure spike analysis can be performed in near real time according to the description in this article, the memory cost associated with storing the current build operation failure for later manual code diagnosis work is reduced. For example, before the current solution, the operation error and report are usually stored for human engineers to analyze, and when the operation error occurs, the report cannot be used for such analysis. By utilizing the system, method and device described in this article, the processing cost associated with the release of the new software build is also reduced. For example, before the current solution, when the engineer determines that the operation failure may be related to the code defect in the current software build, the error rate in those determinations is relatively high compared to what can be achieved according to the current description. Therefore, when such a determination is made, usually at least a portion of the new software build code is rolled back from the new software build to determine whether the code is indeed defective and causes an operation error, and / or which parts of the code are defective and cause an operation error. Due to the lower error rate achieved according to the system, method and device described in this article, the need to remove code and replace code is greatly reduced, thereby significantly reducing the processing, storage and labor costs associated with this.

[0021] Figure 1 1 is a schematic diagram illustrating an exemplary distributed computing environment 100 for detecting code defects associated with a cloud-based application service. Distributed computing environment 100 includes a first development ring 102, which is the outermost development ring for a current software build prior to a production environment; a second development ring 104, which is a development ring prior to first development ring 102 (e.g., second development ring 104 may include a first level of testing with an engineering group of users for the software build, and first development ring 102 may include a second level of testing with a larger internal test user group for the software build); and a production environment 106, which is a ring corresponding to releasing the current software build to users external to the organization developing the current software build. Distributed computing environment 100 also includes a development ring 108, which is the outermost development ring for a previous version of the current software build prior to the production environment; a development ring 110, which is a development ring prior to development ring 108 in a previous version of the software build; and a production environment 112, which is a ring corresponding to releasing a previous software build to users external to the organization developing the previous software build. That is, development ring 108 corresponds to a previous software build of first development ring 102 , development ring 110 corresponds to a previous software build of second development ring 104 , and production environment 112 corresponds to a previous software build of production environment 106 .

[0022] The distributed computing environment 100 also includes a network and processing sub-environment 114, which includes a network 116, via which any computing device in the distributed computing environment 100 can communicate with each other; a server computing device 118; and a previously deployed data storage device 120. A monitoring service that monitors operational data from one or more cloud-based applications can reside on one or more computing devices in the network and processing sub-environment 114. For example, one or more cloud-based applications can report operational errors to the monitoring service. The operational errors may be associated with software builds of cloud-based applications deployed to one or more rings and deployment environments. In other examples, the monitoring service can detect operational errors associated with software builds for cloud-based applications deployed to one or more rings and deployment environments. The monitoring service can determine whether a degradation in the quality of service associated with one or more cloud-based applications corresponds to a code defect associated with one or more operations in a new software build for one or more application operations, or whether the degradation in the quality of service is related to noise in the system.

[0023] The system monitor can detect and / or receive information associated with operational errors of the current software build from users and devices that execute application software of the current software build in the first development ring 102. In some examples, when an operational error occurs, the cloud-based application can automatically report the operational error to the system monitor. In other examples, there can be an opt-in system so that, at least in a production environment, the user must opt-in to allow the operational errors to be automatically reported to the system monitor. In yet other examples, in addition to or instead of receiving operational error notifications, the system monitor can detect operational errors.

[0024] The system monitor also detects and / or receives notifications associated with operational errors of the current software build from users and devices on which application software of the current software build in the second development ring 104 is executed. In some examples, the current software build in the second development ring 104 can be executed simultaneously with the current software build in the first development ring 104 (i.e., the current software build can be deployed in both rings / environments at the same time). In other examples, the current software build in the second development ring 104 can be executed after the current software build has been tested in the first development ring 104.

[0025] The system monitor may analyze data from operational errors detected and / or received from the first development ring 102 and determine whether an operational spike associated with the first development ring 102 occurs and / or satisfies a threshold. In some examples, the threshold may include a number of operational failures for a particular operation within a specified duration. In other examples, the threshold may include a percentage of users in the first development ring 102 who have a particular operational failure within a specified duration. In still other examples, the threshold may include a percentage of a number of operational failures for a particular operation compared to a number of operational successes for the particular operation.

[0026] Regardless, after determining that an operational spike associated with first development ring 102 has occurred, the system monitor may analyze data from operational errors detected and / or received from second development ring 104 and, based on the analysis, determine whether an operational spike associated with second development ring 104 for the same particular operation occurred at a time corresponding to the time when the spike occurred in first development ring 102. That is, the system monitor may: analyze operational failure information from second development ring 104 for an approximate time window in which operational failures occurred for first development ring 102; and determine: whether a same or similar percentage of operational errors were reported for a same or similar duration during the time window with respect to first development ring 102; and / or whether a same or similar number of operational errors were reported for a same or similar duration during the time window with respect to first development ring 102; and / or whether a smaller number of operational errors were reported for a smaller number of test users in second development ring 104 as compared to a larger number of test users in first development ring 102 for a same or similar duration during the time window with respect to first development ring 102. In some examples, the system monitor may look for an operational failure spike in first development ring 102 that is lagging in time from a time window in which an operational failure spike occurs in second development ring 104. That is, based on the number of users in each of first development ring 102, second development ring 104, and / or a delayed release between second development ring 104 and first development ring 102, the system monitor may account for a certain lag time between operational spikes.

[0027] According to some examples, before reporting that an operation spike for a particular operation within a duration corresponds to a code error for the current software build, corresponding to detecting an operation failure spike for the particular operation in each of the first development ring 102 and the second development ring 104 for the current software build, the system monitor may verify that the operation failure spike does not exist in the corresponding ring for the previous software build (e.g., the development ring 108 in the previous software build corresponds to the first development ring 102 in the current software build; and the development ring 110 in the previous software build corresponds to the second development ring 104 in the current software build). That is, the system monitor may analyze the operation error data from the development ring 110 in the previous software build to verify that a failure spike corresponding to the failure spike detected in the second development ring 104 in the current software build has not occurred; and the system monitor may analyze the operation error data from the development ring 108 in the previous software build to verify that a failure spike corresponding to the failure spike detected in the first development ring 102 in the current software build has not occurred. In some examples, the operation data for the previous software build may be stored in the previously deployed data storage device 120 for later analysis by the system monitor. For example, operational data (including operational failure data) for each of development ring 108, development ring 110, and / or production environment 112 may be stored in previously deployed data storage 120. In additional examples, operational data for the current software build may also be stored in previously deployed data storage 120, or in the current software deployment data storage. In other examples, operational data from the current software build and / or previous software builds may be analyzed in real time or near real time.

[0028] According to some examples, if the system monitor determines that an operation spike for a particular operation occurred in a time window of the first development ring 102, and a corresponding spike 104 for the particular operation occurred in the same time window of the second development ring 104, or the two operation spikes are correlated based on lag time as described above, the system monitor can determine that a code error associated with the particular operation has occurred in the current software build. In some examples, the system monitor must first verify that the operation spike corresponding to the spike identified / determined in the first development ring 102 and the second development ring 104 has not occurred in the corresponding production ring (i.e., development ring 108 and development ring 110) in the previous software build before determining that a code error associated with the particular operation has occurred in the current software build. Once this determination has been made, the system monitor can mark the error and notify a user in the failed operation marking environment 122 for diagnosis and / or repair. The user can be an IT support user associated with the cloud-based application service. The marking of the error can include sending a report of the operation failure spike from the first development ring 102 and the second development ring 104 to the user in the failed operation marking environment 122. In some examples, the report may include a graphical representation of the number and / or percentage of operational failures for a particular operation over a duration for each of the first development ring 102 and the second development ring 104. Figure 3 Examples of reports are illustrated and described.

[0029] According to some examples, upon determining that a code error associated with a particular operation has occurred in the current software build, the system monitor may confirm the code error by performing an additional step of determining that an operation spike of the particular operation has occurred for the current software build in the production software environment 106, and / or that a corresponding operation spike of the particular operation has not occurred for a previous version of the software build in the production environment 112. That is, in some examples, in addition to determining that a failure spike of the particular operation has occurred in the first development ring 102 and the second development ring 104, and determining that a corresponding failure spike of the particular operation has not occurred in corresponding development rings (e.g., development ring 108 and development ring 110) of a previous version of the software build, the system monitor may additionally verify that a corresponding operation spike has not occurred in the production environment 112 for the previous version of the software build, and / or a corresponding operation spike has not occurred in the production environment 106 for the current version of the software build. In the example where the system monitor verifies that a corresponding operation spike has indeed occurred in the production environment 106 for the current software build, a graphical representation of the failure spike may be provided to the IT user as part of a report sent in the failed operation tagging environment 122.

[0030] Figure 2Illustrated is a software development ring 200 that can be monitored as part of a software build implemented in a cloud-based application service. The software development ring 200 includes an engineering build ring 202, an internal verification ring one 204, an internal verification ring two 206, and a production environment ring 208. Typically, the software development ring will include at least one engineering ring, such as engineering build 202, in which the software build is released to a relatively small number of software engineers in the organization that created the software build; at least one internal verification ring, such as internal verification ring one 204 and internal verification ring two 206, in which the software build is released to a relatively larger number of users in the organization; and at least one production environment ring, such as production environment ring 208, in which the software build is released for consumption by users outside the organization that developed the software build. Although there is a lot of data available before the production environment ring 208, the software development ring 202 may be used to monitor the software build. Figure 2 In the example of FIG. 2 , there are three rings (i.e., engineering construction ring 202, internal verification ring 1 204, and internal verification ring 2 206), but it should be understood that there may be more or fewer rings before the production environment ring. For example, if there is no internal verification ring 1 204, the internal verification ring 2 206 (because it is the last ring before the production environment) can still be used in combination with the engineering construction ring 202 to implement the present invention.

[0031] Engineering build ring 202 includes a first test ring for a software build. For example, when a software build is first created, it can be sent to a small group of users in a software engineering group that develops the software build. The group of users can include engineering build ring 202. Figure 1 , the engineering build ring 202 may correspond to the second development ring 104 in the current software build and / or the development ring 110 in a previous software build.

[0032] Internal verification ring one 204 includes a second test ring for the software build. For example, when the software build has been verified in a first group of users (e.g., users in the engineering build ring 202), the software build can be released to a relatively larger group of users who are part of the organization that developed the software build. The group of users can include internal verification ring one 204. Internal verification ring one 204 is Figure 2 There is no corresponding ring in , because in Figure 1 There are not two inner test loops in . Instead, Figure 1 A single inner test ring is illustrated (eg, a first development ring 102 in the current version of the software build and a corresponding development ring 108 in the previous version of the software build), and is the outermost development ring prior to the production environment ring.

[0033] Internal verification ring two 206 includes a third test ring for the software build. For example, when the software build has been verified between a first group of users (e.g., users in engineering build ring 202) and a second group of users (e.g., users in internal verification ring one 204), the software build can be released to a larger group of users who are part of the organization that developed the software build. This group of users can include internal verification ring 206. Figure 1 , the internal verification ring two 206 may correspond to the first development ring 102 in the current software build and / or the development ring 108 in a previous version of the software build.

[0034] The production environment ring 208 includes releasing the current software build to users outside the organization that developed the current software build. For example, when the software build has been verified among users in one or more internal and / or engineering rings, the software build can be released as part of a standard product release or as part of a test group of consumers for consumption by users outside the organization. This group of users can include the production environment ring 208.

[0035] Figure 3 An exemplary user interface on computing device 302 is illustrated for indicating that operational failure spikes in the external development ring and the engineering development ring are correlated with each other and indicate code defects. Although graphs for operational failure spikes in the external development ring and the engineering development ring are illustrated, it should be understood that a graph showing operational failure spikes over time for a production environment can similarly be displayed on computing device 302.

[0036] The user interface on the computing device 302 includes a graphical representation 304 of operational errors for a particular operation in the current software build in the external development ring, and a graphical representation 306 of operational errors for a particular operation in the current software build in the engineering development ring. The Y-axis of each of the graphical representation 304 and the graphical representation 306 represents a number of operational errors received / determined by the system monitor for the particular operation, and the X-axis of each of the graphical representation 304 and the graphical representation 306 represents time. The time and scale represented by the X-axis of the graphical representation 304 and the graphical representation 306 are the same.

[0037] In some examples, the scale of the Y axis may be the same for the external development ring and the engineering development ring. This may occur when the number of users in the external development ring and the engineering development ring is the same or similar. In other examples, the scale of the Y axis may be different for the external development ring and the engineering development ring. This may occur when the number of users in the external development ring and the engineering development ring is different. For example, in most cases, the number of users in the engineering development ring will be less than the number of users in the external development ring. Therefore, the scale for the external development ring (e.g., each horizontal line in the graphical representation 304 may represent one thousand users, and each horizontal line in the graphical representation 304 may represent ten thousand users) may be larger than the scale for the engineering development ring (e.g., each horizontal line in the graphical representation 306 may represent ten users, and each horizontal line in the graphical representation 306 may represent one hundred users). Therefore, in order for the system monitor to detect corresponding operational failure spikes between the external development ring and the engineering development ring, the operational failures of duration in the engineering development ring may be significantly less than in the external development ring.

[0038] In additional examples, the Y-axis for the external development ring graphic (graphical representation 304) and the engineering development ring graphic (graphical representation 306) can represent the percentage of users who experienced errors for a particular operation at any given time in each respective ring compared to the percentage of users who did not experience errors for the particular operation at any given time. In other examples, the Y-axis for the external development ring graphic and the engineering development ring graphic can represent the percentage of the number of failed operations for a given operation compared to the number of successfully executed operations for the given operation. In such examples, the scale of the Y-axis in each of the graphical representations 304 and 306 can be the same (e.g., 25%, 50%, 75% for the horizontal lines in each of the external development ring graphic and the engineering development ring graphic).

[0039] In this example, at portion 310 of graphical representation 306, a spike of operational errors for a particular operation is shown to have occurred in the engineering development ring. At portion 308 of graphical representation 304, a corresponding spike of operational errors for a particular operation is shown to have occurred in the external development ring. In this example, the operational failure spike that occurred in the external development ring is later than the operational failure spike in the engineering development ring (i.e., there is a lag between the spikes). In some examples, the lag may be due to the software build being first released to users in the engineering development ring and then being released to the external development ring. In other examples, the lag may be due to initially releasing the software build in the external development ring to a smaller number of users than the engineering development ring, and then releasing the software build to additional users in the external development ring over time.

[0040] Figure 4An exemplary method 400 for detecting code defects associated with a cloud-based application service begins at a start operation, and flow continues to operation 402 .

[0041] At operation 402, the system monitor monitors a plurality of operations associated with a cloud-based application service. The cloud-based application service may host one or more applications, including productivity applications, such as word processing applications, spreadsheet applications, email applications, presentation applications, video applications, calling applications, instant messaging applications, and the like. New versions, patches, and other updates associated with those applications and / or cloud-based services may be rolled out as new software builds that are tested in a plurality of development rings before being released to users outside the development organization in a generation environment for those software builds. According to some examples, the monitoring service may receive notifications when an operation associated with a cloud-based application fails and / or causes one or more elements of the application to fail. In other examples, the monitoring service itself may detect when an operation associated with a cloud-based application fails and / or causes one or more elements of the application to fail. Examples of operations that may be monitored by the system monitor include, but are not limited to, save operations, new document operations, copy operations, paste operations, and the like.

[0042] The process proceeds from operation 402 to operation 404, where a failure spike of a specific operation among a plurality of operations is identified in a first development ring (i.e., the outermost development ring before the production environment) of a software build associated with a cloud-based application service. In some examples, the failure spike includes a threshold number of failed operations of any given type within a threshold duration (e.g., one hundred save operation failures occurred within twenty minutes, fifty create new document operation failures occurred within thirty minutes). In other examples, the failure spike includes a threshold percentage of users who experienced failed operations for any given operation type within a threshold duration (e.g., 10% of users who attempted a save operation within a duration of twenty minutes experienced a failure, and 5% of all users experienced a save operation failure within a duration of thirty minutes). In some examples, the outermost development ring is one of a plurality of internal development rings used to test a new software build before releasing the software to users outside the organization. Prior to this last development ring before the production environment, there may be one or more levels of internal testing. Typically, these internal test rings include at least one engineering user ring, which is less than the number of users of at least one subsequent user ring working for the organization developing the new software build. However, the present invention may be applied in a testing environment with only the engineering ring in the inner ring and / or only organizational users outside the engineering department testing new software builds.

[0043] From operation 404, the flow proceeds to operation 406, where a failure spike of a particular operation in a plurality of operations is identified for a software build in a second development ring preceding the first development ring. As with identifying the failure spike at operation 404, the failure spike identified at operation 406 may include a threshold number of operation failures within a threshold duration and / or a threshold percentage of users experiencing operation failures within a threshold duration. In some examples, the failure spike identified at operation 406 may overlap at least partially in time with the failure spike identified at operation 404. In some examples, the failure spike identified at operation 406 may begin before the spike identified at operation 404. For example, where the second development ring includes a group of engineering users, those users may begin testing new software builds before the time of the next ring of development testing, and therefore, a failure spike in the engineering ring will likely occur before a spike in a ring external to the engineering level.

[0044] Flow proceeds from operation 406 to operation 408, where for the previous version of the software build, it is determined that the failure spike for a particular operation in the plurality of operations did not occur in a development ring corresponding to the first development ring. In some examples, the system monitor may analyze the stored operation data of the previous version of the software build for the ring corresponding to the first development ring in the previous software build. In other examples, the previous version of the software build may be executed by a user concurrently with the current version of the software build, and the system monitor may analyze the data of the previous version of the software build and / or the current version of the software build in real time or near real time. The system monitor may determine that a spike corresponding to the failure spike identified at operation 404 (e.g., a threshold number of operation failures for the operation within a threshold duration, a threshold percentage of users experiencing operation failures for the operation within a threshold duration, a threshold percentage of failed operations for the operation compared to successful operations for the operation within a threshold duration) did not occur in the previous version of the software build for the ring (i.e., a ring in the previous version of the software build corresponding to the outermost first development ring before the production environment in the current software build) and the particular operation being viewed.

[0045] Flow proceeds from operation 408 to operation 410, where, for the previous version of the software build, it is determined that a spike in failures for a particular operation in the plurality of operations did not occur in a development ring corresponding to the second development ring. In some examples, the system monitor may analyze the stored operational data of the previous version of the software build for a ring corresponding to the first development ring in the previous software build. In other examples, the previous version of the software build may be executed by a user concurrently with the current version of the software build, and the system monitor may analyze the data of the previous version of the software build and / or the current version of the software build in real time or near real time. The system monitor may determine that a spike corresponding to the spike in failures identified at operation 406 (e.g., a threshold number of operation failures for the operation within a threshold duration, a threshold percentage of users experiencing operation failures for the operation within a threshold duration, a threshold percentage of failed operations for the operation compared to successful operations for the operation within a threshold duration) did not occur in the previous version of the software build for the ring (i.e., a ring in the previous version of the software build corresponding to the second development ring in the current software build—a ring before the outermost development ring before the production environment) and the particular operation being viewed.

[0046] In some examples, the threshold number of operational errors that must occur within a threshold duration may be different than the number of errors that constitute an operational failure spike in the first and second rings. For example, the closer the development ring is to the center (first level testing), the lower the number and / or percentage may be due to the smaller number of users in those inner rings (e.g., the engineering ring typically has a smaller number of test users than a ring that includes testing with a broader group of users in the organization).

[0047] The process proceeds from operation 410 to operation 412, where a specific operation in the operation is marked as being associated with a code defect. In some examples, before marking the operation as being associated with a code defect, the system monitor can verify that there is no operational failure spike in a specific operation in the production environment ring for a previous version of the software build. In other examples, before marking the operation as being associated with a code defect, the system monitor can identify that there is an operational failure spike in a specific operation in the production environment ring for the current version of the software build. The marking of a specific operation may include sending a notification to one or more IT users associated with the current software build. The notification may include or provide access to one or more graphical representations of operational failure spikes identified / detected in the current and / or previous software builds. In additional examples, the system monitor may generate a real-time graph that includes operational failures of one or more operations over time, and may provide snapshots of operational failures corresponding to code defects and identified spikes as part of the marking process.

[0048] Flow continues from operation 412 to an end operation, and method 400 ends.

[0049] Figure 5 Another exemplary method 500 for detecting code defects associated with a cloud-based application service is shown. The method 500 begins at a start operation, and flow proceeds to operation 502.

[0050] At operation 502, a correlation is identified between an operation failure spike in a first development ring for software build and an operation failure spike in a second development ring for software build. The first development ring for software build is the outermost development ring before the production environment ring. In some examples, the operation failure spike may include a threshold number of operation failures for a given operation within a threshold duration. In other examples, the operation failure spike may include a threshold percentage of users who experienced an operation failure for an operation within a threshold duration. In yet other examples, the operation failure spike may include a threshold percentage of operation failures for a particular operation compared to successful operations for the particular operation within a threshold duration.

[0051] Flow continues from operation 502 to operation 504, where it is determined that there is no operational failure spike corresponding to the first failure spike for the previous version of the software build. The determination may be based on analysis of data stored by a system monitor for the previous version of the software build, or based on analysis of real-time or near real-time data for the previous version of the software build.

[0052] Flow continues from operation 504 to operation 506, where it is determined that there is no operational failure spike corresponding to the second failure spike for the previous version of the software build. As with the determination made at operation 504, it may be determined by the software monitor based on stored data and / or based on real-time or near real-time data for the previous version of the software build that there is no operational failure spike corresponding to the second failure spike for the previous version of the software build.

[0053] In the example where a threshold number of operational errors within a threshold duration indicates a spike in operational failures, the threshold number of operational errors in the outermost development ring preceding the production environment ring may be higher than the ring preceding the outermost development ring preceding the production environment ring because the outermost development ring preceding the production environment ring may have a larger test user group.

[0054] Flow proceeds from operation 506 to operation 508, where the operations associated with the failure spikes are marked as being associated with code defects in the current software build. The marking may include sending an email or other electronic message to one or more users indicating that a code defect associated with the current software build and / or an operation of the current software build has been detected by the system monitor. In an example, a diagram of a timeline associated with the operation failure spikes in one or more development rings and / or production rings may be included with identification of the marked operations.

[0055] Flow continues from operation 508 to an end operation, and method 500 ends.

[0056] Figure 6 and Figure 7 A mobile computing device 600 is illustrated, such as a mobile phone, a smart phone, a wearable computer, a tablet computer, an e-reader, a laptop computer, an AR-compatible computing device, or a VR computing device, with which embodiments of the present disclosure may be practiced. Figure 6 , illustrates one aspect of a mobile computing device 600 for implementing various aspects. In a basic configuration, the mobile computing device 600 is a handheld computer having both input elements and output elements. The mobile computing device 600 typically includes a display 605 and one or more input buttons 610, which allow a user to input information into the mobile computing device 600. The display 605 of the mobile computing device 600 can also be used as an input device (e.g., a touch screen display). If included, the optional side input element 615 allows further user input. The side input element 615 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 600 can incorporate more or fewer input elements. For example, in some embodiments, the display 605 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 600 is a portable telephone system, such as a cellular phone. The mobile computing device 600 may also include an optional keypad 635. The optional keypad 635 may be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, output elements include a display 605 for showing a graphical user interface (GUI), a visual indicator 620 (e.g., a light emitting diode), and / or an audio transducer 625 (e.g., a speaker). In some aspects, the mobile computing device 600 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 600 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0057] Figure 7 700 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, a mobile computing device 700 may incorporate a system (e.g., architecture) 702 to implement some aspects. In one embodiment, the system 702 is implemented as a "smart phone" capable of running one or more applications (e.g., browsers, email, calendars, contact managers, messaging clients, games, and media clients / players). In some aspects, the system 702 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0058] One or more application programs 766 may be loaded into the memory 762 and run on or in association with the operating system 864. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 702 also includes a non-volatile storage area 768 within the memory 762. The non-volatile storage area 768 may be used to store persistent information that should not be lost when the system 702 is powered off. The application program 766 may use information and store it in the non-volatile storage area 768, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 702 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize information stored in the non-volatile storage area 768 with corresponding information stored on the host computer. It should be understood that other applications may be loaded into the memory 762 and run on the mobile computing device 700, including instructions for providing and operating a digital assistant computing platform.

[0059] System 702 has a power source 770, which can be implemented as one or more batteries. Power source 770 can also include an external power source, such as an AC adapter or a powered docking cradle that replenishes or charges the batteries.

[0060] The system 702 may also include a radio interface layer 772 that performs the function of sending and receiving radio frequency communications. The radio interface layer 772 facilitates wireless connections between the system 702 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 772 are performed under the control of the operating system 764. In other words, communications received by the radio interface layer 772 can be disseminated to the application 766 via the operating system 764, and vice versa.

[0061] The visual indicator 620 can be used to provide a visual notification, and / or the audio interface 774 can be used to generate an auditory notification via an audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to the power supply 770 so that when activated, even if the processor 760 and other components may be turned off to save battery power, they remain on for the duration specified by the notification mechanism. The LED can be programmed to remain on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 774 is used to provide an auditory signal to the user and receive an auditory signal from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 774 can also be coupled to a microphone to receive an audible input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to facilitate control of the notification, as will be described below. The system 702 can also include a video interface 776, which enables the operation of the onboard camera 630 to record still images, video streams, etc.

[0062] The mobile computing device 700 implementing the system 702 may have additional features or functionality. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be stored in the mobile computing device 700. Figure 7 This is illustrated by the non-volatile storage area 768.

[0063] As described above, data / information generated or captured by the mobile computing device 700 and stored via the system 702 may be stored locally on the mobile computing device 700, or the data may be stored on any number of storage media accessed by the device via the radio interface layer 722 or via a wired connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be appreciated that such data / information may be accessed via the mobile computing device 700, via the radio interface layer 772, or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transmission and storage units including electronic mail and collaborative data / information sharing systems.

[0064] Figure 8is a block diagram illustrating the physical components (e.g., hardware) of a computing device 800 in which aspects of the present disclosure may be practiced. The computing device components described below may have computer executable instructions for detecting code defects associated with cloud-based application services. In a basic configuration, the computing device 800 may include at least one processing unit 802 and a system memory 804. Depending on the configuration and type of the computing device, the system memory 804 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 804 may include an operating system 805 suitable for running one or more code defect system monitoring programs. The operating system 805 may, for example, be suitable for controlling the operation of the computing device 800. In addition, embodiments of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and embodiments of the present disclosure are not limited to any particular application or system. This basic configuration is Figure 8 808. The computing device 800 may have additional features or functionality. For example, the computing device 800 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage devices may be Figure 8 It is implemented by a removable storage device 809 and a non-removable storage device 810.

[0065] As described above, a plurality of program modules and data files may be stored in the system memory 804. When executed on the processing unit 802, the program module 806 (e.g., the real-time code defect telemetry application 820) may perform processes including, but not limited to, the various aspects described herein. According to an example, when one or more operations associated with the executed software build occur, the operation failure monitor engine 811 may provide a notification to the system monitor. The notification may include additional information about the failure, such as which user experienced the failure, which operation failed, the system crashed or failed in other ways, etc. The spike identification engine 813 may determine whether a threshold number of operations have occurred for a given operation in a given ring within a threshold duration, and / or whether a threshold percentage of users have experienced an operation failure for a given operation in a given ring within a threshold duration. The spike correlation engine 815 may compare spike data from current and previous software builds for the same and different development and production rings. The operation defect marking engine 817 may perform one or more operations associated with notifying one or more users when a code defect has been identified as being associated with an operation failure in one or more development rings for a software build.

[0066] In addition, embodiments of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or microprocessors. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC), wherein the Figure 8 Each or many components illustrated in the figure can be integrated into a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into the chip substrate as a single integrated circuit. When operated via the SOC, the functions described in this article regarding the ability of the client to switch protocols can be operated via dedicated logic integrated with other components of the computing device 800 on a single integrated circuit (chip). Other technologies capable of performing logical operations can also be used to practice the embodiments of the present disclosure, such as, for example, AND, OR, and NOT, including but not limited to: mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure can be practiced in a general-purpose computer or in any other circuit or system.

[0067] The computing device 800 may also have one or more input devices 812, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. It may also include (one or more) output devices 814, such as a display, a speaker, a printer, etc. The aforementioned devices are examples, and other devices may be used. The computing device 800 may include one or more communication connections 816 that allow communication with other computing devices 850. Examples of suitable communication connections 816 include, but are not limited to: radio frequency (RF) transmitters, receivers and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0068] The term "computer-readable medium" as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures or program modules). System memory 804, removable storage device 809 and non-removable storage device 810 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, magnetic tape, disk storage or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 800. Any such computer storage medium may be part of computing device 800. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0069] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode the information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media.

[0070] Fig. 9 One aspect of the architecture of a system for processing data received from a remote source (such as, for example, a personal / general purpose computer 904, a tablet computing device 906, or a mobile computing device 908 as described above) on a computing system is illustrated. The content displayed at the server device 902 may be stored in different communication channels or other storage types. For example, a directory service 922, a web portal 924, a mailbox service 926, an instant messaging storage 928, or a social networking site 930 may be used to store various documents. The program module 806 may be employed by a client communicating with the server device 902, and / or the program module 806 may be employed by the server device 902. The server device 902 may provide data to or from a client computing device (such as a personal / general purpose computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smart phone)) via a network 915. For example, the above reference Figure 6-8The described computer system may be embodied in a personal / general purpose computer 904, a tablet computing device 906, and / or a mobile computing device 908 (e.g., a smartphone). In addition to receiving graphics data that may be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device may obtain content from storage 916.

[0071] For example, the above text describes various aspects of the present disclosure with reference to the block diagrams and / or operational descriptions of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions indicated in the blocks may not occur in the order shown in any flow chart. For example, depending on the functions / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or sometimes the blocks may be executed in the reverse order.

[0072] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed for protection in any way. The various aspects, examples and details provided in this application are considered to be sufficient to convey all and enable others to make and use the best mode of the disclosure claimed for protection. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is combined or individually displayed and described, various features (in structure and method) should be selectively included or omitted to produce an embodiment with a specific feature set. After the description and explanation of the disclosure have been provided, those skilled in the art can envision various variations, modifications and replacement aspects falling within the spirit of the broader aspects of the general concept of the present invention embodied in this application, which do not deviate from the wider scope of the disclosure claimed for protection.

[0073] The various embodiments described above are provided by way of illustration only and should not be construed as limiting the appended claims. Those skilled in the art will readily appreciate that various modifications and changes may be made without following the exemplary embodiments and applications illustrated and described herein and without departing from the true spirit and scope of the appended claims.

Claims

1. A method for detecting code defects associated with a cloud-based application service, the method comprising: identifying a correlation between a first operational failure spike in a first development ring for a software build and a second operational failure spike in a second development ring for the software build, wherein the first development ring is an outermost development ring prior to a production environment for the software build and includes a level of testing for the software build, and wherein the second development ring is prior to the first development ring and includes another level of testing for the software build; determining that an operation failure spike corresponding to the first operation failure spike does not exist for a previous version of the software build; determining that an operational failure spike corresponding to the second operational failure spike does not exist for a previous version of the software build; and marking an operation associated with the failure spike as being related to the code defect; Wherein, the identifiable correlation between a first operation failure spike in the first development ring built for the software and a second operation failure spike in the second development ring built for the software includes: a ratio of the number of operation failures for the operation within a specified duration in the first development ring to the number of operation failures for the operation within a specified duration in the second development ring.

2. The method according to claim 1, wherein: Each of the operation failure spikes includes a plurality of operation failures for the operation within a specified time duration.

3. The method according to claim 1, wherein: The outermost development ring is the last ring in which the software build is tested before it is released in a production environment.

4. The method according to claim 1, wherein: The second development ring is a development ring immediately preceding the first development ring.

5. A system for detecting code defects associated with a cloud-based application service, comprising: a memory for storing executable program code; as well as one or more processors functionally coupled to the memory, the one or more processors responsive to computer-executable instructions contained in the program code and operative to: identifying a correlation between a first operational failure spike in a first development ring for a software build and a second operational failure spike in a second development ring for the software build, wherein the first development ring is an outermost development ring prior to a production environment for the software build and includes a level of testing for the software build, and wherein the second development ring is prior to the first development ring and includes another level of testing for the software build; determining that an operation failure spike corresponding to the first operation failure spike does not exist for a previous version of the software build; determining that an operational failure spike corresponding to the second operational failure spike does not exist for a previous version of the software build; and marking an operation associated with the failure spike as being related to the code defect; Wherein, the identifiable correlation between a first operation failure spike in the first development ring built for the software and a second operation failure spike in the second development ring built for the software includes: a ratio of the number of operation failures for the operation within a specified duration in the first development ring to the number of operation failures for the operation within a specified duration in the second development ring.

6. The system according to claim 5, wherein: Each of the operation failure spikes includes a plurality of operation failures for the operation within a specified time duration.

7. The system according to claim 5, wherein: The outermost development ring is the last ring in which the software build is tested before it is released in a production environment.

8. The system according to claim 5, wherein: The second development ring is a development ring immediately preceding the first development ring.

9. A computer-readable storage device comprising executable instructions that, when executed by one or more processors, assist in detecting code defects associated with a cloud-based application service, the computer-readable storage device comprising instructions executable by the one or more processors to: identifying a correlation between a first operational failure spike in a first development ring for a software build and a second operational failure spike in a second development ring for the software build, wherein The first development ring is an outermost development ring prior to a production environment for the software build and includes a test level for the software build, and wherein the second development ring precedes the first development ring and includes another test level for the software build; determining that an operation failure spike corresponding to the first operation failure spike does not exist for a previous version of the software build; determining that an operational failure spike corresponding to the second operational failure spike does not exist for a previous version of the software build; and marking an operation associated with the failure spike as being related to the code defect; Wherein, the identifiable correlation between a first operation failure spike in the first development ring built for the software and a second operation failure spike in the second development ring built for the software includes: a ratio of the number of operation failures for the operation within a specified duration in the first development ring to the number of operation failures for the operation within a specified duration in the second development ring.

10. The computer-readable storage device of claim 9, wherein: The outermost development ring is the last ring in which the software build is tested before it is released in a production environment.

11. The computer-readable storage device of claim 9, wherein: The second development ring is a development ring immediately preceding the first development ring.

12. The computer-readable storage device of claim 9, wherein: Each of the operation failure spikes includes a plurality of operation failures for the operation within a specified time duration.

Citation Information

Patent Citations

  • Defect localization in software integration tests

    CN105279084A

  • Early risk identification in DevOps environments

    US9921952B2

  • Automatic regression identification

    WO2017044069A1