Method, device and computer readable medium for evaluating production quality

Abstract

The application provides a method, device and equipment for evaluating production quality and a computer readable medium. In the method, the information of self-service with execution errors is recorded in an error detail table during the updating of each application version. Then, the number of error systems of the version publisher can be calculated according to the error detail table, and the total number of systems of the version publisher can be calculated according to the self-service change list. Then, the production quality score of the version publisher is calculated according to at least the number of error systems and the total number of systems. Since the total number of systems and the number of error systems of the version publisher are considered simultaneously when the production quality is evaluated, the production quality of the version publisher with a large number of application versions can be evaluated more accurately, and the problem that the production quality is not accurately evaluated due to repeated statistics of the number of self-services with common errors is avoided, and the accuracy of the evaluation of the production quality is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and computer-readable medium for evaluating production quality. Background Technology

[0002] In various business sectors such as banking and telecommunications, as developers continuously develop and improve the functionality of numerous application systems, the deployment of application versions is becoming increasingly frequent. With the increasing number of deployments, the risk of deployment failure also increases. Therefore, to mitigate this risk, it's common practice to aggregate the number of self-service errors for each release party within a preset timeframe. This aggregated number of self-service errors is used to assess the deployment quality of that release party. Here, "self-service" refers to the application change self-service process, which comprises individual, complete procedures broken down from the application version release process.

[0003] However, research has revealed that existing methods for evaluating deployment quality are not entirely accurate. Specifically, some release providers release a large number of application versions, resulting in a correspondingly higher number of self-service errors. Existing evaluation methods are inadequate for these release providers with a large number of application versions. Furthermore, a single self-service may exhibit multiple common errors, and different self-services may also exhibit common errors due to objective reasons. Because many of the aggregated self-service errors are common (i.e., errors of the same type), the aggregated number of self-service errors cannot accurately reflect the deployment quality of the release provider. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for evaluating production quality, so as to improve the production quality of the release party.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, embodiments of this application disclose a method for evaluating production quality, including:

[0007] Update each application version according to the self-service change list; wherein, the self-service change list includes: relevant information of each application version that needs to be updated; the relevant information of the application version includes: the identifier of the version release party to which the application version belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the production release machine identifier corresponding to the self-service; the self-service is a program after the application version release process is broken down;

[0008] During the update of each application version, the relevant information of the self-service that failed to execute is recorded in the error details table; wherein, the relevant information of the self-service that failed to execute includes: the identifier of the version release party to which the self-service that failed to execute belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service that failed to execute, and the error details of the self-service that failed to execute;

[0009] Based on the error details table, the number of error-prone systems of the version publisher is calculated; and based on the self-service change list, the total number of systems of the version publisher is calculated; wherein, the number of error-prone systems is the total number of application systems with self-services that have execution errors published by the version publisher; and the total number of systems is the total number of application systems published by the version publisher.

[0010] The release quality score of the version release party is calculated based at least on the number of faulty systems and the total number of systems; wherein the release quality score of the version release party is used to evaluate the release quality of the version release party.

[0011] Optionally, in the above-described method for evaluating production quality, calculating the number of error-prone systems for the version release provider based on the error details table includes:

[0012] Deduplication is performed on the same version releaser, application system identifier, and application version deployment time in the error details table to obtain a deduplicated error details table.

[0013] The number of application systems included in the deduplicated error details table is summed to obtain the number of error systems for the version release.

[0014] The step of calculating the total number of systems of the version release party based on the self-service change list includes:

[0015] Deduplication is performed on the same version publisher, application system identifier, and application version deployment time in the self-service change list to obtain a deduplicated self-service change list.

[0016] The total number of application systems included in the deduplicated self-service change list is summed to obtain the total number of systems released by the version publisher.

[0017] Optionally, in the above-described method for evaluating production quality, after recording the relevant information of the self-service that malfunctions in execution to the error detail table, the method further includes:

[0018] If the self-service that caused the execution error is determined to be a fatal factor, then a fatal factor mark is added to the self-service that caused the execution error in the error details table; wherein, a fatal factor is a self-service that causes the application version to be rolled back due to the execution error;

[0019] Based on the error details table, the number of fatal factors is calculated; wherein, the number of fatal factors is the total number of application systems with fatal factors;

[0020] The step of calculating the release quality score for the version release party, based at least on the number of faulty systems and the total number of systems, includes:

[0021] The release quality score of the version release party is calculated based at least on the number of faulty systems, the total number of systems, and the number of fatal factors.

[0022] Optionally, in the above-described method for evaluating production quality, calculating the number of fatal factors based on the error details table includes:

[0023] A fatal factor table is generated from the error details table; wherein, the fatal factor table includes: relevant information of the self-services identified as fatal factors; the relevant information of the self-services identified as fatal factors includes: the identifier of the version release party to which the self-services identified as fatal factors belong, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-services identified as fatal factors, and the error details information of the self-services identified as fatal factors;

[0024] Deduplication is performed on the same version publisher, application system identifier, and application version deployment time in the fatal factor table to obtain a deduplicated fatal factor table.

[0025] The number of application systems included in the deduplicated fatal factor table is summed to obtain the number of fatal factors.

[0026] Optionally, in the above-described method for evaluating the quality of production deployment, after updating each application version according to the self-service change list, the method further includes:

[0027] The performance evaluation factors for the release of statistical versions; wherein, the performance evaluation factors are indicators that affect the production and implementation process of application versions;

[0028] The step of calculating the release quality score of the version release party based at least on the number of faulty systems, the total number of systems, and the number of fatal factors includes:

[0029] The production quality score of the version release party is calculated based on at least the number of faulty systems, the total number of systems, the number of fatal factors, and the statistically obtained assessment factors of the version release party.

[0030] Optionally, in the above-mentioned method for evaluating production quality, the step of calculating the production quality score of the version release party based at least on the number of faulty systems, the total number of systems, the number of fatal factors, and the statistically obtained assessment factors of the version release party includes:

[0031] The self-service failure rate of the version publisher is calculated based on the number of systems that malfunction and the total number of systems; wherein, the self-service failure rate of the version publisher is the probability that the application system released by the version publisher fails to execute.

[0032] A self-service metric score is calculated based on the self-service failure rate of the version release provider; wherein, the self-service metric score is used to illustrate the production quality reflected by the version release provider in the execution of self-service.

[0033] The release quality score of the version release provider is calculated based on the scores corresponding to the assessment factors, the scores corresponding to the critical factors, and the self-service indicator scores. The scores corresponding to the assessment factors are calculated based on the statistically analyzed assessment factors of the version release provider, and these scores illustrate the release quality reflected by the assessment factors. The scores corresponding to the critical factors are calculated based on the number of critical factors, and these scores illustrate the release quality reflected by the critical factors.

[0034] Optionally, the above-mentioned methods for evaluating the quality of production launch may also include:

[0035] During the update of each application version, the execution status of the self-service is recorded in the self-service change list, and the solutions for self-services that fail to execute are recorded in the error details table; wherein, the execution status of the self-service is used to describe the execution status of the self-service on the production machine corresponding to the self-service.

[0036] Secondly, this application discloses a device for evaluating production quality, comprising:

[0037] The update unit is used to update each application version according to the self-service change list; wherein, the self-service change list includes: relevant information of each application version that needs to be updated; the relevant information of the application version includes: the identifier of the version release party to which the application version belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the production release machine identifier corresponding to the self-service; the self-service is a program after the application version release process has been broken down;

[0038] The first recording unit is used to record relevant information of self-services that fail to execute during the process of updating various application versions to an error detail table; wherein, the relevant information of the self-services that fail to execute includes: the identifier of the version publisher to which the self-services that fail to execute belong, the identifier of the application system, the identifier of the application version, the production time of the application version, the identifier of the self-services that fail to execute, and the error details of the self-services that fail to execute.

[0039] The first calculation unit is used to calculate the number of error-prone systems of the version publisher based on the error details table; and to calculate the total number of systems of the version publisher based on the self-service change list; wherein, the number of error-prone systems is the total number of application systems with self-services that have execution errors published by the version publisher; and the total number of systems is the total number of application systems published by the version publisher.

[0040] The second calculation unit is used to calculate the production quality score of the version release party based at least on the number of faulty systems and the total number of systems; wherein the production quality score of the version release party is used to evaluate the production quality of the version release party.

[0041] Thirdly, this application discloses a production quality evaluation device, including a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program, specifically to implement the production quality evaluation method as described in any of the first aspects above.

[0044] Fourthly, this application discloses a computer-readable medium for storing a computer program, which, when executed, is specifically used to implement the production quality evaluation method as described in any of the first aspects above.

[0045] Based on the production quality assessment method provided in the above embodiments of the present invention, each application version is updated according to the self-service change list. Then, during the update process, information related to self-services that malfunctioned is recorded in an error detail table. Based on the error detail table, the number of malfunctioning systems of the version publisher can be calculated, and based on the self-service change list, the total number of systems of the version publisher can be calculated. The number of malfunctioning systems is the total number of application systems with malfunctioning self-services released by the version publisher, and the total number of systems is the total number of application systems released by the version publisher. Then, at least based on the number of malfunctioning systems and the total number of systems, a production quality score for the version publisher is calculated. This production quality score is used to evaluate the production quality of the version publisher. Since the embodiments of this application consider both the total number of systems and the number of malfunctioning systems when assessing production quality, the production quality assessment is more accurate for version publishers with a large number of released application versions compared to existing technologies. Furthermore, by using the number of faulty systems and the total number of systems to assess the quality of production deployment, compared to the method of summarizing the number of self-service errors, the problem of inaccurate assessment of production deployment quality caused by repeatedly counting the number of self-services with common errors can be avoided, thus improving the accuracy of the assessment of production deployment quality. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart illustrating the production quality evaluation method proposed in this application. Figure 1 ;

[0048] Figure 2 This is a schematic diagram illustrating a process for implementing and deploying an application version according to an embodiment of this application.

[0049] Figure 3 This is a flowchart illustrating a method for determining the number of error systems proposed in an embodiment of this application;

[0050] Figure 4 This is a flowchart illustrating a method for determining the total number of systems according to an embodiment of this application;

[0051] Figure 5 A flowchart illustrating the method for determining the number of lethal factors proposed in this application. Figure 1 ;

[0052] Figure 6A flowchart illustrating the method for determining the number of lethal factors proposed in this application. Figure 2 ;

[0053] Figure 7 This is a flowchart illustrating a method for determining production quality score according to an embodiment of this application.

[0054] Figure 8 A flowchart illustrating the production quality evaluation method proposed in this application. Figure 2 ;

[0055] Figure 9 This is a schematic diagram of the structure of a production quality evaluation device proposed in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] See Figure 1 This application proposes a method for evaluating production quality, which can be applied to the version production implementation process and production management platform (hereinafter referred to as the management platform). The management platform is used to implement production management for each application version, and specifically includes the following steps:

[0059] S101. Update each application version according to the self-service change list. The self-service change list includes: relevant information of each application version that needs to be updated. The relevant information of the application version includes: the identifier of the version release party to which the application version belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the production release machine identifier corresponding to the self-service. The self-service is the program after the application version release process is broken down.

[0060] The self-service change list includes information about each application version that needs to be updated. This information includes: the identifier of the version publisher of the application version to be updated, the identifier of the application system to which the application version to be updated belongs, the identifier of the application version to be updated, the deployment time (or update time) of the application version to be updated, the identifier of the self-service corresponding to the application version to be updated, the execution time of the self-service corresponding to the application version to be updated, and the deployment machine identifier of the self-service.

[0061] The self-service change list includes information about each application version that needs to be updated (which can also be understood as information about each application version that needs to be released and put into production). It is mainly used to explain the execution time of the self-service corresponding to the application version released by the version publisher. Then, when the management platform updates the application version released by the version publisher, it can schedule the execution of the self-service according to the execution time of the self-service corresponding to the application version. By executing the self-service corresponding to the application version, the production work of the application version can be completed.

[0062] The following provides a detailed explanation of each item included in the self-service change list:

[0063] The version releaser refers to the individual or team that releases an application version, such as the organization, company, department, or R&D team that releases the application version.

[0064] The publisher's identifier refers to information unique to the publisher, such as the publisher's name or ID. The application system identifier published by the publisher refers to the identifier of the application system to which the published application version belongs. This identifier is application system-specific information, such as the application system's name or ID.

[0065] The application version identifier refers to the identifier of the application version released by the version publisher. Because application systems need continuous updates and iterations, new application versions need to be released constantly. Different application version identifiers distinguish the released application versions. The application version identifier refers to unique information about the application version, such as the version number or the application version name. For example, if a version publisher releases version 1.0 of system A, then the application system name is system A, and the application version number is 1.0.

[0066] The production launch time of an application version can be understood as the time when the application version update is initiated, which can be presented in the form of a date and time.

[0067] The identifier for a self-service corresponding to an application version refers to unique information about that self-service, such as its ID. A self-service corresponding to an application version refers to an independent, complete program formed by breaking down the application version's release process. An application version may have one or more self-services; by executing all the self-services corresponding to that application version, the release process for that application version can be completed.

[0068] The execution time of a self-service refers to the time it takes for the management platform to execute the self-service. The production machine identifier corresponding to the self-service refers to the unique information of the production machine that executes the self-service, such as the name and address of the production machine.

[0069] It should be noted that the number of version publishers included in the self-service change list is not limited in this embodiment of the application. For example, corresponding self-service change lists can be established for different version publishers. That is, the self-service change list corresponding to a certain version publisher only includes information related to the application version released by that version publisher, while information related to the application systems released by other version publishers is included in the self-service change lists corresponding to those other version publishers. Alternatively, the self-service change list can include information related to the application versions released by all version publishers.

[0070] Optionally, in addition to the above information, the self-service change list may also include the self-service's execution status (also known as the self-service's state), planned end time, and performance evaluation factors. The execution status describes the self-service's status on the corresponding production machine. This status can include: not executed (new), failed, or completed. If the execution status changes, it can be updated in the self-service change list. Performance evaluation factors are indicators that affect the application version's production implementation process, such as the application version's lateness rate or rollback rate. The lateness rate refers to the probability that the actual production of an application version is later than its expected production time. The rollback rate refers to the probability of rolling back an application version. Specifically, rolling back an application version refers to the probability of rolling back the application version shown in the self-service change list that needs updating.

[0071] For example, Table 1 below shows a portion of the self-service change list, illustrating information about an application version that needs updating. As shown in Table 1, this application version is divided into self-service 1, self-service 2, self-service 3, self-service 4, self-service 5, and self-service 6. The planned start time in Table 1 can be understood as the execution time of the aforementioned self-services. The planned end time refers to the expected completion time of the self-service execution. The status in Table 1 can be understood as the execution status of the aforementioned self-services. A status of "execution completed" indicates that the self-service has been successfully executed on the production machine; a status of "execution failed" indicates that the self-service was not successfully executed (i.e., an error occurred during the execution); and a status of "new" indicates that the self-service has not started execution. The change operation content in Table 1 can be understood as the production machine corresponding to the aforementioned self-services. Taking self-service 1 as an example, as shown in Table 1, the management platform scheduled self-service 1 to be executed on the gray-scale cluster AP in the production center of City A at 19:00 on August 26, 2022. The expected end time for the execution of self-service 1 is 7:00 on August 27, 2022. The current status of self-service 1 is "execution completed".

[0072] Table 1

[0073]

[0074] Specifically, the process of executing step S101 can be as follows: Read the self-service change list; for each application system displayed in the self-service change list, start updating the application version according to the production deployment time of the corresponding application version. Specifically, read the execution time of each self-service corresponding to the application version in the self-service change list. For each self-service corresponding to the application version, at the execution time of the self-service, schedule the self-service to be executed on the corresponding production machine according to the self-service's identifier and the production machine identifier. After executing each self-service corresponding to the application version in the order of their execution times, the update of the application version is completed.

[0075] Optionally, during the execution of the self-service, the process nodes may include: version acquisition and general backup, application stop, backup, pre-release script execution, version release, post-release script execution, pre-startup script execution, application start, post-startup script execution, version check, green light test, health check, etc. These process nodes may encounter execution errors, leading to self-service failure. Therefore, if the self-service execution status is "execution failed" (i.e., execution error), the cause of the error can be analyzed, and then a response (i.e., a solution) can be decided. If the solution requires reverting the application version to the release party for modification, the self-service execution is forcibly terminated. If the solution does not require reverting the application version to the release party, the handling measures are determined based on the actual situation (e.g., the cause of the error, the impact of the self-service on version updates, etc.). The handling measures include: first measure and second measure. First measure: Re-execute the self-service. Second measure: Skip the error-prone process node in the self-service and continue executing the next process node of the self-service.

[0076] It should be noted that the steps of analyzing the cause of the error and deciding on the corresponding measures can be performed manually, or the relevant programs can be pre-deployed on the management platform and then executed by the management platform, or they can be performed jointly by humans and the management platform. This application embodiment does not limit this.

[0077] Optionally, in a specific embodiment of this application, if there are dependent self-services among multiple self-services corresponding to an application version, the dependency relationships between the self-services can be pre-configured in the management platform. A self-service with dependencies will be invoked after all preceding self-services have been successfully implemented.

[0078] For example, such as Figure 2As shown, the update process for a certain application version during step S101 of the management platform is as follows: When the current time is detected as the production launch time of a certain application version, the version update is initiated, and the management platform begins updating the application version. Specifically, self-service 1 is executed at time point 1. Then, it is determined whether self-service 1 has completed execution. Similarly, self-service 2 is executed at time point 2, and its completion is determined. Self-service 3 is executed at time point 3, and its completion is determined… The dependencies between self-service 1, self-service 2, and self-service 12 are pre-set. When self-service 1 and self-service 2 both execute successfully, self-service 12, which has a dependency on it, is executed, and its completion is determined. If self-service 12 completes, the process ends. When it is determined that self-service 2 has failed, manual intervention is performed to analyze the error cause of self-service 2 and decide on countermeasures. After deciding on the response measures for Service 2, it is determined whether a version rollback is necessary. If so, Service 2 is forcibly terminated; otherwise, a handling measure is decided based on the actual situation (i.e., the error cause of Service 2 and its impact on the overall application version update). The handling measures are Measures 1 (equivalent to the first measure mentioned above) and Measures 2 (equivalent to the second measure mentioned above). When Measures 1 is adopted, the process node where Service 2 reported the error is re-executed. When Measures 2 is adopted, the process node where Service 2 reported the error is skipped, and the next process node of Service 2 is executed. After all process nodes of Service 2 have been executed sequentially, it is determined again whether Service 2 has completed execution.

[0079] Optionally, in a specific embodiment of this application, before executing step S101, the following preparatory work may be included (which can also be understood as preparation work for application version deployment): determining the production environment and parameters of the application version, and preparing the application version to be presented in a self-service form. Here, the production environment refers to the machine on which the application version is deployed, such as the machine running the service in the production server room of a bank's Beijing data center. Parameters refer to the operating parameters of the production environment, such as CPU, Network Attached Storage (NAS), operating system permissions, etc.

[0080] The process of preparing an application version to be presented as a self-service can be as follows: The application version release process is pre-divided according to types such as multi-site active-active and disaster recovery deployment, version content, and version structure, resulting in individual, complete self-services. Multi-site active-active refers to multiple physical addresses (such as Beijing and Wuhan) that can independently provide services (the application in any physical location is active and can provide services at any time). Disaster recovery deployment refers to switching production to a disaster recovery environment when a location experiences an uncontrollable disaster such as an earthquake that prevents normal service from being provided.

[0081] With the self-service change list, the management platform can automatically update each application version on a regular basis according to the production launch time of each application version, effectively saving manpower consumed in the process of updating application versions.

[0082] S102. During the process of updating each application version, the relevant information of the self-service that failed to execute is recorded in the error details table. The relevant information of the self-service that failed to execute includes: the identifier of the version publisher to which the self-service that failed to execute belongs, the identifier of the application system, the identifier of the application version corresponding to the application system, the production time of the application version, the identifier of the self-service that failed to execute, and the error details of the self-service that failed to execute.

[0083] The error details table records the following information about self-services that failed to execute: the identifier of the version publisher, the identifier of the application system published by the version publisher, the identifier of the application version corresponding to the application system, the production release time of the application version, the identifier of the self-service that failed to execute the application version, and the error details of the self-service that failed to execute. The error details table is mainly used to record information about self-services that failed to execute in the self-service change list. All items included in the error details table are related to self-services that failed to execute. That is, if all application versions published by a certain version publisher are successfully updated and there are no self-services that failed to execute, then the identifier of that version publisher, the identifier of the application system published by the version publisher, the identifier of the application version corresponding to the application system, and the production release time of the application version will only exist in the self-service change list used to update the application version, and will not be recorded in the error details table for the information of each application version published by that version publisher.

[0084] Among them, the identifier of the version publisher, the identifier of the application system, the identifier of the application version corresponding to the application system, and the production time of the application version in the error details table are all information related to the self-service that failed to execute. For specific meanings, please refer to the relevant introduction in step S101 above, which will not be repeated here.

[0085] The error details table records the identifier of the self-service that failed to execute the application version. This can be understood as the unique information of the self-service that failed to execute during the execution of step S101. For example, it may be the ID or name of the self-service that failed to execute. The error details of the self-service that failed to execute refers to the error details of the self-service that failed to execute. Specifically, it may be the error reason or error type of the self-service. For example, the error details of the self-service that failed to execute may include: unable to find the keyword XXX, the XXX directory does not exist, the XXX directory does not have write permissions, the XXX database table does not exist, and unable to connect to the network.

[0086] Optionally, one implementation of step S102 includes: for each self-service in the self-service change list, during the execution of the self-service, detecting whether the self-service has been successfully executed; if the self-service is detected to have failed to execute (i.e., an execution error), then the error details of the self-service, the identifier of the version publisher to which the self-service belongs, the identifier of the application system, the identifier of the application version, the production time of the application version, and the identifier of the self-service are all recorded in the error details information until it is verified that the self-service has been successfully executed.

[0087] Optionally, solutions to self-service failures can also be recorded in the error details table. Specifically, when a self-service fails during execution, the cause of the error needs to be analyzed and a corresponding solution developed. To provide more detailed reference information for subsequent evaluation of production quality, the solution can also be recorded in the error details table.

[0088] It should be noted that if the same self-service fails to execute multiple times, the relevant information of the self-service at each time an error occurs can be recorded in the error details table. This will provide the error details table with detailed information about each error of the self-service, which will facilitate subsequent evaluation of production quality and rectification of production implementation.

[0089] It should also be noted that the relevant information of the self-service that failed to execute includes, but is not limited to, the content proposed in the embodiments of this application. For example, it may also include the time point of the execution error, the production machine identifier corresponding to the self-service, and other information.

[0090] S103. Based on the error details table, calculate the number of error-prone systems of the version publisher, where the number of error-prone systems is the total number of self-service application systems with execution errors published by the version publisher.

[0091] Specifically, since the error details table includes information about the self-services that malfunctioned, and as described in step S102 above, this information includes the identifier of the version releaser of the self-service that malfunctioned, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service that malfunctioned, and the error details of the self-service that malfunctioned. In other words, all application systems to which the self-services that malfunctioned are located are recorded in the error details table. Therefore, the number of application systems with malfunctioning self-services can be counted through the error details table, i.e., the number of malfunctioning systems can be calculated. The number of malfunctioning systems can also be understood as the total number of application systems whose self-services malfunctioned.

[0092] Optionally, in a specific embodiment of this application, one implementation of step S103 is as follows: Based on the error detail table, calculate the number of error-prone systems for the version release party within a preset time period. That is, only count the number of error-prone systems within a preset time period (e.g., a certain month) from the error detail table.

[0093] Optionally, see Figure 3 In one specific embodiment of this application, another implementation of step S103 is as follows:

[0094] S301. Perform deduplication on the same version publisher, application system identifier, and application version launch time in the error details table to obtain a deduplicated error details table.

[0095] Specifically, the rows in the error details table that have the same version publisher, application system identifier, and application version release time are deduplicated (i.e., duplicate rows with the same "version publisher, application system identifier, and application version release time"), and finally the deduplicated error details table is obtained.

[0096] For example, as shown in Table 2 below, the error details table shows two duplicate rows for the version release provider (Unit A), application system (Application A), and production launch date (August 27, 2022). Therefore, after deduplication, Table 2 will only retain one of these duplicate rows. However, there are no duplicate rows for the version release provider (Unit A), application system (Application B), and production launch date (August 27, 2022). After deduplication, the row containing these duplicate rows in Table 2 remains unchanged.

[0097] Table 2

[0098]

[0099] S302. Sum the number of application systems included in the deduplicated error details table to obtain the number of error systems for the version release party.

[0100] After deduplication, for each release provider, the deduplicated error details table only retains the row number of self-service application systems with execution errors deployed by that release provider at each deployment time. The total number of error systems for that release provider is obtained by summing the number of application systems included in the deduplicated error details table (i.e., directly counting the number of rows for that release provider in the deduplicated error details table). For example, after deduplication in Table 2 above, only one row remains: one containing Unit A, Application A, with a deployment time of 2022 / 8 / 27, and another row containing Unit A, Application B, with a deployment time of 2022 / 8 / 27. This means that Unit A had a total of 2 error systems on August 27, 2022 (i.e., Application A and Application B).

[0101] Optionally, one implementation of steps S301 to S302 is as follows: The error detail table is calculated using a first formula to obtain the number of error-prone systems. The first formula is: Number of error-prone systems = SUM(Number of rows in the error detail table after deduplication based on (Version Releaser & Production Time & Application System)). Here, "SUM()" represents summation, and "&" signifies "sum". The first formula calculates the total number of rows in the error detail table after deduplicating rows with the same version releaser, production time, and application system.

[0102] Optionally, if the number of error systems for the version release party is the number of error systems for the version release party within a preset time period, then before executing step S301, it may also include: filtering out the number of rows in the error details table whose production time is within the preset time period to obtain the filtered error details table, and then subsequently executing steps S301 to S302 on the filtered error details table to obtain the number of error systems for the version release party within the preset time period.

[0103] The number of error systems of the version release party counted in step S103 can reflect the production quality of the version release party from the perspective of self-service errors. Compared with the existing technology of summarizing the number of self-service errors, it can avoid the problem of inaccurate evaluation of production quality caused by repeatedly counting the number of self-services with common errors, and improve the accuracy of evaluating production quality.

[0104] For example, in the prior art, Table 2 above directly summarizes the number of self-service errors. That is, the number of self-service errors for Unit A is 3, meaning that the common error "Cannot find XXX keyword" that existed when deploying Application A in Unit A is repeatedly accumulated. Using the summarized number of self-service errors to evaluate the deployment quality of Unit A is obviously not very accurate. However, in this embodiment, after the calculation in step S103, the number of error systems for Unit A is calculated to be 2. By calculating the number of error systems, the problem of inaccurate deployment quality evaluation caused by repeatedly counting self-service errors with common errors can be avoided, thus improving the accuracy of deployment quality evaluation.

[0105] S104. Based on the self-service change list, calculate the total number of systems of the version publisher, where the total number of systems is the total number of application systems published by the version publisher.

[0106] Specifically, as described in step S101 above regarding the self-service change list, the list includes information about each application version that needs updating. This information includes: the identifier of the version publisher, the identifier of the application system, the identifier of the application version, the deployment time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the deployment machine identifier of the self-service. Therefore, all application systems that need updating (i.e., those published by the version publisher) are recorded in the self-service change list. Thus, the total number of systems published by the version publisher can be calculated from the self-service change list. The total number of systems published by the version publisher can also be understood as the total number of application systems published by the version publisher that require application version updates.

[0107] Optionally, see Figure 4 In a specific embodiment of this application, one implementation of step S104 is as follows:

[0108] S401. Perform deduplication on the same version publisher, application system identifier, and application version deployment time in the self-service change list to obtain a deduplicated self-service change list.

[0109] The execution process and principle of step S401 can be referred to the aforementioned step S301. The only difference is that step S301 deduplicates the error details table, while step S401 deduplicates the self-service change list.

[0110] S402. Sum the number of application systems included in the deduplicated self-service change list to obtain the total number of systems of the version release party.

[0111] The execution process and principle of step S402 can be referred to the aforementioned step S302. The only difference is that step S302 is to sum the error details after deduplication, while step S402 is to sum the self-service change list after deduplication.

[0112] Optionally, one implementation of steps S401 to S402 is as follows: The self-service change list is calculated using a second formula to obtain the total number of systems. The second formula is: Total number of systems = SUM(Number of rows in the self-service change list after deduplication based on (Version Releaser & Deployment Time & Application System)). Here, "SUM()" represents summation, and "&" signifies "sum". The second formula calculates the total number of rows in the self-service change list after deduplicating rows with the same version releaser, deployment time, and application system.

[0113] The deduplicated self-service change list only retains the number of application systems released by the release party at each production launch time. The total number of systems included in the deduplicated self-service change list is calculated by summing the number of application systems in the release party's list.

[0114] Optionally, if the total number of systems released by the version publisher is the total number of systems released by the version publisher within a preset time period, then before executing step S401, the process may further include: filtering out the number of rows in the self-service change list whose production time is within the preset time period to obtain the filtered self-service change list, and then subsequently executing steps S401 to S402 on the filtered self-service change list to obtain the total number of systems released by the version publisher within the preset time period.

[0115] It should be noted that the execution order of steps S103 and S104 is not limited in this embodiment of the application, and steps S103 and S104 can also be executed simultaneously.

[0116] In existing technologies, the quality of a release from a publisher is directly assessed using the aggregated number of self-service errors. This can lead to a negative outcome for publishers releasing a large number of application versions: more work, more errors, resulting in a lower assessed quality. Publishers may then reduce the number of application versions they release to improve their quality assessment. However, in this embodiment, the quality assessment considers both the total number of systems and the number of error-prone systems, making the assessment more accurate for publishers with a large number of application versions compared to existing technologies.

[0117] Optionally, see Figure 5 In a specific embodiment of this application, after performing step S102, the method further includes:

[0118] S501. Determine whether the self-service that failed to execute is a fatal factor, where a fatal factor is a self-service that causes the application version to be rolled back due to an execution error.

[0119] Specifically, there are two types of self-services that fail during execution: one is a fatal factor, which requires a version rollback if an error occurs during implementation; the other is a recoverable factor, which can be skipped or re-executed after error analysis to achieve the purpose of application version release and update. By analyzing the error causes and corresponding solutions of self-services that fail during execution, it can be determined whether the self-service that fails is a fatal factor.

[0120] If it is determined that the self-service that failed to execute is a fatal factor, then step S502 is executed. If it is determined that the self-service that failed to execute is not a fatal factor, then no other operations need to be performed.

[0121] S502. Add a fatal factor flag to the self-service that failed to execute in the error details table.

[0122] There are many ways to mark fatal factors. For example, you can mark the relevant information of self-services that belong to fatal factors recorded in the error details table with a specific color, or you can directly add a list item to the error details table indicating whether it is a fatal factor. Self-services that belong to fatal factors will be marked at the corresponding list item. This application embodiment does not limit the way to mark fatal factors for self-services that have execution errors.

[0123] Optionally, a fatal factor table can be created separately. If a self-service that malfunctions is identified as a fatal factor, that self-service is recorded in the fatal factor table. Specifically, the fatal factor table may include: the identifier of the version publisher, the identifier of the application system published by the version publisher, the identifier of the application version corresponding to the application system, the production deployment time of the application version, the identifier of the self-service corresponding to the application version that is a fatal factor, and the error details of the self-service that is a fatal factor.

[0124] S503. Calculate the number of fatal factors based on the error details table, where the number of fatal factors is the total number of application systems with fatal factors.

[0125] Specifically, the fatal factor count refers to the number of application systems with fatal factors released by the version release provider. As mentioned in step S502 above, self-services identified as fatal factors will be marked in the error detail table. Therefore, the number of application systems with fatal factors can be calculated statistically from the error detail table.

[0126] Optionally, in a specific embodiment of this application, one implementation of step S503 is as follows: Based on the error detail table, calculate the number of fatal factors for the version release party within a preset time period. That is, only count the total number of application systems with fatal factors within the preset time period (e.g., a certain month) from the error detail table.

[0127] As explained above regarding fatal factors, a fatal factor is a self-service failure that necessitates the rollback of an application version, representing a significant error during the application update process. Therefore, the larger the total number of application systems with fatal factors released by a release party, the worse the release party's deployment quality will be. The deployment quality of a release party can be assessed based on the number of fatal factors. By calculating the number of fatal factors for a release party, and combining this with the total number of systems and the number of faulty systems calculated in steps S104 and S105, the deployment quality of a release party can be accurately assessed.

[0128] Optionally, see Figure 6 In one specific embodiment of this application, one implementation of step S503 includes:

[0129] S601. Generate a fatal factor table from the error details table. The fatal factor table includes: relevant information of the self-service identified as a fatal factor. The relevant information of the self-service identified as a fatal factor includes: the identifier of the version publisher to which the self-service identified as a fatal factor belongs, the identifier of the application system, the identifier of the application version, the production time of the application version, the identifier of the self-service identified as a fatal factor, and the error details of the self-service identified as a fatal factor.

[0130] Specifically, as described above regarding the error detail table, it includes information about all self-services that encountered execution errors, while the generated fatal factor table only includes information about self-services that are considered fatal factors. In step S502, the self-services that encountered execution errors and were classified as fatal factors were already marked; therefore, information about self-services identified as fatal factors can be generated from the error detail table.

[0131] For example, rows containing self-services marked as fatal factors can be filtered from the error details table, and the filtered error details table can be identified as the fatal factor table.

[0132] S602. Perform deduplication on the same version publisher, application system identifier, and application version launch time in the fatal factor table to obtain the deduplicated fatal factor table.

[0133] The execution process and principle of step S602 can be referred to the aforementioned step S301. The only difference is that step S301 removes duplicates from the error details table, while step S602 removes duplicates from the fatal factor table.

[0134] S603. Sum the number of application systems included in the deduplicated fatal factor table to obtain the number of fatal factors.

[0135] The execution process and principle of step S603 can be referred to the aforementioned step S302. The only difference is that step S302 is to sum the error details table after deduplication, while step S602 is to sum the fatal factors table after deduplication.

[0136] Optionally, one implementation of steps S602 to S603 is as follows: The critical factor table is calculated using a third formula to obtain the total number of systems. The third formula is: Critical Factor Count = SUM(Number of rows in the critical factor table after deduplication based on (Version Releaser & Production Time & Application System)). Here, "SUM()" represents summation, and "&" signifies "sum". The third formula calculates the total number of rows in the critical factor table after deduplicating rows with the same version releaser, production time, and application system.

[0137] S105. Calculate the release quality score of the version releaser based at least on the number of faulty systems and the total number of systems. The release quality score of the version releaser is used to evaluate the release quality of the version releaser.

[0138] With the same total number of systems, fewer faulty systems result in higher release quality for the release provider. Conversely, with the same number of faulty systems, a higher total number of systems results in higher release quality. Based on the relationship between the number of faulty systems, the total number of systems, and release quality, corresponding release quality calculation formulas or release quality evaluation models can be established. By substituting the number of faulty systems and the total number of systems into the established release quality calculation formulas or release quality evaluation models, a release quality score can be obtained for the release provider.

[0139] For example, the self-service failure rate of a release provider can be calculated based on the number of systems experiencing errors and the total number of systems. Then, the release provider's deployment quality score can be calculated based on the self-service failure rate. For instance, the ratio of the number of systems experiencing errors to the total number of systems can be determined as the release provider's self-service failure rate.

[0140] Optionally, if step S503 was executed before step S105 to obtain the number of critical factors, then when executing step S105, the release quality score of the version release party can be calculated at least based on the number of faulty systems, the total number of systems, and the number of critical factors. The number of critical factors is negatively correlated with the release quality score of the version release party, and the ratio of the number of faulty systems to the total number of systems is also negatively correlated with the release quality score.

[0141] Optionally, if after executing step S102, the method further includes: calculating the performance evaluation factors of the version release provider, where the performance evaluation factors are indicators that affect the production implementation process of the application version, then when executing step S105, the production quality score of the version release provider can be calculated at least based on the number of error systems, the total number of systems, the number of critical factors, and the number of performance evaluation factors of the version release provider obtained from the statistics.

[0142] In this embodiment, the release quality of the version release party is evaluated from several aspects, including the number of faulty systems, the total number of systems, the number of fatal factors, and the number of assessment factors of the version release party obtained by statistics. Compared with the existing methods of evaluating release quality, it takes into account more factors affecting release quality, and thus the accuracy of the evaluation of release quality is also higher.

[0143] For example, see Figure 7 In a specific embodiment of this application, an implementation method for calculating the production quality score of a version release party based at least on the number of faulty systems, the total number of systems, the number of fatal factors, and the statistically obtained assessment factors of the version release party includes:

[0144] S701. Based on the number of systems that have errors and the total number of systems, calculate the self-service failure rate of the version publisher, where the self-service failure rate of the version publisher is the probability that the application system released by the version publisher fails to execute.

[0145] For example, the self-service failure rate can be the ratio of the number of faulty systems to the total number of systems. That is, the self-service failure rate is calculated by substituting the number of faulty systems and the total number of systems into the fourth formula. The fourth formula is: FalRate is the self-service failure rate of the version release provider, A is the number of systems with errors in the version release provider, and B is the total number of systems in the version release provider.

[0146] S702. Calculate the self-service metric score based on the self-service failure rate of the version release party. The self-service metric score is used to illustrate the production quality reflected by the version release party in the execution of self-service.

[0147] The higher the self-service failure rate, the lower the self-service metric score. For example, the self-service failure rate can be substituted into the fifth formula to calculate the self-service metric score. The fifth formula is: D = C × (1 - FalRate). D is the self-service metric score, FalRate is the self-service failure rate of the version release provider, and C is the maximum score for the self-service metric. The value of C can be set manually.

[0148] It should be noted that there are many ways to calculate the self-service metric score based on the self-service failure rate of the version release party, including but not limited to the method proposed in the embodiments of this application.

[0149] S703. Based on the scores corresponding to the assessment factors, the scores corresponding to the critical factors, and the self-service indicator scores, the production quality score of the version release party is calculated. The scores corresponding to the assessment factors are calculated based on the statistical assessment factors of the version release party and are used to illustrate the production quality reflected by the version release party on the assessment factors. The scores corresponding to the critical factors are calculated based on the number of critical factors and are used to illustrate the production quality reflected by the version release party on the critical factors.

[0150] The score corresponding to the assessment factor is calculated based on the assessment factors of the version release party. For example, the product of the statistically obtained assessment factor and the assessment coefficient can be used as the score corresponding to the assessment factor. The score corresponding to the assessment factor can be positively correlated with the production quality of the version release party. The method of setting the assessment coefficient is not limited in this embodiment. The specific calculation method of the score corresponding to the assessment factor is also not limited in this embodiment.

[0151] The score corresponding to a fatal factor can be negatively correlated with the release quality of the version. For example, the score corresponding to a fatal factor can be equal to the product of the number of fatal factors and the fatal factor coefficient. The value of the fatal factor coefficient can be arbitrarily set.

[0152] For example, one implementation of step S703 includes: substituting the scores corresponding to the assessment factors, the scores corresponding to the critical factors, and the self-service indicator scores into the sixth formula to calculate the release quality score of the version release party. The sixth formula is: E + FG = H. Where E is the score corresponding to the assessment factors, F is the self-service indicator score, G is the score corresponding to the critical factors, and H is the release quality score of the version release party.

[0153] Optionally, in a specific embodiment of this application, after each version releaser calculates its production quality score using the production quality evaluation method proposed in this application, the version releasers can be ranked based on their production quality scores. Version releasers with lower rankings are prompted to make improvements to enhance the production efficiency of the subsequent management platform.

[0154] like Figure 8 As shown below, to further explain in detail the production quality evaluation method proposed in the aforementioned embodiments of this application, the production quality evaluation process is illustrated with an example based on the production quality evaluation method proposed in the embodiments of this application: First, accurate version production is pre-configured in the management platform, that is, the application versions split into multiple self-services are pre-configured, and corresponding self-service change lists are created. Then, according to the self-service change lists, version updates are initiated periodically, and the management platform schedules and executes the self-services corresponding to each application version. Then, during the update of each referenced version, the version production implementation process is recorded. Specifically, the self-service error details (i.e., error details information of the self-service) and the solutions (or solutions) are recorded in the error details table, and the implementation results of the self-services (i.e., the execution status of the aforementioned self-services) are also recorded. Then, based on the error details table and the self-service change lists, the self-service failure rate is calculated, and other assessment factors of version production quality (i.e., the assessment factors mentioned above) are also calculated. Finally, based on the self-service failure rate and assessment factors, the production quality score of each version release unit is calculated, and the production quality scores of each version release unit are ranked. Units with lower rankings are reminded to rectify, so as to improve the efficiency of subsequent production.

[0155] The deployment quality assessment method provided in this invention updates each application version according to a self-service change list. During the update process, information related to self-services that malfunctioned is recorded in an error detail table. Based on the error detail table, the number of malfunctioning systems for the version publisher can be calculated, and based on the self-service change list, the total number of systems for the version publisher can be calculated. The number of malfunctioning systems refers to the total number of application systems with malfunctioning self-services published by the version publisher, and the total number of systems refers to the total number of application systems published by the version publisher. Then, based at least on the number of malfunctioning systems and the total number of systems, a deployment quality score for the version publisher is calculated. This deployment quality score is used to assess the deployment quality of the version publisher. Because this embodiment considers both the total number of systems and the number of malfunctioning systems when assessing deployment quality, it provides a more accurate assessment of deployment quality for version publishers with a large number of application versions compared to existing technologies. Furthermore, by using the number of faulty systems and the total number of systems to assess the quality of production deployment, compared to the method of summarizing the number of self-service errors, the problem of inaccurate assessment of production deployment quality caused by repeatedly counting the number of self-services with common errors can be avoided, thus improving the accuracy of the assessment of production deployment quality.

[0156] See Figure 9 Based on the production quality evaluation method proposed in the above embodiments of this application, this application also discloses a production quality evaluation device, including: an update unit 901, a first recording unit 902, a first calculation unit 903, and a second calculation unit 904.

[0157] Update unit 901 is used to update each application version according to the self-service change list. The self-service change list includes relevant information for each application version that needs updating. The application version information includes: the identifier of the version release provider, the identifier of the application system, the identifier of the application version, the deployment time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the deployment machine identifier of the self-service. A self-service is a program that breaks down the application version release process.

[0158] The first recording unit 902 is used to record relevant information of self-services that malfunction during the update of various application versions to an error detail table. This information includes: the identifier of the version publisher to which the malfunctioning self-service belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the malfunctioning self-service, and the error details of the malfunctioning self-service.

[0159] The first calculation unit 903 is used to calculate the number of error-prone systems of the version publisher based on the error details table, and to calculate the total number of systems of the version publisher based on the self-service change list. The number of error-prone systems refers to the total number of self-service application systems published by the version publisher that have execution errors. The total number of systems refers to the total number of application systems published by the version publisher.

[0160] Optionally, the first calculation unit 903 includes: a first calculation subunit, a second calculation subunit, a third calculation subunit, and a fourth calculation subunit.

[0161] The first calculation subunit is used to perform deduplication on the same version releaser, application system identifier, and application version launch time in the error details table to obtain a deduplicated error details table.

[0162] The second calculation subunit is used to sum the number of application systems included in the deduplicated error details table to obtain the number of error systems of the version release party.

[0163] The third calculation subunit is used to perform deduplication on the same version publisher, application system identifier, and application version deployment time in the self-service change list to obtain a deduplicated self-service change list.

[0164] The fourth calculation subunit is used to sum the number of application systems included in the deduplicated self-service change list to obtain the total number of systems of the version release party.

[0165] The second calculation unit 904 is used to calculate the release quality score of the version release provider based at least on the number of faulty systems and the total number of systems. The release quality score is used to evaluate the release quality of the version release provider.

[0166] Optionally, the production quality assessment device may also include a marking unit and a third calculation unit.

[0167] The tagging unit is used to add a fatal factor tag to the error details table if a self-service that failed to execute is determined to be a fatal factor. A fatal factor is a self-service whose execution error necessitates a rollback of the application version.

[0168] The third calculation unit is used to calculate the number of fatal factors based on the error details table. The number of fatal factors is the total number of application systems with fatal factors. The second calculation unit 904 includes a fifth calculation subunit, used to calculate the release quality score of the version release party based at least on the number of error-prone systems, the total number of systems, and the number of fatal factors.

[0169] Optionally, the third calculation unit includes: a generation subunit, a sixth calculation subunit, and a seventh calculation subunit.

[0170] The generation subunit is used to generate a fatal factor table from the error details table. The fatal factor table includes information about the self-services identified as fatal factors. This information includes: the identifier of the version release party to which the self-service belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service identified as a fatal factor, and the error details of the self-service identified as a fatal factor.

[0171] The sixth calculation subunit is used to perform deduplication on the same version publisher, application system identifier, and application version launch time in the fatal factor table to obtain the deduplicated fatal factor table.

[0172] The seventh calculation subunit is used to sum the number of application systems included in the deduplicated fatal factor table to obtain the number of fatal factors.

[0173] Optionally, the equipment for evaluating the quality of production launch also includes:

[0174] The statistics unit is used to calculate the performance evaluation factors for the version release provider. These performance evaluation factors are indicators that influence the deployment and implementation process of the application version. The second calculation unit 904 includes an eighth calculation subunit, used to calculate the deployment quality score of the version release provider based at least on the number of error-prone systems, the total number of systems, the number of critical factors, and the statistically obtained performance evaluation factors.

[0175] Optionally, the eighth calculation subunit includes: the ninth calculation subunit, the tenth calculation subunit, and the eleventh calculation subunit.

[0176] The ninth calculation subunit is used to calculate the self-service failure rate of the version release provider based on the number of systems experiencing errors and the total number of systems. The self-service failure rate of the version release provider is the probability that the application system released by the version release provider will fail.

[0177] The tenth calculation subunit is used to calculate the self-service metric score based on the self-service failure rate of the version release provider. The self-service metric score is used to illustrate the production quality reflected by the version release provider in the execution of self-service.

[0178] The eleventh calculation subunit is used to calculate the release quality score of the version release provider based on the scores corresponding to the assessment factors, the scores corresponding to the critical factors, and the self-service indicator scores. The scores corresponding to the assessment factors are calculated based on the statistically collected assessment factors of the version release provider. These scores illustrate the release quality reflected by the assessment factors. The scores corresponding to the critical factors are calculated based on the number of critical factors. These scores illustrate the release quality reflected by the critical factors.

[0179] Optionally, the production quality assessment device further includes a second recording unit, used to record the execution status of self-services in the self-service change list and record the solutions of self-services that fail to execute in the error detail table during the update of each application version. The execution status of the self-services describes the execution status of the self-services on the corresponding production machine.

[0180] The execution principles of each unit and subunit in the production quality evaluation device proposed in this application can be referred to the production quality evaluation method proposed in the embodiments of this application, and will not be repeated here.

[0181] In the production quality assessment apparatus proposed in this application, the update unit 901 updates each application version according to the self-service change list. Then, during the update process, the first recording unit 902 records the relevant information of the self-services that malfunctioned into an error detail table. Subsequently, the first calculation unit 903 calculates the number of malfunctioning systems of the version publisher based on the error detail table, and calculates the total number of systems of the version publisher based on the self-service change list. The number of malfunctioning systems refers to the total number of application systems with malfunctioning self-services published by the version publisher, and the total number of systems refers to the total number of application systems published by the version publisher. Then, the second calculation unit 904 calculates the production quality score of the version publisher based at least on the number of malfunctioning systems and the total number of systems. This production quality score is used to assess the production quality of the version publisher. Since this embodiment considers both the total number of systems and the number of malfunctioning systems of the version publisher when assessing production quality, it provides a more accurate assessment of the production quality for version publishers with a large number of published application versions compared to existing technologies. Furthermore, by using the number of faulty systems and the total number of systems to assess the quality of production deployment, compared to the method of summarizing the number of self-service errors, the problem of inaccurate assessment of production deployment quality caused by repeatedly counting the number of self-services with common errors can be avoided, thus improving the accuracy of the assessment of production deployment quality.

[0182] This application also discloses a production quality evaluation device, including a memory and a processor; wherein the memory is used to store a computer program; and the processor is used to execute the computer program, specifically to implement any production quality evaluation method as described in this application.

[0183] This application also discloses a computer-readable medium for storing a computer program, which, when executed, is specifically used to implement any of the production quality evaluation methods described in this application.

[0184] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0185] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0186] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the quality of production, characterized in that, include: Update each application version according to the self-service change list; wherein, the self-service change list includes: relevant information of each application version that needs to be updated; the relevant information of the application version includes: the identifier of the version release party to which the application version belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the production release machine identifier corresponding to the self-service; the self-service is a program after the application version release process is broken down; During the update of each application version, the relevant information of the self-service that failed to execute is recorded in the error details table; wherein, the relevant information of the self-service that failed to execute includes: the identifier of the version release party to which the self-service that failed to execute belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service that failed to execute, and the error details of the self-service that failed to execute; Based on the error details table, the number of error-prone systems of the version publisher is calculated; and based on the self-service change list, the total number of systems of the version publisher is calculated; wherein, the number of error-prone systems is the total number of application systems with self-services that have execution errors published by the version publisher; and the total number of systems is the total number of application systems published by the version publisher. The release quality score of the version release party is calculated based at least on the number of faulty systems and the total number of systems; wherein the release quality score of the version release party is used to evaluate the release quality of the version release party. The step of calculating the number of error-prone systems for the version release provider based on the error details table includes: Deduplication is performed on the same version releaser, application system identifier, and application version deployment time in the error details table to obtain a deduplicated error details table. The number of application systems included in the deduplicated error details table is summed to obtain the number of error systems for the version release.

2. The method according to claim 1, characterized in that, The total number of systems for version release providers is calculated based on the self-service change list, including: Deduplication is performed on the same version publisher, application system identifier, and application version deployment time in the self-service change list to obtain a deduplicated self-service change list. The total number of application systems included in the deduplicated self-service change list is summed to obtain the total number of systems released by the version publisher.

3. The method according to claim 1, characterized in that, After recording the relevant information of the self-service that failed to execute to the error details table, the method further includes: If the self-service that caused the execution error is determined to be a fatal factor, then a fatal factor mark is added to the self-service that caused the execution error in the error details table; wherein, a fatal factor is a self-service that causes the application version to be rolled back due to the execution error; Based on the error details table, the number of fatal factors is calculated; wherein, the number of fatal factors is the total number of application systems with fatal factors; The step of calculating the release quality score for the version release party, based at least on the number of faulty systems and the total number of systems, includes: The release quality score of the version release party is calculated based at least on the number of faulty systems, the total number of systems, and the number of fatal factors.

4. The method according to claim 3, characterized in that, The calculation of the number of fatal factors based on the error details table includes: A fatal factor table is generated from the error details table; wherein, the fatal factor table includes: relevant information of the self-services identified as fatal factors; the relevant information of the self-services identified as fatal factors includes: the identifier of the version release party to which the self-services identified as fatal factors belong, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-services identified as fatal factors, and the error details information of the self-services identified as fatal factors; Deduplication is performed on the same version publisher, application system identifier, and application version deployment time in the fatal factor table to obtain a deduplicated fatal factor table. The number of application systems included in the deduplicated fatal factor table is summed to obtain the number of fatal factors.

5. The method according to claim 4, characterized in that, After updating each application version according to the self-service change list, the process also includes: The performance evaluation factors for the release of statistical versions; wherein, the performance evaluation factors are indicators that affect the production and implementation process of application versions; The step of calculating the release quality score of the version release party based at least on the number of faulty systems, the total number of systems, and the number of fatal factors includes: The production quality score of the version release party is calculated based on at least the number of faulty systems, the total number of systems, the number of fatal factors, and the statistically obtained assessment factors of the version release party.

6. The method according to claim 5, characterized in that, The step of calculating the release quality score of the version release provider based at least on the number of faulty systems, the total number of systems, the number of fatal factors, and the statistically obtained assessment factors of the version release provider includes: The self-service failure rate of the version publisher is calculated based on the number of systems that malfunction and the total number of systems; wherein, the self-service failure rate of the version publisher is the probability that the application system released by the version publisher fails to execute. A self-service metric score is calculated based on the self-service failure rate of the version release provider; wherein, the self-service metric score is used to illustrate the production quality reflected by the version release provider in the execution of self-service. The release quality score of the version release provider is calculated based on the scores corresponding to the assessment factors, the scores corresponding to the critical factors, and the self-service indicator scores. The scores corresponding to the assessment factors are calculated based on the statistically analyzed assessment factors of the version release provider, and these scores illustrate the release quality reflected by the assessment factors. The scores corresponding to the critical factors are calculated based on the number of critical factors, and these scores illustrate the release quality reflected by the critical factors.

7. The method according to claim 1, characterized in that, Also includes: During the update of each application version, the execution status of the self-service is recorded in the self-service change list, and the solutions for self-services that fail to execute are recorded in the error details table; wherein, the execution status of the self-service is used to describe the execution status of the self-service on the production machine corresponding to the self-service.

8. A device for evaluating production quality, characterized in that, include: The update unit is used to update each application version according to the self-service change list; wherein, the self-service change list includes: relevant information of each application version that needs to be updated; the relevant information of the application version includes: the identifier of the version release party to which the application version belongs, the identifier of the application system, the identifier of the application version, the production release time of the application version, the identifier of the self-service corresponding to the application version, the execution time of the self-service, and the production release machine identifier corresponding to the self-service; the self-service is a program after the application version release process has been broken down; The first recording unit is used to record relevant information of self-services that fail to execute during the process of updating various application versions to an error detail table; wherein, the relevant information of the self-services that fail to execute includes: the identifier of the version publisher to which the self-services that fail to execute belong, the identifier of the application system, the identifier of the application version, the production time of the application version, the identifier of the self-services that fail to execute, and the error details of the self-services that fail to execute. The first calculation unit is used to calculate the number of error-prone systems of the version publisher based on the error details table; and to calculate the total number of systems of the version publisher based on the self-service change list; wherein, the number of error-prone systems is the total number of application systems with self-services that have execution errors published by the version publisher; and the total number of systems is the total number of application systems published by the version publisher. The second calculation unit is used to calculate the release quality score of the version release party based at least on the number of faulty systems and the total number of systems; wherein the release quality score of the version release party is used to evaluate the release quality of the version release party; The step of calculating the number of error-prone systems for the version release provider based on the error details table includes: Deduplication is performed on the same version releaser, application system identifier, and application version deployment time in the error details table to obtain a deduplicated error details table. The number of application systems included in the deduplicated error details table is summed to obtain the number of error systems for the version release.

9. A production quality assessment device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program, specifically to implement the production quality evaluation method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, Used to store computer programs, which, when executed, are specifically used to implement the production quality evaluation method as described in any one of claims 1 to 7.

Images