Change service evaluation methods, devices, equipment, media and program products

Abstract

This disclosure provides a change service evaluation method applicable to the fields of IT operations and maintenance and financial technology. The change service evaluation method comprises: obtaining business factors from a data sample, including the change execution time concentration, the change execution success rate, and the change operation compliance rate; calculating first, second, and third scores corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate, respectively; and evaluating the change service based on the first, second, and third scores. This disclosure also provides a change service evaluation device, apparatus, storage medium, and program product.

Description

Technical Field

[0001] The present disclosure relates to the fields of IT operation and maintenance and finance, and more specifically to changes in IT operation and maintenance services, and more specifically to a method, apparatus, device, medium, and program product for evaluating change services. Background Art

[0002] Change services refer to services that provide daily change capabilities for operating systems, databases, and middleware in the field of IT operations and maintenance. Generally, change services will follow certain change specifications and standards and can be automatically executed after being instantiated on the operations and maintenance platform. Change services generally correspond to actual change scenarios and have corresponding business significance. They are usually composed of one or more atomic operations, each of which corresponds to a command or a script. After the change service is instantiated, the atomic operations are executed one by one in sequence. After all atomic operations are completed, the execution of the change service is completed, and the corresponding business purpose is usually achieved. Depending on the business purpose, the design of the change service will also be different. Similarly, the complexity of the execution steps and processes will also vary, and the corresponding service success rate will also vary.

[0003] With the development of technologies and tools related to IT operations and maintenance, IT operations and maintenance work has become increasingly complex. The interaction between different technologies and tools has become more common, and changes have become increasingly complicated and trivial. For example, upgrading agent software in a virtual machine may be affected by multiple factors such as the operating system, file media, and network storage. Therefore, in order to complete the change work safely and efficiently, the requirements for change services have become more stringent. However, how to judge whether a change service is a high-quality service has become a difficult problem.

[0004] Currently, service change evaluations in the operations and maintenance field are primarily based on personal experience and familiarity. Lack of an objective and effective evaluation / scoring method leads to one-sided conclusions, making it difficult to provide referenceable quantitative data, or even incompletely representative data. For example, a general software upgrade change service may achieve the desired change objectives with a high success rate, but it may also encounter security and stability issues in certain scenarios. In this case, it's difficult to quantify whether the service is high-quality or low-quality. Therefore, a precise, effective, and quantifiable digital evaluation / scoring method is needed. Summary of the Invention

[0005] In view of the above problems, the present disclosure provides a method, apparatus, device, medium and program product for evaluating a change service to improve the comprehensiveness of the evaluation.

[0006] According to a first aspect of the present disclosure, a change service evaluation method is provided, comprising: obtaining business factors of a data sample, the business factors including change execution time concentration, change execution success rate and change operation compliance rate; calculating a first score, a second score and a third score corresponding to the change execution time concentration, the change execution success rate and the change operation compliance rate, respectively; and evaluating the change service according to the first score, the second score and the third score.

[0007] According to an embodiment of the present disclosure, calculating the first score of the change execution time concentration includes: calculating the average change execution time; calculating the change execution time standard deviation of the data sample based on the average change execution time; and obtaining the first score based on the change execution time standard deviation.

[0008] According to an embodiment of the present disclosure, calculating the second score of the change execution success rate includes: obtaining the number of successful change executions; setting a first default score, and calculating the second score based on the first default score and the number of successful executions.

[0009] According to an embodiment of the present disclosure, calculating the third score of the change operation compliance rate includes: detecting the change operation maintenance rule keywords of the data sample; and calculating the third score according to the change operation maintenance rule keywords.

[0010] According to an embodiment of the present disclosure, the business factor also includes the manual intervention rate of change execution, and calculating the second score also includes: obtaining the number of manual interventions for change execution; and calculating the second score based on the first default score, the number of successes, and the number of manual interventions.

[0011] According to an embodiment of the present disclosure, the business factor also includes the robustness of the change operation, and calculating the third score also includes: obtaining the number of abnormal rollbacks of the change operation; and calculating the third score based on the operation and maintenance rule keyword and the number of abnormal rollbacks.

[0012] According to an embodiment of the present disclosure, the business factor also includes the change execution failure rate, and calculating the second score also includes: obtaining the number of failures in the change execution; and calculating the second score based on the first default score, the number of successes, the number of manual interventions and the number of failures.

[0013] According to an embodiment of the present disclosure, the business factor also includes the host usage ratio of the changed service. Calculating the fourth score corresponding to the host usage ratio of the changed service includes: obtaining the number of times the host is used in different time periods; setting the first weight corresponding to the number of times used; and calculating the fourth score based on the number of times used and the first weight.

[0014] According to an embodiment of the present disclosure, the business factor also includes the change service user satisfaction, and calculating the fifth score corresponding to the change service user satisfaction includes: obtaining the number of user feedbacks, the number of feedbacks includes the number of positive feedbacks and the number of negative feedbacks; setting the second default score, and calculating the fifth score based on the second default score and the number of feedbacks.

[0015] According to an embodiment of the present disclosure, evaluating the changed service based on the first score, the second score and the third score includes: setting the second weight of each business factor; calculating the total score of the changed service based on the first score, the second score, the third score and the second weight; and evaluating the changed service based on the total score.

[0016] According to an embodiment of the present disclosure, the first score, second score and third score corresponding to the change execution time concentration, change execution success rate and change operation compliance rate are calculated, and the extreme values ​​in the processing scores are also included, including 0 and the highest weight score of each business factor.

[0017] The second aspect of the present disclosure provides a change service evaluation device, including: an acquisition module for acquiring business factors of data samples, the business factors including change execution time concentration, change execution success rate and change operation compliance rate; a calculation module for calculating a first score, a second score and a third score corresponding to the change execution time concentration, the change execution success rate and the change operation compliance rate respectively; and an evaluation module for evaluating the change service according to the first score, the second score and the third score.

[0018] The third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned change service evaluation method.

[0019] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned change service evaluation method.

[0020] The fifth aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned change service evaluation method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1Schematically illustrates an application scenario diagram of a change service evaluation method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a method for changing service evaluation according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A flowchart of a method for calculating a first score according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 4 Schematically shows a flow chart of a method for calculating a second score according to an embodiment of the present disclosure;

[0026] Figure 5 Schematically shows a flow chart of a method for calculating a third score according to an embodiment of the present disclosure;

[0027] Figure 6 A diagram schematically illustrates a newly created file system change scoring result according to an embodiment of the present disclosure;

[0028] Figure 7 Schematically shows a structural block diagram of a device for evaluating service changes according to an embodiment of the present disclosure; and

[0029] Figure 8 The block diagram schematically shows an electronic device suitable for implementing the service evaluation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] It should be noted that the change service evaluation method and device determined in the embodiments of the present disclosure can be used in the financial field for IT operation and maintenance, and can also be used in any field other than the financial field. The application field of the change service evaluation method and device disclosed in the present disclosure is not limited.

[0035] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0036] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0037] Figure 1 The application scenario diagram of the change service evaluation method, apparatus, device, medium and program product according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0039] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0040] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0041] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the service change evaluation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the service change evaluation device provided in the embodiment of the present disclosure can generally be set in the server 105. The service change evaluation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the service change evaluation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0044] The following will be based on Figure 1 The scene described by Figures 2 to 6 The change service evaluation method of the disclosed embodiment is described in detail.

[0045] Figure 2 The flowchart of the method for changing service evaluation according to an embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the change service evaluation of this embodiment includes, for example, operations S210 to S230:

[0047] In operation S210 , business factors of the data sample are obtained, where the business factors include change execution time concentration, change execution success rate, and change operation compliance rate.

[0048] In operation S220 , the first score, the second score, and the third score corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate are calculated.

[0049] In operation S230 , the change service is evaluated according to the first score, the second score, and the third score.

[0050] According to the embodiments of the present disclosure, business factors are application data related to change services, such as operational data from application systems in the field of operation and maintenance tools. By combing and integrating application data related to change services in the operation and maintenance system, and comprehensively considering and applying various static, dynamic, objective, and subjective data, an evaluation model is constructed to score and evaluate change services and assign star ratings. This allows for a refined assessment of change services, helping system administrators gain a comprehensive understanding of change services from multiple dimensions such as success rate, stability, and security, laying the foundation for further targeted optimization and iteration, and ultimately completing the upgrade and improvement of the entire change system.

[0051] According to an embodiment of the present disclosure, for example, an evaluation model may be constructed based on the scores of various business factors, and the evaluation model may be used to evaluate the changed service.

[0052] Figure 3 A flowchart of a method for calculating a first score according to an embodiment of the present disclosure is schematically shown.

[0053] According to the embodiments of the present disclosure, Figure 3 As shown, for example, through operations S2201 to S2203 , a first score corresponding to the concentration of execution time changes is calculated.

[0054] According to an embodiment of the present disclosure, business factors include, for example, the concentration of change execution time, the change execution success rate, and the change operation compliance rate. The concentration of change execution time can, for example, reflect the stability of the change service. Stability can explain the degree of influence of external factors on the service time consumption. The higher the score, the smaller the degree of influence of external factors on its time consumption. The data for measuring stability comes from, for example, a historical data set of change instance execution, which is a dynamic and objective data, wherein the execution time is, for example, the instance execution time divided by the number of hosts changed. Based on the statistics, analysis, and calculation of application data to obtain objective results, scientific data calculation ensures that the evaluation conclusion is effective and objective, and will not vary from person to person. In this embodiment, for example, the execution time is concentrated by using the probability model normal distribution, mean, variance and other numerical values. The more concentrated it is, the closer the execution time of the service under various business conditions is, and the more stable it is. The stability score can be calculated, for example, through steps S2201 to S2203.

[0055] S2201, calculating the average change execution time.

[0056] According to an embodiment of the present disclosure, the average change time from changing a service instance to changing the host is:

[0057]

[0058] Where n is the number of hosts that were changed (i.e., the number of samples), and x is the change time on each host that was changed.

[0059] S2202: Calculate the standard deviation of the change execution time of the data sample based on the average change execution time.

[0060] According to an embodiment of the present disclosure, the standard deviation of the data is changed:

[0061]

[0062] Among them, x i is the single implementation time, This is the mean calculated by formula (1), where n is the number of samples. The smaller the standard deviation, the more sample data is concentrated around the mean of the change time, and the better the stability.

[0063] S2203: Calculate a first score based on the standard deviation of the change execution time.

[0064] According to the embodiment of the present disclosure, the standard deviation σ calculated by formula (2) cannot directly reflect the stability of the data. In order to eliminate the numerical difference of the standard deviation between cloud services, the standard deviation σ is multiplied by the coefficient E to obtain the stability score S1:

[0065] S1=E·σ (3)

[0066] in, It is a non-negative monotonically decreasing function related to the mean of the change time, and then a score is calculated based on the weighted standard deviation, which is the stability score S1. For example, the weighting factor (i.e. coefficient) N is a natural number greater than 1. Since the execution time distributions of different change services are different, for example, the execution time distributions of service A and service B are the same, but the average execution time of service A is larger than that of service B, the standard deviation calculated for A is higher than the standard deviation calculated for B, which will result in calculation bias. Therefore, we choose a natural number greater than 1, such as N=2, to process change services with different average values ​​and obtain the weighting factor E. After the weighted calculation is completed, the average values ​​are the same, and the obtained standard deviations are also comparable. It should be noted that when obtaining stability calculation samples, extreme values, such as 0, should be removed so that the stability calculation model can always return a stability score.

[0067] Figure 4 A flowchart of a method for calculating a second score according to an embodiment of the present disclosure is schematically shown.

[0068] According to the embodiments of the present disclosure, Figure 4 As shown, for example, the second score corresponding to the change execution success rate is calculated through operations S2204 to S2205.

[0069] S2204, obtaining the number of successful executions of the change.

[0070] According to the embodiments of the present disclosure, the change execution success rate and the interruption rate can, for example, reflect the correctness of the change service. Correctness can indicate whether the change service can be successfully executed. The higher the score, the better the correctness. Data reflecting correctness, for example, comes from the execution history data set of the change instance, which is a dynamic and objective data. Generally speaking, the sum of the success rate and the termination rate is not always equal to 1. In most cases, it is close to 1. In order to simplify the model and facilitate calculation, it can be regarded as summing up to 1 to improve the speed of the correctness calculation model in processing samples. For example, the number of successes and failures of each change service execution can be counted, or the number of failures can be calculated based on the number of successes counted.

[0071] S2205: Set a first default score, and calculate a second score based on the first default score and the number of successes.

[0072] According to an embodiment of the present disclosure, for example, a default score can be set, and then based on the default score, the change service is scored and subtracted according to the execution status. The calculated correctness score S2 is as follows:

[0073]

[0074] Where O1 is the first default score, m and p are the sample sizes, a represents the number of successful executions, and b represents the number of failed executions. To more comprehensively reflect the correctness of the change service, for example, the number of manual interventions can also be included in the correctness score calculation:

[0075]

[0076] Where q is the number of samples, and c represents the number of manual interventions. The automation rate, which represents the number of manual interventions, is usually close to 1.

[0077] Figure 5 A flowchart of a method for calculating a third score according to an embodiment of the present disclosure is schematically shown.

[0078] According to the embodiments of the present disclosure, Figure 5 As shown, for example, the third score corresponding to the change operation compliance rate is calculated through operations S2206 to S2207.

[0079] S2206, detecting the keywords of the change operation and maintenance rules of the data sample.

[0080] According to the embodiments of the present disclosure, the compliance rate of the change operation can, for example, reflect the security of the change service. And security can be explained by testing and evaluating factors such as security, compliance, and potential risks of the change service based on standard rules related to operation and maintenance. The higher the score, the better the security. The data for measuring security comes from, for example, the operation and maintenance rule library data and the operation definition data. It is a static and objective data that depends on the degree of perfection of the operation and maintenance rule library. In this embodiment, for example, the keywords in the operation execution script can be detected, and different keywords can be distinguished according to the built-in risk level. In order to further improve the quality of security assessment, the robustness of the service can also be introduced into the security score calculation, that is, the abnormal rollback situations during the execution process are counted.

[0081] S2207: Calculate a third score based on the keyword of the changed operation and maintenance rule.

[0082] According to the embodiments of the present disclosure, data processing is performed based on the risk detection of operations in the change service in the rule base. For example, it includes processing keywords, deducting scores based on the number of keywords detected in the change service, and counting the rollback of operations when the service is abnormal. If the rollback is performed normally, the score is increased, otherwise the score is deducted. The security score S3 is obtained by weighted processing based on the keyword detection and abnormal rollback situation:

[0083]

[0084] Here, u and v are the number of samples, d represents the keyword detection status in the current service operation, and p1 represents the keyword detection weight. e represents the statistics of abnormal rollbacks in the current service, and p2 represents the weight of abnormal rollbacks. The sum of p1 and p2 is, for example, 100%.

[0085] According to an embodiment of the present disclosure, for example, the entire service change can be rated based on the scores of multiple dimensions (business factors) plus the weights of the corresponding dimensions, with a full score of five stars. The score obtained for each dimension is multiplied by the corresponding weight to obtain the score S corresponding to the entire service change rating model:

[0086]

[0087] Among them, w is the number of dimensions (ie, business factors), f represents the score of each dimension, such as S1, S2 and S3 in this embodiment, and p represents the weight corresponding to each dimension.

[0088] According to the embodiments of the present disclosure, different users have different service change requirements, and their focus on the results of the change services is also different. Therefore, other business factors can be introduced to improve the quality of the change service score in more dimensions, and provide operation and maintenance personnel with an overall evaluation result of the change system. In addition to the stability, security and correctness in the above embodiments, for example, versatility (popularity) and business performance can also be introduced. Versatility refers to the number of hosts used and the proportion of use within a certain time range. The data for measuring versatility comes from, for example, the number of change applications and the number of hosts selected by the change application. It is a dynamic and objective data, and the number of applications and the number of hosts changed are both integers greater than or equal to 0. Versatility mainly reflects the use of the change service within a period of time, so in the time dimension, for example, five time periods are divided, and each time period has a different weight for the calculation of data. The specific time periods are divided as follows:

[0089] Table 1 Change service usage time weight

[0090] Time period Weight 0-3 months ago 1.0 4-6 months ago 0.8 7-9 months ago 0.5 10-12 months ago 0.2 12 months ago 0.1

[0091] The universality score S4 can be calculated based on different weights and data statistics of the corresponding time period:

[0092]

[0093] Among them, g is the host usage data in each stage, p j Indicates the weight corresponding to each time period.

[0094] According to an embodiment of the present disclosure, the business score is, for example, a score calculated based on user evaluation feedback information, and the higher the popularity, the higher the score. The subjective evaluation of the user can be reflected by the number of likes / dislikes of the changed service, highlighting the business nature of the service. It is a dynamic, subjective data that changes dynamically over time. There are two main sources of business data, for example: after the execution of the changed service is completed, the user makes a simple feedback on the execution of the work order; the user makes a simple feedback on the changed service on the work order application page. For example, a default score is set, and then based on the default score, the changed service is added or subtracted based on the user's positive or negative feedback, and the business score S5 is calculated:

[0095]

[0096] Among them, O2 is the second default score, r and t are the number of samples, h represents the positive feedback statistics of users, and l represents the negative feedback statistics of users. In the process of correctness and business calculation, due to negative feedback from users or during execution, if there is no default value, it may always be 0 points. Therefore, a default value is set to determine the passing score. Later, according to the execution situation or user feedback, it will not always be 0. The specific O1 or O2 value (passing score) needs to be determined according to the overall operation and maintenance standards of the data center. It can be accurately measured using big data statistics, or multiple preliminary estimates can be given based on the overall requirements of the data center, and then tested multiple times to finally determine the value based on the actual situation. The calculation of each business factor is triggered by a timer and is not real-time. The calculation of each business factor is a percentage system, that is, the highest score is 100 points and the lowest score is 0 points. Combined with the number of business factors that need to be introduced, the scoring result of the corresponding changed service scoring model can be obtained according to formula (7).

[0097] Figure 6 The figure schematically shows a new file system change scoring result diagram according to an embodiment of the present disclosure.

[0098] According to the embodiment of the present disclosure, taking the above five business factors as an example, the graphical scoring results of the newly created file system change service are obtained, such as Figure 6 As shown. Figure 6 It can be seen that the change stability and correctness of the newly built file system are between 2 stars and 3 stars, the business performance and security are both over 3 stars, while the popularity (versatility) is less than 1 star, and the comprehensive evaluation is 2.2 stars. That is, the change service evaluation method proposed in this disclosure sorts out the actual business of the change service and summarizes the five business factors for evaluation in different dimensions, thus ensuring the comprehensiveness of the evaluation principle. For the entire data center, various other change services are also involved, such as adding operating system users, modifying user permissions, expanding the file system, modifying middleware initialization parameters, configuring database user roles, etc. The service evaluation results calculated based on the method disclosed in this disclosure will gradually accumulate and iterate over time, and eventually form a complete evaluation system for all change services, giving a digital (including factor data, model data, system data, etc.) evaluation result for the entire change system.

[0099] According to an embodiment of the present disclosure, when calculating the score of each business factor, it is necessary to process the respective extreme values, such as deducting the score to 0 or reaching the maximum score. For example, in the calculation of the correctness score, when the number of successes a reaches 20 times, S2 reaches the maximum value. Then, when the number of subsequent successes exceeds 20 times, S2 is still processed as the maximum score (the score weight of the business factor among all business factors) and does not continue to increase according to formula (4) or formula (5).

[0100] Based on the above-mentioned change service evaluation method, the present disclosure also provides a change service evaluation device. Figure 7 The device is described in detail.

[0101] Figure 7 The following schematically shows a structural block diagram of a device for evaluating service changes according to an embodiment of the present disclosure.

[0102] like Figure 7 As shown, the change service evaluation device 700 of this embodiment includes an acquisition module 710 , a calculation module 720 and an evaluation module 730 .

[0103] The acquisition module 710 is used to obtain the business factors of the data sample, which include the concentration of change execution time, change execution success rate and change operation compliance rate. In one embodiment, the acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0104] The calculation module 720 is used to calculate the first score, the second score, and the third score corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate. In one embodiment, the calculation module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0105] The evaluation module 730 is used to evaluate the modified service according to the first score, the second score, and the third score. In one embodiment, the evaluation module 730 can be used to perform the operation S230 described above, which will not be described in detail here.

[0106] According to an embodiment of the present disclosure, any multiple modules in the acquisition module 710, the calculation module 720, and the evaluation module 730 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 710, the calculation module 720, and the evaluation module 730 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 710, the calculation module 720, and the evaluation module 730 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0107] Figure 8 The block diagram schematically shows an electronic device suitable for implementing the service evaluation method according to an embodiment of the present disclosure.

[0108] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0109] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0110] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 808 including a hard disk; and a communication portion 809 including a network interface card such as a LAN card or a modem. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 810 as needed, so that a computer program read therefrom can be installed into the storage portion 808 as needed.

[0111] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0112] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0113] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the item recommendation method provided by the embodiments of the present disclosure.

[0114] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 801 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0115] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0116] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0117] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0120] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A change service evaluation method, comprising: Obtaining business factors of the data sample, wherein the business factors include change execution time concentration, change execution success rate, and change operation compliance rate; Calculate the first score, the second score, and the third score corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate, respectively; Evaluate the change service according to the first score, the second score, and the third score; Calculating the first score of the change execution time concentration includes: Calculate the average change execution time; Calculating a standard deviation of the change execution time of the data sample based on the average change execution time; Calculating the first score based on the standard deviation of the change execution time; Calculating the second score of the change execution success rate includes: Get the number of successful change executions; Setting a first default score, and calculating the second score based on the first default score and the number of successes; The third score for calculating the compliance rate of the change operation includes: Detecting keywords of change operation and maintenance rules of the data sample; The third score is calculated based on the keyword of the change operation and maintenance rule.

2. The change service evaluation method according to claim 1, wherein the business factor further comprises a change execution manual intervention rate, and calculating the second score further comprises: Get the number of manual interventions for change execution; The second score is calculated based on the first default score, the number of successes, and the number of manual interventions.

3. The change service evaluation method according to claim 1, wherein the business factor further comprises change operation robustness, and calculating the third score further comprises: Get the number of abnormal rollbacks of the change operation; The third score is calculated based on the operation and maintenance rule keyword and the number of abnormal rollbacks.

4. The change service evaluation method according to claim 2, wherein the business factor further comprises a change execution failure rate, and calculating the second score further comprises: Get the number of failed change executions; The second score is calculated based on the first default score, the number of successes, the number of manual interventions, and the number of failures.

5. The method for evaluating a change service according to claim 1, wherein the business factor further comprises a host usage ratio of the change service, and calculating the fourth score corresponding to the host usage ratio of the change service comprises: Obtain the number of times the host is used in different time periods; Setting a first weight corresponding to the number of times used; The fourth score is calculated based on the number of times used and the first weight.

6. The change service evaluation method according to claim 1, wherein the business factor further comprises change service user satisfaction, and calculating the fifth score corresponding to the change service user satisfaction comprises: Obtaining the number of feedbacks from the user, including the number of positive feedbacks and the number of negative feedbacks; A second default score is set, and the fifth score is calculated based on the second default score and the number of feedbacks.

7. The method for evaluating a change service according to claim 1, wherein evaluating the change service according to the first score, the second score, and the third score comprises: Setting a second weight for each of the business factors; Calculate the total score of the change service according to the first score, the second score, the third score and the second weight; The change service is evaluated according to the total score.

8. The change service evaluation method according to claim 1, wherein the calculation of the first score, the second score, and the third score corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate, respectively, further comprises: The extreme values ​​in the scores are processed, where the extreme values ​​include 0 and the highest weighted score of each of the business factors.

9. A device for evaluating a service change, comprising: An acquisition module is used to acquire business factors of data samples, wherein the business factors include change execution time concentration, change execution success rate and change operation compliance rate; a calculation module, configured to calculate a first score, a second score, and a third score corresponding to the change execution time concentration, the change execution success rate, and the change operation compliance rate, respectively; as well as, an evaluation module, configured to evaluate the change service according to the first score, the second score, and the third score; Calculating the first score of the change execution time concentration includes: Calculate the average change execution time; Calculating a standard deviation of the change execution time of the data sample based on the average change execution time; Calculating the first score based on the standard deviation of the change execution time; Calculating the second score of the change execution success rate includes: Get the number of successful change executions; Setting a first default score, and calculating the second score based on the first default score and the number of successes; The third score for calculating the compliance rate of the change operation includes: Detecting keywords of change operation and maintenance rules of the data sample; The third score is calculated based on the keyword of the change operation and maintenance rule.

10. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the service change evaluation method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the service change evaluation method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for evaluating a service change according to any one of claims 1 to 8 is implemented.

Images