Open source big data system software reliability modeling method based on troubleshooting efficiency

Abstract

The invention discloses an open source big data system software reliability modeling method based on troubleshooting efficiency, and belongs to the technical field of open source software reliability models. The problem that a traditional software reliability model is not suitable for the development and test environment of open-source big data system software, so that the reliability evaluation requirement of the open-source big data system software cannot be met is solved. The invention provides an open source big data system software reliability model considering complex changes such as fault detection rate, fault introduction and fault elimination efficiency, the model can accurately predict the number of residual faults of the software, and the reliability of the open source big data system software is effectively evaluated. Through comparison with the established software reliability model, the fault prediction accuracy of the proposed model is verified, which indicates that the proposed model can effectively evaluate the reliability of the open source big data system software. The provided model can be used for fault prediction and reliability evaluation in actual open source big data system software development.

Description

Technical Field

[0001] The present invention belongs to the technical field of open source software reliability models, and in particular relates to an open source big data system software reliability modeling method based on troubleshooting efficiency. Background Art

[0002] In recent years, big data technology has developed rapidly and has been used to serve human society and daily life. Big data system is a distributed parallel processing system. Hadoop is an important part of big data system, which includes HDFS, MapReduce, HBase, Hive and other functional components. Big data system can process massive amounts of information data in parallel, collect and analyze useful and valuable information from massive amounts of data, and bring benefits and convenience to people's business activities and daily life. Due to the complexity of big data system, its reliability is also increasingly valued.

[0003] Big data systems are generally developed and tested in an open source environment. There are also various faults in open source big data system software. These faults mainly include six categories: code defects, application software faults, system design limitations, configuration errors, operation faults, and hard faults. Cao and Gao pointed out that the main types of software faults in big data systems are operation faults, configuration errors, application software faults, and code defects. These types of faults have a significant impact on the reliability of open source big data system software. Therefore, studying the reliability modeling and evaluation of open source big data system software is of great significance to improving software quality.

[0004] Generally speaking, since the modeling environment of closed-source software reliability models is different from the development and testing environment of open-source big data system software, closed-source software reliability models are not suitable for reliability assessment of open-source big data system software. The development process of closed-source software is relatively closed, and the testing process has fixed testers and hierarchical management. The development process of open-source software is more open and collaborative, allowing a large number of developers to contribute code together, accelerating software iteration and innovation, and the testing process is more transparent and open. The developer community can participate in testing together and provide feedback and suggestions.

[0005] Compared with the research on closed-source software reliability modeling, the research on open-source big data system software reliability modeling is relatively small. Tamura et al. proposed a three-dimensional software reliability model using stochastic differential equations, considering the relationship between fault detection, big data system software failures and cloud computing network factors. Tamura and Yamada proposed a two-dimensional Wiener process software reliability model considering jump diffusion, big data failures and network factors in cloud computing environment. Cao and Gao analyzed the causes of big data system software failures, compared the proportions of various types of failures, and proposed a big data system software reliability model based on fault tree. Tamura et al. proposed a risk rate-based reliability assessment method considering big data failures, cloud computing environment and network factors through fault data clustering. Tamura and Yamada used deep learning to classify the severity of big data system software failures in a defect tracking system. Tamura et al. proposed a software reliability model considering cloud computing, big data and jump diffusion process, and used neural networks for optimal data partitioning. Kumar et al. compared some traditional closed-source software reliability models using open-source big data system software failure data. The experimental results show that software reliability models with good fitting performance do not necessarily have good prediction performance.

[0006] Due to the complexity of software development and testing of open source big data systems, studying fault detection, fault introduction phenomena, and troubleshooting efficiency is of great significance to establishing a high-quality open source big data system software reliability model. The complexity of fault detection includes: 1) Open source big data systems usually contain multiple subsystems and components, and there are complex interactions and possible dependencies between these subsystems and components. Therefore, when performing fault detection, it is necessary to comprehensively consider the operating status of each subsystem and component and their mutual influence. 2) Big data systems process huge amounts of data with diverse types, including structured data and unstructured data. These data may have various faults during storage, processing, and analysis, such as data loss, data corruption, and data inconsistency. In addition, big data systems usually need to process a large amount of data in real time in a short period of time, which further increases the complexity of fault detection. 3) Big data systems usually adopt a distributed architecture, and data is distributed on multiple nodes. This distributed feature requires fault detection across nodes, which increases the complexity and cost of detection.

[0007] In the process of fault repair of open source big data system software, there is a risk of introducing new faults. For example, when repairing existing faults, if the code is not understood deeply enough or modified improperly, new logical errors or performance issues may be introduced. Especially when dealing with the core components or critical paths of big data systems, any minor changes may have a significant impact on the stability and performance of the entire system. In addition, studying the efficiency of fault repair has an important impact on the reliability modeling of open source big data system software, because it can quantify the relationship between fault detection and fault repair. Summary of the invention

[0008] In view of the problem that the traditional software reliability model is not suitable for the development and testing environment of open source big data system software, and therefore cannot meet the reliability evaluation requirements of open source big data system software, this paper proposes an open source big data system software reliability model that considers complex changes such as fault detection rate, fault introduction and fault elimination efficiency. This model can accurately predict the number of remaining software faults and effectively evaluate the reliability of open source big data system software. The accuracy and effectiveness of the proposed model are verified through corresponding experiments.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The open source big data system software reliability modeling method based on troubleshooting efficiency includes the following steps:

[0011] Step 1: Consider the failures in the open source big data system software development and testing process as a non-homogeneous Poisson process NHPP. The probability distribution of NHPP is defined as follows:

[0012]

[0013] Among them, {N(t), t≥0} is a fault counting process, which represents the cumulative number of faults detected by time t and obeys NHPP; ζ(t) represents the mean function MVF, that is, the cumulative number of faults expected to be detected by time t; n represents a positive integer; ! represents factorial;

[0014] Step 2: Considering the complexity of fault detection, fault introduction, and troubleshooting efficiency during the development and testing of open source big data system software, the following model assumptions are proposed:

[0015] Assumption (1), open source big data system software failures follow NHPP;

[0016] Assumption (2) that fault detection during the software testing of open source big data systems exhibits complex variations and follows a generalized inflection point S-shaped GISS distribution;

[0017] Assumption (3): When the detected faults are eliminated, new faults will be introduced; during the debugging process of open source big data system software, the fault introduction rate is a constant α ;

[0018] Assumption (4), the number of faults detected within the time (t, t + Δt) is related to the number of faults remaining in the software after the software is corrected;

[0019] Assumption (5): The troubleshooting efficiency is the ratio of the number of fault detections to the number of fault repairs. In the open source big data system software testing, the troubleshooting efficiency is a constant p.

[0020] Step 3, based on the above assumptions:

[0021]

[0022] Where: θ(t), υ(t) and p represent the fault detection rate function, fault content function and fault elimination efficiency respectively;

[0023] The mathematical expression of the fault detection rate function is:

[0024]

[0025] Among them, β represents the inflection point factor;

[0026] The fault content function is expressed as:

[0027] υ(t)=υ+αζ(t)

[0028] Where υ and α represent the expected total number of initially detected faults and the fault introduction rate, respectively;

[0029] Step 4: Use maximum likelihood estimation (MLE) to estimate the model parameter values. The maximum likelihood function is expressed as:

[0030]

[0031] Among them, Pr{} represents the probability, N(t 1 )=y 1 ,N(t 2 )=y 2 ,...,N(t n )=y n is the fault counting process, which is expressed as 1 ,t 2 ,...,t n The cumulative number of faults detected at the moment y 1 ,y 2 ,...,y n ;m(t i) represents the mean function, that is, as of time t i The cumulative number of faults expected to be detected;

[0032] Taking the base e logarithm of both sides gives:

[0033]

[0034] Taking partial derivatives we get:

[0035]

[0036] Solve to obtain the parameter values ​​of the proposed model.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] This paper establishes a new software reliability model based on the characteristics of open source big data system software development and testing, such as the complexity of fault detection, the possibility of introducing new faults during fault debugging, and the efficiency factor of fault elimination. By comparing with the established software reliability model, the accuracy of the proposed model in fault prediction is verified, indicating that the proposed model can effectively evaluate the reliability of open source big data system software. The proposed model can be used for fault prediction and reliability evaluation in actual open source big data system software development. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 The following is a schematic diagram comparing the cumulative number of failures estimated by all models. (a) (b) and (c) respectively represent the comparison of the cumulative number of failures estimated by the models using 100% of the failure data sets (DS1-1, DS1-2 and DS1-3). (d) (e) and (f) respectively represent the comparison of the cumulative number of failures estimated by the models using 100% of the failure data sets (DS2-1, DS2-2 and DS2-3).

[0041] Figure 2 This is a schematic diagram of the comparison results of the cumulative number of model-predicted faults using 90% of DS1-3 fault data;

[0042] Figure 3 This is a diagram showing the comparison of the cumulative number of failures predicted by the model using 90% of the DS2-3 failure data. DETAILED DESCRIPTION

[0043] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0044] The present invention uses a big data fault dataset from the Apache Software Foundation website (https: / / issues.apache.org). In the issue tracking system, there are various faults from open source software projects. We collected two fault datasets from big data system components (such as Hadoop Common and Hive) in the fault tracking system. The first fault dataset (DS1) contains three fault sub-datasets from consecutive software versions, such as Hadoop Common 0.21.0, Hadoop Common 0.22.0, and Hadoop Common 0.23.0, which we named DS1-1, DS1-2, and DS1-3 respectively. The second fault dataset (DS2) contains three fault sub-datasets from Hive components, such as Hive 0.5.0, Hive 0.6.0, and Hive 0.7.0, which we named DS2-1, DS2-2, and DS2-3 respectively. For DS1-1, a total of 309 faults were detected from December 2007 to July 2010. For DS1-2, a total of 120 faults were collected from October 2009 to September 2011. For DS1-3, a total of 153 faults were obtained from October 2007 to December 2012. In addition, for DS2-1, DS2-2, and DS2-3, a total of 110, 115, and 153 faults were collected from September 2008 to February 2010, February 2009 to October 2010, and September 2008 to March 2011, respectively.

[0045] We collected a fault dataset using the fixed version in the issue tracking system. For the issue type, we selected "Bug", "Task", and "Sub-task" as the collected faults. For the issue status, we collected faults with the status of "Closed" and "Resolved". Finally, among the selected faults, we deleted the faults labeled "Unresolved", "Won'tfix", "Duplicate", "Invalid", and "Cannot Reproduce". We used the collected fault dataset to conduct model fitting and prediction performance comparison experiments.

[0046] 1. Comparison Models:

[0047] To fully validate the goodness of fit and predictive performance of the proposed model, we compare it with various established software reliability models, including Perfect Debug (PD) models such as the Goel-Okumoto (GO) model, Delayed S-shaped (DSS) model, Inflection S-shaped (ISS) model, Samal-kumar model, Wang model, and Li model, and Imperfect Debug (ID) models such as the Yamada Imperfect-1 model and Yamada Imperfect-2 model, Zhang-Teng-Pham model, and the proposed model.

[0048] From different perspectives of software development and testing, these models can be divided into closed source software (CSS) reliability models, such as the GO model, DSS model, ISS model, Yamada Imperfect-1 model, Yamada Imperfect-2 model, Samal-Kumar model, Zhang-Teng-Pham model, etc.; open source software (OSS) reliability models include Wang model, Li model and the proposed model. From the perspective of debugging efficiency, the Samal-Kumar model, Zhang-Teng-Pham model and the model proposed in this paper all take debugging efficiency factors into consideration. Table 1 lists various software reliability models.

[0049] Table 1. Various software reliability models

[0050]

[0051] Among them, the G-O model is derived from Y. Tamura and S. Yamada. Reliability analysis based on three-dimensional stochastic differential equation for big data on cloud computing. Proceedings of the IEEE International Conference on Industrial Engineering and Engineering Management, 2014, CDROM (Quality Control & Management III); the DSS model is derived from Y. Tamura and S. Yamada, "Reliability analysis based on a jump diffusion model with two Wiener processes for cloud computing with big data," Entropy, Vol. 17, No. 7, pp. 4533–4546, 2015; the ISS model is derived from Y. Tamura, Y. Nobukawa, S. Yamada. A method of reliability assessment based on hazard rate by clustering approach for cloud computing with big data. 2015 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM). IEEE, 2015: 732-736; the Yamada Imperfect-1 model and the Yamada Imperfect-2 model are derived from Y. Tamura, S. Yamada. Comparison of big data analyses for reliable open source software. 2016 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM). IEEE, 2016: 1345-1349; the Samal-Kumar model is derived from Y. Tamura, T. Takeuchi, S.Yamada.Software reliability and cost analysis consideringservice user for cloud with big data.International Journal ofReliability,Quality and Safety Engineering,2017,24(02):1750009;Wang model is derived from R.Kumar,S.Kumar,SKTiwari.A study of software reliability on big data open sourcesoftware.International Journal of System Assurance Engineering andManagement, 2019, 10: 242-250; Li model is derived from ALGoel, K. Okumoto, Time-dependent error-detection rate model for software reliability and other performance measures. IEEE Transactions on Reliability, 1979, R28: 206–211; Zhang-Teng-Pham model is derived from S. Yamada, M. Ohba, S. Osaki, S-shaped reliability growth modeling for software error detection. IEEE Transactions on Reliability,1983,R-32:475–484.

[0052] 2. Model comparison criteria:

[0053] We used six model comparison criteria, including mean square error (MSE) (from M. Ohba, Inflection S-shaped software reliability growth model, Stochastic Models in Reliability Theory, Berlin, Springer, 1984: 144-162.), R 2(from M.Ohba, Inflection S-shapedsoftware reliability growth model, Stochastic Models in Reliability Theory, Berlin, Springer, 1984: 144-162.) and Akaike Information Criterion (AIC) (from S.Yamada, K.Tokuno, and S.Osaki, Imperfect debugging models with fault introduction rate for software reliability assessment. Int. J.Syst.Sci., 1992, 23 (12): 2241-2252.) to measure goodness of fit, and mean square error prediction (MSEP), variance (Variance) and root mean square prediction error (RMSPE) to measure prediction performance. These model criteria are defined as follows:

[0054] 1) MSE: Calculates the deviation between the estimated value and the actual observed value.

[0055]

[0056] where ζ(t i ) and O(t i ) represent the number of faults estimated by the model and the number of faults actually detected, respectively. k1 and k2 represent the sample size of the fault dataset and the number of model parameters, respectively.

[0057] 2) R 2 : Measures the fitting effect of the model, defined as,

[0058]

[0059] 3) AIC: It uses the maximum likelihood function to measure the goodness-of-fit performance of the model and penalizes models with more parameters.

[0060] AIC=-2log(maximum likelihood estimated values)+2*k2

[0061] 4)MSEP: It measures the deviation between the predicted values ​​and the actual observed values.

[0062]

[0063] We use t 1 to m Failure data is used to estimate the model parameter values ​​and the m+1 tok1 The fault data is compared with the model prediction value, i=1,2,3...,m....k1.

[0064] 5) Variance: It measures the standard deviation between the estimated and actual values.

[0065]

[0066] 6)RMSPE: can be used to measure model prediction performance.

[0067]

[0068] R 2 The larger the value, the better the model fit. The smaller the value of the other model criteria mentioned above, the better the model fit.

[0069] 3. Comparison of model fitting performance

[0070] The parameter values ​​of the model are estimated using 100% of the fault data, and the fitting effect of the model is compared. As can be seen from Table 2, the fitting effect of the proposed model is better than that of other models using 100% DS1-1. 2 The values ​​of MSE, R 2 The MSE and AIC values ​​of the DSS model are 7 times and 1.5 times of those of the model in this paper, respectively.

[0071] Using 100% of DS1-2, the MSE and R 2 and AIC values ​​were 7.5, 0.9967, and 131.87, respectively; the second was the DSS model, with MSE, R 2 and AIC values ​​are 75.56, 0.9632, and 160.57, respectively. Among all the models, the Zhang-Teng-Pham model is the worst. The MSE and AIC values ​​of the DSS model are nearly 10 times and 1.2 times of those of the proposed model, respectively. The proposed model is significantly better than other models.

[0072] Table 2 Comparison of model fitting performance using 100% fault datasets DS1 and DS2

[0073]

[0074]

[0075] Using 100% of DS1-3, the MSE and R 2The MSE, R 2 The MSE and AIC values ​​of the DSS model are 9 times and 2 times of those of the model in this paper, respectively. The fitting effect of the model of the present invention is better than that of other models.

[0076] For 100% DS2-1, the MSE and R 2 The MSE, R 2 and AIC values ​​are 89.14, 0.9586 and 129.16 respectively; among all the models, the Yamada Imperfect-2 model is the worst, and the MSE and AIC values ​​of the Zhang-Teng-Pham model are approximately 5 times and 1.2 times of those of the proposed model respectively; the fitting effects of the remaining models are worse than those of the proposed model.

[0077] Taking DS2-2 as 100%, the MSE and R 2 and AIC values ​​were 52.91, 0.9809, and 138.26, respectively. The DSS model ranked second with its MSE, R 2 The MSE, R 2 The MSE and AIC values ​​of the DSS model are 5 times and 1.3 times of those of the proposed model, respectively. Compared with other models, the performance of the proposed model is better than that of other models.

[0078] For 100% DS2-3, the MSE and R 2 The MSE, R 2 The MSE, R 2 The MSE and AIC values ​​of the DSS model are approximately 52 and 3.5 times that of the proposed model, respectively. The fitting performance of the proposed model is better than that of other models.

[0079] from Figure 1We can also see that the proposed model has better fitting performance than other models. Overall, the proposed model has better fitting performance compared to other models.

[0080] 4. Comparison of model prediction performance

[0081] In order to verify the prediction performance of the proposed model, 90% of the fault data sets are used to estimate the model parameter values, and the remaining fault data sets are used to compare the prediction performance of the model. Prediction experiments are conducted on 9 software reliability models and the proposed model. Prediction experiments are conducted on 9 software reliability models and the proposed model. The experimental results are shown in Table 3, and the model estimation parameters are shown in Table 4.

[0082] Table 3. Comparison of model prediction performance using 90% failure datasets DS1-3 and DS2-3.

[0083]

[0084] Table 4. Parameter estimates of the proposed model

[0085]

[0086]

[0087] As can be seen from Table 3, using 90% of DS1-3, the MSEP, Variance, and RMSPE values ​​of the proposed model are 79.22, 3.1752, and 3.2794, respectively. The second place is the Li model, whose MSEP, Variance, and RMSPE values ​​are 97.98, 3.493, and 3.5949, respectively. The MSEP, Variance, and RMSPE values ​​of the Li model are approximately 1.23, 1.1, and 1.1 times those of the proposed model, respectively. The worst is the Yamada Imperfect-1 model. Using 90% of DS1-1, the prediction performance of the proposed model is better than that of other models. Figure 2 It shows that the proposed model has better prediction performance than other models.

[0088] For 90% of DS2-3, the MSEP, Variance, and RMSPE values ​​of the Zhang-Teng-Pham model are 79.58, 2.8298, and 2.938, respectively, and its prediction performance ranks second compared with other models. The prediction performance of our model ranks first, and the MSEP, Variance, and RMSPE values ​​of our model are 21.72, 1.7515, and 1.8109, respectively. The MSEP, Variance, and RMSPE values ​​of the Zhang-Teng-Pham model are nearly 4 times, 2 times, and 2 times of those of our model, respectively. The third place is the ISS model. The worst is the Samal-Kumar model. Figure 3 It shows that the prediction performance of our model is the best among all models, and for 90% of DS2-3, its fitting performance is generally good.

[0089] In summary, the model proposed in this paper is superior to other models in both fitting performance and prediction performance. Whether it is a closed-source software reliability model or an open-source software reliability model, their fitting performance and prediction performance are unstable and poor for fault fitting and prediction of big data system software. In the reliability evaluation of open-source big data system software, it is necessary and important to establish a reliability model that adapts to the development and testing environment of big data system software.

[0090] This paper uses two open source big data system software fault data sets, six model comparison criteria, and ten software reliability models to conduct fault fitting and prediction comparison experiments, and uses the maximum likelihood estimation method to estimate model parameters. The experimental results show that the proposed model has better fault fitting and fault prediction performance than other models. In future research, we will consider the dynamic changes in troubleshooting efficiency and establish a corresponding open source big data system software reliability model.

[0091] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. An open source big data system software reliability modeling method based on troubleshooting efficiency, characterized by: The following steps are involved: Step 1: Consider the failures in the open source big data system software development and testing process as a non-homogeneous Poisson process NHPP. The probability distribution of NHPP is defined as follows: Among them, {N(t), t≥0 } is a fault counting process, which represents the cumulative number of faults detected by time t and obeys NHPP; ζ(t) represents the mean function MVF, that is, the cumulative number of faults expected to be detected by time t; n represents a positive integer; ! represents factorial; Step 2: Considering the complexity of fault detection, fault introduction, and troubleshooting efficiency during the development and testing of open source big data system software, the following model assumptions are proposed: Assumption (1), open source big data system software failures follow NHPP; Assumption (2) that fault detection during the software testing of open source big data systems follows a generalized inflection point S-shaped GISS distribution; Assumption (3), when the detected faults are eliminated, new faults will be introduced; during the debugging process of open source big data system software, the fault introduction rate is a constant α; Assumption (4), the number of faults detected within the time (t, t + Δt) is related to the number of faults remaining in the software after the software is corrected; Assumption (5): The troubleshooting efficiency is the ratio of the number of fault detections to the number of fault repairs. In the open source big data system software testing, the troubleshooting efficiency is a constant p. Step 3, based on the above assumptions: Where: θ(t), υ(t) and p represent the fault detection rate function, fault content function and fault elimination efficiency respectively; The mathematical expression of the fault detection rate function is: Among them, β represents the inflection point factor; The fault content function is expressed as: υ(t)=υ+αζ(t) Where υ and α represent the expected total number of initially detected faults and the fault introduction rate, respectively; Step 4: Use maximum likelihood estimation (MLE) to estimate the model parameter values. The maximum likelihood function is expressed as: Where Pr{} represents probability, N(t1)=y1, N(t2)=y2,..., N(t n )=y n is the fault counting process, representing the time t1, t2, ..., t n The cumulative number of faults detected at the moment y1,y2,...,y n ;m(t i ) represents the mean function, that is, as of time t i The cumulative number of faults expected to be detected; Taking the base e logarithm of both sides gives: Taking partial derivatives we get: Solve to obtain the parameter values ​​of the proposed model.

Images