Open source big data system software reliability modeling method considering complex test and troubleshooting efficiency

Abstract

The invention belongs to the technical field of software reliability models, and particularly relates to an open source big data system software reliability modeling method considering complex testing and troubleshooting efficiency. Aiming at the problem of reliability evaluation of open source big data system software, software reliability modeling is carried out by considering a complex test environment and troubleshooting efficiency. Related experiments verify the accuracy of the provided model in the aspect of predicting residual faults of the software and the effectiveness of evaluating the reliability of the software of the open source big data system. The model can help software developers to carry out fault prediction and reliability evaluation in the actual development and test process of open source big data system software.

Description

Technical Field

[0001] This invention belongs to the field of software reliability modeling technology, specifically involving an open-source big data system software reliability modeling method that considers complex testing and troubleshooting efficiency. Background Technology

[0002] In recent years, big data technology has been widely applied in the field of human social services. While enjoying the benefits and convenience brought by big data technology, people have also raised questions about the reliability of big data system software. The software testing complexity of open-source big data systems, represented by Hadoop, Spark, Flink, and HBase, mainly stems from the system's own architectural features, the characteristics of the open-source ecosystem, and the special requirements of big data application scenarios. This complexity manifests in multiple dimensions, including system design, data characteristics, testing scenarios, and other levels.

[0003] First, to handle massive amounts of data, open-source big data systems typically employ distributed architectures and component-based designs, introducing multi-layered complexity to testing. Second, open-source projects generally adopt a community-driven model, resulting in rapid system version iterations and severe version fragmentation, further increasing testing complexity. Third, the core of big data systems lies in "data processing," and the scale, data types, and timeliness requirements of the data directly increase the difficulty of software testing. Fourth, performance is a key indicator for big data systems (such as throughput, latency, and concurrency), and testing performance and scalability is far more challenging than for traditional software. Fifth, open-source big data systems often require integration with upstream and downstream tools (such as relational databases, message queues, and BI tools), making compatibility testing extremely extensive. Sixth, "fault tolerance" (such as automatic recovery after node failure) is one of the core characteristics of distributed systems, but constructing relevant test scenarios is extremely difficult. Finally, the lack of mature standardized tools for testing open-source big data systems makes it difficult to effectively improve automated test coverage.

[0004] To address the reliability challenges of open-source big data systems, various software reliability models have been proposed in previous studies. Tamura et al. proposed a three-dimensional stochastic differential equation software reliability model that considers big data characteristics, failure factors, and network factors. Wang et al. proposed an open-source big data software reliability model based on the Weibull-Weibull distribution. Cao and Gao proposed a method for evaluating big data systems using fault tree analysis. Wang et al. further considered the efficiency of fault introduction and elimination, proposing a relative software reliability model for open-source big data systems. Tamura and Yamada et al. proposed a software reliability assessment method suitable for cloud computing and big data environments based on fault data clustering and risk rate models. Kumar et al. used software reliability growth models (SRGMs) based on non-homogeneous Poisson processes (NHPP) to evaluate the software reliability of open-source data systems. Although the above models can predict the number of remaining software faults and effectively evaluate the software reliability of open-source big data systems under certain conditions, they still do not fully consider the complexity of the software testing process for open-source big data systems. In addition, fault elimination efficiency also plays a crucial role in software reliability modeling. Fault elimination efficiency is defined as the ratio of the number of faults eliminated to the number of faults detected during software testing, used to quantify the proportion of detected faults that can actually be repaired. Summary of the Invention

[0005] Due to the relative complexity of testing open-source big data system software, including distributed system design, data complexity, the dynamic development of the open-source ecosystem, the skills of community testers, and limitations of testing resources, the reliability of open-source big data system software has received widespread attention. Currently, research on the reliability of open-source big data system software is limited. To address the reliability assessment problem of open-source big data system software, this invention provides a reliability modeling method that considers both complex testing and troubleshooting efficiency.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] An open-source big data system software reliability modeling method that considers the efficiency of complex testing and troubleshooting includes the following steps:

[0008] Assume the testing process of an open-source big data system follows a non-homogeneous Poisson process with the following mean function: During testing, fault detection can be viewed as a counting process, using... Indicates the time Number of faults detected so far:

[0009] (1);

[0010] Where i is an integer, i = 0, 1, 2, 3, ..., n;

[0011] Assume that this complex testing process follows a Weibull distribution. Its complex test rate function Represented as:

[0012] (2);

[0013] in, For scale parameters, For shape parameters;

[0014] Considering the learning process of fault detection during the testing of open-source big data system software, the fault detection rate function... Represented as:

[0015] (3);

[0016] in, For fault detection rate, Inflection point coefficient;

[0017] Assuming that in the fault detection process of an open-source big data system, the number of faults detected instantaneously is related to the actual number of remaining faults in the software, the following formula is established:

[0018] (4);

[0019] in, This indicates the total number of faults that the software can initially detect. Indicates fault clearing efficiency;

[0020] Substituting equations (2) and (3) into equation (4), we get:

[0021] (5);

[0022] Equation (5) is the expression for the proposed software reliability model.

[0023] Furthermore, the parameters of the software reliability model are estimated using the maximum likelihood estimation method.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] Based on a systematic analysis of the testing complexity characteristics and fault-solving efficiency impact mechanisms of open-source big data systems, this invention proposes a software reliability model for open-source big data systems that simultaneously considers testing complexity and fault-solving efficiency. Experimental results show that this model exhibits good performance in residual fault prediction and reliability assessment. Attached Figure Description

[0026] Figure 1 It is the Weibull probability density function;

[0027] Figures 2(a) to (f) show the cumulative number of faults detected over time using 100% of DS1-1, 100% of DS1-2, 100% of DS1-3, 100% of DS2-1, 100% of DS2-2, and 100% of DS2-3, respectively.

[0028] Figures 3(a) and (b) show the cumulative number of faults detected using 80% of DS1-3 and 80% of DS2-1 over time, respectively.

[0029] Figures 4(a) and (b) show the 95% confidence intervals for 80% and 80% of DS1-3 and DS2-1, respectively.

[0030] Figures 5(a) to (f) show the model parameters proposed using 100% DS1-2. , , d and Sensitivity analysis plot. Detailed Implementation

[0031] To further illustrate the technical solution of the present invention, the present invention will be further described below through embodiments.

[0032] Example 1

[0033] An open-source big data system software reliability modeling method that considers the efficiency of complex testing and troubleshooting includes the following steps:

[0034] Assume the testing process of an open-source big data system follows a non-homogeneous Poisson process with the following mean function: During testing, fault detection can be viewed as a counting process, using... Indicates time Number of faults detected previously:

[0035] (1);

[0036] The testing process for open-source big data system software is significantly complex due to the influence of various factors. Assume this complex testing process follows a Weibull distribution. Its complex test rate function It can be represented as:

[0037] (2);

[0038] in, For scale parameters, For shape parameters.

[0039] In the testing process of open-source big data system software, testing complexity may fall into two categories. One is that testing complexity may gradually decrease over time. The other is that testing complexity initially increases and then decreases over time. The first scenario might be due to the testers (contributors) being highly skilled and familiar with the software, and encountering no major problems during testing. The second scenario might be due to the testers (contributors) being unfamiliar with the software, or it might be due to their limited skill level and the need for time to learn and understand the software. Figure 1 It can be seen that the probability density function (PDF) of the Weibull distribution can effectively simulate the changes in the complexity of these two types of tests.

[0040] Considering the learning process of fault detection during the testing of open-source big data system software, the fault detection rate function can be expressed as:

[0041] (3);

[0042] in, For fault detection rate, This represents the inflection point coefficient. Generally, the fault detection rate of open-source big data systems is affected by test complexity; higher test complexity results in a lower fault detection rate, and vice versa. Furthermore, the possibility of faults detected during testing being actually eliminated is also considered. It is assumed that the number of faults detected instantaneously during fault detection in an open-source big data system is related to the actual number of remaining faults in the software. Therefore, the following formula is established:

[0043] (4);

[0044] in, This indicates the total number of faults that the software can initially detect. This indicates the efficiency of fault elimination.

[0045] Substituting equations (2) and (3) into equation (4), we get:

[0046] (5);

[0047] Equation (2) is the expression of the proposed software reliability model.

[0048] This embodiment uses the maximum likelihood estimation method to estimate the parameters of the proposed model. The maximum likelihood function can be expressed as:

[0049] (6);

[0050] Taking the logarithm of both sides of equation (6), we obtain the log-likelihood function:

[0051]

[0052] (7);

[0053] By taking the partial derivatives of equation (7) with respect to the model parameters and setting them to zero, we can obtain the maximum likelihood estimates of each parameter of the proposed model:

[0054] (8).

[0055] Example 2

[0056] 1. Fault dataset, model comparison criteria, and comparison of software reliability growth models

[0057] The fault dataset used in this embodiment comes from the website (https: / / issues.apache.org). Fault data from open-source big data systems was collected from two released software components: MapReduce and Spark. In MapReduce, three fault datasets (DS1-1, DS1-2, and DS1-3) were collected from MapReduce versions 0.23.5, 0.23.6, and 0.23.7, respectively. Additionally, three fault datasets (DS2-1, DS2-2, and DS2-3) were collected from Spark versions 0.9.0, 0.9.1, and 0.9.2, respectively. The collected fault datasets include fault types of "Bug," "Task," and "Sub-tas"; fault statuses of "Resolved" and "Closed"; and error resolution of "fixed." For the first fault dataset (DS1-1), 30 faults were collected from May 2010 to November 2012. For the second fault dataset (DS1-2), 21 faults were collected from May 2012 to January 2013. For the third fault dataset (DS1-3), 24 faults were collected from January 2012 to April 2013. For the fourth fault dataset (DS2-1), 30 faults were collected from October 2012 to March 2014. For the fifth fault dataset (DS2-2), 41 faults were collected from June 2013 to April 2014. For the last fault dataset (DS2-3), 28 faults were collected from January 2014 to July 2014.

[0058] To fully compare the performance of the proposed models, comparative experiments were conducted using different software reliability growth models. For example, fully debugged models included the GO model, DSS model, ISS model, Wang model, and Li model; incompletely debugged models included the Yamada incomplete-1 model, Yamada incomplete-2 model, Zhang-Teng-Pham model, and Verma model; open-source software reliability growth models included the Wang model and Li model; and other software reliability growth models were traditional closed-source software reliability growth models. In addition, models that considered fault elimination efficiency, such as the Zhang-Teng-Pham model and Verma model, were also considered. Table 1 lists the various software reliability growth models used. Table 2 lists the parameter values ​​of the proposed models obtained using maximum likelihood estimation.

[0059] Table 1 shows the various software reliability growth models used.

[0060]

[0061] Table 2 shows the parameter estimation of the model using the maximum likelihood estimation method.

[0062]

[0063] To fully validate the model's fitting and predictive performance, mean squared error (MSE) and coefficient of determination (R²) were used. 2 The model evaluation criteria include eight metrics: Root Mean Square Error (RMSE), Predictive Power (PP), Theil Statistic (TS), Variance, Root Mean Square Prediction Error (RMSPE), and Akaike Information Criterion (AIC). Detailed explanations are provided below:

[0064] 1) The mean squared error (MSE) is defined as follows:

[0065] (9);

[0066] in, and They represent time respectively The expected number of faults detected and the actual number of faults observed are compared. and These represent the sample size of the fault dataset and the number of model parameters, respectively.

[0067] 2) R 2 The definition is as follows:

[0068] (10);

[0069] 3) The Akaike Information Criterion (AIC) is defined as follows:

[0070] (11);

[0071] 4) The root mean square error (RMSE) is defined as follows:

[0072] (12);

[0073] 5) The Theil statistic (TS) is defined as follows:

[0074] (13);

[0075] 6) Variance is calculated as follows:

[0076] ;

[0077] (14);

[0078] 7) Predictive power (PP) is defined as follows:

[0079] (15);

[0080] 8) The root mean square prediction error (RMSPE) is defined as follows:

[0081] (16);

[0082] For MSE, RMSE, PP, TS, Variance, RMSPE, and AIC, the smaller their values, the better the model's performance. For R... 2 The larger the value, the better the model's fitting performance.

[0083] 2. Comparison of model fitting performance

[0084] As shown in Table 3, the proposed model exhibits the best fitting performance among all models when using 100% DS1-1. The MSE and R-squared values ​​of the proposed model are... 2 The AIC values ​​were 11.25, 0.9133, and 103.86, respectively. In R... 2 The GO model, Yamada imperfection-1 model, Yamada imperfection-2 model, and Wang model all have negative values. This indicates that their fitting performance is poor and they are not suitable for fitting the OSBDS software fault dataset (DS1-1). As shown in Figure 2(a), the proposed model has a significantly better fitting performance than the other models.

[0085] Table 3 shows that when using 100% DS1-2, the proposed model outperforms other models in terms of fit. The proposed model's MSE and R-values ​​are also shown. 2 The AIC values ​​are 4.34, 0.9194, and 76.88, respectively. Note the R-values ​​of the GO model, ISS model, Wang model, and Li model. 2 All values ​​are negative. This indicates that using these models to fit the fault dataset (DS1-2) is inappropriate. Figure 2(b) shows that PM outperforms all other models.

[0086] Using 100% of DS1-3, Table 3 shows that the MSE of the DSS model and the AIC of the Yamada imperfection-2 model are both lower than those of the proposed model. However, the R-value of the proposed model is lower. 2 R compared to other models 2 Overall, the proposed model's fitting performance is no worse than that of the DSS model and the Yamada Imperfect-2 model (AIC model), and it outperforms other models. Figure 2(c) shows a comparison of the fitting performance of each model.

[0087] As shown in Table 4, the PM model has the best fit performance among all models. When using 100% DS2-2, the R² of the Wang model is -0.3831. This indicates that the Wang model is not suitable for fitting the fault dataset (DS2-2). Furthermore, when using 100% DS2-3, the R² of the ZhangTeng-Pham model and the Verma model... 2 The value is negative. This indicates that neither of these two models is suitable for fitting the fault dataset (DS2-3). From Figures 2(d) to (f), it can be seen that the proposed model is the best among all models.

[0088] In summary, the proposed model can fit fault data from open-source big data system software very well. Compared with other models, the proposed model has better fitting performance. Besides the DSS model and the proposed model, other models have higher R-values. 2 All values ​​are negative. This indicates that other models cannot fit open-source big data fault datasets well. Furthermore, as shown in Tables 3 and 4, the fitting performance of the DSS model fluctuates significantly across different fault datasets. This method demonstrates good and stable fitting performance on various open-source big data fault datasets. The results show that this model can fit fault datasets of open-source big data system software very well.

[0089] Table 3. Performance Comparison of DS1 with 100% Fit of Software Reliability Model

[0090]

[0091] Table 4. Performance Comparison of DS2 Software Reliability Model Fitting at 100%

[0092]

[0093] 3. Comparison of Model Predictive Performance

[0094] As can be seen from Table 5, using 80% of DS1-3, the proposed model has lower MSE, RMSPE, TS, RMSE, Variance, and PP than other models.

[0095] The model with the best predictive performance is the Verma model. The third is the DSS model. The worst is the Zhang Dengfan model. As can be seen from Figure 3(a), the predictive performance of this model is better than the other models. Similarly, when using 80% of DS2-1, the MSE, RMSPE, TS, RMSE, variance, and PP of the proposed model are also lower than those of the other models. The proposed model is better than the other models, followed by the GO model, then the Yamada imperfect-2 model, and the worst is the DSS model. As can be seen from Figure 3(b), the proposed model has the best predictive performance among all models.

[0096] To quantify the uncertainty of the estimation, we calculated the 95% confidence intervals using 80% of DS1-3 and DS2-1, respectively. Figures 4(a) and 4(b) show that the estimates of the proposed model fall entirely within the upper and lower bounds of the 95% confidence intervals. Overall, the proposed model exhibits good predictive performance and is stable on various open-source big data system software. However, the predictive performance of other models on open-source big data systems is not stable, exhibiting significant fluctuations.

[0097] Table 5 Comparison of Predictive Performance of Software Reliability Models

[0098]

[0099] Example 3

[0100] To investigate the changes in the model parameters proposed in the study, a sensitivity analysis was conducted. Parameter sensitivity analysis refers to changing one parameter while keeping other parameters constant. As shown in Figures 5(a) to (f), the parameter... , , and It has a significant impact on the model, while the parameters and d are not sensitive parameters; their changes have no significant impact on the model. The reasons for this are analyzed as follows:

[0101] parameter This represents the initial expected total number of faults to be detected. Variations in this value significantly affect the accuracy of the model's fault predictions in the software, substantially impacting the model's fitting and predictive performance.

[0102] parameter This represents the fault detection rate. Its changes have a significant impact on the number of faults estimated by the model. Therefore, it is a parameter with a significant impact.

[0103] parameter This is the curvature factor. Its variation significantly affects the shape of the fault number curve estimated by the model. It reflects the trend of the fault number curve estimated by the model.

[0104] This parameter This represents the fault elimination efficiency. Its variation has a significant impact on the estimation of the actual number of remaining faults in the software.

[0105] parameter denoted by , and denoted by d, which is the scale parameter. These are not sensitivity parameters. However, they play a crucial role in stabilizing model performance.

[0106] In summary, this invention establishes a reliability model for open-source big data system software that considers complex testing environments and fault elimination efficiency. Six fault datasets were collected from two consecutive releases of open-source big data system software. Eight model comparison standards and nine software reliability models were used for fitting and prediction performance comparison. The model parameters were estimated using the maximum likelihood estimation method. Experimental results show that, compared with other models, the proposed model has better fitting and prediction performance and can effectively evaluate the reliability of open-source big data system software. Parameter sensitivity analysis shows that most parameters in the model are significant.

Claims

1. A reliability modeling method for open-source big data system software that considers the efficiency of complex testing and troubleshooting, characterized in that, Includes the following steps: Assume the testing process of an open-source big data system follows a non-homogeneous Poisson process with the following mean function: During testing, fault detection can be viewed as a counting process, using... Indicates the time Number of faults detected so far: （1）； Where i is an integer; Assume that this complex testing process follows a Weibull distribution. Its complex test rate function Represented as: （2）； in, For scale parameters, For shape parameters; Considering the learning process of fault detection during the testing of open-source big data system software, the fault detection rate function... Represented as: （3）； in, For fault detection rate, Inflection point coefficient; Assuming that in the fault detection process of an open-source big data system, the number of faults detected instantaneously is related to the actual number of remaining faults in the software, the following formula is established: （4）； in, This indicates the total number of faults that the software can initially detect. Indicates fault clearing efficiency; Substituting equations (2) and (3) into equation (4), we get: （5）； Equation (5) is the expression for the proposed software reliability model.

2. The open-source big data system software reliability modeling method considering complex testing and troubleshooting efficiency as described in claim 1, characterized in that, The parameters of the software reliability model are estimated using the maximum likelihood estimation method.

Images