Software development big data analysis system

Abstract

The invention discloses a software development big data analysis system, which belongs to the technical field of data analysis, and comprises the following steps: comprehensively collecting various data in a software development process, and carrying out standardization processing on the collected various data to obtain development processing data; performing feature extraction on the development processing data to screen out defect associated features related to software defects; based on the development processing data, constructing a software defect prediction model by using a multiple linear regression algorithm, and training and optimizing the model by adjusting parameters of the model; the method comprises the following steps: acquiring corresponding new data in a target software development process, performing processing and feature extraction on the new data, inputting acquired data values corresponding to defect associated features into a software defect prediction model to obtain a prediction result of the number of software defects, and displaying the prediction result in a visual manner. Developers are facilitated to optimize software design and development processes, and software quality and development efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a software development big data analysis system. Background Art

[0002] During software development, the presence of software defects can severely impact software quality and reliability, increasing development and maintenance costs. Timely and accurate prediction of software defects helps development teams allocate resources rationally and proactively implement risk prevention and quality control. Furthermore, existing analysis systems lack the ability to adapt to the ever-changing environment and requirements of software development, resulting in low prediction accuracy and an inability to effectively support software development decisions.

[0003] Therefore, there is an urgent need for a software development big data analysis system that can accurately predict the number of software defects through scientific data analysis models and optimization methods. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a software development big data analysis system to solve the problems raised in the above background technology.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A software development big data analysis system, comprising:

[0006] The data collection module is used to comprehensively collect various types of data during the software development process, including code data, developer operation data, project requirement data, and test data;

[0007] The data processing module performs standardized processing on all types of collected data, including cleaning and preprocessing, to obtain development processing data. It also performs feature extraction on the development processing data to screen out defect-related features related to software defects.

[0008] The modeling module uses the multivariate linear regression algorithm to build a software defect prediction model based on development and processing data, and trains and optimizes the model by adjusting the model parameters;

[0009] The prediction module obtains various new data corresponding to the target software development process, processes and extracts features from the new data, and inputs the data values ​​corresponding to the defect-related features into the software defect prediction model to obtain the prediction results of the number of software defects.

[0010] Preferably, the specific process of collecting various types of data includes:

[0011] Use code static analysis tools to regularly analyze the code in the code base, obtain the number of code lines, code complexity, and code structure, and store them in the database; use performance monitoring tools to collect dynamic data of the code and transmit it to the data acquisition module in real time;

[0012] Integrate data collection plug-ins in the development environment to record developers' development time, modification frequency, and operation records; and obtain developers' code submission records through the version control system Git, including submission time, submission content, and modified file lists;

[0013] Furthermore, project requirement data is obtained through project requirement management tools to obtain the functional requirements, performance requirements, business rules, and changes in user requirements of the project, and is regularly updated and maintained;

[0014] Automatically collect test data through test scripts and test frameworks, including test case design, execution results, and number of defects.

[0015] Preferably, the standardized processing of various types of collected data includes:

[0016] Clean the collected data, remove duplicate data, fill missing values, and smooth noisy data to ensure data quality;

[0017] When implementing preprocessing, data from different sources and in different formats are processed uniformly and converted into the corresponding preset standard format, and the maximum-minimum normalization method is used to normalize the numerical data separately.

[0018] Preferably, the method for obtaining defect correlation features is as follows:

[0019] Traverse the development and processing data to obtain various factors, and use the correlation coefficient method to calculate the Pearson correlation coefficient between each factor and the number of defects;

[0020] According to the calculated Pearson correlation coefficient, factors corresponding to which the absolute value of the Pearson correlation coefficient is greater than or equal to the correlation threshold are screened out as defect correlation features.

[0021] Preferably, the specific implementation steps of the modeling module include:

[0022] Based on the development and processing data, the extracted defect correlation features and defect counts are obtained and integrated into training samples. There are m training samples in total, each of which is (X, Y), where X represents the matrix of defect correlation features and Y represents the number of defects. Furthermore, X = (X1, X2, ..., X n ), X1, X2, ..., X n is the data value corresponding to each defect-related feature, and n is the total number of defect-related features;

[0023] Based on the multiple linear regression algorithm, the multiple linear regression model is obtained, and the expression of the multiple linear regression model is:

[0024] Y=β0+β1X1+β2X2+...+β n X n +ε;

[0025] Where β0 is the intercept term, β1, β2, ..., β n is the regression coefficient corresponding to each defect correlation feature; ε is the random error term.

[0026] Preferably, the specific implementation steps of the modeling module also include:

[0027] The least squares method is used to solve the model parameters, including the intercept term and the regression coefficient corresponding to each defect correlation feature;

[0028] By defining a loss function to measure the difference between the model's predicted value and the true value, and continuously adjusting the model parameters for derivative calculation, the loss function is minimized and the model parameters at this time are obtained as the optimal parameters;

[0029] Among them, the expression of the loss function is as follows:

[0030]

[0031] Where J is the loss function, i = 1, 2, ..., m, i represents the index of the training sample; X i1 ,X i2 ,...,X in represents the defect-related features corresponding to the i-th training sample; Y i represents the number of defects corresponding to the i-th training sample;

[0032] The specific formula for the derivative calculation is:

[0033]

[0034] in, represents the optimal parameter; X T The transpose of the matrix formed by defect-related features;

[0035] Based on the optimal parameters, the corresponding multivariate linear regression model is obtained, which is the software defect prediction model. Its specific expression is:

[0036]

[0037] in, is the predicted value of the number of defects; is the corresponding model parameter in the optimal parameter; Represents the input features that need to be predicted, that is, the data values ​​corresponding to each defect-related feature.

[0038] Preferably, evaluating and optimizing the software defect prediction model includes:

[0039] Obtain the number of defects and the predicted value of the number of defects corresponding to each training sample, and evaluate the software defect prediction model by calculating the determination coefficient;

[0040] The calculation formula of the coefficient of determination is:

[0041]

[0042] Where R 2 is the coefficient of determination;

[0043] If the calculated determination coefficient is greater than or equal to the preset fitting threshold, it means that the software defect prediction model meets the prediction criteria; otherwise, the model structure is further optimized by adjusting the model parameters or using the stepwise regression method to achieve the optimization of the software defect prediction model.

[0044] Preferably, the target software development refers to a specific software development project for which the number of defects currently needs to be predicted.

[0045] Preferably, it also includes: a display and analysis module, which displays the obtained prediction results in a visual manner and performs further analysis and decision-making based on the prediction results.

[0046] In order to solve the above problems, the present invention also provides a software development big data analysis method, comprising the following steps:

[0047] Comprehensively collect all types of data during the software development process, including code data, developer operation data, project requirement data, and test data;

[0048] Standardize all types of collected data, including cleaning and preprocessing, to obtain development and processing data. Feature extraction is then performed on the development and processing data to screen out defect-related features related to software defects.

[0049] Based on development and processing data, a software defect prediction model is constructed using a multivariate linear regression algorithm, and the model is trained and optimized by adjusting the model parameters;

[0050] Obtain various new data corresponding to the target software development process, process and extract features from the new data, input the acquired data values ​​corresponding to the defect-related features into the software defect prediction model, and obtain the prediction results of the number of software defects;

[0051] The obtained prediction results are displayed in a visual way, and further analysis and decision-making are carried out based on the prediction results.

[0052] Compared with the existing solutions, the present invention achieves the following beneficial effects:

[0053] The present invention ensures the integrity and quality of data by comprehensively collecting various types of data during the software development process and performing standardization processing such as cleaning and normalization, thus providing a reliable basis for subsequent analysis.

[0054] The present invention uses the Pearson correlation coefficient method to screen out features that are highly correlated with software defects, avoiding the interference of redundant features and improving the prediction accuracy and efficiency of the model;

[0055] The present invention builds a software defect prediction model based on development and processing data, and through parameter adjustment and training optimization, enables the model to accurately capture the complex relationship between defects and features, thereby improving the accuracy of prediction;

[0056] The present invention obtains new data of target software development in real time, processes it quickly and predicts the number of defects in advance, and displays the results in a visual manner, which helps developers intuitively understand the distribution and trends of defects, assists in optimizing the design and development process, reduces the cost of later repairs, and significantly improves software quality and development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described below with reference to the accompanying drawings.

[0058] Figure 1 This is a module structure diagram of a software development big data analysis system proposed by the present invention.

[0059] Figure 2 This is a flowchart of the software development big data analysis method proposed by the present invention. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0061] Example 1, as Figure 1 As shown, the present invention is a software development big data analysis system, including a data acquisition module, a data processing module, a modeling module, a prediction module, and a display and analysis module;

[0062] The data collection module is used to comprehensively collect various types of data during the software development process, including code data, developer operation data, project requirement data, and test data;

[0063] Code data includes but is not limited to the number of code lines, code complexity, code structure, execution path, and call frequency. The specific collection process includes:

[0064] Use static code analysis tools such as Lint and Checkstyle to regularly analyze the code in the code base, obtain the number of code lines, code complexity, and code structure, and store them in the database. Use performance monitoring tools such as JProfiler and VisualVM to collect dynamic code data, such as execution path and call frequency, and transmit it to the data collection module in real time.

[0065] The process of collecting developer operation data includes: integrating a data collection plug-in into the development environment to record the developer's development time, modification frequency, and operation history; for example, recording the time and operation content of each time the developer opens a file, edits code, and commits code; and obtaining the developer's code submission record through the version control system Git, including submission time, submission content, and modified file list;

[0066] Furthermore, project requirement data is obtained through project requirement management tools such as Jira and Trello to obtain the functional requirements, performance requirements, business rules, and changes in user requirements of the input project, and is regularly updated and maintained;

[0067] Automatically collect test data through test scripts and test frameworks, including test case design, execution results, and number of defects.

[0068] The data processing module performs standardized processing on all types of collected data, including cleaning and preprocessing, to obtain development processing data. It also performs feature extraction on the development processing data to screen out defect-related features related to software defects.

[0069] Standardization of various types of collected data includes:

[0070] Clean the collected data, remove duplicate data, fill missing values, and smooth noisy data to ensure data quality;

[0071] During preprocessing, data from different sources and formats are processed uniformly and converted into the corresponding preset standard format, and the maximum-minimum normalization method is used to normalize the numerical data.

[0072] Furthermore, the method for obtaining defect-related features is as follows:

[0073] Traverse the development and processing data to obtain various factors, and use the correlation coefficient method to calculate the Pearson correlation coefficient between each factor and the number of defects;

[0074] Based on the calculated Pearson correlation coefficient, factors corresponding to the absolute value of the Pearson correlation coefficient being greater than or equal to the correlation threshold are screened out as defect correlation features; wherein, the correlation threshold is obtained by experts in this field through practice through multiple test data.

[0075] It should be noted that feature extraction is performed on the development processing data to screen out defect-related features related to software defects, such as code complexity and frequency of requirement changes, to provide effective data input for subsequent modeling.

[0076] The modeling module uses the multivariate linear regression algorithm to build a software defect prediction model based on development and processing data, and trains and optimizes the model by adjusting the model parameters;

[0077] Specifically, based on the development and processing data, the extracted defect correlation features and defect counts are obtained and integrated into training samples; there are m training samples in total, each training sample is (X, Y), X represents the matrix composed of defect correlation features, and Y is the number of defects; further, X=(X1,X2,...,X n ), X1, X2, ..., X n is the data value corresponding to each defect-related feature, and n is the total number of defect-related features;

[0078] Based on the multiple linear regression algorithm, the multiple linear regression model is obtained, and the expression of the multiple linear regression model is:

[0079] Y=β0+β1X1+β2X2+...+β n X n +ε;

[0080] Where β0 is the intercept term, β1, β2, ..., β n is the regression coefficient corresponding to each defect correlation feature, reflecting the influence of each defect correlation feature on the number of defects; ε is the random error term, which obeys the normal distribution with a mean of 0;

[0081] The least squares method is used to solve the model parameters, including the intercept term and the regression coefficient corresponding to each defect correlation feature;

[0082] Specifically, a loss function is defined to measure the difference between the model's predicted value and the true value, and the model parameters are continuously adjusted to perform derivative calculations to minimize the loss function, and the model parameters at this time are obtained as the optimal parameters;

[0083] Among them, the expression of the loss function is as follows:

[0084]

[0085] Where J is the loss function, i = 1, 2, ..., m, i represents the index of the training sample; X i1 ,X i2 ,...,X in represents the defect-related features corresponding to the i-th training sample; Y i represents the number of defects corresponding to the i-th training sample;

[0086] The specific formula for the derivative calculation is:

[0087]

[0088] in, represents the optimal parameter; X T The transpose of the matrix formed by defect-related features;

[0089] Based on the optimal parameters, the corresponding multivariate linear regression model is obtained, which is the software defect prediction model. Its specific expression is:

[0090]

[0091] in, is the predicted value of the number of defects; is the corresponding model parameter in the optimal parameter; Represents the input features that need to be predicted, that is, the data values ​​corresponding to each defect-related feature;

[0092] Obtain the number of defects and the predicted value of the number of defects corresponding to each training sample, and evaluate the software defect prediction model by calculating the determination coefficient;

[0093] The calculation formula of the coefficient of determination is:

[0094]

[0095] Where R 2 is the coefficient of determination;

[0096] It should be noted that the coefficient of determination R 2 Used to measure the degree of fit of the model to the data; its value range is between [0,1], and the closer it is to 1, the better the model fit;

[0097] If the calculated coefficient of determination is greater than or equal to the preset fitting threshold, it means that the software defect prediction model meets the prediction standard; otherwise, the model structure is further optimized by adjusting the model parameters or using the stepwise regression method to achieve the optimization of the software defect prediction model; wherein, the preset fitting threshold can be set to 0.75; further, the stepwise regression method dynamically retains the factors that contribute significantly to the model through the F test, which is an existing technology and will not be described in detail here;

[0098] The prediction module obtains various new data corresponding to the target software development process, processes and extracts features from the new data, and inputs the obtained data values ​​corresponding to the defect-related features into the software defect prediction model to obtain a prediction result for the number of software defects. The target software development refers to the specific software development project for which the number of defects needs to be predicted.

[0099] The display and analysis module displays the obtained prediction results in a visual manner and conducts further analysis and decision-making based on the prediction results. The visualization methods include but are not limited to bar charts, line charts, pie charts, and heat maps.

[0100] Specifically, the prediction results will be presented to the development team, and an in-depth analysis will be conducted on the prediction results. Combined with the specific data of the project, resource allocation suggestions and risk prevention measures will be provided to the development team to help the development team reasonably arrange the development progress, allocate human resources, and carry out quality control and risk management in advance.

[0101] Example 2, as Figure 2 As shown, the present invention is a software development big data analysis method, comprising the following steps:

[0102] Comprehensively collect all types of data during the software development process, including code data, developer operation data, project requirement data, and test data;

[0103] Standardize all types of collected data, including cleaning and preprocessing, to obtain development and processing data. Feature extraction is then performed on the development and processing data to screen out defect-related features related to software defects.

[0104] Based on development and processing data, a software defect prediction model is constructed using a multivariate linear regression algorithm, and the model is trained and optimized by adjusting the model parameters;

[0105] Obtain various new data corresponding to the target software development process, process and extract features from the new data, input the acquired data values ​​corresponding to the defect-related features into the software defect prediction model, and obtain the prediction results of the number of software defects;

[0106] The obtained prediction results are displayed in a visual way, and further analysis and decision-making are carried out based on the prediction results.

[0107] In the several embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative. For example, the division of modules is only a logical function division, and other division methods may be used in actual implementation.

[0108] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0109] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0110] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A software development big data analysis system, characterized in that: include: The data collection module is used to comprehensively collect various types of data during the software development process, including code data, developer operation data, project requirement data, and test data; The data processing module performs standardized processing on all types of collected data, including cleaning and preprocessing, to obtain development processing data. It also performs feature extraction on the development processing data to screen out defect-related features related to software defects. The modeling module uses the multivariate linear regression algorithm to build a software defect prediction model based on development and processing data, and trains and optimizes the model by adjusting the model parameters; The prediction module obtains various new data corresponding to the target software development process, processes and extracts features from the new data, and inputs the data values ​​corresponding to the defect-related features into the software defect prediction model to obtain the prediction results of the number of software defects.

2. A software development big data analysis system according to claim 1, characterized in that: The specific process of collecting various types of data includes: Use code static analysis tools to regularly analyze the code in the code base, obtain the number of code lines, code complexity, and code structure, and store them in the database; use performance monitoring tools to collect dynamic data of the code and transmit it to the data acquisition module in real time; Integrate data collection plug-ins in the development environment to record developers' development time, modification frequency, and operation records; and obtain developers' code submission records through the version control system Git, including submission time, submission content, and modified file lists; Furthermore, project requirement data is obtained through project requirement management tools to obtain the functional requirements, performance requirements, business rules, and changes in user requirements of the project, and is regularly updated and maintained; Automatically collect test data through test scripts and test frameworks, including test case design, execution results, and number of defects.

3. A software development big data analysis system according to claim 2, characterized in that: Standardization of various types of collected data includes: Clean the collected data, remove duplicate data, fill missing values, and smooth noisy data to ensure data quality; When implementing preprocessing, data from different sources and in different formats are processed uniformly and converted into the corresponding preset standard format, and the maximum-minimum normalization method is used to normalize the numerical data separately.

4. A software development big data analysis system according to claim 3, characterized in that: The method for obtaining defect-related features is as follows: Traverse the development and processing data to obtain various factors, and use the correlation coefficient method to calculate the Pearson correlation coefficient between each factor and the number of defects; According to the calculated Pearson correlation coefficient, factors corresponding to which the absolute value of the Pearson correlation coefficient is greater than or equal to the correlation threshold are screened out as defect correlation features.

5. A software development big data analysis system according to claim 1, characterized in that: The specific implementation steps of the modeling module include: Based on the development and processing data, the extracted defect correlation features and defect counts are obtained and integrated into training samples. There are m training samples in total, each of which is (X, Y), where X represents the matrix of defect correlation features and Y represents the number of defects. Furthermore, X = (X1, X2, ..., X n ), X1, X2, ..., X n is the data value corresponding to each defect-related feature, and n is the total number of defect-related features; Based on the multiple linear regression algorithm, the multiple linear regression model is obtained, and the expression of the multiple linear regression model is: Y=β0+β1X1+β2X2+...+β n X n +e; Where β0 is the intercept term, β1, β2, ..., β n is the regression coefficient corresponding to each defect correlation feature; ε is the random error term.

6. A software development big data analysis system according to claim 5, characterized in that: The specific implementation steps of the modeling module also include: The least squares method is used to solve the model parameters, including the intercept term and the regression coefficient corresponding to each defect correlation feature; By defining a loss function to measure the difference between the model's predicted value and the true value, and continuously adjusting the model parameters for derivative calculation, the loss function is minimized and the model parameters at this time are obtained as the optimal parameters; Among them, the expression of the loss function is as follows: Where J is the loss function, i = 1, 2, ..., m, i represents the index of the training sample; X i1 ,X i2 ,...,X in represents the defect-related features corresponding to the i-th training sample; Y i represents the number of defects corresponding to the i-th training sample; The specific formula for the derivative calculation is: in, represents the optimal parameter; X T The transpose of the matrix formed by defect-related features; Based on the optimal parameters, the corresponding multivariate linear regression model is obtained, which is the software defect prediction model. Its specific expression is: in, is the predicted value of the number of defects; is the corresponding model parameter in the optimal parameter; Represents the input features that need to be predicted, that is, the data values ​​corresponding to each defect-related feature.

7. A software development big data analysis system according to claim 6, characterized in that: Evaluate and optimize software defect prediction models, including: Obtain the number of defects and the predicted value of the number of defects corresponding to each training sample, and evaluate the software defect prediction model by calculating the determination coefficient; The calculation formula of the coefficient of determination is: Where R 2 is the coefficient of determination; If the calculated determination coefficient is greater than or equal to the preset fitting threshold, it means that the software defect prediction model meets the prediction criteria; otherwise, the model structure is further optimized by adjusting the model parameters or using the stepwise regression method to achieve the optimization of the software defect prediction model.

8. The software development big data analysis system according to claim 1, characterized in that: Target software development refers to a specific software development project that currently requires a forecast of the number of defects.

9. A software development big data analysis system according to claim 8, characterized in that: Also includes: The display and analysis module displays the obtained prediction results in a visual way and conducts further analysis and decision-making based on the prediction results.

Images