Python language defect prediction method in combination with function-level dynamic operation characteristics

Abstract

The invention discloses a Python language defect prediction method combined with function-level dynamic operation characteristics, which comprises the following steps of: acquiring a Python language code item by utilizing an open source website, and constructing a Python language defect data set through item community information multi-layer screening; using a function-level dynamic instrumentation technology and a test case generator to count function-level dynamic operation characteristics; calculating a function-level feature weight by adopting an improved SXGB method based on a game theory; introducing the code version time difference information into a bidirectional long short-term memory network to construct an asymmetric sampling time sequence data adaptive model; performing defect prediction on the high-version Python project based on the constructed model; the method makes substantial progress in the aspects of dynamic feature interpretability, version evolution adaptability, complex asynchronous behavior capture and the like.

Description

Technical Field

[0001] This invention relates to the field of software defect prediction technology, and specifically to a Python language defect prediction method that combines function-level dynamic runtime characteristics. Background Technology

[0002] Software defects, also known as faults or bugs, are errors or functional flaws in computer software or programs that impair their normal operation. To prevent software defects from having a serious impact on software or programs, software defect prediction technology has become a research direction for addressing software defects. Software defect prediction technology constructs a software defect prediction dataset for a project by mining relevant features from historical version data. Machine learning or deep learning methods are then used to build a software defect prediction model based on this dataset, thereby predicting potential software defects that may appear in subsequent versions of the project. Software defect prediction not only helps developers identify potential problems in advance but also provides strong support for defect localization and repair, thereby improving development efficiency and reducing later maintenance costs.

[0003] Existing methods mainly rely on static metrics of code for feature extraction, lacking dynamic features of code, and most research languages ​​lack relevant research on Python in C and JAVA. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a Python language defect prediction method that combines function-level dynamic runtime features, which solves the problem that existing methods mainly rely on static code metrics for feature extraction, lack dynamic code features, and most research languages ​​lack relevant research on Python in C and JAVA.

[0005] Technical Solution: The present invention provides a Python language defect prediction method combining function-level dynamic runtime features, comprising the following steps:

[0006] (1) Obtain Python language code projects from open source websites and construct a Python language defect dataset by filtering project community information in multiple layers.

[0007] (2) Use function-level dynamic instrumentation technology and test case generator to statistically analyze function-level dynamic runtime characteristics;

[0008] (3) The SXGB method based on game theory is used to calculate the function-level feature weights;

[0009] (4) Introduce code version time difference information into a bidirectional long short-term memory network to construct an adaptive model for asymmetric sampling time series data;

[0010] (5) Based on the constructed model, perform defect prediction on high-version Python projects.

[0011] Further, step (1) is as follows: Based on the GitHub platform, select Python projects with ≥2000 stars and frequent version iterations; parse the defect tag identifiers in the project commit logs; and construct a structured defect dataset containing version time sequence information.

[0012] Furthermore, in step (2), the function-level dynamic runtime characteristics include: function call depth: obtaining the maximum call stack depth when the function is called using the inspect tool; exception propagation path distance: counting the number of propagation layers after an exception is thrown by the function without being caught using the inspect tool, with cross-module propagation being calculated twice; parameter type dynamic change: calculating the coefficient of variation of parameter types when the same function is called multiple times using the inspect tool; coroutine blocking time: counting the proportion of total blocking time of await statements in async functions using the cProfile tool; memory leak slope: calculating the linear regression slope of memory usage after multiple function calls using the cProfile tool; reflection call density: counting the proportion of dynamic execution calls to total function calls using the inspect tool; cross-thread contention intensity: calculating the ratio of the average waiting time to the execution time of locks in functions using the cProfile tool; generator state residue: counting the number of instances that are not recycled after the generator function closes() using the inspect tool; decorator chain overhead: calculating the proportion of additional time overhead caused by multi-level decorators using the cProfile tool; C extension call risk: counting the ratio of the number of C extension module calls to the number of Python code lines using the inspect tool.

[0013] Furthermore, in step (3), the specific details are as follows:

[0014] (31) Calculate the XGBoost Gain value using the following formula:

[0015]

[0016] (32) Calculate the SHAP value using the following formula:

[0017]

[0018] (33) Combine the Gain value and SHAP value, the formula is as follows: Final=α·Gain+(1-α)·SHap

[0019] (33) Compare the Gain value and SHAP value to identify and remove inconsistent features.

[0020] Furthermore, step (4) includes the following steps:

[0021] (41) Embed the time difference, concatenate the time difference with the original features and input the input into BiLSTM; the time difference formula is as follows:

[0022] E Δt =ReLU(W e ·ln(1+Δt)+b e )

[0023] Among them W e E is a learnable parameter. Δt The data is concatenated with the original features and then input into the BiLSTM layer, thereby increasing the perceived span of time intervals.

[0024] (42) Time gating mechanism, the formula is as follows:

[0025] α t =σ(W α ·Δt t +b α )

[0026] (43) Update the state of the memory cell, using the following formula: c t =f t ⊙c t-1 +α t ⊙i t ⊙g t

[0027] (44) The time-decay attention mechanism, the formula is as follows:

[0028]

[0029] Further, step (5) is as follows: obtain the prior parameter probabilities of static code features through XGBoost training; use the SXGB method to screen important static features; combine the prior results of static features with dynamic running features and input them into the TIE-BiLSTM model; train the model using multiple versions of data and predict defects in higher versions of the same project.

[0030] The present invention discloses a Python language defect prediction system that combines function-level dynamic execution features, comprising:

[0031] Dataset processing module: configured to filter Python projects from GitHub and build a defect dataset;

[0032] Dynamic feature extraction module: configured to extract function-level dynamic runtime features through dynamic instrumentation and test case generator;

[0033] Feature processing module: configured to use the SXGB method to calculate feature weights;

[0034] Model building module: configured to build a time-aware TIE-BiLSTM model;

[0035] Training prediction module: Configured to train the model and perform high-version defect prediction;

[0036] Evaluation module: Configured to evaluate prediction results using F-value and KUR indicators.

[0037] An electronic device according to the present invention includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the steps of any of the methods described herein.

[0038] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described herein.

[0039] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: Firstly, based on the dynamic characteristics of the Python language, it proposes a targeted extraction system for ten dynamic runtime features, including function call depth, exception propagation paths, and coroutine blocking time. Through dynamic instrumentation and adaptive test case generation, it accurately captures runtime defect causes that are difficult to cover using traditional static analysis. Secondly, it proposes an improved SXGB feature selection method based on game theory, which effectively solves the problems of noise interference and multicollinearity in high-dimensional dynamic features by combining SHAP values ​​and the Gain value of gradient boosting decision trees for dual weight optimization. Finally, it introduces the characteristics of version update time intervals in model prediction to solve the limitation of defect propagation pattern learning caused by uneven time spans in version iterations. Compared with existing technologies, this invention not only improves defect prediction accuracy but also achieves substantial progress in dynamic feature interpretability, version evolution adaptability, and the capture of complex asynchronous behaviors. Attached Figure Description

[0040] Figure 1 This is an overall flowchart of the present invention;

[0041] Figure 2 This is a schematic diagram of the overall model of the SXGB-TIE-BiLSTM of this invention;

[0042] Figure 3 This is a breakdown model diagram of the SXGB feature filtering module of the present invention;

[0043] Figure 4 This is a flowchart of the SXGB feature filtering module of the present invention;

[0044] Figure 5 This is a flowchart illustrating the selection process for Python source code projects on the open-source website of this invention.

[0045] Figure 6 This is a flowchart of the dynamic feature filtering and extraction process at the function level of this invention;

[0046] Figure 7 This is the original model diagram of the XGBoost algorithm of this invention;

[0047] Figure 8 This is a breakdown model diagram of the TIE-BiLSTM algorithm module of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0049] like Figure 1-8 As shown, this embodiment of the invention proposes a Python language defect prediction method that combines function-level dynamic runtime features, including the following steps:

[0050] (1) Collecting Python Software Defect Datasets: The data source is crucial in constructing a Python software defect prediction dataset. High-quality open-source software projects, after careful selection, not only provide abundant defect examples but also have clear defect labels, facilitating defect collection and statistical analysis, and ensuring data representativeness and diversity. The Python project selection process is attached. Figure 4 As shown: To ensure data representativeness, the proportion of Python code in the projects needs to be considered. The selected open-source projects should have at least 95% Python code to ensure that the extracted data primarily comes from Python code and accurately reflects the limitations of the Python language. For each project, the file structure should be analyzed to confirm whether the core code is largely implemented in Python, ensuring that the collected data is not influenced by other programming languages.

[0051] A project's community activity directly impacts the frequency and quality of bug fixes. Therefore, project activity includes the frequency of developer submissions, the quality of submitted content, and the community's response speed. The most direct criterion is the project's Stars (number of favorites); a project with over 2000 favorites can be considered active. Software version update history is crucial for bug analysis. Projects with complete version history and detailed commit records should be selected to ensure clear and traceable code change information between versions. Projects with explicit bug tags for each version's commit information are particularly important, as these facilitate regularization matching of bug identification in commit comments. These records roughly identify which code modifications were made to address existing bugs, thus providing a basis for bug labeling in the dataset.

[0052] (2) Function-level dynamic instrumentation technology and a self-built test case generator are used to statistically analyze function-level dynamic runtime characteristics. Specifically: First, the collected Python projects are sorted by version, and version sequence information is recorded. For each version, the core code AST is segmented at the function level. Second, a function-level dynamic instrumentation tool is used to automatically inject monitoring and decorating code into the segmented functions. Then, the hypothesis library, a test case generator, is used for automatic boundary generation and parameter combination. Finally, a feature monitor is used to perform general feature statistics, asynchronous feature statistics, and thread feature statistics. The features include the following ten types:

[0053] Table 1 Function-level dynamic operation characteristics

[0054]

[0055]

[0056] Under dynamic typing mechanisms, parameter type mutation (feature 3) and reflection call density (feature 6) can better reflect type safety defects and code maintainability risks; function call depth (feature 1) and exception propagation path (feature 2) focus on the unique risks of context pollution and error propagation in dynamic scopes. Memory leak statistical features (feature 5) and generator state residue (feature 8) detect defects in the garbage collection mechanism by quantifying resource management anomalies. For asynchronous and concurrent scenarios, coroutine blocking time (feature 4) and thread lock contention intensity (feature 7) reveal performance defects under event-driven architecture and global lock mechanisms, respectively. Decorator chaining overhead (feature 9) and C extension call risk (feature 10) capture design defects and system-level crash defects in the blind spots of traditional static analysis from the perspective of metaprogramming and hybrid programming.

[0057] Based on the differences in the execution mechanism, the above ten types of features can be divided into three main categories for extracting dynamic execution features of Python functions: general dynamic execution features, asynchronous dynamic execution features, and thread dynamic execution features.

[0058] The general dynamic features are mainly achieved through runtime context monitoring and dynamic code injection technology: the interpreter's built-in code inspection module captures the runtime status such as function call hierarchy and reflection operation frequency in real time, and combines performance analysis tools to statistically analyze memory leak trends and decorator stacking overhead; for cross-language extended call behavior, module loading tracking and symbol table dynamic resolution technology are used to identify potential risks in mixed programming.

[0059] Extracting asynchronous dynamic features requires embedding an asynchronous event loop monitoring mechanism. By using timestamp marking and coroutine status tracking, the blocking ratio of input and output operations can be quantified, and abnormal blocking modes can be verified by combining task scheduling logs.

[0060] Thread dynamic features require overcoming the limitations of global lock mechanisms. This paper employs thread-local state recording and lock operation timing analysis to calculate the correlation between synchronization wait time and code execution efficiency, revealing resource contention patterns in multi-threaded scenarios. The introduction of dynamic features makes the feature extraction method in this paper more complete. By statistically analyzing dynamic features in software defect prediction metrics and combining them with static code metrics, a more comprehensive set of Python language defect features can be constructed to capture potential defects that may be exposed during code execution. This analysis method, combining static and dynamic features, provides a more accurate basis for software defect prediction, thereby improving the accuracy and reliability of defect prediction. The overall dynamic feature extraction process is as follows: Figure 5 As shown.

[0061] (3) The SXGB method based on game theory is used to calculate the function-level feature weights, as follows: The method of calculating the Gain value of the weight parameters of the original features is shown in Table 2.

[0062] Table 2. Gain Value Calculation Method

[0063]

[0064] After calculating the Gain value, the XGboost machine learning algorithm performs a simple partitioning and sorting, but it has the following three limitations: The Gain value only reflects the contribution of a feature at the split node, ignoring the global impact. It is difficult to quantify the synergistic or offsetting effects between features.

[0065] To address the limitations of the XGBoost method's simple ranking of gain values ​​to remove redundant features, this invention introduces the SHAP model, which focuses on global interpretability, to improve the accuracy of key feature selection. The overall calculation formula is shown in Equation 4.2, where α is the weight coefficient, and SHap is the mean SHap value of the feature, representing the global importance of the feature.

[0066] Final = α·Gain + (1-α)·SHap

[0067] The SHap value is calculated as shown in Formula 4.3: where N represents the set of all features, M represents the total number of features, S represents the subset excluding feature i, and f(S) represents the model prediction based on subset S. The result is the weighted average of the contributions of feature i across all possible subsets.

[0068]

[0069] The specific process of introducing the SHAP method for feature selection is as follows: Figure 4 As shown.

[0070] (4) Introducing code version time difference information into a bidirectional long short-term memory network to construct an adaptive model for asymmetric sampling time series data.

[0071] Because the release intervals of project versions are inconsistent during the update process, the traditional BiLSTM model alone cannot capture features in scenarios with irregular time intervals. Therefore, this invention proposes a bidirectional long short-term memory network that includes time difference interval management based on the BiLSTM model. By integrating time difference embedding, time gating mechanism and attention mechanism, it can achieve adaptive modeling of non-uniformly sampled time series data.

[0072] Temporal difference embedding and feature fusion

[0073] To address the issue of missing time interval information, a time difference feature encoding layer is designed. For the input sequence timestamp set {t1, t2, t3, ... t...} n} Calculate the time interval Δt between adjacent time steps. i =t i -t i-1 The formula maps it to an embedding vector aligned with the original feature space through a linear transformation:

[0074] E Δt =ReLU(W e ·ln(1+Δt)+b e )

[0075] Among them W e E is a learnable parameter. Δt The data is concatenated with the original features and then input into the BiLSTM layer, thereby increasing the perceived span of time intervals.

[0076] Time gate status update

[0077] After adding time interval awareness, the BiLSTM computation module is updated as follows:

[0078] α t =σ(W α ·Δt t +b α (Time gating coefficient)

[0079] c t =f t ⊙c t-1 +α t ⊙i t ⊙g t (BiLSTM parameter update)

[0080] Time decay attention weight allocation

[0081] A time-decay attention module is designed after the BiLSTM output layer to address the decay effect of time intervals in long-distance dependencies. A time decay factor is defined. Where γ controls the decay rate. The attention scoring function is then changed accordingly:

[0082]

[0083] This mechanism enables the model to focus more on temporally adjacent feature states when aggregating contextual information, thereby reducing ineffective long-range associations under non-uniform intervals.

[0084] (5) Predict defects in high-version Python projects based on the construction model

[0085] This invention uses the F-value evaluation metric to measure the model's predictive performance. The KUR metric is used to measure the correlation between the predicted results and known repair paths in the knowledge base, verifying the usability of the predicted results for defect repair solutions. KUR is an evaluation metric that quantifies the actual repair value of defect prediction results, measuring the degree of matching between the predicted results and historical defect repair experience in the knowledge base. The formula is as follows:

[0086]

[0087] To demonstrate whether the model improves upon adding dynamic features: Table 3 compares data from five projects on the AEEEM public dataset.

[0088] Table 3 Comparison of Evaluation Parameters for the AEEEM Dataset

[0089]

[0090]

[0091] Table 4 compares the data of the four projects on the Python dataset.

[0092] Table 4 Comparison of Evaluation Parameters for Python Datasets

[0093]

[0094] Table 5 KUR Parameter Comparison Table

[0095]

[0096] The data in Table 5 shows that most of the KUR parameter indicators have shown significant improvement.

Claims

1. A Python language defect prediction method combining function-level dynamic execution features, characterized in that, Includes the following steps: (1) Obtain Python language code projects from open source websites and construct a Python language defect dataset by filtering project community information in multiple layers. (2) Use function-level dynamic instrumentation technology and test case generator to statistically analyze function-level dynamic runtime characteristics; (3) The SXGB method based on game theory is used to calculate the function-level feature weights; (4) Introduce code version time difference information into a bidirectional long short-term memory network to construct an adaptive model for asymmetric sampling time series data; (5) Based on the constructed model, perform defect prediction on high-version Python projects.

2. The Python language defect prediction method combining function-level dynamic execution features as described in claim 1, characterized in that, Step (1) is as follows: Select Python projects with ≥2000 stars and frequent version iterations on the GitHub platform; parse the defect tags in the project commit logs; and construct a structured defect dataset containing version time information.

3. The Python language defect prediction method combining function-level dynamic execution features as described in claim 1, characterized in that, In step (2), the function-level dynamic runtime characteristics include: function call depth: the maximum call stack depth when the function is called is obtained through the inspect tool; exception propagation path distance: the number of propagation layers that are not caught after the function throws an exception is counted through the inspect tool, and the calculation is doubled for cross-module propagation; parameter type dynamic change: the coefficient of variation of parameter type when the same function is called multiple times is calculated through the inspect tool; coroutine blocking time: the proportion of total blocking time of await statements in async functions is counted through the cProfile tool; memory leak slope: the linear regression slope of memory usage after multiple function calls is calculated through the cProfile tool; reflection call density: the proportion of dynamic execution calls to total function calls is counted through the inspect tool; cross-thread contention intensity: the ratio of the average waiting time of locks in functions to the execution time is calculated through the cProfile tool; generator state residue: the number of instances that are not recycled after the generator function closes() is counted through the inspect tool; decorator chain overhead: the proportion of additional time overhead caused by multi-level decorators is calculated through the cProfile tool; C extension call risk: the ratio of the number of C extension module calls to the number of Python code lines is counted through the inspect tool.

4. The Python language defect prediction method combining function-level dynamic execution features as described in claim 1, characterized in that, In step (3), the specific details are as follows: (31) Calculate the XGBoost Gain value using the following formula: (32) Calculate the SHAP value using the following formula: (33) Combine the Gain value and SHAP value, the formula is as follows: Final=α·Gain+(1-α)·SHap (33) Compare the Gain value and SHAP value to identify and remove inconsistent features.

5. The Python language defect prediction method combining function-level dynamic execution features as described in claim 1, characterized in that, Step (4) includes the following steps: (41) Embed the time difference, concatenate the time difference with the original features and input the input into BiLSTM; the time difference formula is as follows: HAVE BEEN Δt =ReLU(W e ·ln(1+Δt)+b e ) Among them W e E is a learnable parameter. Δt The data is concatenated with the original features and then input into the BiLSTM layer, thereby increasing the perceived span of time intervals. (42) Time gating mechanism, the formula is as follows: a t =σ(W α ·Δt t +b α ) (43) Update the state of the memory cell, using the following formula: c t =f t ⊙c t-1 +α t ⊙i t ⊙g t (44) The time-decay attention mechanism, the formula is as follows:

6. The Python language defect prediction method combining function-level dynamic execution features according to claim 1, characterized in that, Step (5) is as follows: Obtain prior parameter probabilities by training static code features through XGBoost; use the SXGB method to screen important static features; combine the prior results of static features with dynamic running features and input them into the TIE-BiLSTM model; train the model using multiple versions of data and predict defects in higher versions of the same project.

7. A Python language defect prediction system that combines function-level dynamic execution features, characterized in that, include: Dataset processing module: configured to filter Python projects from GitHub and build a defect dataset; Dynamic feature extraction module: configured to extract function-level dynamic runtime features through dynamic instrumentation and test case generator; Feature processing module: configured to use the SXGB method to calculate feature weights; Model building module: configured to build a time-aware TIE-BiLSTM model; Training prediction module: Configured to train the model and perform high-version defect prediction; Evaluation module: Configured to evaluate prediction results using F-value and KUR indicators.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images