Anomaly behavior detection method in open source community PR process

Abstract

The application provides an abnormal behavior detection method in an open source community PR process, comprising the following steps: obtaining original event logs of the PR process; performing data cleaning, activity type division and key activity extraction on the original event logs to obtain standard event logs; identifying control flow abnormalities in the standard event logs based on process mining technology; and identifying semantic abnormalities in the standard event logs based on semantic abnormality identification rules. The method provided by the application can help project managers to comprehensively understand abnormal behaviors in the PR process, provide support for managers to improve the internal PR collaboration process, and thus promote the healthy and stable development of open source projects.

Description

Technical Field

[0001] This invention belongs to the field of process mining and anomaly detection, specifically a method for detecting abnormal behavior in the open-source community PR process. Background Technology

[0002] In open-source communities, pull requests (PRs) are a typical software development process. They facilitate collaboration and communication among team members, improve codebase quality and maintainability, and enhance the efficiency and transparency of software development. Through PRs, team members can review and discuss code changes, ensuring they meet code quality standards and reviewing and testing changes before merging them into the main branch, thus minimizing errors and vulnerabilities. However, the ever-increasing number of PRs poses a challenge to the standardization of the PR collaboration process. Faced with a daily surge in PR merge requests, project members experience increased pressure to review them, potentially leading to non-standard practices such as reviewers failing to respond to contributors' review requests for extended periods or maintainers skipping the review process and directly merging PRs. These non-standard practices can reduce developers' enthusiasm for contributing to open-source projects and may also result in malicious code being merged into the repository.

[0003] Due to the openness of the open-source community, the PR process is highly complex, making it difficult to detect abnormal developer behavior. To detect abnormal behavior during the PR process, existing research mostly employs time-consuming and labor-intensive non-automated methods such as questionnaires and interviews, resulting in high implementation costs. Secondly, although some research utilizes deep learning technology to detect abnormal business execution processes, deep learning is a black-box model with poor interpretability for anomalies. Specifically, while deep learning models can provide anomaly detection results, their lack of transparency prevents them from pinpointing the exact location of the anomaly. Compared to deep learning, process mining techniques can comprehensively analyze the root causes of anomalies and provide better interpretability. However, existing research using process mining techniques to detect abnormal behavior in business processes has two shortcomings. First, process mining requires a set of normal cases to build a reference model. Existing research does not discuss how to obtain this set of normal cases when building the reference model, but instead assumes that a set of normal cases exists beforehand. However, obtaining a set of normal cases is difficult in most business scenarios, thus limiting the practical application of process mining techniques. On the other hand, most existing studies ignore the semantic information of business processes and cannot detect abnormal behaviors that violate the security constraints of business processes, which can lead to missed detections. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a method for detecting abnormal behavior in the open-source community PR process. It mines the PR process from both control flow and semantic perspectives, detects abnormal behavior of developers during the PR process, and helps open-source project teams improve their internal PR collaboration process.

[0005] The technical solution of this invention is to provide a method for detecting abnormal behavior during the open-source community PR process, the method comprising the following steps:

[0006] S1: Obtain the raw event logs of the PR process;

[0007] S2: Perform data cleaning, activity type classification, and key activity extraction on the raw event logs to obtain standard event logs;

[0008] S3: Identify control flow anomalies in standard event logs based on process mining techniques;

[0009] S4: Identify semantic anomalies in standard event logs based on semantic anomaly recognition rules.

[0010] Preferably, the event log in S1 or S2 is a set of cases, each case being the execution process of a business process. Each case contains a set of events, and each event contains multiple attributes, namely event ID, activity, timestamp, and executor.

[0011] Preferably, step S2 includes the following steps:

[0012] Remove all incomplete cases from the original event log; incomplete cases refer to business processes that have not been completed.

[0013] Based on the execution status or stage of the activity in the original event log, activities are categorized into different types. Review activities have three statuses: commented, changes_requested, and approved. These statuses are further divided into three different types of activities: ReviewComment, ReviewRejected, and ReviewApproved. ReviewComment represents a code review activity with the status of commented; ReviewRejected represents a code review activity with the status of changes_requested; and ReviewApproved represents a code review activity with the status of approved. Code commit activities are executed in two phases: before and after PR creation. Based on these two different execution phases, code commit activities are divided into two types of activities: SubmitCommit and Revise. SubmitCommit is a code commit activity executed before PR creation, and Revise is a code commit activity executed after PR creation.

[0014] Based on the set of critical activities, all critical activities are extracted from the raw event log. Critical activities are those that will affect the project's code quality, code review, and PR decisions, including code commit activities, PR creation activities, review request activities, review request removal activities, code review activities, PR decision activities, branch deletion activities, and reference activities.

[0015] Preferably, step S3 includes the following steps:

[0016] Using a log partitioning algorithm based on minimum overall coverage, standard event logs are divided into a set of quasi-normal cases and a set of candidate abnormal cases.

[0017] Mining reference models of business processes from a set of quasi-normal cases. The reference models are process models that conform to business process specifications. The process models contain logical structure information between business events, namely sequential structure, parallel structure, selection structure or loop structure.

[0018] A consistency test algorithm is applied to the candidate abnormal case set to calculate the goodness of fit between each candidate abnormal case and the reference model. Cases with a goodness of fit below a threshold are regarded as control flow anomalies. The goodness of fit measures the degree of fit between the actual business process executed in the case and the standardized business execution process in the reference model. The threshold is the minimum goodness of fit in the quasi-normal case set.

[0019] Preferably, the log partitioning algorithm based on minimum overall coverage includes the following steps:

[0020] Calculate all process variants in the event log and their frequencies, and denote the set of process variants in the event log as . A process variant represents a trajectory type. A trajectory is a sequence of activities in a case ordered by execution time. The frequency of a process variant is the number of cases with the same trajectory in the event log.

[0021] Each time from Select the process variant v that occurs most frequently and add it to the set. In the meantime, process variant v is transferred from... Remove from, then update This step, which checks the overall event log coverage (cover_rate), is performed repeatedly until cover_rate exceeds a threshold. The ratio of the number of cases contained in all process variants to the total number of cases in the event log;

[0022] Filter from event logs The set of cases corresponding to all process variants constitutes the quasi-normal case set. The case set corresponding to the variants of the remaining process constitutes the candidate anomaly case set.

[0023] Preferably, the consistency algorithm is used to calculate the goodness of fit between the case and the reference model. This invention uses an alignment algorithm to perform consistency checks. When the actual business execution process in the case does not conform to the ideal business execution process in the reference model, the case needs to be moved to match the ideal business execution process in the reference model. This process is called alignment. The goodness of fit calculation formula based on the alignment algorithm is as follows:

[0024]

[0025] In the above formula, t represents a case and M represents a process model. This indicates the best alignment between the case study and the reference model. This represents the worst alignment between the case and the reference model. The best alignment is the minimum movement cost required for the case to perfectly match the reference model, and the worst alignment is the maximum movement cost required for the case to perfectly match the reference model.

[0026] Preferably, step S4 includes the following steps:

[0027] By establishing a mapping relationship between security constraints in the PR process and specific activity transitions, a set of semantic anomaly identification rules is obtained. Each rule corresponds to an anomalous activity transition, which is used to identify a specific type of semantic anomaly. The activity transition is a local behavior pattern in the form of "Activity A → Activity B", which indicates that Activity B and Activity A are adjacent in execution order and Activity B is executed after Activity A.

[0028] Iterate through all activity transitions contained in the case, and check whether the case contains abnormal activity transitions. Cases containing abnormal activity transitions are considered semantically abnormal.

[0029] Preferably, the security constraints in the PR process are the normative guidelines in the open source community's PR collaboration process, and a PR process that violates the security constraints will pose a risk to the project;

[0030] Preferably, the abnormal activity transitions include four types: ReviewRequested→ClosePR indicates that the maintainer closed the PR without the reviewer responding to the review request; ReviewRequested→MergePR indicates that the maintainer merged the PR without the reviewer responding to the review request; ReviewApproved→ClosePR indicates that the maintainer closed the PR after the reviewer approved the merge for the last time; and ReviewRejected→MergePR indicates that the maintainer merged the PR after the reviewer rejected the merge for the last time.

[0031] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The method proposed in this invention utilizes process mining technology to mine the PR process, providing detailed anomaly diagnostic information and facilitating project managers to analyze the root causes of abnormal behavior in the PR process; the method proposed in this invention can build a reference model of the business process in the absence of normal case sets, overcoming the problem of the inability to obtain normal data in the actual implementation of process mining technology; the method proposed in this invention considers both control flow and semantic dimensions, enabling comprehensive identification of abnormal behavior in the PR process and reducing the false negative rate; the method proposed in this invention can help project managers fully understand abnormal behavior in the PR process, providing support for managers to improve the internal PR collaboration process, thereby promoting the healthy and stable development of open source projects. Attached Figure Description

[0032] Figure 1 This is a flowchart of an abnormal behavior detection method in the open-source community PR process according to Embodiment 1 of the present invention.

[0033] Figure 2 This is a schematic diagram of the log partitioning algorithm based on minimum overall coverage in Embodiment 1 of the present invention. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged.

[0037] The term "control flow anomaly" as used in this article refers to cases that occur infrequently and deviate significantly from the reference model, where the discrepancy lies between the case and the reference model in the business execution process.

[0038] As used in this paper, the term "semantic anomaly" refers to a case that occurs infrequently and contains an anomalous activity transition, which is a local behavioral pattern of the form "activity A → activity B", indicating that activity B and activity A are adjacent in execution order and activity B is executed after activity A.

[0039] Example 1:

[0040] This invention provides a method for detecting abnormal behavior during the open-source community pull request (PR) process. Figure 1 This is a flowchart of the method, which specifically includes the following steps:

[0041] Step 110: Obtain the raw event logs of the PR process;

[0042] We crawled all activity information within the PR lifecycle using the Timeline API (Application Programming Interface) provided by GitHub, such as code commit activities, code review activities, and PR decision activities. Table 1 shows the field information in the raw event logs crawled using the Timeline API.

[0043] Table 1 Field information from the original event log

[0044]

[0045] Step 120: Perform data cleaning, activity type classification, and key activity extraction on the raw event logs to obtain standard event logs. The specific steps are as follows:

[0046] Step 121: The data cleaning step involves removing all incomplete cases from the original event log. Incomplete cases refer to business processes that have not yet been completed. In a PR scenario, an incomplete case refers to a PR that is still open. For example, the case with id 1 in Table 2 is incomplete because it has not yet been closed or merged and is still open; while the case with id 2 is complete because it has already been merged.

[0047] Table 2 Example of event log during the PR process

[0048]

[0049] Step 122: Activity Type Classification. This step classifies activities into different types based on their status or execution stage in the original event log. Review activities have three statuses (corresponding to the state field in Table 1): commented, changes_requested, and approved. Based on these three statuses, review activities are divided into three different types: ReviewComment, ReviewRejected, and ReviewApproved. ReviewComment is a code review activity with the status of commented, ReviewRejected is a code review activity with the status of changes_requested, and ReviewApproved is a code review activity with the status of approved. Code submission activities are executed in two phases: before and after PR creation. Based on these two different execution phases, code submission activities are divided into two types: SubmitCommit and Revise. SubmitCommit is a code submission activity executed before PR creation, and Revise is a code submission activity executed after PR creation.

[0050] Step 123: The critical activity extraction step involves extracting all critical activities from the original event log based on the set of critical activities. The critical activities in the PR process are those that will affect the project's code quality, code review, and PR decisions. Table 3 shows all the critical activities in the PR process.

[0051] Table 3 Key Activities in the PR Process

[0052]

[0053] Step 130: Identify control flow anomalies in standard event logs based on process mining techniques. The specific steps are as follows:

[0054] Step 131: Using a log partitioning algorithm based on minimum overall coverage, divide the standard event log into a quasi-normal case set and a candidate anomaly case set. A schematic diagram of this step in this embodiment is shown below. Figure 2 As shown, the specific steps of the log partitioning algorithm based on minimum overall coverage are as follows:

[0055] Calculate all process variants in the event log and their frequencies, and denote the set of process variants in the event log as . A process variant represents a trajectory type. A trajectory is a sequence of activities in a case ordered by execution time. The frequency of a process variant is the number of cases with the same trajectory in the event log.

[0056] Each time from Select the process variant v that occurs most frequently and add it to the set. In the meantime, process variant v is transferred from... Remove from, then update This step, which checks the overall event log coverage (cover_rate), is performed repeatedly until cover_rate exceeds a threshold. The ratio of the number of cases contained in all process variants to the total number of cases in the event log;

[0057] Filter from event logs The set of cases corresponding to all process variants constitutes the quasi-normal case set. The case set corresponding to the variants of the remaining process constitutes the candidate anomaly case set.

[0058] Step 132: Mine reference models of business processes on the quasi-normal case set. The reference model is a process model that conforms to the business process specification. The process model contains logical structure information between business events, namely sequential structure, parallel structure, selection structure or loop structure.

[0059] Process discovery algorithms can be used to construct process models of business processes from event logs. Process discovery refers to building a process model using event logs without any prior information. This model can intuitively express or interpret the behavior recorded in the event logs, helping practitioners understand the actual business processes. In this embodiment, an inductive miner-infrequency (IMi) algorithm is used to perform process discovery. The IMi algorithm adopts a divide-and-conquer approach, splitting the event log into multiple sub-logs, applying the process discovery algorithm to each sub-log to obtain a substructure of the process model, and finally merging these multiple substructures into a complete process model.

[0060] Step 133: Apply the consistency test algorithm to the candidate anomaly case set to calculate the goodness of fit between each candidate anomaly case and the reference model. Cases with a goodness of fit below the threshold are regarded as control flow anomalies.

[0061] The fit measure quantifies the degree of fit between the actual business process executed in the case and the standardized business execution process in the reference model;

[0062] The consistency algorithm is used to calculate the goodness of fit between the case and the reference model. In this embodiment, the alignment algorithm is used to perform the consistency check. When the actual business execution process in the case does not conform to the ideal business execution process in the reference model, the case needs to be moved to match the ideal business execution process in the reference model. This process is called alignment. The goodness of fit calculation formula based on the alignment algorithm is as follows:

[0063]

[0064] In the above formula, t represents a case and M represents a process model. This indicates the best alignment between the case study and the reference model. This represents the worst alignment between the case and the reference model. The best alignment is the minimum movement cost required for the case to perfectly match the reference model, and the worst alignment is the maximum movement cost required for the case to perfectly match the reference model.

[0065] The output of the alignment algorithm contains two key pieces of information: one is the fit between the case and the process model, which can be used to distinguish between normal and abnormal cases; the other is the alignment diagnostic information between the case and the reference model, which can be used to determine the type of deviation in the case and locate the specific location of the deviation in the case, and can clearly show the cause of the anomaly.

[0066] The deviation refers to the inconsistency between the case and the reference model in the business execution process. For example, activity A must be executed in the reference model, but activity A is not executed in the case. The types of deviations that this embodiment focuses on are shown in Table 4, where each type of deviation corresponds to a control flow anomaly.

[0067] The threshold is used to distinguish between normal cases and abnormal cases. Setting the threshold too high will lead to more missed cases, while setting the threshold too low will lead to more false alarms. In this embodiment, the minimum fit of the quasi-normal case set is used as the threshold. Cases with a fit lower than the threshold are regarded as control flow anomalies.

[0068] Table 4. Control flow anomaly types during the PR process

[0069]

[0070] Step 140: Identify semantic anomalies in standard event logs based on semantic anomaly recognition rules. The specific steps are as follows:

[0071] Step 141: Establish a mapping relationship between the security constraints in the PR process and specific activity transitions to obtain a set of semantic anomaly identification rules. Each rule corresponds to an anomalous activity transition and is used to identify a specific type of semantic anomaly.

[0072] The safety constraints mentioned in the PR process are the normative guidelines for the PR collaboration process in the open source community. PR processes that violate safety constraints will pose risks to the project. The activity transition is a local behavior pattern in the form of "Activity A → Activity B", which means that Activity B and Activity A are adjacent in execution order and Activity B is executed after Activity A.

[0073] Table 5 summarizes four safety constraints in the PR process based on best practice papers. The following transitions are listed: ReviewRequested→ClosePR indicates that the maintainer closed the PR without the reviewer responding to the review request; ReviewRequested→MergePR indicates that the maintainer merged the PR without the reviewer responding to the review request; ReviewApproved→ClosePR indicates that the maintainer closed the PR after the reviewer approved the merge for the last time; and ReviewRejected→MergePR indicates that the maintainer merged the PR after the reviewer rejected the merge for the last time.

[0074] Table 5. Types of semantic anomalies during the PR process

[0075]

[0076] Step 142: Traverse all activity transitions contained in the case and check whether the case contains abnormal activity transitions. Cases containing abnormal activity transitions are considered semantically abnormal.

[0077] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for detecting abnormal behavior during the open-source community PR process, characterized in that, The method includes: S1: Obtain the raw event logs of the PR process; S2: Perform data cleaning, activity type classification, and key activity extraction on the raw event logs to obtain standard event logs; S3: Identify control flow anomalies in standard event logs based on process mining techniques; S4: Identify semantic anomalies in standard event logs based on semantic anomaly recognition rules; S3 includes the following steps: Using a log partitioning algorithm based on minimum overall coverage, standard event logs are divided into a set of quasi-normal cases and a set of candidate abnormal cases. Mining reference models of business processes from a set of quasi-normal cases. The reference models are process models that conform to business process specifications. The process models contain logical structure information between business events, namely sequential structure, parallel structure, selection structure or loop structure. A consistency test algorithm is applied to the candidate abnormal case set to calculate the goodness of fit between each candidate abnormal case and the reference model. Cases with a goodness of fit below a threshold are regarded as control flow anomalies. The goodness of fit measures the degree of fit between the actual business process executed in the case and the standard business execution process in the reference model. The threshold is the minimum goodness of fit in the quasi-normal case set. The log partitioning algorithm based on minimum overall coverage includes the following steps: Calculate all process variants in the event log and their frequencies, and denote the set of process variants in the event log as . A process variant represents a trajectory type. A trajectory is a sequence of activities in a case ordered by execution time. The frequency of a process variant is the number of cases with the same trajectory in the event log. Each time from Select the process variant v that occurs most frequently and add it to the set. In the meantime, process variant v is transferred from... Remove from, then update This step, which checks the overall event log coverage (cover_rate), is performed repeatedly until cover_rate exceeds a threshold. The ratio of the number of cases contained in all process variants to the total number of cases in the event log; Filter from event logs The set of cases corresponding to all process variants constitutes the quasi-normal case set. The case set corresponding to the variants of the remaining process constitutes the candidate anomaly case set; S4 includes the following steps: By establishing a mapping relationship between security constraints in the PR process and specific activity transitions, a set of semantic anomaly identification rules is obtained. Each rule corresponds to an anomalous activity transition, used to identify a specific type of semantic anomaly. The activity transition is a local behavioral pattern of the form "Activity A → Activity B", which indicates that Activity B and Activity A are adjacent in execution order and Activity B is executed after Activity A. Traverse all activity transitions contained in the case and check whether the case contains abnormal activity transitions. Cases containing abnormal activity transitions are considered semantically abnormal.

2. The method for detecting abnormal behavior during the open-source community PR process as described in claim 1, characterized in that, The event log in S1 or S2 is a set of cases, each case being the execution of a business process. Each case contains a set of events, and each event contains multiple attributes, namely event ID, activity, timestamp, and executor.

3. The method for detecting abnormal behavior during the open-source community PR process as described in claim 1, characterized in that, S2 includes the following steps: Remove all incomplete cases from the original event log; incomplete cases refer to business processes that have not been completed. Based on the execution status or stage of the activity in the original event log, activities are categorized into different types. Review activities have three statuses: commented, changes_requested, and approved. These statuses are further divided into three different types of activities: ReviewComment, ReviewRejected, and ReviewApproved. ReviewComment represents a code review activity with the status of commented; ReviewRejected represents a code review activity with the status of changes_requested; and ReviewApproved represents a code review activity with the status of approved. Code commit activities are executed in two phases: before and after PR creation. Based on these two different execution phases, code commit activities are divided into two types of activities: SubmitCommit and Revise. SubmitCommit is a code commit activity executed before PR creation, and Revise is a code commit activity executed after PR creation. Based on the set of critical activities, all critical activities are extracted from the raw event log. Critical activities are those that will affect the project's code quality, code review, and PR decisions, including code commit activities, PR creation activities, review request activities, review request removal activities, code review activities, PR decision activities, branch deletion activities, and reference activities.

4. The method for detecting abnormal behavior during the open-source community PR process as described in claim 3, characterized in that, The consistency check algorithm is used to calculate the goodness of fit between the case and the reference model. An alignment algorithm is used to perform the consistency check. When the actual business execution process in the case does not conform to the ideal business execution process in the reference model, the case needs to be moved to match the ideal business execution process in the reference model. This process is called alignment. The goodness of fit calculation formula based on the alignment algorithm is as follows: In the above formula, t represents a case and M represents a process model. This indicates the best alignment between the case study and the reference model. This represents the worst alignment between the case and the reference model. The best alignment is the minimum movement cost required for the case to perfectly match the reference model, and the worst alignment is the maximum movement cost required for the case to perfectly match the reference model.

5. The method for detecting abnormal behavior during the open-source community PR process as described in claim 4, characterized in that, The security constraints mentioned in the PR process are the guidelines and norms for PR collaboration in the open source community. PR processes that violate security constraints will pose risks to the project.

Images