System and method for evaluating code contributions of software developers

Abstract

A tool for evaluating the contributions of different software developers to a software project is provided, which takes into account both the structural value and the non-structural value of each developer's contribution. The source code of the project is parsed at different times during the project to generate an abstract syntax tree (AST) representing the code. The tool can perform both structural value analysis and non-structural value analysis on the code contributions of each developer. The structural value analysis of each developer can include performing a call-commit analysis on a call-commit graph generated from the AST to analyze the function calls and commits attributable to each developer. The non-structural value analysis of each software developer can include identifying and classifying such commits based on the impact of such commits on the project. The tool can then calculate a development value score for each developer based on the structural and non-structural metrics for each developer.

Description

Technical Field

[0001] The present disclosure relates to microcontrollers and, more particularly, to systems and methods for evaluating respective code contributions of different software developers to a software project. Background Art

[0002] In a typical software development environment, developers contribute code to a software project repository. Each developer's code contribution is typically characterized by a simple metric, such as the number of commits (NOC) or lines of code (LOC) attributable to each developer. For example, GitHub uses NOC to rank developers on a software project. As another example, Expertise Browser, a classic tool for identifying developer expertise, uses the number of changed LOCs as an indicator of each developer's contribution. Such a metric measures the amount of code contribution by each developer, rather than the value of the code contribution. For example, functions that are at the core of the application logic are typically more valuable than auxiliary scripts, but conventional code contribution metrics cannot capture this distinction.

[0003] In many cases, it is useful to identify and compare the value of the contributions of different developers in addition to conventional quantity-based metrics. While traditional value-based software engineering typically focuses on creating economic value as a way to prioritize resource allocation and scheduling, in some cases other value measures are more relevant. For example, (in addition to non-code contributions), a programming teacher may want to evaluate the code contributions of individual students to a group project, even though such code contributions are typically not associated with financial rewards. As another example, an engineering manager may need to perform quantitative measurements of the corresponding performance of individual team members. In addition, software engineering researchers typically observe the development activity itself, but not necessarily its financial rewards. As yet another example, for free and open source software (FOSS) projects, the contributions of different developers greatly influence the collaboration, coordination, and leadership aspects of the related projects.

[0004] Therefore, there is an object to provide tools for analyzing code contributions of software developers, which capture the intrinsic value of code contributions - such as the impact of contributed code on development activity, rather than just counting lines of code or commits, for example to more effectively evaluate and / or rank individual developers and improve understanding of the software development process. Summary of the Invention

[0005] In general, and in one aspect, a system for evaluating code contributions of each of a plurality of software developers to a software project is provided, the system considering both the structural value and the non-structural value of each developer's contribution. The system may include at least one memory device and at least one processor communicatively coupled to the memory device. The memory device may store a code repository including a code collection including code portions developed by each of the plurality of software developers, the code portions including a plurality of functions. The memory device may also store a list of defined code impact categories for classifying submissions according to their impact.

[0006] The processor may be configured to access a code collection from a code repository and parse the code collection at multiple different times during a software project to generate an abstract syntax tree (AST) representing the code collection or a portion thereof. The processor may perform both a structural value analysis and a non-structural value analysis on the code contributions of each software developer. The structural value analysis for each software developer may include (a) generating at least one call-commit graph from the AST, (b) performing a call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer, and (c) calculating a structural value score for each software developer based at least on the call-commit analysis. The non-structural value analysis for each software developer may include (a) analyzing the AST to identify commits attributable to each software developer, (b) performing commit classification based on a list of defined code impact categories to classify each identified commit, and (c) calculating a non-structural code impact score for the software developer based at least on the commit classification of the commits attributable to each software developer.

[0007] The processor may then calculate a development value score for each software developer based at least on (a) the structural value score of each software developer and (b) the non-structural code impact score of each software developer. The processor may display a representation of the development value score calculated for each software developer, for example, by generating a graph, chart, or other displayable visual material.

[0008] A fundamental aspect of the present invention is to provide a system and method for automatically quantifying the value of a developer's code contributions to a project based on a combination of: (a) the structural value of each developer's code contributions, such as the structural impact of each developer's code, and (b) the non-structural value of each developer's code contributions, such as the impact of each developer's code on the development of the software project. These two components are calculated and combined to calculate the overall development value for each developer. The inventors have found that the evaluation of software developers according to the present invention provides better and more useful results than those obtained by human ratings or by simple conventional counting methods such as simply counting lines of code or commits.

[0009] In another aspect, a method for evaluating the code contribution of each of a plurality of software developers is provided, the method considering both the structural value and the non-structural value of each developer's contribution. A code repository may store a code collection comprising a portion of code developed by each of the plurality of software developers, the code portion comprising a plurality of functions. A list of defined code impact categories for classifying submissions according to their impact may also be stored. The code collection may be accessed from the code repository and parsed at a plurality of different times during a software project to generate an abstract syntax tree (AST) representing the code collection or a portion thereof.

[0010] Both a structural value analysis and a non-structural value analysis of each software developer's code contribution can be performed based on the AST. The structural value analysis of each software developer can include generating at least one call-commit graph from the AST, performing call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer, and calculating a structural value score for each software developer based at least on the call-commit analysis. The non-structural value analysis of each software developer can include analyzing the AST to identify commits attributable to each software developer, performing commit classification based on a list of defined code impact categories to classify each identified commit, and calculating a non-structural code impact score for the software developer based at least on the commit classification of the commits attributable to each software developer. A development value score can be calculated for each software developer based on multiple development value score inputs, including (a) a structural value score for each software developer and (b) a non-structural code impact score for each software developer. A representation of the development value score calculated for each software developer can then be displayed or otherwise output. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be readily understood from the following detailed description taken in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements.

[0012] Figure 1 is a diagram of an example system for evaluating code contributions of software developers that considers both the structural value and the non-structural value of each developer's contribution according to an embodiment of the present invention;

[0013] Figure 2 An example embodiment of the present invention is shown. Figure 1 The illustrated system performs an example method for evaluating code contributions to a software project by each of a plurality of software developers.

[0014] Figure 3 Represents an example file with two code commits;

[0015] Figure 4 and Figure 5 Indicates that Figure 3 Abstract syntax trees (ASTs) for the two commits in the example file shown;

[0016] Figure 6-1 and Figure 6-2 Indicates from Figure 4 and Figure 5 Example call-commit graph generated by the AST shown in ;

[0017] Figure 7 An example displayable view according to one embodiment of the present invention is shown, including an example development history graph indicating the development progress of a software project over time;

[0018] Figure 8 shows an example displayable view of a developer skill set graphic indicating the skill set of each respective developer according to one embodiment of the present invention;

[0019] Figure 9 shows an example displayable view of a development value score graphic indicating a development value score and its components for a selected developer according to one embodiment of the present invention; and

[0020] Figures 10A to 10C An example displayable view of an example labor allocation graph providing a visual indication of the relationship between developers and technical categories (or tasks) associated with a software project is shown in accordance with one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The specific embodiments of the present invention are described in detail below. In the following description of the embodiments of the present invention, specific details are described to provide a thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be practiced without these specific details. In other cases, well-known features are not described in detail to avoid unnecessarily complicating this specification.

[0022] There is an object to provide tools for analyzing code contributions of software developers, which capture the intrinsic value of code contributions - such as the impact of contributed code on development activity, rather than just counting lines of code or commits, for example to more effectively evaluate and / or rank individual developers and improve understanding of the software development process. Accordingly, the present invention provides systems and methods for evaluating code contributions of software developers based on a combination of: (a) the structural value of each developer's code contributions, such as the structural impact of each developer's code, and (b) the non-structural value of each developer's code contributions, such as the impact of each developer's code on the development of a software project. The inventors have found that such an evaluation of software developers provides better and more useful results than those obtained using conventional evaluation techniques.

[0023] In one aspect, a system for evaluating code contributions of each of a plurality of software developers to a software project is provided, the system considering both the structural value and the non-structural value of each developer's contribution. For example, the system can calculate a development value score for each developer that combines (a) at least one structural value score representing the structural value of the corresponding developer's code contribution and (b) at least one non-structural value score representing the non-structural value of the corresponding developer's code contribution.

[0024] The system may include a memory device and a processor connected to the memory device. The memory device may store a code repository including a source code collection of a software project, the source code collection including code portions developed by each of different developers. The code portions attributed to each developer may include various code segments called functions. Each function attributed to each developer may have dependencies or other relationships with other functions attributed to the same developer or other developers.

[0025] In some embodiments, the memory device may also store algorithms, rules, and / or other data for evaluating each developer's code contribution, as disclosed herein. For example, in some embodiments, the memory device may store a list of defined code impact categories for categorizing and scoring each developer's submissions according to the impact attributed to such submissions, as described below.

[0026] As another example, the memory device may store rules and / or weights for evaluating the structural value of code attributable to each developer, such as by evaluating the relationship between functions and commits attributable to each developer, as discussed below.

[0027] As another example, the memory device may store a set of code quality rules for evaluating the quality of code attributable to each developer.

[0028] As another example, the memory device may store an algorithm and / or a set of weights for calculating a development value score for each developer, the set of weights including, for example, a weight for each structural value score representing the structural value of the corresponding developer's code contribution and a weight for each non-structural value score representing the non-structural value of the corresponding developer's code contribution.

[0029] In some embodiments, the system may store and utilize each of the various types of data listed above. In other embodiments, the system may store and utilize only some of the data listed above. For example, the system may store and utilize a list of defined code impact categories for categorizing commits attributable to each developer and a set of code quality rules for evaluating the quality of code attributable to each developer, but may not store or utilize weights for calculating a development value score for each developer.

[0030] The system may include a source code parser that is configured to access the software project source code from a code repository and convert the source code into a set of abstract syntax trees (ASTs) representing the code or parts thereof at multiple different times during the software project. Each AST includes a tree structure representing the logic of the corresponding code. The parser may include or utilize any suitable tools and techniques known in the art for generating ASTs from source code. Typically, different programming languages ​​such as C++, Javascript, etc. have different AST expressions. Therefore, in some embodiments, the parser is configured to generate a general, language-independent AST representing an abstract version of multiple language-specific ASTs.

[0031] The system may also include (a) a structural value analysis tool for analyzing and scoring the structural aspects of each developer's code contribution and (b) a non-structural value analysis tool for analyzing and scoring the non-structural aspects of each developer's code contribution. In some embodiments, both the structural value analysis tool and the non-structural value analysis tool may perform their respective analysis and scoring based on the AST generated by the parser.

[0032] In some embodiments, the structural value analysis tool may include one or both of (a) a workload calculator configured to calculate a development equivalent score representing the amount of work or labor attributed to each developer based on the code attributed to the developer, and (b) a structural value scoring unit configured to calculate a structural value score for each developer.

[0033] The workload calculator can be configured to measure the labor of coding reflected by the AST. In one embodiment, the workload calculator is configured to count the number of logical operations expressed or performed by the code attributable to each developer, and calculate a development equivalent score for the corresponding developer based on the number of logical operations. The number of logical operations expressed by the code represents the complexity of the code developed by the corresponding developer, and therefore represents the amount of labor of the developer. The development equivalent score is generally similar to counting lines of code (LOC), but provides a more accurate and useful measure of the amount of work performed by each developer. For example, LOC is often arbitrary (e.g., due to the number of returns or other format-type details in the code) and may also be exploited by developers. In contrast, the development equivalent score calculated by the development equivalent score can avoid such issues common to LOC and other conventional work estimation techniques.

[0034] The structural value scoring unit may be configured to calculate a structural value score for each developer. The structural value scoring unit may be configured to generate a call-commit graph for each developer from a related AST, perform call-commit analysis on the call-commit graph to analyze function calls and commits attributable to each developer, and calculate a structural value score for each software developer based at least on the call-commit analysis.

[0035] In one embodiment, the structural value scoring unit includes a graph builder that is configured to generate a call-commit graph from a selected AST. The call-commit graph may include two types of nodes: (1) function nodes representing functions (code segments) and (2) commit nodes representing code commits. Each commit in the code may introduce a new function or make changes to an existing function. The call-commit graph may also define two types of node relationships between various nodes: (1) function call relationships between function nodes (code segments), where one function calls another function (or multiple other functions) to implement certain functionality, and (2) commit change relationships between commit nodes and function nodes, representing commits that change existing functions; each such relationship may be defined by an edge from a corresponding commit node to a function node of a function that is changed by the corresponding commit.

[0036] The structure value scoring unit may also include a structure value calculator configured to calculate a structure value score for each developer based on the nodes and / or node relationships in the call-commit graph generated by the graph builder. In one embodiment, the structure value calculator includes a development ranking algorithm configured to calculate a structure value score, also referred to herein as a "development ranking" score, in a manner similar to known page ranking algorithms. The algorithm assumes that a visitor has accessed the call-commit graph and applies rules to visitor behavior. The algorithm calculates the probability of a visitor arriving at each node (including function nodes and commit nodes), which represents the importance of the node. Therefore, the algorithm assigns a structure value to each node, defined by or based on the determined probability of arriving at each node. The algorithm then sums the structure values ​​of all nodes attributed to each developer in the graph to calculate a structure value score for the corresponding developer. In another embodiment, the algorithm sums only the structure values ​​of the commit nodes attributed to each developer to calculate the structure value score for the corresponding developer.

[0037] As described above, in addition to the example structural value analysis tool discussed above, the system also includes a non-structural value analysis tool for analyzing and scoring non-structural aspects of each developer's code contribution. In some embodiments, the non-structural value analysis tool may include any one, two, or all three of (a) an impact coding analysis unit, (b) a code quality analyzer, and / or (c) a project quality analyzer, wherein the impact coding analysis unit is configured to classify and score the submissions attributed to each developer based on the type of the developer's submissions, the code quality analyzer is configured to calculate a code quality score representing the quality of the code attributed to each developer, and the project quality analyzer is configured to calculate a project quality score for each developer based on a set of software engineering metrics.

[0038] The impact coding analysis unit is configured to classify and score commits in the code. Each commit in the code (including commits that create new functions and commits that change existing functions) changes the code base, which is converted into an AST. Each code commit produces two AST versions, one AST version before the commit and one AST version after the commit. The impact coding analysis unit may include a commit difference generator that is configured to compare the AST versions to identify differences representing code commits. The commit difference generator may be configured to identify code commits attributable to each developer.

[0039] The impact code analysis unit may further include a commit classifier configured to classify each commit attributed to each developer based on a list of defined code impact categories. The code impact categories may define code categories that typically have different levels of impact on the software project. For example, the code impact categories may include the following categories: (a) feature commits, (b) enhancement commits, (c) bug fix commits, and (d) maintenance commits.

[0040] The impact coding analysis unit may further include an impact coding calculator configured to calculate an impact coding score for each developer based on the classification category of the submissions attributed to the developer. The impact coding calculator may apply a weighted score to each submission category and sum the weighted scores of all submissions attributed to each developer to calculate the impact coding score for each developer. The weighted score for each submission category may represent a typical level of impact of that category of submissions on the software project.

[0041] The code quality analyzer can be configured to calculate a code quality score, which represents the quality of the code attributed to each developer. The code quality analysis unit can calculate a code quality score for each corresponding developer based on a set of defined code quality rules applied to the code attributed to the developer. The code quality rules can include any rules that define the quality of the code. An example code quality rule checks whether each variable defined in the code is actually used in the code. Another example code quality rule checks whether each section of code that opens a file (e.g., to read certain data) has a corresponding code section that subsequently closes the file, for example, to release processing resources. In one embodiment, the code quality analyzer is configured to identify and count the number of times each developer violates the code quality rules, and scores each developer based on the number of violations.

[0042] The project quality analyzer can be configured to calculate the project quality score for each developer based on the software engineering metrics of a set of definitions. For example, the project quality analyzer can determine and quantify the extent to which the submissions attributed to each developer affect each software engineering metric, and calculate a score for each developer based on their impact on the entire set of software engineering metrics. In some embodiments, the project quality analyzer can analyze the source code itself (by applying the software engineering metrics to the source code), or can analyze the AST representing the source code, or can analyze both the source code and the AST to determine the impact of each developer's code contribution on each corresponding software engineering metric. The software engineering metrics defined in this set can include any quantifiable metrics for analyzing the code attributed to each developer.

[0043] Example software engineering metrics may include:

[0044] ■ Modularity: Indicates how easy it is to partition the code into cohesive, separate components. This metric is calculated as the modularity score of the best partition of the call-commit graph of the code.

[0045] ■Documentation Coverage: Indicates the percentage of functions that have formatted comments.

[0046] ■ Velocity: Indicates how quickly the development team has progressed over the last defined time period (e.g., last month). This metric is calculated as the slope of the development value curve over the last defined time period (e.g., last month).

[0047] ■Test coverage: Indicates the percentage of functions covered by test cases.

[0048] ■ Robustness: Indicates the percentage of developers whose development value is lower than the average value of the developers being analyzed.

[0049] ■ Dryness: Indicates the percentage of code that is not repeated.

[0050] The system may further include a development value calculation unit configured to calculate a development value score for each developer based on (a) a score output by the structural value analysis tool and (b) a score output by the non-structural value analysis tool for the corresponding developer. For example, in one embodiment, the development value calculation unit may calculate the development value score for each developer based on any combination of the following development value score inputs: (a) the developer's development equivalence score calculated by the workload calculator (structural value input), (b) the structural value score calculated by the structural value scoring unit (structural value input), (c) the developer's impact coding score generated by the impact coding analysis unit (non-structural value input), and (d) the developer's code quality score calculated by the code quality analysis unit.

[0051] In some embodiments, the system further includes a development value weighting unit configured to score and / or dynamically adjust weights of various development value score inputs used to calculate a development value score for each respective developer. The development value weighting unit may include a weight learning unit configured to implement a ranking learning algorithm to dynamically determine and adjust weights of different development value score inputs over time, for example, based on user input indicating a comparison or ranking of submissions in the code.

[0052] Since structural analysis and non-structural analysis capture two fundamental aspects of development value, the development value weighting unit combines the two to calculate the overall development value. Suppose a submission has a structural value d and a non-structural value t. In some embodiments, the goal is to find a function that combines the two d can represent each structural score or a combination of structural scores (e.g., development equivalence score and / or structural value score), and t can represent each non-structural score or a combination of non-structural scores (e.g., code impact score, code quality score, and / or project quality score).

[0053] If reliable ground truth—that is, a large set of submissions with known overall exploitation value—is available, the task can be posed as an optimization problem: determine from the dataset the weight vector w in:

[0054]

[0055] Make the true value of each submission Minimize the average error between .

[0056] Unfortunately, developers find it difficult to directly rate the value of code in a free form, e.g., assigning 0.17 to one commit and 0.06 to another, and therefore it is difficult to obtain reliable ground truth in this form. Therefore, in one embodiment, developers are asked to compare pairs of commits from the same developer and identify which of each pair is more valuable, and such manual input is used for weight learning. This technique helps to eliminate the influence of personal interests and interpersonal relationships. Based on this "pairwise ground truth", a learning to rank (L2R) algorithm can be used to determine The example algorithm uses d and t as input features for the sorting SVM. After training, the weight vector of the SVM is considered to be This approach allows the structural and non-structural value scores of each submission to be combined to determine an overall development value score for each developer.

[0057] After calculating the development value score for each developer, the system may display or otherwise output a representation of the development value score, for example, by generating a graph, chart, or other visual for display on any suitable display device.

[0058] Each component of the above system—including at least the source code parser, workload calculator, structural value scoring unit, call-commit graph builder, structural value calculator, impact coding analysis unit, commit difference generator, commit classifier, impact coding calculator, code quality analysis unit, code quality checker, development value calculation unit, and development value weighting unit—includes or has access to any suitable hardware, software, and / or firmware elements for performing the functions of the corresponding system component. In some embodiments, one, some, or all of the system components are implemented in software stored in a memory and executable by one or more processors to perform the functions of the corresponding system component.

[0059] Figure 1 An example system 100 for evaluating the code contributions of each of a plurality of software developers to a software project is shown, according to an example embodiment of the present invention. The system 100 considers both the structural and non-structural value of each developer's contribution. The example system 100 includes a memory 102, a developer analysis tool 104, and a user interface 106. The memory 102 can be implemented by one or more data storage devices, such as any one or a combination of a hard drive, RAM, ROM, EEPROM, flash memory, a removable storage device (e.g., a USB drive), etc. The memory 102 can store a code repository 110 that stores source code 112 for a software project. The source code 112 includes a collection of functions 114, i.e., code segments, developed by a plurality of different software developers. The memory 102 can also store a set of abstract syntax trees (ASTs) 120 representing the source code or portions thereof at various times during the software project. The memory 102 can also store a set of call-commit graphs 122 that include and define relationships between function nodes and code commit nodes, as discussed herein.

[0060] The memory 102 may also store a development ranking algorithm 124 for calculating a structure value score for each developer. The development ranking algorithm 124 may be configured to calculate the probability of a visitor arriving at each node in the call-commit graph 122 and calculate a score for each developer by adding the probabilities corresponding to the code developed by the corresponding developer.

[0061] The memory 102 may also store a list or set of code impact categories 130 for classifying code submissions and a set of impact category weights 132 for corresponding code impact categories 130 , where the weights 132 may indicate the importance or impact of each category 130 of the submission on the software project.

[0062] The memory 102 may also store a set of code quality rules 136 and software engineering metrics 138 for analyzing the quality of the code contributed by each developer. Finally, the memory 102 may also store a development value weight learning algorithm 140, which is configured to dynamically determine and / or adjust a set of weights 142 to weight various inputs to each developer's overall development value score, and the memory 102 may also store development value user input 144 used by the weight learning algorithm 140 to adjust the weights 142. The development value user input 144 may include, for example, manual input regarding the relative value of different commits or different types of commits in the analyzed code.

[0063] The developer analysis tool 104 includes any suitable tool for analyzing the source project code 112 to calculate a development value score for each developer, the development value score representing the value of the code contribution of each respective developer. The developer analysis tool 104 is configured to access, utilize, and store any relevant data and / or algorithms in the memory 102. In this example embodiment, the developer analysis tool 104 includes a source code parser 150, a workload calculator 152, a structural value scoring unit 160, an impact coding analysis unit 170, a code quality checker 180, a project quality checker 182, a development value weighting unit 190, and a development value calculation unit 192. The structural value scoring unit 160 may include a call-commit graph builder 162 and a structural value calculator 164. The impact coding analysis unit 170 may include a commit difference generator 172, a commit classifier 174, and an impact coding calculator 176. Refer to the discussion below Figure 2 The processing flow shown is described in detail Figure 1 The functional operation of each of the developer analysis tools 104 shown in FIG.

[0064] The user interface 106 may include one or more user input devices 194 and one or more display devices 196. The input devices 194 may include any device configured to allow a user to interface with the developer analysis tool 104, such as at least one keyboard, keypad, mouse, touchpad, touch screen, voice-based input device, etc. The display device 196 may include any device for displaying information generated by or related to the system 100, such as at least one monitor, touch screen or other display screen, etc.

[0065] Figure 2 An example method 200 for evaluating the code contribution of each of a plurality of software developers to a software project is shown, according to an example embodiment of the present invention, the method 200 considering both the structural value and the non-structural value of each developer's contribution. The example method 200 may be performed by Figure 1 The illustrated example system 100 performs the following. Therefore, the method 200 illustrates the operational relationship of the various components of the system 100.

[0066] First, a source code parser 150 can access the source code 112 of a software project from a code repository 110 and convert the source code into a set of ASTs 120 representing the code or portions thereof at various times during the software project. Each AST includes a tree structure representing the logic of the corresponding code. The parser 150 can generate a language-independent AST representing an abstract version of a plurality of language-specific AST formats.

[0067] The system can perform a variety of different analyses based on the generated AST 120. The various analyses can be performed in any order or in whole or in part simultaneously. As shown at 210, a structural workload analysis can be performed to analyze the amount of work or effort expended by the respective developers. In this example, the workload calculator 152 can count the number of logical operations expressed or performed by the code attributable to each developer and calculate a development equivalence score 212 for the developer based on the number of counted logical operations.

[0068] As shown in 220, a structural value analysis of the functions (code segments) and commits attributable to each developer may be performed to calculate a structural value score for each developer. The graph builder 162 may generate a call-commit graph 122 from the selected AST 120. As described above, each call-commit graph 122 may include two types of nodes: (1) function nodes representing functions (code segments) and (2) commit nodes representing code commits, and may define two types of node relationships: (1) function call relationships between function nodes, and (2) commit change relationships between commit nodes and function nodes.

[0069] The figures provide examples to illustrate some of the concepts discussed above.

[0070] first, Figure 3 Shown is an example file "patent.py" with two commits 310 and 320. In the first commit, patent.py has four functions: "fibonacci," "bubble_sort," "reverse_array," and "main." In the second commit, both "fibonacci" and "bubble_sort" have been renamed and modified; "reverse_array" has been removed entirely; and "main" has been modified so that it no longer calls "reverse_array."

[0071] Next, Figure 4 and Figure 5 Respectively Figure 3 The ASTs of two example submissions 310 and 320 in patent.py are shown. In the AST, a "term" specifies the type of node; "lhs" / "rhs" refers to the left / right side.

[0072] at last, Figure 6-1 and Figure 6-2 Indicates from Figure 4 and Figure 5. The call-commit graph 500 is composed of "commits", "nodes" which are functions, and "links" between nodes. Each commit can be represented by its "id", such as "ff6f447c7d8a0a97d0ea472b4ca6b3f5edc59d93". Each node can be represented by the function path, name and location, such as "patent.py::fibonacci((n):: <global>::3::11::186"; it also contains the "history" of all commits that modified the function.

[0073] Return to Figure 1 For reference, the structural value calculator 164 can calculate a structural value score 222 for each developer based on the nodes and / or node relationships in the call-commit graph 112. For example, the structural value calculator 164 can execute the development ranking algorithm 124 to calculate the structural value score 222 in a manner similar to known page ranking algorithms. Generally speaking, the algorithm 124 assumes that a visitor accesses the call-commit graph and applies rules to visitor behavior. The algorithm calculates the probability that the visitor will reach each node (including function nodes and commit nodes), which represents the importance of the node. Therefore, the algorithm assigns a structural value to each node, which is defined by or based on the determined probability of reaching the node. The algorithm then sums the structural values ​​of all nodes in the graph attributed to each developer to calculate the structural value score 222 for the corresponding developer.

[0074] A specific example of a development ranking algorithm 124 is described below. In most imperative programming languages, functions (procedures, methods) are the fundamental unit of program structure. The development value of a function is based not only on the labor expended in creating the function, but also on the labor saved when other code calls the function. The structural component of development value (structural value) is captured by an example development ranking algorithm, referred to herein as the "DevRank algorithm," which is an extension of the original PageRank algorithm known in the art.

[0075] PageRank is the basis of Google Web Search and can be found in various fields. The algorithm operates on a directed graph of web pages. It assumes that web surfers have assumed access behavior and iteratively calculates the probability that the surfer visits each page. The meaning of the calculated probability depends on the surfer's behavior. In the original PageRank, the surfer performs two random actions: (1) After arriving at a page, the surfer randomly selects a link on that page with probability α and visits the linked page; (2) the surfer transfers to a random page with probability 1-α and continues. The damping factor α is a pre-selected fixed probability. Based on the behavior, the resulting probability reflects the probability of visiting the page based on the page's link structure. Intuitively, it reflects the popularity or importance of the page on the network.

[0076] To calculate the structural value of each function, the example DevRank algorithm constructs a static function-call graph of the code. Although program execution never randomly jumps to unrelated functions as in PageRank, the inventors have found PageRank to be a very convenient model for characterizing code development. Therefore, the inventors interpret random transitions as development activities of navigating code, rather than execution activities.

[0077] In the DevRank algorithm, the hypothetical "surfer" becomes a development sniffer whose task is to detect development labor. The algorithm assumes that the development labor spent on a function is shown by the total LOC of all changes that produced the function throughout its development history. The inventors believe that this technique quantifies development labor more accurately than counting the LOC at the latest snapshot. Based on this assumption, the behavior of the sniffer is constructed as follows: (1) After arriving at a function, the sniffer visits one of the called functions with probability α, whose probability is proportional to the development labor of these functions. Since the inventors believe that calling a function is a way to save development labor on the caller's side, this behavior reflects how much development labor is saved by encoding each function call. (2) The sniffer transfers to a random function with probability 1-α, whose probability is proportional to the development labor of the function. Such a transfer can be interpreted as a search for development labor by the sniffer. In general, it can be seen that the resulting probability of the sniffer appearing on each function reflects the development labor spent on that function and that the function saves other developers. Therefore, it reflects the development value of the function.

[0078] After calculating the DevRank score 222 for a function, the structure value calculator 164 can distribute the development value of the function to the commits, and further to the developers, thereby calculating a structure value score 222 for each developer. This can be done by distributing the value of the function to all commits that change the function - the value is proportional to the size of their changes (i.e., the number of LOCs changed) - and then distributing the value of the commits to their corresponding authors. In this way, developers receive credit for their contributions.

[0079] Workload analysis 210 and structural value analysis 220 represent structural analyses of code contributions by different developers. In addition to these structural analyses, method 200 may include the following non-structural analyses of developers' respective contributions to the project: (a) impact coding analysis 230, (b) code quality analysis 240, and (c) project quality analysis 250.

[0080] For impact coding analysis 230, the impact coding analysis unit 170 can, for example, classify and score the commits attributed to each developer based on the type of each developer's commits, since different types of commits typically have different levels of importance or impact on the project. The commit difference generator 172 of the impact coding analysis unit 170 can compare different AST versions representing code snapshots at different times to identify differences 232 in the AST versions, thereby identifying code commits attributable to each developer. The commit classifier 174 can then classify each commit attributed to each developer based on a list of defined code impact categories 130, such as feature commits, improvement commits, bug fix commits, or maintenance commits.

[0081] The impact coding calculator 176 can then calculate an impact coding score 236 for each developer based on the classification categories of the submissions attributed to that developer. The impact coding calculator can apply a code impact category weight 132 to each submission based on the submission category and sum the weighted scores of all submissions attributed to each developer to calculate the impact coding score 236 for each developer. The weight 132 for each submission category can represent the typical level of impact that type of submission has on the software project.

[0082] For code quality analysis 240, the code quality analyzer 180 can be configured to calculate a code quality score representing the quality of the code attributed to each developer by applying a set of code quality rules 136 to the code attributed to each developer. In one embodiment, the code quality analyzer 180 can identify and count the number of times the corresponding developer violates each code quality rule 136 and score the developer based on the total number of violations.

[0083] For code quality analysis 250, project quality analyzer 182 can be configured to calculate a project quality score 252 for each developer based on a defined set of software engineering metrics 138 applied to the code attributed to the respective developer. Project quality analyzer 182 can apply software engineering metrics 138 directly to project source code 112 rather than analyzing AST 120.

[0084] A development value score 270 may be calculated for each developer based on the various development value score inputs calculated as described above, including one, some, or all of the following: development equivalence score 212 (structural), structural value score 222 (structural), impact coding score 232 (non-structural), code quality score 242 (non-structural), and project quality score 252 (non-structural).

[0085] In some embodiments, for example, as shown in method 200, score input weighting can be performed at 260. For example, the development value weighting unit 190 can execute the weight learning algorithm 140 based on at least the user input 144 to determine and dynamically adjust a set of weights 142 for various development value score inputs (e.g., the corresponding weights of scores 212, 222, 232, 242, and 252). The user input 144 can include manual input indicating a comparison or ranking of submissions or submission types in the code over time.

[0086] The development value calculation unit 192 can then calculate a development value score 270 for each developer based on the development value score inputs (e.g., scores 212, 222, 232, 242, and 252) and the score input weights 142 determined at 260. Thus, the development value score 270 for each developer considers both structural and non-structural aspects of the contribution provided by the developer and, therefore, provides various advantages over conventional developer analysis techniques.

[0087] After calculating the development value score 270 for each developer, the system 100 may display or otherwise output a representation of the development value score, for example, by generating a graph, chart, or other visual for display on any suitable display device 196 .

[0088] Figures 7 to 10C Example graphs and views are shown that may be generated by system 100 and displayed on any suitable display device 196 . Figure 7 13. Example views are shown that include (a) an example development history chart 700 indicating the development progress of a software project over time, and (b) an example spider chart (or radar chart) 702 indicating the measurement of six example software engineering metrics 138 applied to the software project code. The project quality analyzer 182 can generate a spider chart 702 for each different developer that indicates, for example, the impact of that developer's commits on the overall project code. In one embodiment, the project quality analyzer 182 allows a user to select (e.g., click) each software engineering metric 138 in the spider chart 702 for a particular developer and automatically displays suggestions for improving the developer's code related to the selected metric 138. For example, a user can click on the "Dryness" metric in the spider chart 702, and the project quality analyzer 182 can automatically display examples of duplicate or redundant code segments that affect the Dryness metric.

[0089] Figure 8 An example view of a developer skill set graph 800 generated by the system 100 is shown. The system can categorize project code into different technical categories based on a set of defined tags added by developers when writing code, and determine each developer's skill level in each technical category based on the amount of code tagged with the corresponding technical category tag. As shown, the graph 800 can display the various skill levels of the selected developers in any suitable manner.

[0090] Figure 9 An example view of a development value score graph 900 generated by the system 100 is shown. The development value score graph 900 can display the development value score 270 and / or selected components thereof for a selected developer. In the example shown, the development value score graph 900 displays the developer with the highest development value score 270 and the following component scores: development equivalence score 212, impact coding score 236, and code quality score 242.

[0091] Figures 10A to 10C An example view of an example labor allocation graph generated by the system 100 is shown. Figure 10A An example labor allocation graph 1000A is shown that provides a visual indication of the relationship between various developers associated with a project and various technical categories (or tasks), the visual indication being indicated by lines of varying widths. As shown, each developer may be associated with one or more technical categories, and each technical category may be associated with one or more developers. In one embodiment, the relationship may be determined by the technical category label in each developer code, for example, as described above with respect to Figure 8 Thus, workforce allocation graph 1000A may represent a compilation of developer skill data indicated for each respective developer in developer skill set graph 800. The width of each line in workforce allocation graph 1000A may indicate the amount of code or commit value attributed to each developer for each respective skill category.

[0092] In one embodiment, a user may select an individual developer or an individual technology category to highlight data related to that developer or technology category. Figure 10B An example graph 1000B resulting from selecting a particular developer is shown, which illustrates the developer's skill set, and Figure 10C An example graph 1000C resulting from selecting a particular technology category is shown, which shows the developers contributing to that technology category and the relative amount of contribution from each developer (indicated by line width). For example, the labor allocation graphs 1000A to 1000C can be used to manage and adjust the allocation of labor to various aspects of a software project.

[0093] Although exemplary embodiments of the present invention are described and illustrated herein, it should be understood that they are illustrative only and that modifications may be made to these embodiments without departing from the spirit and scope of the present invention. The actions described herein may be computer-readable and executable instructions that may be implemented by one or more processors and stored in a computer-readable memory or article. Computer-readable and executable instructions may include, for example, application programs, program modules, routines and subroutines, execution threads, etc. In some cases, not all actions need to be implemented in the methods described herein.

[0094] Terms such as "component," "module," "unit," and "system" are intended to encompass hardware, software, firmware, or any combination thereof. For example, a system or component can be a process, a process executing on a processor, or a processor. Furthermore, functionality, a component, or a system can be located on a single device or distributed across several devices. The described subject matter can be implemented as an apparatus, method, or article of manufacture using standard programming or engineering techniques to produce software, firmware, hardware, or any combination thereof to control one or more computing devices.

[0095] Thus, the scope of the invention is intended to be limited only by the appended claims, as they may be amended, with each claim being expressly incorporated into this description of the detailed description as an embodiment of the invention.< / global>

Claims

1. A system for evaluating code contributions of each of a plurality of software developers to a software project, the system comprising: At least one memory device storing: a code repository comprising a code collection, the code collection comprising code portions developed by each of the plurality of software developers, the code portions comprising a plurality of functions; A list of defined code impact categories used to categorize submissions based on their impact; at least one processor communicatively coupled to the at least one memory device and configured to: accessing the code collection from the code repository; parsing the code set at a plurality of different times during the software project to generate an abstract syntax tree (AST) representing the code set or a portion of the code set; Perform a structural value analysis for each software developer through a structural value analysis, including: Generating at least one call-commit graph from the AST, wherein generating at least one call-commit graph from the AST comprises: generating a call-commit graph comprising (a) first nodes representing functions in the code collection and (b) second nodes representing commits; performing a call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer; and calculating a structural value score for each software developer based on the nodes and / or node relationships in the at least one call-commit graph based at least on the call-commit analysis; Perform a non-structural value analysis for each software developer through a structural value analysis, including: analyzing the AST to identify commits attributable to each software developer; performing submission classification based on the defined list of code impact categories to classify each identified submission; and calculating a non-structural code impact score for each software developer based at least on the submission classification of submissions attributable to the software developer; and calculating a development value score for each software developer based on a plurality of development value score inputs, the plurality of development value score inputs including (a) the structural value score for each software developer and (b) the non-structural code impact score for each software developer; and displaying a representation of said development value score calculated for each software developer, Wherein, performing the structural value analysis further comprises counting the number of logical operations expressed or performed by code attributable to each developer and calculating a development equivalence score for the corresponding developer based on the number of logical operations, and wherein the plurality of development value score inputs further comprises the development equivalence scores.

2. The system of claim 1, wherein: The at least one memory device stores a set of code quality rules; as well as Performing the non-structural value analysis for each software developer through the structural value analysis further includes: performing code quality analysis by applying the set of code quality rules to the code portions developed by each software developer; and calculating a non-structural code quality score for each software developer based at least on the code quality analysis; and The development value score of each software developer is calculated based on development value score inputs, wherein the development value score inputs include (a) the structural value score of each software developer, (b) the non-structural code impact score of each software developer, and (c) the non-structural code quality score of each software developer.

3. System according to one of the preceding claims, wherein Generating at least one call-commit graph from the AST includes generating a call-commit graph representing (a) function call relationships between selected functions in the code set and (b) relationships between selected commits and functions affected by the selected commits.

4. The system according to claim 1 or 2, wherein: Calculating the development value score for each software developer based on the plurality of development value score inputs includes: accessing or determining a set of weighted values ​​for the plurality of development value score inputs; and The development value score is calculated for each software developer based on (a) the plurality of development value score inputs and (b) the set of weighted values.

5. The system according to claim 4, wherein: The processor is configured to dynamically adjust the set of weighted values ​​over time based on user input regarding the comparison or ranking of the submissions.

6. The system according to claim 1 or 2, wherein: Performing the call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer includes determining a number of function calls or a probability of function calls to functions attributable to each software developer.

7. The system according to claim 1 or 2, wherein: Analyzing the AST to identify commits attributable to each software developer includes: Compare multiple ASTs representing snapshots of the code at different times; Identify commits based on the differences of multiple compared ASTs.

8. The system according to claim 1 or 2, wherein: The defined list of code impact categories includes at least one of the following categories: features, improvements, bug fixes, and maintenance.

9. A method for evaluating code contributions of each of a plurality of software developers to a software project, the method comprising: storing a code repository including a code collection, the code collection including code portions developed by each of the plurality of software developers, the code portions including a plurality of functions; storing a list of defined code impact categories used to classify submissions based on their impact; accessing the code collection from the code repository; parsing the code set at a plurality of different times during the software project to generate an abstract syntax tree (AST) representing the code set or a portion of the code set; Perform a structural value analysis for each software developer through a structural value analysis, including: Generating at least one call-commit graph from the AST, wherein generating at least one call-commit graph from the AST comprises: generating a call-commit graph comprising (a) first nodes representing functions in the code collection and (b) second nodes representing commits; performing a call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer; and calculating a structural value score for each software developer based on the nodes and / or node relationships in the at least one call-commit graph based at least on the call-commit analysis; Perform a non-structural value analysis for each software developer through a structural value analysis, including: analyzing the AST to identify commits attributable to each software developer; performing submission classification based on the defined list of code impact categories to classify each identified submission; and calculating a non-structural code impact score for each software developer based at least on the submission classification of submissions attributable to the software developer; and calculating a development value score for each software developer based on a plurality of development value score inputs, the plurality of development value score inputs including (a) the structural value score for each software developer and (b) the non-structural code impact score for each software developer; and displaying a representation of said development value score calculated for each software developer, Wherein, performing the structural value analysis further comprises counting the number of logical operations expressed or performed by code attributable to each developer and calculating a development equivalence score for the corresponding developer based on the number of logical operations, and wherein the plurality of development value score inputs further comprises the development equivalence scores.

10. The method according to claim 9, wherein: Performing the non-structural value analysis for each software developer through the structural value analysis further includes: performing code quality analysis by applying a set of stored code quality rules to the code portions developed by each software developer; and calculating a non-structural code quality score for each software developer based at least on the code quality analysis; and The development value score of each software developer is calculated based on development value score inputs, wherein the development value score inputs include (a) the structural value score of each software developer, (b) the non-structural code impact score of each software developer, and (c) the non-structural code quality score of each software developer.

11. The method according to claim 9, wherein: Generating at least one call-commit graph from the AST includes generating a call-commit graph representing (a) function call relationships between selected functions in the code set and (b) relationships between selected commits and functions affected by the selected commits.

12. The method according to claim 9, wherein: Calculating the development value score for each software developer based on the plurality of development value score inputs includes: accessing or determining a set of weighted values ​​for the plurality of development value score inputs; and The development value score is calculated for each software developer based on (a) the plurality of development value score inputs and (b) the set of weighted values.

13. The method of claim 12, further comprising dynamically adjusting the set of weighted values ​​over time based on user input regarding the comparison or ranking of submissions.

14. The method according to claim 9, wherein: Performing the call-commit analysis on the at least one call-commit graph to analyze function calls and commits attributable to each software developer includes determining a number of function calls or a probability of function calls to functions attributable to each software developer.

15. The method according to one of claims 9 to 10, wherein Analyzing the AST to identify commits attributable to each software developer includes: Compare multiple ASTs representing snapshots of the code at different times; Identify commits based on the differences of multiple compared ASTs.

16. The method according to one of claims 9 to 10, wherein The defined list of code impact categories includes at least one of the following categories: features, improvements, bug fixes, and maintenance.

Images