Large model-based open source community developer behavior influence assessment method and system

Abstract

The invention discloses an open source community developer behavior influence assessment method and system based on a large model, and belongs to the technical field of developer assessment, and the method comprises the steps: obtaining the record and statistical information of developers in a software development project; performing multi-dimensional analysis on the text data in the recorded and statistical information by using a large model, and integrating the scores of all dimensions to obtain a first score of the developer; wherein the dimensions comprise technology, communication, cooperation, continuous contribution and code quality; constructing a knowledge graph according to the record and statistical information, and obtaining a second score of the developer based on the knowledge graph; and obtaining a behavior influence evaluation result of the developer based on the first score and the second score. According to the method, the behavior influence of the developer can be accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the field of developer evaluation technology, and in particular to a method and system for evaluating the influence of developer behavior in an open source community based on a large model. Background Art

[0002] Traditional technical approaches primarily rely on simple statistical metrics to assess the impact of developer behavior, such as the number of commits, number of pull requests (PRs), and frequency of issue discussions. While these methods can reflect developer activity to a certain extent, they often overlook the value of non-coding behaviors, such as the quality of developers' contributions and collaborative abilities in actual projects. Therefore, some existing methods consider the characteristics of collaborative networks among developers during the evaluation process.

[0003] For example, Chinese patent application CN111191882A discloses a method and device for identifying influential developers in a heterogeneous information network. The method constructs a heterogeneous information network, comprehensively calculates the multi-faceted influence values of developers on GitHub (including interactions between individuals, projects, and companies), and integrates these values to generate a final score to identify the most influential developers.

[0004] However, the existing methods still have the following shortcomings:

[0005] 1) Existing evaluation systems have difficulty distinguishing between real developer contributions and automated behavior by bots (such as mechanically submitting pull requests and repeatedly generating issues), leading to the risk of data contamination in evaluation results.

[0006] 2) Existing methods fail to consider the breadth and depth of developers’ technology stacks, making it difficult to assess their ability to adapt to multi-technology scenarios.

[0007] 3) Existing evaluation models do not adequately assess time decay and cannot accurately reflect the higher value weight of recent active contributions in open source communities.

[0008] In summary, existing developer evaluation methods still have large errors in assessing the influence of developer behavior. Summary of the Invention

[0009] The present invention provides a method and system for evaluating the behavioral influence of developers in an open source community based on a large model, which can accurately evaluate the behavioral influence of developers.

[0010] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0011] A method for evaluating the influence of developer behavior in an open source community based on a large model, the method comprising:

[0012] Obtain records and statistics of developers in software development projects;

[0013] A large model is used to perform multi-dimensional analysis on the text data in the records and statistical information, and the scores of each dimension are comprehensively evaluated to obtain the developer's first score. The dimensions include: technology, communication, collaboration, continuous contribution, and code quality.

[0014] Constructing a knowledge graph based on the records and statistical information, and obtaining a second score of the developer based on the knowledge graph;

[0015] An evaluation result of the developer's behavior influence is obtained based on the first score and the second score.

[0016] Furthermore, the big model is used to perform technical dimension analysis on the text data in the records and statistical information, including:

[0017] Generate a technical scoring prompt; wherein the content of the technical scoring prompt includes: the technical dimension task description, text data in the record and statistical information, and the technical dimension scoring criteria, the technical dimension task description defines technical sub-dimensions, and the technical sub-dimensions include: technical complexity, code innovation, and technical breadth;

[0018] Use the technical score prompt to guide the large model to generate the technical complexity score, code innovation score, and technical breadth score for the text data;

[0019] The technical dimension score is obtained based on the technical complexity score, code innovation score, and technical breadth score of the text data.

[0020] Furthermore, the big model is used to analyze the communication dimension of the text data in the records and statistical information, including:

[0021] Generate a communication scoring prompt; wherein the content of the communication scoring prompt includes: a communication dimension task description, text data in the record and statistical information, and a communication dimension scoring standard, wherein the communication dimension task description defines communication sub-dimensions, and the communication sub-dimensions include: clarity of expression, emotional tendency, and logical rigor;

[0022] Use the communication score prompt to guide the large model to generate the expression clarity score, sentiment tendency score, and logical rigor score of the text data;

[0023] The communication dimension score is obtained based on the expression clarity score, emotional tendency score and logical rigor score of the text data.

[0024] Furthermore, the large model is used to conduct collaborative dimension analysis on the text data in the records and statistical information, including:

[0025] Generate a collaboration scoring prompt; wherein the content of the collaboration scoring prompt includes: a collaboration dimension task description, text data in the record and statistical information, and a collaboration dimension scoring standard; the collaboration dimension task description defines collaboration sub-dimensions, and the collaboration sub-dimensions include: response time, opinion acceptance and feedback quality, and proactive assistance;

[0026] Use the collaborative scoring prompt to guide the large model to generate the response time score, opinion acceptance and feedback quality score, and proactive help score for the text data;

[0027] The collaboration dimension score is obtained based on the response time score, opinion acceptance and feedback quality score, and active help score of the text data.

[0028] Furthermore, the large model is used to conduct continuous contribution dimension analysis on the text data in the records and statistical information, including:

[0029] Generate a continuous contribution scoring prompt; wherein the content of the continuous contribution scoring prompt includes: the continuous contribution dimension task description, text data in the record and statistical information, and the continuous contribution dimension scoring criteria. The continuous contribution dimension task description defines continuous contribution sub-dimensions, and the continuous contribution sub-dimensions include: submission frequency and submission interval, activity of long-term maintenance projects, and document updates and bug fixes;

[0030] Use the continuous contribution scoring prompt to guide the large model to generate submission frequency and submission interval scores for the text data, activity scores for long-term maintenance projects, and scores for document updates and bug fixes;

[0031] The continuous contribution dimension score is obtained based on the submission frequency and submission interval score of the text data, the activity score of the long-term maintenance project, and the document update and bug fix score.

[0032] Furthermore, the large model is used to analyze the code quality dimensions of the text data in the records and statistical information, including:

[0033] Generate a code quality scoring prompt; wherein the content of the code quality scoring prompt includes: a code quality dimension task description, text data in records and statistical information, and a code quality scoring standard; the code quality dimension task description defines code quality sub-dimensions, and the code quality sub-dimensions include: code readability and comment quality, code consistency, and code reliability and stability;

[0034] Use the code quality score prompt to guide the large model to generate code readability and comment quality scores, code consistency scores, and code reliability and stability scores for the text data;

[0035] The code quality dimension score is obtained based on the code readability and comment quality score, code consistency score, and code reliability and stability score of the text data.

[0036] Furthermore, the nodes in the knowledge graph include: developer nodes, warehouse nodes, organization nodes, technology stack nodes, and collaboration event nodes;

[0037] The obtaining of a second rating of the developer based on the knowledge graph includes:

[0038] Use the PageRank algorithm to calculate the importance score of the developer node;

[0039] Analyze the knowledge graph to obtain the quality score of developers' contributions to the code repository and the developer's technology coverage score;

[0040] The second score of the developer is obtained based on the importance score of the developer's node, the quality score of the developer's contribution to the code repository, and the developer's technology coverage score.

[0041] Furthermore, the developer's contribution quality score to the code repository Among them, d represents the target developer node, r represents the code repository, and R d A represents the set of all repositories in which developer d participates. r represents the PR adoption rate of developer d in repository r, S r represents the number of stars of warehouse r, T r Indicates the time of the developer d's most recent contribution to repository r.

[0042] Furthermore, the developer's technical coverage score Among them, d represents the target developer node, t represents the technology stack, and T d represents the technology stack set used by developer d, p t represents the frequency of use of technology stack t in all contributions of developer d.

[0043] A large-scale model-based open source community developer behavior influence assessment system, comprising:

[0044] Information acquisition module, used to obtain records and statistical information of developers in software development projects;

[0045] A first scoring calculation module is used to use a large model to perform multi-dimensional analysis on the text data in the records and statistical information, and to comprehensively score each dimension to obtain a first score for the developer; wherein the dimensions include: technology, communication, collaboration, continuous contribution, and code quality;

[0046] A second score calculation module is used to construct a knowledge graph based on the records and statistical information, and obtain a second score of the developer based on the knowledge graph;

[0047] An evaluation result generating module is used to obtain an evaluation result of the developer's behavior influence based on the first score and the second score.

[0048] Compared with the prior art, the present invention has at least the following beneficial effects.

[0049] 1. By introducing LLM multi-dimensional text analysis technology, it effectively solves the semantic understanding problem of unstructured communication data (such as PR comments, issue discussions, and code analysis), and realizes the quantitative evaluation of technical communication score (C) and collaboration score (Co), thus overcoming the problem of single developer evaluation dimension in existing technologies.

[0050] 2. By using the large natural language model (LLM) to perform semantic detection on the content of the text, effective filtering of false contributions can be achieved.

[0051] 3. Propose a technology coverage score (Diversity(T)) to quantify the full-stack technical capabilities of developers.

[0052] 4. Design a nonlinear attenuation function (1 / log(1+T_r)). By introducing a logarithmic function, we enhance the sensitivity of the recent time window, making the evaluation results more in line with the rapid iteration requirements of the open source community. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall flow chart of the technical solution of this application.

[0054] Figure 2 This is an example diagram of the knowledge graph. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0056] In order to facilitate a deeper understanding of the technical solutions of the present invention, relevant concepts will be explained in detail below.

[0057] Github: In this invention, GitHub is a Git-based code hosting platform that supports version control, collaborative development, and project management. It allows users to store and manage code, track modification history, and facilitate team collaboration. By creating repositories, developers can share and maintain project code, track issues, and conduct code reviews. In this embodiment of the invention, data is obtained from the GitHub platform.

[0058] Large Language Model (LLM): In this context, large models specifically refer to large natural language models (LLMs), deep learning models with hundreds of millions of parameters specifically designed for natural language processing. By leveraging massive amounts of text data and powerful computing resources, large natural language models are able to capture complex patterns and nuances in language.

[0059] Prompt: In this paper, a large model's prompt is a structured set of instructions designed to guide a large-scale language model to generate a specific response or perform a specific task. Prompts consist of input text, control parameters, and contextual information. By precisely configuring these elements, the content, style, and complexity of the model's output can be effectively controlled.

[0060] A knowledge graph is a structured representation of knowledge that describes entities and their relationships through nodes and edges. Nodes represent entities (e.g., developers, code repositories), while edges represent relationships between entities (e.g., affiliations or associations). Knowledge graphs can integrate multi-source data, provide rich semantic expressions and complex reasoning capabilities, and support efficient query and analysis. In this paper, a knowledge graph is constructed based on data obtained from Github.

[0061] PageRank: It is a link analysis algorithm used to measure the importance of web pages and rank them in search engine results. In this invention, the PageRank algorithm is applied to the knowledge graph to evaluate the importance of nodes. The algorithm calculates the weight of each node by analyzing the link relationships between nodes. Through this mechanism, the technical solution of the present invention can more accurately evaluate the importance of nodes in the knowledge graph.

[0062] The present invention first obtains and cleans data from GitHub, then uses a large language model (LLM) to process text data and scores it based on five dimensions: technology (T), communication (C), collaboration (Co), continuous contribution (D), and code quality (Q). Secondly, a knowledge graph containing entities such as developers, repositories, and organizations is constructed, and the importance of nodes is evaluated using the PageRank algorithm. The contribution quality is calculated by combining the PR adoption rate, number of stars, and time decay coefficient, and the coverage score is calculated using the technology stack distribution calculation technology. Finally, the LLM score is combined with the knowledge graph score to comprehensively derive the developer influence score.

[0063] Specifically, the open source community developer behavior influence evaluation method based on the large model of the present invention is as follows: Figure 1 As shown, the following steps are included.

[0064] Step S1: Data acquisition.

[0065] The present invention relates to a method for extracting specific data through a programming interface. Specifically, the method uses the application programming interface (API) provided by GitHub to systematically retrieve various records and statistical information in a software development project. First, log in to your GitHub account and generate a Personal Access Token. (1) Get code commit records: send a GET request to the / repos / :owner / :repo / commits endpoint, where :owner and :repo are placeholders for the target repository owner and repository name, respectively. (2) Get Pull Request records: use the / repos / :owner / :repo / pulls endpoint to initiate a GET request to retrieve all Pull Requests. (3) Get Issue records: get all Issues data by calling the / repos / :owner / :repo / issues endpoint. (4) Get repository metadata: send a GET request to the / repos / :owner / :repo endpoint to obtain.

[0066] The above code submission records, Pull Request records, Issue records and warehouse metadata provide the basic data source for the construction of large models and knowledge graphs.

[0067] For example, the text comments used in the subsequent large model mainly come from:

[0068] The title, description, and review comments in the Pull Request;

[0069] The title, description, and discussion comments in the issue;

[0070] The commit message in the commit record.

[0071] The data used in the knowledge graph is further extracted and modeled based on the above structured data, mainly including:

[0072] Developer identity (determined by committer, PR / Issue creator, etc.);

[0073] Warehouse and organization information (obtained from warehouse metadata);

[0074] Technology stack information (which can be analyzed from repository language annotations or dependency files);

[0075] Collaboration events (such as commits, PRs, issue discussions, and other behavioral nodes and their relationships).

[0076] Therefore, large models rely on text-based field content, while knowledge graphs model relational networks based on structured metadata. Both can extract the required information from the data obtained in this step.

[0077] Step S2: Prompt construction.

[0078] This paper proposes a method that uses a large natural language model to comprehensively analyze text data (including code and text comments) and score them based on multiple dimensions. This method scores the following five aspects and assigns appropriate weights to the scores to comprehensively evaluate the developer's performance and contribution. The details of each score are as follows:

[0079] S201 Technical Score (T): The technical score is designed to comprehensively assess the developer's performance in both code writing and technical problem solving. This score is based on three dimensions: (1) Technical Complexity: the complexity of the code and the difficulty of the problem solved. (2) Code Innovation: the uniqueness and novelty of the solution. (3) Technical Breadth: the diversity of the technology stack involved.

[0080] S202 Communication Score (C) is used to evaluate the effectiveness of developers' communication and expression behaviors in written comments and discussions. This score is based on three dimensions: (1) Clarity of Expression: Whether the description is clear and understandable. (2) Emotional Tendency: The emotional tone (positive / negative) in comments and discussions. (3) Logical Rigor: Whether the argument is reasonable and well-founded.

[0081] S203 Collaboration Score (Co): The collaboration score focuses on the developer's behavior in project collaboration, especially their contribution to teamwork. This score considers three dimensions: (1) Response time in PR discussions and issue comments; (2) Receptivity to others' opinions and the quality of feedback; and (3) Whether they proactively help other developers solve problems.

[0082] S204 Sustained Contribution Score (D) is designed to assess the developer's long-term contribution behavior and activity to the project. This score is based on three dimensions: (1) commit frequency and commit interval; (2) long-term project maintenance activity; and (3) whether documentation is updated and bugs are fixed regularly.

[0083] S205 Code Quality Score (Q): The code quality score is used to evaluate the quality and maintainability of the code written by developers. This score is considered from three dimensions: (1) Code readability and comment quality to ensure that the code is easy to understand and maintain. (2) Code consistency to ensure a unified code style and facilitate team collaboration. (3) Code reliability and stability.

[0084] S206 Prompt design.

[0085] (1) Technical score (T) prompt.

[0086] Please evaluate the following code snippet comprehensively and rate each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0087] 1. Technical Complexity (TC):

[0088] -Logical complexity of the code implementation.

[0089] -The difficulty and challenge of solving the problem.

[0090] -The complexity of the data structures and algorithms used.

[0091] 2. **Code Innovation (CI)**:

[0092] - Uniqueness of the solution.

[0093] -Whether new methods or technologies have been introduced.

[0094] - Whether it is a significant improvement over existing solutions.

[0095] 3. **Technology Breadth (TB)**:

[0096] -The number of different technologies, frameworks, or libraries used.

[0097] -Programming languages and technology areas involved.

[0098] -Multidisciplinary knowledge demonstrated in the code.

[0099] {text}

[0100] Scoring criteria:

[0101] -Technical Complexity (TC):

[0102] 1- Very simple, with almost no logical complexity.

[0103] 10-Extremely complex, involving advanced data structures and algorithms, and solving very challenging problems.

[0104] -Code Innovation (CI):

[0105] 1- No innovation, completely following common solutions.

[0106] 10 - Very innovative, introducing a completely new approach or technology that significantly improves existing solutions.

[0107] -Technology Breadth (TB):

[0108] 1-Only involves a single technology or framework.

[0109] 10-Demonstrates broad interdisciplinary knowledge across multiple technologies, frameworks, or libraries.

[0110] Please give scores according to the three sub-dimensions mentioned above.

[0111] (2) Communication score (C)-Prompt.

[0112] Please provide an overall evaluation of the following comments or discussions and rate each sub-dimension on a scale of 1 to 10.

[0113] Consider the following three sub-dimensions:

[0114] 1. **Expression Clarity (EC)**:

[0115] - Is the description concise and clear?

[0116] -Whether easy-to-understand language and terminology are used.

[0117] -Whether vague or ambiguous statements are avoided.

[0118] 2. **Emotional Tendency (ES)**:

[0119] - Whether the sentiment expressed in the comment is positive or negative.

[0120] -Whether it contains encouragement, support, criticism or complaints.

[0121] - Whether the overall tone and attitude is positive or negative.

[0122] 3. **Logical Rigor (EL)**:

[0123] -Whether the argument is well-founded and justified.

[0124] - Whether the reasoning process is logical and has no obvious loopholes.

[0125] - Whether specific evidence or examples are provided to support the viewpoint.

[0126] {text}

[0127] Scoring criteria:

[0128] -Expression Clarity (EC):

[0129] 1- Very unclear and difficult to understand.

[0130] 10- Very clear and easy to understand without any ambiguity.

[0131] -Emotional tendency (ES):

[0132] 1- Very negative, full of criticism and complaints.

[0133] 10- Very positive, encouraging and supportive.

[0134] -Logical rigor (EL):

[0135] 1- Very imprecise, lacking basis and logic.

[0136] 10- Very rigorous, with arguments that are well-founded and well-reasoned.

[0137] Please give scores according to the three sub-dimensions mentioned above.

[0138] (3) Collaborative scoring (Co)-Prompt.

[0139] Please provide a comprehensive evaluation of the following PR discussion or issue comments and give a score for each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0140] 1. **Response Time (RT)**:

[0141] -How quickly developers respond in pull request discussions or issue comments.

[0142] - Whether the response is timely and whether it demonstrates a positive communication attitude.

[0143] 2. **Opinion acceptance and feedback quality (AQ)**:

[0144] -How receptive the developer is to the opinions of others.

[0145] -The quality of the feedback, whether it provides constructive opinions and suggestions.

[0146] - Whether the issues under discussion can be effectively resolved.

[0147] 3. **Active Help (AH)**:

[0148] -Whether developers proactively help other developers solve problems.

[0149] - Demonstrated teamwork and willingness to share knowledge and experience.

[0150] {text}

[0151] Scoring criteria:

[0152] -Response Time (RT):

[0153] 1- The response is very slow, almost no reply in time.

[0154] 10- Very responsive, always replied promptly.

[0155] - Opinion acceptance and feedback quality (AQ):

[0156] 1-Almost no acceptance of others’ opinions and low-quality feedback.

[0157] 10-Highly receptive to others' opinions and provide high-quality and constructive feedback.

[0158] -Active Help (AH):

[0159] 1-Hardly takes the initiative to help others and lacks teamwork spirit.

[0160] 10-Take the initiative to help others and demonstrate a strong team spirit.

[0161] Please give scores according to the three sub-dimensions mentioned above.

[0162] (4)Continuous Contribution Score (D)-Prompt.

[0163] Please provide a comprehensive evaluation of the developer's ongoing contributions to the project and rate each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0164] 1. **Submission frequency and submission interval (SF)**:

[0165] -How often developers commit code.

[0166] - Whether the time between submissions is reasonable and whether it maintains continuous contribution.

[0167] 2. **Activity of long-term maintenance projects (PA)**:

[0168] -The developer's long-term activity in the project.

[0169] -Whether to continue to participate in the development and maintenance of the project.

[0170] 3.**Document updates and bug fixes (DB)**:

[0171] -Whether developers regularly update project documentation.

[0172] -Whether to actively fix discovered bugs and keep the code base healthy.

[0173] {text}

[0174] Scoring criteria:

[0175] -Submission frequency and submission interval (SF):

[0176] 1- There are almost no submissions, or submissions are very irregular.

[0177] 10-Commit frequently and regularly, maintaining continuous contribution.

[0178] -Activity of long-term maintenance projects (PA):

[0179] 1-Almost no participation in the long-term maintenance of the project and lack of activity.

[0180] 10-Continuously be active in project maintenance and actively participate in development and improvement.

[0181] -Documentation updates and bug fixes (DB):

[0182] 1- Documentation is rarely updated or bugs are fixed, resulting in outdated project documentation and accumulation of bugs.

[0183] 10-Update documents regularly, actively fix bugs, and maintain the accuracy of project documents and the health of the code base.

[0184] Please give scores according to the three sub-dimensions mentioned above.

[0185] (5) Code quality score (Q)-Prompt.

[0186] Please evaluate the following code snippet comprehensively and rate each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0187] 1. **Code readability and comment quality (QR)**:

[0188] - Are variable and function names descriptive and easy to understand?

[0189] - Are there appropriate comments to explain complex logic or key steps?

[0190] - Is the code structure clear and easy to follow?

[0191] 2. **Code consistency (QC)**:

[0192] -Whether a consistent coding style is followed (such as indentation, naming conventions, etc.).

[0193] -Whether unified design patterns and architectural principles are used.

[0194] -Whether duplicate code is avoided (DRY principle).

[0195] 3. **Code reliability and stability (QRb)**:

[0196] -Are there adequate input validation and exception handling mechanisms?

[0197] -Whether various boundary conditions and extreme situations are considered.

[0198] -Are there unit tests or other forms of automated tests covering key functionality?

[0199] {text}

[0200] Scoring criteria:

[0201] -Code readability and comment quality (QR):

[0202] 1- The code is hard to read and has almost no comments.

[0203] 10- The code is very readable and the comments are detailed and helpful in understanding the code logic.

[0204] -Code consistency (QC):

[0205] 1- The coding style is inconsistent and there is a lot of duplicate code.

[0206] 10-The code style is highly consistent, follows a unified design pattern, and there is no duplication of code.

[0207] -Code reliability and stability (QRb):

[0208] 1- No input validation and exception handling, lack of test coverage.

[0209] 10- Has comprehensive input validation and exception handling mechanisms, and extensive test coverage.

[0210] Please give scores according to the three sub-dimensions mentioned above.

[0211] Step S3: Large model scoring.

[0212] S301 uses the Qwen2.5-72B-Instruct natural language model for scoring. This model has strong natural language processing capabilities and extensive domain knowledge, making it suitable for complex scoring tasks.

[0213] S302 inputs text construction, embedding the code data to be evaluated and text comments into the "{text}" position in the pre-designed prompt template. For example, for the technical score (T), the prompt will clearly instruct the model to evaluate technical complexity, code innovation, and technical breadth.

[0214] S303 Feature Extraction and Preliminary Scoring: A large model is used to deeply analyze the input text, extracting feature information relevant to the scoring dimensions and generating a preliminary score (using the technical scoring (T) process as an example): Technical Complexity (TC) Scoring: The model generates a technical complexity score (a) based on factors such as code structure and algorithm difficulty; Code Innovation (CI) Scoring: The model generates a code innovation score (b) based on the uniqueness and novelty of the solution; and Technical Breadth (TB) Scoring: The model generates a technical breadth score (c) based on the diversity of the technology stack involved.

[0215] S304 Weighted and Comprehensive Scoring: Weights are assigned to each dimension based on its importance. In this example, all dimensions are assumed to have equal weights, so the average is taken. The final technical score (T) is calculated by averaging the three scores above: (a+b+c) / 3.

[0216] S305 repeats S302-S304 to generate a technical score (T), a communication score (C), a collaboration score (Co), a continuous contribution score (D), and a code quality score (Q) respectively.

[0217] Step S4: Score aggregation based on large model.

[0218] In this paper, we use a systematic approach to aggregate and weight the scores of each dimension to comprehensively evaluate the performance of various developer behaviors. This method ensures a comprehensive and fair evaluation of all aspects of performance and provides a scientific basis for subsequent project management and team collaboration.

[0219] S401 Weight setting: Set appropriate weights according to the importance of each scoring dimension and perform weighted summary of each score. The specific weight settings are as follows (can be flexibly adjusted according to actual conditions): Technical score weight (ω T )=0.3, communication score (ω C )=0.2, collaboration score (ω Co )=0.2, continuous contribution score (ω D )=0.15、Code quality score(ω Q )=0.15.

[0220] S402 score weighted summary, S LLM =ω T ·T+ω C C+ωCo CO+ω D ·D+ω Q ·Q.

[0221] Step S5: Knowledge graph construction.

[0222] In this invention, in order to comprehensively and systematically represent the complex relationships between developers, repositories, organizations, technology stacks (including programming languages and frameworks) and collaborative events (such as submissions, Pull Requests, Issue discussions, etc.), a knowledge graph construction method based on entity nodes is used. This method forms a structured and easy-to-query knowledge graph by using the above elements as entity nodes. Specific entity nodes include developers, repositories, organizations, technology stacks (programming languages, frameworks), collaborative events (submissions, PRs, Issue discussions, etc.), as shown in the example diagram. Figure 2 shown.

[0223] Step S6: Score the developer based on the knowledge graph.

[0224] S601 In the present invention, in order to evaluate the importance and influence of developers in the network, the PageRank algorithm based on the knowledge graph calculates the developer node PageRank (d) to quantify its relative importance in the network.

[0225] S602 Contribution Quality Score: Quality (T) calculation. In this paper, in order to evaluate the quality of developers' contributions to the code repository, a comprehensive scoring method based on multiple factors is proposed. This score describes the quality of the contribution and is calculated using a specific formula. The specific calculation method is as follows:

[0226]

[0227] Among them, d represents the target developer node, r represents the code repository, and R d A represents the set of all repositories in which developer d participates. r represents the PR adoption rate of developer d in repository r, S r represents the number of stars of warehouse r, T r Indicates the time (in months) when developer d made his most recent contribution to repository r. For example, the current month is 1 and the previous month is 2. T indicates the relative time of the most recent contribution.

[0228] In order to strengthen the influence of recent active records and weaken the influence of long-term active records, a time decay coefficient can be set for dynamic modeling and evaluation based on active records. This coefficient gives higher weight to recent active contributions.

[0229] S603 Technology Coverage Score: Diversity (T) calculation. In this paper, a technology coverage scoring method is proposed to evaluate the distribution of developers' technology stacks and the comprehensiveness of their skills. The higher the score, the more dispersed the technology stack distribution and the more comprehensive the developer's skills. The specific calculation method is as follows:

[0230]

[0231] Where d represents the target developer node, t represents the technology stack (such as programming language), and T d represents the technology stack set used by developer d, p t Indicates the frequency of use of technology stack t in all contributions of developer d, for example, the percentage of times Java was used.

[0232] Step S7: Score the developer behavior based on the knowledge graph.

[0233] In this invention, in order to comprehensively evaluate the comprehensive capabilities and importance of developers, multiple key indicators are combined, including PageRank value, contribution quality score and skill coverage score, to calculate the importance score of the developer node. Combining PageRank (d) obtained in S601, the contribution quality score Quality (T) obtained in S602, and the technology coverage score Diversity (T) obtained in S603, the importance score of the developer node based on the knowledge graph is calculated as follows:

[0234]

[0235] Where: α, β, γ represent weight coefficients, which can be α = 0.5, β = 0.3, γ = 0.2; PageRank (d), Quality (T), Diversity (T) are calculated in step S6, Indicates the normalization of the contribution quality score Quality(T).

[0236] Step S8: Comprehensive scoring.

[0237] In this paper, in order to comprehensively evaluate the comprehensive ability and influence of developers, a method combining large model scoring S LLM and the knowledge graph-based scoring S KG The final comprehensive scoring method. This method introduces the weight coefficient θ to flexibly adjust the contribution ratio of the two scoring sources to ensure that the scoring results can adapt to different data integrity and application scenarios. The specific calculation method is as follows:

[0238] S Final =θ·S LLM +(1-θ)SKG

[0239] Where: S LLM The score generated based on the large model in step S4 reflects the developer's technical level, communication skills, collaboration, continuous contribution and code quality; S KG is the score calculated based on the knowledge graph in step S7, including PageRank value, contribution quality score and technology coverage score; θ is the weight coefficient, which can be adjusted based on the completeness of the knowledge graph data and the usage scenario. The default value is 0.5, which determines the relative importance of the large model score and the knowledge graph score in the final comprehensive score. If the knowledge graph data is relatively complete and accurate, the θ value can be appropriately reduced and S can be increased. KG Weight; in application scenarios where the developer's technical ability needs to be emphasized, such as technical review or code review, the θ value can be increased to increase S LLM Weight.

[0240] In summary, the present invention uses a multi-dimensional LLM scoring model and a dynamic prompt design. It uses a large language model (LLM) to perform semantic analysis on developer text data (such as issue discussions, PR descriptions, code, etc.), and combines it with the prompt design to generate fine-grained indicators such as technical score (T), communication score (C), collaboration score (Co), continuous contribution score (D), and code quality score (Q). Compared with traditional open source community influence assessments that rely heavily on single statistical indicators such as the number of code submissions (such as commits and PRs), the present invention uses the semantic understanding ability of LLM to model developer behavior characteristics from multiple dimensions such as technical depth, communication efficiency, and willingness to collaborate, deeply mining the deep semantics in natural language data, breaking through the limitations of traditional statistical indicators and improving the accuracy and applicability of scoring.

[0241] This paper designs a cross-modal fusion scoring architecture of large model LLM and knowledge graph, and designs a dual-channel scoring fusion function S Final =θ·S LLM +(1-θ)S KG Dual-channel scoring allows for a more comprehensive assessment of developer influence. The weight parameter θ can be dynamically adjusted based on the actual use case and different priorities, allowing for flexible assessment of developer influence based on the actual use case. Simultaneously, the fusion model can enhance model robustness and avoid bias from a single model.

[0242] The present invention designs targeted prompt templates to guide LLMs to fully understand the scoring criteria, generate standardized scoring outputs from different perspectives, and improve the objectivity of the evaluation.

[0243] The design contribution quality formula of the present invention It integrates the time decay factor and the warehouse influence parameter through a nonlinear decay function Strengthen the weight of recent active contributions.

[0244] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A method for evaluating the influence of developer behavior in an open source community based on a large model, characterized by: The method comprises: Obtain records and statistics of developers in software development projects; A large model is used to perform multi-dimensional analysis on the text data in the records and statistical information, and the scores of each dimension are comprehensively evaluated to obtain the developer's first score. The dimensions include: technology, communication, collaboration, continuous contribution, and code quality. Constructing a knowledge graph based on the records and statistical information, and obtaining a second score of the developer based on the knowledge graph; An evaluation result of the developer's behavior influence is obtained based on the first score and the second score.

2. The method according to claim 1, characterized in that Use large models to perform technical dimension analysis on text data in records and statistical information, including: Generate a technical scoring prompt; wherein the content of the technical scoring prompt includes: the technical dimension task description, text data in the record and statistical information, and the technical dimension scoring criteria, the technical dimension task description defines technical sub-dimensions, and the technical sub-dimensions include: technical complexity, code innovation, and technical breadth; Use the technical score prompt to guide the large model to generate the technical complexity score, code innovation score, and technical breadth score for the text data; The technical dimension score is obtained based on the technical complexity score, code innovation score, and technical breadth score of the text data.

3. The method according to claim 1, characterized in that Use the big model to analyze the communication dimension of text data in records and statistical information, including: Generate a communication scoring prompt; wherein the content of the communication scoring prompt includes: a communication dimension task description, text data in the record and statistical information, and a communication dimension scoring standard, wherein the communication dimension task description defines communication sub-dimensions, and the communication sub-dimensions include: clarity of expression, emotional tendency, and logical rigor; Use the communication score prompt to guide the large model to generate the expression clarity score, sentiment tendency score, and logical rigor score of the text data; The communication dimension score is obtained based on the expression clarity score, emotional tendency score and logical rigor score of the text data.

4. The method according to claim 1, wherein Use the big model to conduct collaborative dimensional analysis of text data in records and statistical information, including: Generate a collaboration scoring prompt; wherein the content of the collaboration scoring prompt includes: a collaboration dimension task description, text data in the record and statistical information, and a collaboration dimension scoring standard; the collaboration dimension task description defines collaboration sub-dimensions, and the collaboration sub-dimensions include: response time, opinion acceptance and feedback quality, and proactive assistance; Use the collaborative scoring prompt to guide the large model to generate the response time score, opinion acceptance and feedback quality score, and proactive help score for the text data; The collaboration dimension score is obtained based on the response time score, opinion acceptance and feedback quality score, and active help score of the text data.

5. The method according to claim 1, wherein Use large models to conduct continuous contribution dimension analysis on text data in records and statistical information, including: Generate a continuous contribution scoring prompt; wherein the content of the continuous contribution scoring prompt includes: the continuous contribution dimension task description, text data in the record and statistical information, and the continuous contribution dimension scoring criteria. The continuous contribution dimension task description defines continuous contribution sub-dimensions, and the continuous contribution sub-dimensions include: submission frequency and submission interval, activity of long-term maintenance projects, and document updates and bug fixes; Use the continuous contribution scoring prompt to guide the large model to generate submission frequency and submission interval scores for the text data, activity scores for long-term maintenance projects, and scores for document updates and bug fixes; The continuous contribution dimension score is obtained based on the submission frequency and submission interval score of the text data, the activity score of the long-term maintenance project, and the document update and bug fix score.

6. The method according to claim 1, characterized in that Use the big model to analyze code quality dimensions on text data in records and statistical information, including: Generate a code quality scoring prompt; wherein the content of the code quality scoring prompt includes: a code quality dimension task description, text data in records and statistical information, and a code quality scoring standard; the code quality dimension task description defines code quality sub-dimensions, and the code quality sub-dimensions include: code readability and comment quality, code consistency, and code reliability and stability; Use the code quality score prompt to guide the large model to generate code readability and comment quality scores, code consistency scores, and code reliability and stability scores for the text data; The code quality dimension score is obtained based on the code readability and comment quality score, code consistency score, and code reliability and stability score of the text data.

7. The method according to claim 1, characterized in that The nodes in the knowledge graph include: developer nodes, warehouse nodes, organization nodes, technology stack nodes and collaboration event nodes; The obtaining of a second rating of the developer based on the knowledge graph includes: Use the PageRank algorithm to calculate the importance score of the developer node; Analyze the knowledge graph to obtain the quality score of developers' contributions to the code repository and the developer's technology coverage score; The second score of the developer is obtained based on the importance score of the developer's node, the quality score of the developer's contribution to the code repository, and the developer's technology coverage score.

8. The method according to claim 7, characterized in that The quality rating of the developer's contribution to the code repository Among them, d represents the target developer node, r represents the code repository, and R d A represents the set of all repositories in which developer d participates. r represents the PR adoption rate of developer d in repository r, S r represents the number of stars of warehouse r, T r Indicates the time of the developer d's most recent contribution to repository r.

9. The method according to claim 7, characterized in that The developer's technical coverage score Among them, d represents the target developer node, t represents the technology stack, and T d represents the technology stack set used by developer d, p t represents the frequency of use of technology stack t in all contributions of developer d.

10. A large-scale model-based open source community developer behavior influence assessment system, characterized by: The system comprises: Information acquisition module, used to obtain records and statistical information of developers in software development projects; A first scoring calculation module is used to use a large model to perform multi-dimensional analysis on the text data in the records and statistical information, and to comprehensively score each dimension to obtain a first score for the developer; wherein the dimensions include: technology, communication, collaboration, continuous contribution, and code quality; A second score calculation module is used to construct a knowledge graph based on the records and statistical information, and obtain a second score of the developer based on the knowledge graph; An evaluation result generating module is used to obtain an evaluation result of the developer's behavior influence based on the first score and the second score.

Images