Method and system for evaluating influence of open source community developer behavior based on large model

Abstract

The application discloses a kind of method and system for evaluating the influence of open source community developers based on large model, belongs to the technical field of developer evaluation, the method includes: obtaining the record and statistical information of developer in software development project;Text data in record and statistical information is analyzed by using large model in multidimension, and the first score of the developer is obtained by comprehensively scoring each dimension;Wherein, the dimension includes: technology, communication, cooperation, continuous contribution and code quality;A knowledge graph is constructed according to the record and statistical information, and the second score of the developer is obtained based on the knowledge graph;Based on the first score and the second score, the behavior influence evaluation result of the developer is obtained.The application can accurately evaluate the behavior influence of developer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of developer evaluation, and in particular to a method and system for evaluating the influence of open source community developers based on large models. BACKGROUND

[0002] In the traditional technical solution, the evaluation of the influence of developer behavior mainly depends on simple statistical indicators, such as the number of submissions, the number of Pull Requests (PRs), and the frequency of Issues discussion. Although these methods can reflect the activity of developer behavior to some extent, they often ignore the non-code behavior values such as the contribution quality and collaboration ability of developers in actual projects. Therefore, some existing methods consider the collaboration network characteristics between developers in the evaluation process.

[0003] For example, Chinese patent application CN111191882A discloses a method and device for identifying influential developers in a heterogeneous information network, which constructs a heterogeneous information network, comprehensively calculates the multi-aspect influence value of developers on GitHub (including the interaction between individuals, projects, and companies), and generates a final score by fusing these values to identify the most influential developers.

[0004] However, the existing methods have the following shortcomings:

[0005] 1) The existing evaluation system cannot distinguish between real developer contributions and automated behavior of robot accounts (such as mechanical submission of PRs and repeated generation of Issues), resulting in a risk of data pollution in the evaluation results;

[0006] 2) The existing methods do not consider the distribution breadth and depth of developers' technical stack, making it difficult to evaluate the ability of developers to adapt to multiple technical scenarios;

[0007] 3) The existing evaluation model is insufficient in time decay evaluation, and cannot accurately reflect the higher value weight of recent active contributions in open source communities.

[0008] In summary, the existing developer evaluation methods still have a large error in evaluating the influence of developer behavior. SUMMARY

[0009] The present application provides a method and system for evaluating the influence of open source community developers based on large models, which can accurately evaluate the influence of developers.

[0010] To achieve the above-mentioned purpose, the technical solution of the present application includes the following contents.

[0011] A method for evaluating the influence of open source community developers based on large models, the method comprising:

[0012] obtaining record and statistical information of a developer in a software development project;

[0013] performing multi-dimensional analysis on text data in the record and statistical information by using a large model, and synthesizing scores of the dimensions to obtain a first score of the developer; wherein the dimensions include technology, communication, collaboration, continuous contribution, and code quality;

[0014] constructing a knowledge graph according to the record and statistical information, and obtaining a second score of the developer based on the knowledge graph;

[0015] obtaining a behavior influence evaluation result of the developer based on the first score and the second score.

[0016] Further, performing technology dimension analysis on the text data in the record and statistical information by using a large model, including:

[0017] generating a technology score Prompt; wherein the content of the technology score Prompt includes a technology dimension task description, text data in the record and statistical information, and technology dimension score standards, and the technology dimension task description defines technology sub-dimensions, including technology complexity, code innovation, and technology breadth;

[0018] using the technology score Prompt to guide the large model to generate a technology complexity score, a code innovation score, and a technology breadth score of the text data;

[0019] obtaining a technology dimension score according to the technology complexity score, the code innovation score, and the technology breadth score of the text data.

[0020] Further, performing communication dimension analysis on the text data in the record and statistical information by using a large model, including:

[0021] generating a communication score Prompt; wherein the content of the communication score Prompt includes a communication dimension task description, text data in the record and statistical information, and communication dimension score standards, and the communication dimension task description defines communication sub-dimensions, including expression clarity, emotional tendency, and logical rigor;

[0022] using the communication score Prompt to guide the large model to generate an expression clarity score, an emotional tendency score, and a logical rigor score of the text data;

[0023] obtaining a communication dimension score according to the expression clarity score, the emotional tendency score, and the logical rigor score of the text data.

[0024] Further, the text data in the records and statistical information is analyzed by the large model in a collaboration dimension, including:

[0025] A collaboration score Prompt is generated, wherein the content of the collaboration score Prompt includes a collaboration dimension task description, the text data in the records and statistical information, and a collaboration dimension score standard, the collaboration dimension task description defines a collaboration sub-dimension, and the collaboration sub-dimension includes response time, opinion acceptance and feedback quality, and active help.

[0026] The large model is guided by the collaboration score Prompt to generate a response time score, an opinion acceptance and feedback quality score, and an active help score of the text data.

[0027] According to the response time score, the opinion acceptance and feedback quality score, and the active help score of the text data, a collaboration dimension score is obtained.

[0028] Further, the text data in the records and statistical information is analyzed by the large model in a continuous contribution dimension, including:

[0029] A continuous contribution score Prompt is generated, wherein the content of the continuous contribution score Prompt includes a continuous contribution dimension task description, the text data in the records and statistical information, and a continuous contribution dimension score standard, the continuous contribution dimension task description defines a continuous contribution sub-dimension, and the continuous contribution sub-dimension includes submission frequency and submission interval, activity of long-term maintenance projects, and document update and Bug repair.

[0030] The large model is guided by the continuous contribution score Prompt to generate a submission frequency and submission interval score, an activity of long-term maintenance projects score, and a document update and Bug repair score of the text data.

[0031] According to the submission frequency and submission interval score, the activity of long-term maintenance projects score, and the document update and Bug repair score of the text data, a continuous contribution dimension score is obtained.

[0032] Further, the text data in the records and statistical information is analyzed by the large model in a code quality dimension, including:

[0033] A code quality score Prompt is generated, wherein the content of the code quality score Prompt includes a code quality dimension task description, the text data in the records and statistical information, and a code quality score standard, the code quality dimension task description defines a code quality sub-dimension, and the code quality sub-dimension includes code readability and annotation quality, code consistency, and reliability and stability of the code.

[0034] The code readability and annotation quality score, the code consistency score, and the reliability and stability score of the code are obtained by prompting the large model to generate the text data using a code quality score prompt.

[0035] According to the code readability and annotation quality score, the code consistency score, and the reliability and stability score of the code, a code quality dimension score is obtained.

[0036] Further, the nodes in the knowledge graph include a developer node, a repository node, an organization node, a technology stack node, and a collaboration event node.

[0037] The second score of the developer is obtained based on the knowledge graph, and the second score of the developer includes:

[0038] The importance score of the developer node is calculated using a PageRank algorithm.

[0039] The knowledge graph is analyzed to obtain a contribution quality score of the developer to the code repository and a technology coverage score of the developer.

[0040] The second score of the developer is obtained based on the importance score of the developer node, the contribution quality score of the developer to the code repository, and the technology coverage score of the developer.

[0041] Further, the contribution quality score of the developer to the code repository Where d represents the target developer node, r represents the code repository, R d represents a set of all repositories in which the developer d participates, A r represents the PR adoption rate of the developer d in the repository r, S r represents the star number of the repository r, T r represents the time of the latest contribution of the developer d in the repository r.

[0042] Further, the technology coverage score of the developer Where d represents the target developer node, t represents the technology stack, T d represents a set of technology stacks used by the developer d, p t represents the frequency of use of the technology stack t in all contributions of the developer d.

[0043] A large model-based open source community developer behavior influence evaluation system, the system includes:

[0044] An information acquisition module for acquiring records and statistical information of developers in a software development project;

[0045] a first score calculation module configured to perform multi-dimensional analysis on the text data in the records and statistical information by using the large model, and to obtain a first score of the developer by synthesizing scores of each dimension, wherein the dimensions include technology, communication, collaboration, continuous contribution, and code quality;

[0046] a second score calculation module configured to construct a knowledge graph according to the records and statistical information, and to obtain a second score of the developer based on the knowledge graph;

[0047] an evaluation result generation module configured to obtain an evaluation result of the behavior influence of the developer based on the first score and the second score.

[0048] Compared with the prior art, the present application has at least the following beneficial effects.

[0049] 1. By introducing the LLM multi-dimensional text analysis technology, the semantic understanding problem of unstructured communication data (such as PR comments, issue discussions, and code analysis) is effectively solved, and the quantitative evaluation of technical communication score (C) and collaboration score (Co) is realized, thereby overcoming the problem of single evaluation dimension of developers in the prior art.

[0050] 2. By using the natural language large model LLM to perform semantic detection on the content of the text, effective filtering of false contributions is realized.

[0051] 3. A technology coverage score (Diversity(T)) is proposed to quantify the full-stack technical ability of the developer.

[0052] 4. A non-linear decay function (1 / log(1+T_r)) is designed to enhance the sensitivity of the recent time window by introducing a logarithmic function, so that the evaluation result is more in line with the characteristics of the rapid iteration of the open source community. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is the general flowchart of the technical scheme of the present application.

[0054] Figure 2 is an example of a knowledge graph. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples.

[0056] In order to facilitate a deeper understanding of the technical scheme of the present application, the related concepts will be explained in detail below.

[0057] GitHub: In this invention, GitHub is a Git-based code hosting platform that supports version control, collaborative development, and project management. It allows users to store and manage code, track modification history, and facilitate team collaboration. By creating repositories, developers can share and maintain project code, conduct issue tracking, and perform code reviews. In the embodiments of this invention, data will be obtained from the GitHub platform.

[0058] Large Language Model (LLM): In this invention, LLM specifically refers to a natural language large model, which is a deep learning model with hundreds of millions of parameters designed for natural language processing tasks. By utilizing vast amounts of text data and powerful computing resources, natural language large models can capture complex patterns and subtle differences in language.

[0059] Prompt: In this invention, the Prompt for a large model is a structured set of instructions designed to guide a large-scale language model to generate specific responses or perform specific tasks. The Prompt consists of input text, control parameters, and contextual information. By precisely configuring these elements, the content, style, and complexity of the model's output can be effectively controlled.

[0060] Knowledge Graph: A structured representation of knowledge, describing entities and their relationships through nodes and edges. Nodes represent entities (such as developers, code repositories), and edges represent relationships between entities (such as membership or association). Knowledge graphs can integrate multi-source data, provide rich semantic expression and complex reasoning capabilities, and support efficient querying and analysis. In this invention, a knowledge graph will be constructed based on the data obtained from GitHub.

[0061] PageRank: It is a link analysis algorithm used to measure the importance of web pages and rank them in search engine results. In this invention, the PageRank algorithm is used in the knowledge graph to evaluate the importance of nodes. The algorithm calculates the weight of each node by analyzing the link relationships between nodes. Through this mechanism, the technical solution of this invention can more accurately evaluate the importance of nodes in the knowledge graph.

[0062] This invention first obtains and cleanses data from GitHub, then uses a large language model (LLM) to process text data, scoring five dimensions: technology (T), communication (C), collaboration (Co), continuous contribution (D), and code quality (Q). Second, a knowledge graph containing entities such as developers, repositories, and organizations is constructed, and the PageRank algorithm is used to evaluate node importance. The contribution quality is calculated by combining the PR adoption rate, star number, and time decay coefficient, and the technology coverage score is calculated by the technology stack distribution. Finally, the LLM score and the knowledge graph score are combined to obtain the developer influence score.

[0063] Specifically, the open source community developer behavior influence assessment method based on a large model of the present application, as shown in Figure 1 includes the following steps.

[0064] Step S1: data acquisition.

[0065] The present application relates to a method for extracting specific data through a programming interface. Specifically, the present method uses the application programming interface (API) provided by GitHub to retrieve various records and statistical information in software development projects in a systematic manner. First, log in to the GitHub account and generate a Personal Access Token, (1) obtain the code submission record: send a GET request to the / repos / :owner / :repo / commits endpoint, where: owner and: repo are placeholders for the target repository owner and repository name, respectively. (2) Obtain Pull Request record: use the / repos / :owner / :repo / pulls endpoint to initiate a GET request to retrieve all Pull Requests. (3) Obtain Issue record: obtain all Issues data by calling the / repos / :owner / :repo / issues endpoint. (4) Obtain repository metadata: send a GET request to the / repos / :owner / :repo endpoint to obtain.

[0066] The above-mentioned code submission record, Pull Request record, Issue record and repository metadata provide a basic data source for the construction of large models and knowledge graphs.

[0067] For example, the text comments used by the subsequent large model mainly come from:

[0068] · The title, description and review comment content in the Pull Request;

[0069] · The title, description and discussion messages in the Issue;

[0070] · The commit information (commit message) in the submission record.

[0071] The data used by the knowledge graph is further extracted and modeled based on the above structured data, mainly including:

[0072] · Developer identity (determined by the submitter, PR / Issue creator, etc.);

[0073] · Repository and organization information (obtained from repository metadata);

[0074] • Technology stack information (analyzable from repository language annotations or dependency files);

[0075] • Collaboration events (such as commit, PR, Issue discussion, etc. behavior nodes and their relationships).

[0076] Therefore, large models rely on text class field content, while knowledge graphs model relationship networks based on structured metadata. Both can extract the required information from the data obtained in this step.

[0077] Step S2: Prompt construction.

[0078] The present application proposes a method for comprehensive analysis of text data (including code and textual comments) using natural language large models, and scoring according to multiple dimensions. This method scores the following five directions and sets appropriate weights to weight the scores to comprehensively evaluate the performance and contribution of developers. The detailed description of each score is as follows:

[0079] S201 Technology Score (T), the technology score aims to comprehensively evaluate the performance of the developer in code writing behavior and technical solution behavior. This score is considered from three dimensions: (1) Technical complexity: the complexity of the code and the difficulty of the problem solved. (2) Code innovation: the uniqueness and novelty of the solution. (3) Technical breadth: the diversity of technology stacks involved.

[0080] S202 Communication Score (C), the communication score is used to evaluate the effectiveness of the developer's expression and communication behavior in textual comments and discussions. This score is considered from three dimensions: (1) Clarity of expression: whether the description is clear and easy to understand. (2) Emotional tendency: the emotional color (positive / negative) in comments and discussions. (3) Logical rigor: whether the argument is reasonable and well-founded.

[0081] S203 Collaboration Score (Co), the collaboration score focuses on the behavior performance of the developer in project collaboration, especially his contribution to team cooperation. This score is considered from three dimensions: (1) Response time in PR discussion and issues comment. (2) Acceptance of others' opinions and feedback quality. (3) Whether to actively help other developers solve problems.

[0082] S204 Continuous Contribution Score (D), the continuous contribution score aims to evaluate the long-term contribution behavior and activity of the developer to the project. This score is considered from three dimensions: (1) Submission frequency and submission interval. (2) Long-term maintenance project activity. (3) Whether to regularly update documents and fix bugs.

[0083] The S205 Code Quality Rating (Q) is used to evaluate the quality and maintainability of the code written by developers. The rating considers three dimensions: (1) Code readability and the quality of comments ensure that the code is easy to understand and maintain. (2) Code consistency ensures a unified coding style and facilitates team collaboration. (3) Code reliability and stability.

[0084] S206 Prompt Design.

[0085] (1) Prompt for technical score (T).

[0086] Please provide a comprehensive evaluation of the following code snippet and assign a score to each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0087] 1. **Technical Complexity (TC)**:

[0088] - The logical complexity of the code implementation.

[0089] - The difficulty and challenge of solving the problem.

[0090] - The complexity of the data structures and algorithms used.

[0091] 2. **Code Innovation (CI)**:

[0092] - The uniqueness of the solution.

[0093] - Have any new methods or technologies been introduced?

[0094] - Is there a significant improvement compared to existing solutions?

[0095] 3. **Technology Breadth (TB)**:

[0096] - The number of different technologies, frameworks, or libraries used.

[0097] - The programming languages ​​and technical fields involved.

[0098] - Multidisciplinary knowledge demonstrated in the code.

[0099] {text}

[0100] Scoring criteria:

[0101] -Technical Complexity (TC):

[0102] 1- It is very simple and has almost no logical complexity.

[0103] 10 - Extremely complex, involving advanced data structures and algorithms, and solving very challenging problems.

[0104] - Code Innovation (CI):

[0105] 1 - No innovation, strictly follows common solutions.

[0106] 10 - Very innovative, introduces entirely new methods or technologies, significantly improves existing solutions.

[0107] - Technical Breadth (TB):

[0108] 1 - Only involves a single technology or framework.

[0109] 10 - Involves multiple technologies, frameworks, or libraries, demonstrating extensive interdisciplinary knowledge.

[0110] Please provide scores for each of the three sub-dimensions.

[0111] (2) Communication Score (C) - Prompt.

[0112] Please evaluate the following comments or discussions comprehensively and give scores for each sub-dimension on a scale of 1 to 10.

[0113] Consider the following three sub-dimensions:

[0114] 1. **Expression Clarity (EC)**:

[0115] - Is the description concise and clear?

[0116] - Are easy-to-understand language and terminology used?

[0117] - Are ambiguous or ambiguous expressions avoided?

[0118] 2. **Emotional Tendency (ES)**:

[0119] - Is the expressed emotion in the comments positive or negative?

[0120] - Does it contain encouragement, support, criticism, or complaints?

[0121] - Is the overall tone and attitude positive or negative?

[0122] 3. **Logical Rigor (EL)**:

[0123] - Are the arguments well-supported and reasoned?

[0124] - Is the reasoning process logical and free of obvious flaws?

[0125] - Are specific evidence or examples provided to support the point of view?

[0126] {text}

[0127] Scoring criteria:

[0128] - Clarity of expression (EC):

[0129] 1 - Very unclear, difficult to understand.

[0130] 10 - Very clear, easy to understand, without any ambiguity.

[0131] - Emotional tone (ES):

[0132] 1 - Very negative, full of criticism and complaints.

[0133] 10 - Very positive, full of encouragement and support.

[0134] - Logical rigor (EL):

[0135] 1 - Very incoherent, lacking in arguments and logic.

[0136] 10 - Very coherent, arguments well-founded and logically reasoned.

[0137] Please give a score for each of the three sub-dimensions.

[0138] (3) Collaborative score (Co) - Prompt.

[0139] Please evaluate the following PR discussion or issues comment and give a score for each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0140] 1. **Response time (RT)**:

[0141] - The speed at which the developer responds in the PR discussion or issues comment.

[0142] - Whether the response is timely and shows a positive communication attitude.

[0143] 2. **Opinion acceptance and feedback quality (AQ)**:

[0144] - The developer's acceptance of others' opinions.

[0145] - The quality of feedback, whether it provides constructive comments and suggestions.

[0146] - The ability to effectively solve the problem discussed.

[0147] 3. **Active help (AH)**:

[0148] - Whether the developer actively helps other developers solve problems.

[0149] - Whether it shows a team spirit, willingness to share knowledge and experience.

[0150] {text}

[0151] Scoring criteria:

[0152] - Response time (RT):

[0153] 1 - Very slow response, almost no timely replies.

[0154] 10 - Very quick response, always replies in time.

[0155] - Opinion receptivity and feedback quality (AQ):

[0156] 1 - Almost no acceptance of others' opinions, low feedback quality.

[0157] 10 - Highly receptive to others' opinions, provides high-quality and constructive feedback.

[0158] - Active help (AH):

[0159] 1 - Almost no active help to others, lack of team spirit.

[0160] 10 - Actively helps others, showing strong team spirit.

[0161] Please give scores according to the above three sub-dimensions respectively.

[0162] (4) Continuous contribution score (D) - Prompt.

[0163] Please evaluate the following developer's continuous contribution in the project and give scores for each sub-dimension within the range of 1 to 10. Consider the following three sub-dimensions:

[0164] 1. **Submission frequency and interval (SF)**:

[0165] - Frequency of developer's code submission.

[0166] - Whether the interval between submissions is reasonable and whether continuous contribution is maintained.

[0167] 2. **Long-term maintenance project activity (PA)**:

[0168] - Long-term activity of the developer in the project.

[0169] - Whether to continuously participate in the development and maintenance work of the project.

[0170] 3. **Document update and bug fixing (DB)**:

[0171] - Whether the developer regularly updates the project documentation.

[0172] - Whether actively fixing discovered bugs and maintaining the health of the codebase.

[0173] {text}

[0174] Scoring Criteria:

[0175] - Submission frequency and interval (SF):

[0176] 1 - Almost no submissions, or submissions are very irregular.

[0177] 10 - Frequent and regular submissions, maintaining consistent contributions.

[0178] - Activity in long-term maintenance of the project (PA):

[0179] 1 - Little to no involvement in long-term maintenance of the project, lacking activity.

[0180] 10 - Continuously active in project maintenance, actively participating in development and improvement.

[0181] - Documentation updates and bug fixes (DB):

[0182] 1 - Almost no updates to documentation or bug fixes, leading to outdated project documentation and accumulated bugs.

[0183] 10 - Regularly updating documentation and actively fixing bugs, maintaining the accuracy of project documentation and the health of the codebase.

[0184] Please provide scores for each of the three sub-dimensions as described above.

[0185] (5) Code Quality Score (Q) - Prompt.

[0186] Please evaluate the following code snippet and provide scores for each sub-dimension on a scale of 1 to 10. Consider the following three sub-dimensions:

[0187] 1. **Code Readability and Comment Quality (QR)**:

[0188] - Are variable and function names descriptive and easy to understand?

[0189] - Are there appropriate comments to explain complex logic or critical steps?

[0190] - Is the code structure clear and easy to follow?

[0191] 2. **Code Consistency (QC)**:

[0192] - Is there adherence to consistent code style (e.g., indentation, naming conventions)?

[0193] - Whether uniform design patterns and architectural principles are used.

[0194] - Whether duplicate code is avoided (DRY principle).

[0195] 3. Reliability and stability of the code (QRb):

[0196] - Whether there are sufficient input validation and exception handling mechanisms.

[0197] - Whether various boundary conditions and extreme cases are considered.

[0198] - Whether there are unit tests or other forms of automated testing covering critical functions.

[0199] {text}

[0200] Scoring criteria:

[0201] - Code readability and quality of comments (QR):

[0202] 1- Code is difficult to read, with almost no comments.

[0203] 10- Code is very readable, with detailed and helpful comments that aid in understanding the code logic.

[0204] - Code consistency (QC):

[0205] 1- Code style is inconsistent, with a lot of duplicate code.

[0206] 10- Code style is highly consistent, following uniform design patterns, with no duplicate code.

[0207] - Reliability and stability of the code (QRb):

[0208] 1- No input validation and exception handling, with lack of test coverage.

[0209] 10- Comprehensive input validation and exception handling mechanisms, with extensive test coverage.

[0210] Please give scores according to the above three sub-dimensions respectively.

[0211] Step S3: Large model scoring.

[0212] S301 Select Qwen2.5-72B-Instruct natural language large model for scoring. This model has strong natural language processing capabilities and extensive domain knowledge, suitable for complex scoring tasks.

[0213] S302 input text construction embeds the code data to be evaluated and text comments into the "{text}" positions in a pre-designed Prompt template. For example, for a technical score (T), the Prompt explicitly instructs the model to evaluate technical complexity, code innovation, and technical breadth.

[0214] S303 Feature Extraction and Preliminary Scoring: A large model is used to perform deep analysis of the input text, extracting feature information related to the scoring dimensions and generating preliminary scores (taking the technical score (T) process as an example): Technical Complexity (TC) Score: The model generates a technical complexity score (a) based on factors such as code structure and algorithm difficulty; Code Innovation (CI) Score: The model generates a code innovation score (b) based on the uniqueness and novelty of the solution; Technical Breadth (TB) Score: The model generates a technical breadth score (c) based on the diversity of the technology stack involved.

[0215] The S304 weighted and comprehensive score assigns weights to each dimension based on their importance. In this example, assuming all dimensions have the same weight, the average value is taken directly. The final technical score (T) = (a+b+c) / 3 is obtained by averaging the three scores.

[0216] S305 repeats S302-S304 to generate the technical score (T), communication score (C), collaboration score (Co), continuous contribution score (D), and code quality score (Q).

[0217] Step S4: Summarize the scores based on the large model.

[0218] In this invention, to comprehensively evaluate the performance of developers across various behaviors, a systematic method is employed to aggregate and weight the scores from each dimension. This method ensures that performance in all aspects receives a comprehensive and fair evaluation, and provides a scientific basis for subsequent project management and team collaboration.

[0219] S401 weight settings assign appropriate weights based on the importance of each scoring dimension and then sum the scores using a weighted average. Specific weight settings are as follows (can be adjusted flexibly according to actual circumstances): Technical score weight (ω) T =0.3, Communication Score (ω) C =0.2, Collaboration score (ω) Co =0.2, Continuous contribution score (ω) D =0.15, Code quality score (ω) Q =0.15.

[0220] S402 weighted score summary, S LLM =ω T ·T+ω C ·C+ωCo • CO + ω D • D + ω Q • Q.

[0221] Step S5: Knowledge graph construction.

[0222] In the present application, in order to comprehensively and systematically represent the complex relationship between developers, repositories, organizations, technology stacks (including programming languages and frameworks) and collaboration events (such as commits, Pull Requests, Issue discussions, etc.), a knowledge graph construction method based on entity nodes is used. This method forms a structured and easily queried knowledge graph by taking the above elements as entity nodes. Specific entity nodes include developers, repositories, organizations, technology stacks (programming languages, frameworks), collaboration events (commits, PRs, Issue discussions, etc.), as shown in the example graph. Figure 2

[0223] Step S6: Scoring developers based on knowledge graph.

[0224] S601 In the present application, in order to evaluate the importance and influence of developers in the network, the PageRank algorithm based on the knowledge graph is used to calculate the PageRank(d) of the developer node, which quantifies its relative importance in the network.

[0225] S602 Contribution quality score: Quality(T) calculation. In the present application, in order to evaluate the contribution quality of developers to code repositories, a comprehensive scoring method based on multiple factors is proposed. This score describes the quality of the contribution and is calculated through a specific formula. The specific calculation method is as follows:

[0226]

[0227] Where d represents the target developer node, r represents the code repository, R d represents the set of all repositories that developer d participates in, A r represents the PR adoption rate of developer d in repository r, S r represents the number of stars of repository r, T r represents the time of the latest contribution of developer d in repository r (unit: month), for example: 1 for this month, 2 for the previous month, T represents the relative time of the latest contribution.

[0228] In order to strengthen the influence of recent active records and weaken the influence of historical long-term active records, a time decay coefficient This coefficient makes the weight of recent active contributions higher.

[0229] ​S603 Technology coverage score: Diversity(T) calculation. In the present application, in order to evaluate the technology stack distribution of the developer and the comprehensiveness of his skills, a method for calculating the technology coverage score is proposed. The higher the score, the more dispersed the technology stack distribution and the more comprehensive the developer's skills. The specific calculation method is as follows:

[0230]

[0231] Where d represents the target developer node, t represents the technology stack (such as programming language), T d represents the set of technology stacks used by developer d, p t represents the frequency of use of technology stack t in all contributions of developer d, for example: the proportion of times using Java.

[0232] Step S7: Score the developer's behavior based on the knowledge graph.

[0233] In the present application, in order to comprehensively evaluate the comprehensive ability and importance of the developer, multiple key indicators are combined, including PageRank value, contribution quality score and skill coverage score, to calculate the importance score of the developer node. Combined with the PageRank(d) obtained in S601, the contribution quality score Quality(T) obtained in S602 and the technology coverage score Diversity(T) obtained in S603, the importance score of the developer node based on the knowledge graph is calculated, and the calculation method is as follows:

[0234]

[0235] Where: α, β, γ represent weight coefficients, and α = 0.5, β = 0.3, γ = 0.2; PageRank(d), Quality(T), Diversity(T) are calculated in step S6, represents the normalized processing of the contribution quality score Quality(T).

[0236] Step S8: Comprehensive score.

[0237] In the present application, in order to comprehensively evaluate the comprehensive ability and influence of the developer, a final comprehensive score method combining the large model score S LLM and the knowledge graph-based score S KG is proposed. This method introduces a weight coefficient θ to flexibly adjust the contribution proportion of the two score sources, ensuring that the score result can adapt to different data completeness and application scenarios. The specific calculation method is as follows:

[0238] S Final = θ·S LLM +(1-θ)SKG

[0239] Wherein: S LLM is the score generated based on the large model in step S4, reflecting the technical level, communication ability, collaboration, continuous contribution and code quality of the developer; S KG is the score calculated based on the knowledge graph in step S7, including the PageRank value, contribution quality score and technical coverage score; theta is a weight coefficient, which can be adjusted comprehensively according to the data integrity of the knowledge graph and the use scene, and is selected as 0.5 by default, which determines the relative importance of the large model score and the knowledge graph score in the final comprehensive score. If the knowledge graph data is relatively complete and accurate, the value of theta can be appropriately reduced, and the weight of S KG is increased; in the application scene where the technical ability of the developer needs to be emphasized, such as technical review or code review, the value of theta can be increased to increase the weight of S LLM .

[0240] In summary, the application uses a multi-dimensional LLM scoring model and a dynamic Prompt design, which analyzes the semantic analysis of the developer's text data (such as Issue discussion, PR description, code, etc.) through a large language model (LLM), and generates fine-grained indicators of technical score (T), communication score (C), collaboration score (Co), continuous contribution score (D) and code quality score (Q) in combination with Prompt design. Compared with the traditional open source community influence evaluation which mainly depends on single statistical indicators such as code submission quantity (such as Commits, PR quantity), the application models the behavior characteristics of the developer from multiple dimensions such as technical depth, communication efficiency and collaboration willingness through the semantic understanding ability of LLM, deeply mines the deep semantics in natural language data, breaks through the limitations of traditional statistical indicators, and improves the accuracy and applicability of the score.

[0241] The application designs a cross-modal fusion scoring architecture of large model LLM and knowledge graph, and designs a double-channel scoring fusion function S Final =θ·S LLM +(1-θ)S KG . The double-channel scoring can comprehensively score the influence of the developer, and the weight parameter theta can be dynamically adjusted according to different emphases according to the actual use scene, so that the influence of the developer can be flexibly evaluated according to the actual use scene. At the same time, the fusion model can enhance the robustness of the model and avoid the deviation of a single model.

[0242] The application designs a targeted Prompt template to guide the LLM to fully understand the scoring standard, generate standardized scoring output from different angles, and improve the objectivity of the evaluation.

[0243] The application designs a contribution quality formula The fusion time decay factor is fused with the warehouse influence parameter through a nonlinear decay function The recent active contribution weight is reinforced.

[0244] The above embodiments are provided merely for the purpose of description of the present application and are not intended to limit the scope of the present application. The scope of the present application is defined by the appended claims. Various equivalents substitutions and modifications which do not depart from the spirit and principles of the present application are intended to be encompassed within the scope of the present application.

Claims

1. A method for evaluating the influence of open-source community developer behavior based on a large model, characterized in that, The method includes: Obtain records and statistics of developers in software development projects; A large model is used to perform multi-dimensional analysis on text data in records and statistical information, and the scores of each dimension are combined to obtain the developer's first score; the dimensions include: technology, communication, collaboration, continuous contribution and code quality. A knowledge graph is constructed based on records and statistical information, and a second rating for the developer is obtained based on this knowledge graph; Based on the first score and the second score, the evaluation result of the developer's behavioral influence is obtained; This includes using large models to perform technical dimension analysis on textual data in records and statistical information, including: Generate a technical score Prompt; wherein, the content of the technical score Prompt includes: a technical dimension task description, text data in the records and statistical information, and technical dimension scoring criteria, wherein the technical dimension task description defines technical sub-dimensions, including: technical complexity, code innovation, and technical breadth; The technical scoring prompt guides a large model to generate technical complexity scores, code innovation scores, and technical breadth scores for the text data. Based on the technical complexity score, code innovation score, and technical breadth score of the text data, a technical dimension score is obtained; Generate a communication score Prompt; wherein the content of the communication score Prompt includes: a description of the communication dimension task, text data in the records and statistics, and the communication dimension scoring criteria. The communication dimension task description defines communication sub-dimensions, which include: clarity of expression, emotional tendency, and logical rigor. The communication scoring prompt guides a large model to generate scores for clarity of expression, sentiment, and logical rigor of the text data. The communication dimension score is obtained based on the text data's clarity of expression, sentiment, and logical rigor. Generate a collaboration score Prompt; wherein the content of the collaboration score Prompt includes: a collaboration dimension task description, text data in the records and statistical information, and collaboration dimension scoring criteria. The collaboration dimension task description defines collaboration sub-dimensions, which include: response time, opinion acceptance and feedback quality, and proactive assistance. The collaborative scoring prompt guides a large model to generate response time scores, opinion acceptance and feedback quality scores, and proactive help scores for this text data. Based on the response time score, opinion acceptance and feedback quality score, and proactive assistance score of the text data, a collaboration dimension score is obtained. Generate a continuous contribution score Prompt; wherein, the content of the continuous contribution score Prompt includes: a description of the continuous contribution dimension task, text data in the records and statistics, and a continuous contribution dimension scoring standard. The continuous contribution dimension task description defines continuous contribution sub-dimensions, which include: submission frequency and submission interval, activity of long-term maintenance projects, and document updates and bug fixes. The continuous contribution score prompt guides the large model to generate scores for the text data's submission frequency and submission interval, the activity score of long-term maintenance projects, and the score for document updates and bug fixes. Based on the submission frequency and interval of the text data, the activity score of the long-term maintenance project, and the score of document updates and bug fixes, a continuous contribution dimension score is obtained. Generate a code quality score Prompt; wherein, the content of the code quality score Prompt includes: a description of the code quality dimension task, text data in the records and statistics, and code quality scoring criteria. The code quality dimension task description defines code quality sub-dimensions, including: code readability and comment quality, code consistency, and code reliability and stability. The code quality scoring prompt guides a large model to generate code readability and annotation quality scores, code consistency scores, and code reliability and stability scores for this text data. Based on the code readability and comment quality scores, code consistency scores, and code reliability and stability scores of the text data, a code quality dimension score is obtained; The nodes in the knowledge graph include: developer nodes, repository nodes, organization nodes, technology stack nodes, and collaboration event nodes; The process of obtaining the developer's second rating based on this knowledge graph includes: The PageRank algorithm is used to calculate the importance score of the developer node; Analyze the knowledge graph to obtain a quality score of developers' contributions to the code repository. and developer's technical coverage score Where d represents the target developer node, r represents the code repository, and R d A represents the set of all repositories in which developer d participates. r S represents the PR adoption rate of developer d in repository r. r T represents the number of stars in warehouse r. r This represents the time when developer d last contributed to repository r, where t represents the technology stack, and T... d p represents the set of technologies used by developer d. t This indicates the frequency of use of the technology stack t in all contributions made by developer d; The developer's second score is obtained based on the importance score of the developer node, the quality score of the developer's contribution to the code repository, and the developer's technical coverage score.

2. A system for evaluating the influence of open-source community developer behavior based on a large model, characterized in that, The system includes: The information acquisition module is used to acquire records and statistical information of developers in software development projects; The first scoring module is used to perform multi-dimensional analysis of text data in records and statistical information using a large model, and to combine the scores of each dimension to obtain the developer's first score; wherein, the dimensions include: technology, communication, collaboration, continuous contribution and code quality. The second rating calculation module is used to construct a knowledge graph based on records and statistical information, and obtain the developer's second rating based on the knowledge graph; The evaluation result generation module is used to obtain the evaluation result of the developer's behavioral influence based on the first score and the second score; This includes using large models to perform technical dimension analysis on textual data in records and statistical information, including: Generate a technical score Prompt; wherein, the content of the technical score Prompt includes: a technical dimension task description, text data in the records and statistical information, and technical dimension scoring criteria, wherein the technical dimension task description defines technical sub-dimensions, including: technical complexity, code innovation, and technical breadth; The technical scoring prompt guides a large model to generate technical complexity scores, code innovation scores, and technical breadth scores for the text data. Based on the technical complexity score, code innovation score, and technical breadth score of the text data, a technical dimension score is obtained; Generate a communication score Prompt; wherein the content of the communication score Prompt includes: a description of the communication dimension task, text data in the records and statistics, and the communication dimension scoring criteria. The communication dimension task description defines communication sub-dimensions, which include: clarity of expression, emotional tendency, and logical rigor. The communication scoring prompt guides a large model to generate scores for clarity of expression, sentiment, and logical rigor of the text data. The communication dimension score is obtained based on the text data's clarity of expression, sentiment, and logical rigor. Generate a collaboration score Prompt; wherein the content of the collaboration score Prompt includes: a collaboration dimension task description, text data in the records and statistical information, and collaboration dimension scoring criteria. The collaboration dimension task description defines collaboration sub-dimensions, which include: response time, opinion acceptance and feedback quality, and proactive assistance. The collaborative scoring prompt guides a large model to generate response time scores, opinion acceptance and feedback quality scores, and proactive help scores for this text data. Based on the response time score, opinion acceptance and feedback quality score, and proactive assistance score of the text data, a collaboration dimension score is obtained. Generate a continuous contribution score Prompt; wherein, the content of the continuous contribution score Prompt includes: a description of the continuous contribution dimension task, text data in the records and statistics, and a continuous contribution dimension scoring standard. The continuous contribution dimension task description defines continuous contribution sub-dimensions, which include: submission frequency and submission interval, activity of long-term maintenance projects, and document updates and bug fixes. The continuous contribution score prompt guides the large model to generate scores for the text data's submission frequency and submission interval, the activity score of long-term maintenance projects, and the score for document updates and bug fixes. Based on the submission frequency and interval of the text data, the activity score of the long-term maintenance project, and the score of document updates and bug fixes, a continuous contribution dimension score is obtained. Generate a code quality score Prompt; wherein, the content of the code quality score Prompt includes: a description of the code quality dimension task, text data in the records and statistics, and code quality scoring criteria. The code quality dimension task description defines code quality sub-dimensions, including: code readability and comment quality, code consistency, and code reliability and stability. The code quality scoring prompt guides a large model to generate code readability and annotation quality scores, code consistency scores, and code reliability and stability scores for this text data. Based on the code readability and comment quality scores, code consistency scores, and code reliability and stability scores of the text data, a code quality dimension score is obtained; The nodes in the knowledge graph include: developer nodes, repository nodes, organization nodes, technology stack nodes, and collaboration event nodes; The process of obtaining the developer's second rating based on this knowledge graph includes: The PageRank algorithm is used to calculate the importance score of the developer node; Analyze the knowledge graph to obtain a quality score of developers' contributions to the code repository. and developer's technical coverage score Where d represents the target developer node, r represents the code repository, and R d A represents the set of all repositories in which developer d participates. r S represents the PR adoption rate of developer d in repository r. r T represents the number of stars in warehouse r. r This represents the time when developer d last contributed to repository r, where t represents the technology stack, and T... d p represents the set of technologies used by developer d. t This indicates the frequency of use of the technology stack t in all contributions made by developer d; The developer's second score is obtained based on the importance score of the developer node, the quality score of the developer's contribution to the code repository, and the developer's technical coverage score.

Images