Artificial intelligence contribution degree automatic measurement method, electronic device, and program product

Abstract

This disclosure provides an automatic measurement method for artificial intelligence contribution, an electronic device, a readable storage medium, and a computer program product. The method includes: acquiring a task set from a software delivery project; determining the execution type of each task in the task set based on a preset task automation level classification standard; for tasks with a fully automated execution type, driving the corresponding code generation agent to execute and collecting process data generated by the code generation agent during task execution; performing multi-dimensional measurements on the collected process data to generate a task-level contribution index for fully automated tasks; calculating a contribution adjustment factor based on the execution complexity analyzed from operation records, and determining the equivalent human workload corresponding to the task based on the basic contribution and the contribution adjustment factor; and summarizing the task-level contribution indices of all fully automated tasks within the software delivery project to generate a project-level contribution index.

Description

Technical Field

[0001] This disclosure relates to an automatic measurement method for artificial intelligence contribution, an electronic device, and a program product. Background Technology

[0002] With advancements in large language models for code generation and natural language understanding, model-based code generation agents have acquired the ability to autonomously understand development requirements, invoke development tools, and generate executable code. This technological innovation is reshaping the traditional software development process, expanding the executor of software development tasks from a single human engineer to a model where both "human" and "AI agents" coexist. However, existing software project management systems and development process architectures still have significant technical shortcomings in adapting to this technological change, mainly in the following aspects: First, existing software project management systems lack a differentiated execution scheduling mechanism based on task characteristics. In traditional software development management architectures, task scheduling logic typically assumes that all development tasks are executed by human engineers. Although existing AI programming aids are widely used in the coding phase, these tools exist only as auxiliary means at the individual engineer level and are not integrated into the core task scheduling logic of the system. The system cannot automatically identify and determine the appropriate level of automation execution based on the technical characteristics of the task during the task creation phase. This results in the inability to route tasks suitable for AI automation to the automation execution channel, and the inability to achieve standardized routing of different execution modes. This lack of a scheduling mechanism means that AI's automation execution capabilities are outside the standardized project management process, preventing system-level resource coordination and process control.

[0003] Second, existing technologies lack a quantifiable model for measuring AI contributions based on objective process data. Current software development metrics primarily rely on manually entered work hours or statistics based on static lines of code to assess workload. This approach fails to objectively reflect the actual contribution of AI agents during task execution. On one hand, simple line-of-code statistics cannot differentiate the actual difficulty of a task; tasks with the same amount of code may have vastly different underlying execution complexities. Existing metrics algorithms lack mechanisms to differentiate and appropriately weight tasks based on varying levels of difficulty, leading to distorted results. On the other hand, existing systems do not structurally collect and analyze the objective data generated during agent execution, making it impossible to establish a quantitative mapping from objective execution data to standardized workload metrics. This prevents project clients from understanding the actual output efficiency of AI based on objective technical indicators and from providing traceable evidence of AI participation for compliance matters.

[0004] Third, the existing development process architecture lacks standardized integration and quality assurance pipelines for AI-generated code. In traditional integrated development environments and delivery processes, code generation and submission are primarily controlled manually by engineers. When the capability of AI to automatically generate code is introduced, the existing system architecture lacks standardized processing and verification procedures for AI-generated code. AI-generated code is often directly mixed into manually written code repositories, lacking clear source marking and independent quality control. The system cannot automatically identify and extract AI-generated code, nor trigger subsequent verification and confirmation processes, resulting in the inability to effectively manage the quality risks of AI-generated code in a closed-loop manner. This lack of a process mechanism makes the incorporation of AI-generated code lack traceability and quality certainty, affecting the overall reliability of the deliverables.

[0005] In summary, how to achieve differentiated execution scheduling based on task characteristics at the architecture level of a software project management system, construct an AI contribution measurement model based on objective execution process data, and establish a standardized integration and quality assurance pipeline for AI-generated code are the technical problems that urgently need to be solved in the current software development field. Summary of the Invention

[0006] This disclosure provides an automatic measurement method for artificial intelligence contribution, an electronic device, a storage medium, and a program product.

[0007] According to one aspect of this disclosure, an automatic measurement method for artificial intelligence contribution is provided, comprising: Obtain the task set in the software delivery project; Based on a preset classification standard for the degree of task automation, the execution type of each task in the task set is determined, and the execution type includes at least the fully automated type that is executed independently by an artificial intelligence agent; For tasks of the fully automated type, the corresponding code generation agent is driven to execute, and process data generated by the code generation agent during task execution is collected. The process data includes at least operation records and code change information. The collected process data is measured in multiple dimensions to generate a task-level contribution index for the fully automated type of task; wherein, the multi-dimensional measurement includes a basic contribution calculated based on the code change information; The contribution adjustment factor is calculated based on the execution complexity analyzed from the operation records, and the equivalent manual workload corresponding to the task is determined according to the basic contribution and the contribution adjustment factor. Summarize the task-level contribution metrics of all fully automated tasks within the software delivery project to generate project-level contribution metrics.

[0008] Optionally, the execution of the code-generating agent corresponding to the driver includes: The fully automated tasks are dispatched to the background task queue by the main project management process to drive the code generation agent to execute in an isolated workspace environment, so that the background task execution process of the code generation agent and the foreground operation process of the main project management process can be carried out in parallel.

[0009] Optionally, the method further includes: For the fully automated type of task, lifecycle state management is performed based on preset state transition rules and state set, wherein the state set includes at least one of the following: pending state, in progress state, pending inspection state, and completed state. Upon receiving a preset trigger event, the task is driven to transition from the current state to the next state according to the state transition rules.

[0010] Optionally, after the code-generating agent corresponding to the driver is executed, the method further includes: The execution results output by the code generation agent are subjected to structured parsing to extract the generated code blocks; The extracted code blocks are marked as pending inspection and pushed to the manual inspection node; In response to receiving a pass inspection instruction, the code block is added to the project deliverables, and the task status is transitioned from pending inspection to completed. Alternatively, in response to receiving an inspection rejection instruction, the task status can be switched from pending inspection back to pending.

[0011] Optionally, before performing multi-dimensional measurements on the collected process data, the method further includes: The validity of the process data is verified. If the verification results show that the code-generating agent has not produced any effective code changes, then the AI ​​contribution index of the task will be set to zero.

[0012] Optionally, the validity verification of the process data includes: Determine whether there are any preset write or edit operations in the operation record; If it does not exist, then it is determined that no valid code change has been generated.

[0013] Optionally, the basic contribution calculated based on the code change information includes: Extract the number of newly added lines and the number of deleted lines from the code change information, and determine the net number of newly added lines of code as the difference between the number of newly added lines and the number of deleted lines. Based on the net increase in lines of code and the preset baseline code productivity, the basic equivalent manual workload is calculated as the basic contribution.

[0014] Optionally, the calculation of the contribution adjustment factor based on the execution complexity analyzed from the operation records includes: Pattern recognition is performed on the operation records to extract complexity features, including iteration depth to characterize the number of "modify-verify" loops and context breadth to characterize the number of different code files involved. Based on the iteration depth and the context breadth, a complexity weight coefficient is calculated as the contribution adjustment factor.

[0015] Optionally, determining the equivalent manual workload corresponding to the task based on the basic contribution and the contribution adjustment factor includes: Multiply the basic contribution by the complexity weighting coefficient to obtain the weighted equivalent human workload; The weighted equivalent human workload is compared with the preset upper limit of the estimated human workload for the task, and the smaller one is taken as the final task-level equivalent human workload for the task.

[0016] Optionally, the step of summarizing the task-level contribution metrics of all fully automated type tasks within the software delivery project to generate project-level contribution metrics includes: The total amount of manual labor saved in the project is obtained by summing up the equivalent manual workload of each fully automated type of task. Based on the total amount of labor saved in the project and the preset unit amount of labor, calculate the equivalent index of the total amount of labor saved in the project; The project efficiency multiplication factor is calculated based on the total saved labor workload and the actual total labor workload of the project.

[0017] Optionally, after generating the project-level contribution metric, the method further includes: One or more of the following can be displayed to the client through a data visualization dashboard: the total amount of labor saved in the project, the equivalent index of the total amount of labor saved in the project, and the efficiency multiplication factor of the project.

[0018] Optionally, the tasks in the task set may also include at least one of the following: assisted tasks completed by AI in collaboration with the user, and purely manual tasks completed independently by the user.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a method according to any embodiment of this disclosure.

[0020] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the method of any embodiment of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method of any embodiment of this disclosure. Attached Figure Description

[0022] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0023] Figure 1 This is a schematic diagram of the overall process of an automatic measurement method for artificial intelligence contribution according to one embodiment of the present disclosure.

[0024] Figure 2 This is a flowchart illustrating the code extraction and inspection steps of one embodiment of this disclosure.

[0025] Figure 3 This is a flowchart illustrating the validity verification of process data according to one embodiment of this disclosure.

[0026] Figure 4 This is a schematic diagram of a process for calculating the basic contribution based on the code change information, according to one embodiment of this disclosure.

[0027] Figure 5 This is a schematic diagram of a process for calculating the contribution adjustment factor based on the execution complexity analyzed from the operation records, according to one embodiment of this disclosure.

[0028] Figure 6 This is a schematic diagram of a process for determining the equivalent manual workload corresponding to the task based on the basic contribution level and the contribution adjustment factor, according to one embodiment of this disclosure.

[0029] Figure 7 This is a flowchart illustrating the process of generating project-level contribution metrics by summarizing the task-level contribution metrics of all fully automated type tasks within the software delivery project, as one embodiment of this disclosure.

[0030] Figure 8This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0032] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 This is a schematic diagram of the overall process of an automatic measurement method for artificial intelligence contribution according to one embodiment of the present disclosure.

[0034] In some embodiments of this disclosure, the automatic measurement method M100 for artificial intelligence contribution includes steps S102 to S112.

[0035] S102, Obtain the set of tasks in the software delivery project.

[0036] A software delivery project is an engineering project in which a software development team delivers a software product to a customer. During project management, the overall project requirements are broken down into multiple specific, actionable development tasks, which together form a task set. Each task typically includes at least one of the following: task description information, technical implementation requirements, and expected delivery criteria.

[0037] S104. Based on the preset task automation level classification standard, determine the execution type of each task in the task set.

[0038] The preset classification criteria for the degree of task automation can be classification rules established in advance based on technical characteristics such as the degree of standardization of the task, logical complexity, and whether it is suitable to be completed independently by artificial intelligence.

[0039] The execution types include at least fully automated types executed independently by an AI agent. Optionally, the execution types also include assisted types completed in collaboration between AI and the user, and purely manual types completed independently by the user.

[0040] It should be noted that the code generation agent is built upon a large language model and possesses the ability to autonomously understand development requirements, invoke development tools, and generate executable code. The agent can autonomously complete coding tasks within an isolated workspace by operating tools such as file reading and writing, command execution, and code searching.

[0041] In this step, when creating or assigning tasks, the main project management process determines the degree of automation for each task based on the aforementioned classification criteria and labels each task with its corresponding execution type. As a result, tasks with different execution types will be routed to different execution channels in subsequent processes, achieving differentiated scheduling.

[0042] S106. For tasks with a fully automated execution type, drive the corresponding code generation agent to execute, and collect the process data generated by the code generation agent during task execution.

[0043] Specifically, process data refers to objectively recordable and traceable data generated by the code generation agent during the autonomous execution of development tasks. Process data includes at least operation records and code change information. Operation records are tool call logs generated by the code generation agent during task execution, recording which tools were called, in what order, and the input / output information for each call. Code change information is statistical information on changes caused by the code generation agent's addition, deletion, or modification of code files, including at least the number of newly added and modified lines of code.

[0044] Based on this step, the workload is no longer estimated by relying on manually entered work hour data. Instead, objective data on the execution process of the intelligent agent is collected in real time, providing an objective and traceable data foundation for subsequent contribution measurement.

[0045] S108. Perform multi-dimensional measurements on the collected process data to generate task-level contribution indicators for fully automated tasks; among which, the multi-dimensional measurements include the basic contribution calculated based on code change information.

[0046] Specifically, the base contribution rate reflects the amount of code output generated by the code generation agent in the task. In one implementation, the number of newly added and deleted lines of code is extracted from code change information. The difference between the number of newly added and deleted lines is determined as the net new lines of code. Then, based on the net new lines of code and a preset baseline code productivity, the basic equivalent human workload can be calculated using the formula Tbase=Lnet / Rbase, which serves as the base contribution rate. Here, Lnet is the net new lines of code, and Rbase is the baseline code productivity. The baseline code productivity can refer to the average number of effective lines of code that a standard human engineer can produce per unit time. This parameter can be preset based on industry statistics or team historical data, both of which fall within the protection scope of this disclosure.

[0047] S110. Calculate the contribution adjustment factor based on the execution complexity analyzed from the operation records, and determine the equivalent manual workload corresponding to the task based on the basic contribution and the contribution adjustment factor.

[0048] It's important to note that measuring AI's contribution solely by code output is insufficient. Tasks with the same amount of code can have vastly different actual execution complexities. For example, a complex task requiring multiple modifications, debugging, and verifications might have the same net number of new lines of code as a task that can be completed in one go, but the former consumes significantly more AI inference and operational resources. Therefore, this step introduces a contribution adjustment factor based on execution complexity to more accurately reflect the actual contribution of AI.

[0049] Specifically, pattern analysis is performed on the collected operation records to extract feature information reflecting the complexity of task execution, and contribution adjustment factors are calculated based on these feature information.

[0050] In one implementation, the complexity features include iteration depth, which characterizes the number of "modify-verify" loops, and context breadth, which characterizes the number of different code files involved. The complexity weight coefficient, i.e., the contribution adjustment factor, is calculated based on the iteration depth and context breadth.

[0051] Furthermore, the base contribution and contribution adjustment factor are combined to obtain the equivalent human workload corresponding to the task. The equivalent human workload refers to the work completed by the AI ​​agent in performing the task, which can be quantified into the workload (e.g., the required working hours) required for a standard human engineer to complete the same task.

[0052] S112. Summarize the task-level contribution metrics of all fully automated tasks within the software delivery project and generate project-level contribution metrics.

[0053] Specifically, after measuring all fully automated tasks, the AI ​​contribution across the entire project is summarized. The summarization method includes, but is not limited to, adding up the task-level equivalent human workload of all tasks to obtain the total saved human workload Htotal; based on the total saved human workload and a preset unit human workload, an equivalent index of the total saved human workload is calculated; based on the total saved human workload and the actual total human workload of the project, the project efficiency multiplier is calculated according to the formula M=Ptotal / (Ptotal-Htotal) (when Ptotal>Htotal and Ptotal>0), where Ptotal is the total estimated project man-hours; when Htotal=0 or Ptotal=0, M=1.0, and an upper limit M≤M_max is set. Further, the cost savings can be calculated according to Csave=Htotal×(Rhuman-Rai), where Rhuman is the hourly cost of human labor and Rai is the hourly cost of AI. In addition, the distribution of task types (the proportion of fully automated, assisted, and purely manual types) can be statistically analyzed as one of the project-level contribution indicators, ultimately generating a project-level contribution indicator that reflects the overall value generated by AI in the entire project.

[0054] This implementation addresses the technical problem of existing software project management systems being unable to quantify the contribution of AI based on objective process data. It integrates the actual code output and execution complexity of the AI ​​agent into a unified measurement model, ensuring that the measurement results accurately reflect the task execution difficulty and actual output efficiency of the AI. Furthermore, this implementation incorporates the scheduling and measurement of AI execution capabilities into a standardized project management process, providing project clients with objective and traceable data for assessing AI participation.

[0055] This implementation method also has the following alternative implementation methods: (1) Replacement of task classification method: The three-level task classification can be replaced by a five-level classification (adding two levels: "AI generated and manually modified" and "AI reviewed") or simplified to a two-level classification (AI executable and AI non-executable).

[0056] (2) Replacement of code change measurement method: Change statistics based on the number of lines of code can be replaced by a measurement method based on the number of changes in the Abstract Syntax Tree (AST) nodes, or a simplified statistical method based on the number of changed files.

[0057] (3) Replacement of equivalent time estimation benchmark: The benchmark code productivity can be replaced by a weighted estimation method based on task cyclomatic complexity or file dependency depth, or a regression model based on the human time of similar historical tasks.

[0058] (4) Replacement of operation sequence complexity features: The two features of iteration depth and context breadth can be replaced by analysis methods based on information entropy of tool call sequence or tool call time interval distribution.

[0059] (5) Replacement of AI code review method: The manual review process can be replaced by automated review (static code analysis + automatic unit test judgment) or hybrid review (automated pre-review + manual final review).

[0060] All the above alternative implementation methods fall within the protection scope of this disclosure, and those skilled in the art can make substitutions or modifications without departing from the spirit of this disclosure.

[0061] For example, a client commissioned the development of an enterprise back-end management system that includes functions such as user management, access control, and data reporting. The project manager broke down the requirements into 20 development tasks. Among them, 8 standardized tasks were set as fully automated, 6 customized logic tasks were set as auxiliary, and 6 core business rule tasks were set as purely manual.

[0062] The system triggers background agents to execute eight fully automated task types. Taking the "user CRUD interface" task as an example: during the agent's execution, a total of 28 tool call sequences were generated (12 file writes, 5 file edits, 3 command executions, 6 file reads, and 2 code searches), identifying two "modify-verify" loop patterns and reading four different files as context. After execution, a net increase of 305 lines of code was detected. The system converts this into basic equivalent working time, then multiplies it by a complexity weighting coefficient calculated based on iteration depth and context breadth to obtain the final contribution.

[0063] Compared to another task, the "data list page" task (which adds 280 lines, has no iterative loops, and reads only one file), its weight coefficient is lower, and its contribution is correspondingly smaller. This clearly shows that the complexity weighting mechanism can effectively distinguish tasks of different difficulties: with the same amount of code, complex tasks that have undergone multiple rounds of debugging receive a higher contribution evaluation, more realistically reflecting the actual time saved by AI.

[0064] In this disclosure, because the system categorizes tasks according to their degree of AI automation, it can automatically identify which tasks are performed by AI, automatically record the time saved by AI on each task, and then aggregate and calculate AI contribution metrics at the project level, presenting them transparently to the client. Without task categorization, the system cannot distinguish between AI-executed tasks and human-executed tasks, and contribution measurement would be impossible. This disclosure solves the problems of opaque AI participation, subjective contribution measurement, and lack of standardized processes in existing technologies through this causal combination of task categorization-driven differentiated execution and automatic contribution measurement.

[0065] Compared with existing technologies, this disclosure forms a complete technical chain of "task classification → differentiated execution → process data collection → code difference analysis → equivalent working time calculation → project-level transparent display". This disclosure also transforms the agent execution process data from internal logs into a quantifiable objective data source of contribution, and distinguishes tasks of different difficulties through complexity features (iteration depth and context breadth), solving the problem that existing technologies rely on subjective estimation or fixed coefficients and cannot reflect the actual execution difficulty of tasks.

[0066] In some embodiments of this disclosure, the execution of the code generation agent corresponding to the driver in step S106 includes: The main project management process dispatches fully automated tasks to a background task queue to drive the code generation agent to execute in an isolated workspace environment, allowing the background task execution process of the code generation agent to run in parallel with the foreground operation process of the main project management process.

[0067] Specifically, the background task queue is a task buffer component situated between the main project management workflow and the code generation agent. After determining the task's execution type, the main project management workflow encapsulates the task as a task message, writes it to the background task queue, and returns without waiting for the execution result. The isolated workspace environment is a runtime environment independently allocated to the code generation agent, isolated from the main project codebase. Within this environment, the agent autonomously completes coding tasks using tools such as file reading and writing, command execution, and code searching. All its operations are fully recorded, and the generated code is not mixed into the final deliverables before being checked.

[0068] Through the mechanism described above in this embodiment, this embodiment solves the technical problem in the traditional synchronous execution mode where the main project management process is blocked while waiting for AI tasks to complete, and cannot respond to user operations in a timely manner. It achieves decoupling between foreground task scheduling and background intelligent agent execution, thereby improving the system's concurrent processing capabilities. It should be noted that the above asynchronous execution method is a preferred embodiment of this disclosure. In other embodiments, the main project management process can also drive the code-generated intelligent agent to execute through other synchronous or asynchronous scheduling methods, all of which fall within the protection scope of this disclosure.

[0069] In some embodiments of this disclosure, the automatic measurement method M100 for artificial intelligence contribution of this disclosure further includes: For fully automated tasks, lifecycle state management is performed based on preset state transition rules and state sets. The state set includes at least one of the following: pending state, in progress state, pending inspection state, and completed state. Upon receiving a preset trigger event, the task is driven to transition from the current state to the next state according to the state transition rules.

[0070] Among them, the state transition rule can refer to the predefined control logic that determines how a task transitions between different states.

[0071] Triggering events are external or internal instructions that drive state transitions and can include one or more of the following: task is triggered to execute, code generation agent executes successfully, code generation agent executes unsuccessfully, manual inspection passes, and manual inspection rejects.

[0072] In one implementation, after the task is marked as fully automated in step S104, its initial state is set to the pending state. This step can be executed in parallel with step S106: when the agent that drives the code generation in step S106 executes, the state transitions from the pending state to the in-progress state; when the agent completes execution, a state transition is triggered based on the execution result: if the execution is successful, it enters the pending inspection state; if the execution fails, it reverts to the pending state; after subsequent inspections are completed, a state transition is triggered based on the inspection result: if the inspection passes, it enters the completed state; if the inspection fails, it reverts to the pending state.

[0073] This implementation method establishes a standardized state management framework for fully automated tasks, enabling the entire lifecycle of AI-executed tasks to be tracked and controlled. It solves the technical problems of opaque AI execution process status and inability to effectively integrate with project management processes in existing systems.

[0074] Figure 2 This is a flowchart illustrating the code extraction and inspection steps of one embodiment of this disclosure.

[0075] refer to Figure 2 In some embodiments of this disclosure, after the code generation agent corresponding to the driver in step S106 is executed, the automatic measurement method M100 for artificial intelligence contribution of this disclosure further includes: S202. Perform structured parsing on the execution results output by the code generation agent to extract the generated code blocks.

[0076] After execution, the code generation agent's output may be unstructured content, possibly mixing natural language descriptions with code snippets. Structured parsing automatically locates and extracts executable code blocks from the execution result by recognizing predefined code block markers or code syntax features. Code block markers can be Markdown-formatted code block identifiers or recognition rules based on programming language syntax features; this disclosure does not impose specific limitations on them.

[0077] S204. Mark the extracted code block as pending inspection and push it to the manual inspection node.

[0078] Specifically, the manual inspection node can be an inspection interface operated by human engineers within the project management system. On this interface, engineers perform quality checks on the AI-generated code, confirming its functional correctness, code style, and compliance with at least one of the project delivery standards. The pending inspection status is consistent with the pending inspection status defined in the aforementioned status management framework, ensuring that the code block inspection process remains synchronized with the task's lifecycle status.

[0079] S206. In response to receiving the inspection pass instruction, the code block is added to the project deliverables, and the task status is transferred from the pending inspection state to the completed state.

[0080] Once a human engineer confirms that the code has passed the inspection, the system receives an inspection pass instruction, officially adds the code block to the project deliverables, and triggers the aforementioned state management mechanism, driving the task status to transition from the pending inspection state to the completed state. The process data of tasks that have passed inspection and whose status has transitioned to the completed state then enters the subsequent contribution measurement steps.

[0081] S208. In response to receiving an inspection rejection instruction, the task status is switched from the pending inspection status back to the pending work status.

[0082] When a human engineer determines that the code has quality issues, the system receives an inspection rejection instruction, triggering the state management mechanism and driving the task status to transition from the pending inspection state back to the pending work state. The task can then be reassigned to the code generation agent for modification, or transferred to a human engineer, forming a closed-loop mechanism for handling quality issues.

[0083] This implementation method establishes a standardized extraction, inspection, and closed-loop circulation mechanism for AI-generated code, solving the technical problems of AI-generated code being directly mixed into deliverables and lacking an independent quality control process in existing technologies. It enables the inclusion of AI-generated code to be traceable and its quality to be controllable, ensuring the overall reliability of delivery.

[0084] Figure 3 This is a flowchart illustrating the validity verification of process data according to one embodiment of this disclosure.

[0085] refer to Figure 3 In some embodiments of this disclosure, before step S108, which involves multi-dimensionally measuring the collected process data to generate a task-level contribution index for fully automated tasks (multi-dimensional measurement includes a basic contribution calculated based on code change information), the automatic measurement method M100 for artificial intelligence contribution of this disclosure further includes: S302. Verify the validity of process data.

[0086] After collecting process data in step S106, some tasks may involve situations where the code-generating agent is driven to execute but does not actually produce effective code. For example, the agent may determine that no coding is needed after analyzing the task requirements, or an anomaly may occur during the agent's execution, resulting in no code changes. Directly including the process data of such tasks in the metric would distort the contribution calculation results. Therefore, this step performs preliminary verification of the process data before proceeding to multi-dimensional metrics.

[0087] S304. If the verification results show that the code generation agent has not produced effective code changes, then the artificial intelligence contribution index of the task shall be set to zero.

[0088] The validity verification can be based on whether there are pre-defined write or edit operations in the operation log. Write or edit operations are tool calls performed by the code generation agent on code files in an isolated workspace environment. The system iterates through the operation logs collected in step S106, searching for records of these tool calls. If no write or edit operations are found in the operation logs, it is determined that the code generation agent has not actually produced code, and the system directly sets the AI ​​contribution index for that task to zero, preventing it from proceeding to the multi-dimensional measurement process in step S108.

[0089] S306. If the verification results show that the code generation agent produces effective code changes, then the process data is sent to step S108 for multi-dimensional measurement.

[0090] If the operation log contains at least one write or edit operation, the code generation agent is deemed to have made a valid code change. At this point, the process data for the task is confirmed as valid, and the process proceeds to the next step S108. Based on the code change information, the basic contribution is calculated, and the contribution adjustment factor is further calculated by combining the execution complexity analyzed from the operation log. Finally, the equivalent manual workload corresponding to the task is determined.

[0091] This implementation method automatically verifies the data based on objective operation records generated during the execution of the intelligent agent, avoiding the drawbacks of relying on manual judgment or subjective estimation, and achieving standardization and automation of the verification process. Furthermore, validity verification, as a pre-filtering step in the measurement process, ensures that contribution calculations are performed only on tasks that actually produce code output, enabling the measurement results to truly reflect the actual contribution of the AI.

[0092] Figure 4 This is a flowchart illustrating the calculation of basic contribution based on code change information, according to one embodiment of this disclosure.

[0093] refer to Figure 4In some embodiments of this disclosure, the basic contribution calculated based on code change information in step S108 described above includes: S402. Extract the number of newly added lines and the number of deleted lines from the code change information, and determine the difference between the number of newly added lines and the number of deleted lines as the net number of newly added lines of code.

[0094] The number of newly added lines is the number of lines of code newly written to the code file by the code generation agent during task execution, while the number of deleted lines refers to the number of lines of code removed from the code file by the agent. Using the difference between the number of newly added lines and the number of deleted lines as the net number of newly added lines of code can more accurately reflect the effective amount of code actually produced by the agent, avoiding the problem of artificially inflating the workload caused by only counting the number of newly added lines and ignoring the deletion operations.

[0095] S406. Based on the net increase in lines of code and the preset baseline code productivity, calculate the basic equivalent human workload as the basic contribution.

[0096] The baseline code productivity can be defined as the average number of lines of effective code a standard human engineer can produce per unit of time. This parameter can be preset based on industry statistics or team historical data, and this disclosure does not impose specific limitations on it. Dividing the net increase in lines of code by the baseline code productivity converts the AI's code output into equivalent human working time, allowing the measurement results to be presented in a workload unit that the project client can understand.

[0097] Figure 5 This is a flowchart illustrating the calculation of the contribution adjustment factor based on the execution complexity derived from operation record analysis, according to one embodiment of this disclosure.

[0098] refer to Figure 5 For the various related implementation methods described above, the contribution adjustment factor for calculating the execution complexity based on the operation record analysis in step S110 described above includes: S502. Perform pattern recognition on the operation records to extract complexity features, including iteration depth to characterize the number of "modify-verify" loops and context breadth to characterize the number of different code files involved.

[0099] Measuring AI's contribution solely through the baseline contribution calculated in step S108 is insufficient. Tasks with the same net increase in lines of code may have significantly different actual execution complexities. For example, a complex task requiring multiple modifications, debugging, and verifications to complete may have the same net increase in lines of code as a task that can be completed in one go, but the former consumes far more AI inference and operational resources. Therefore, this step uses pattern recognition on the operation records to extract feature information that reflects the task's execution complexity.

[0100] The iteration depth refers to the number of times the code generation agent executes the "modify-verify" loop pattern during task execution. This loop pattern can be represented by the agent first performing write or edit operations on the code file, then performing verification operations (such as running tests or code checks), modifying again based on the verification results, and so on. A higher iteration depth indicates a more complex task, requiring more rounds of debugging and correction.

[0101] Context breadth refers to the number of different code files involved in the code generation agent during task execution. Context breadth can be obtained by deduplicating file paths involved in file read operations in the operation log. A higher context breadth indicates a broader range of code contexts that the task needs to understand and manipulate, and consequently, a higher task complexity.

[0102] S504. Based on iteration depth and context breadth, calculate the complexity weight coefficient as a contribution adjustment factor.

[0103] Specifically, the iteration depth and context breadth can be weighted and combined to generate a complexity weight coefficient, which is used to adjust the basic contribution calculated in step S108.

[0104] In one implementation, the complexity weight coefficients can be calculated based on the formula Wi=1+α·di+β·ln(1+bi), where di is the iteration depth, bi is the context breadth, α is the iteration depth weight factor, and β is the context breadth weight factor. α and β can be preset based on actual project experience or statistical data, and this disclosure does not impose specific limitations on them.

[0105] Any adjustments or changes made by those skilled in the art to the calculation method of the complexity weight coefficient, etc., based on the technical solutions disclosed herein, shall fall within the protection scope of this disclosure.

[0106] This implementation incorporates the dynamic complexity characteristics of the task execution process into the measurement model, enabling the contribution calculation results to distinguish tasks of different complexities. This solves the problem that simple code line count statistics in existing technologies cannot reflect the actual difficulty of task execution, and achieves a more accurate measurement of AI contributions.

[0107] Figure 6 This is a schematic diagram of a process for determining the equivalent manual workload corresponding to the task based on the basic contribution level and the contribution adjustment factor, according to one embodiment of this disclosure.

[0108] refer to Figure 6 In some embodiments of this disclosure, determining the equivalent manual workload corresponding to the task based on the basic contribution and the contribution adjustment factor in S110 described above includes: S602. Multiply the basic contribution level by the complexity weighting coefficient to obtain the weighted equivalent human workload.

[0109] It should be noted that the basic contribution calculated in step S108 only reflects the net code output of the code generation agent and does not consider the actual execution complexity of the task. By multiplying the basic contribution by the complexity weight coefficient calculated in step S504, the execution complexity can be incorporated into the measurement result: for complex tasks with high iteration depth and wide context breadth, the complexity weight coefficient is greater than 1, and the weighted equivalent manual workload increases accordingly, more realistically reflecting the actual resources consumed by the agent during execution; for simple tasks that are completed in one go and do not require multiple rounds of debugging, the complexity weight coefficient approaches 1, and the weighted equivalent manual workload is basically equal to the basic contribution.

[0110] S604. Compare the weighted equivalent human workload with the upper limit of the estimated human workload preset for the task, and take the smaller one as the final task-level equivalent human workload for the task.

[0111] The upper limit of the estimated human workload can be the estimated amount of human work required for a task when the project manager assigns the task. This upper limit constraint aims to prevent abnormal situations from distorting the contribution measurement results. For example, when the code generation agent gets stuck in a large number of invalid modification-verification loops due to execution errors, its weighted equivalent human workload may be abnormally high, exceeding the reasonable range for the task. By taking the smaller of the weighted equivalent human workload and the upper limit of the estimated human workload, the measurement results can be kept within a reasonable range, avoiding the problem of inflated contribution due to agent execution errors.

[0112] This implementation method combines the basic contribution level with the execution complexity in a weighted manner and sets an upper limit constraint, so that the final task-level equivalent human workload can reflect the actual execution difficulty of the task and will not deviate from a reasonable range due to abnormal situations, thus achieving an accurate and reliable measurement of AI contribution.

[0113] Figure 7 This is a flowchart illustrating the process of generating project-level contribution metrics by summarizing task-level contribution metrics for all fully automated types of tasks within a software delivery project, as implemented in one embodiment of this disclosure.

[0114] refer to Figure 7 Regarding the various related implementation methods described above, in some implementation methods, S112 described above in this disclosure, which summarizes the task-level contribution indicators of all fully automated type tasks within the software delivery project and generates project-level contribution indicators, includes: S702. The task-level equivalent manual workload of each fully automated task type is summed up to obtain the total manual workload saved in the project.

[0115] It should be noted that the task-level equivalent human workload for each fully automated task is the AI ​​contribution metric for that task, determined after code output calculation, execution complexity weighting, and upper limit constraints in the aforementioned steps. Summing up these metrics for all fully automated tasks yields the total human workload saved by the AI ​​agent replacing human engineers in this project.

[0116] S704. Based on the total amount of labor saved in the project and the preset amount of labor per unit, calculate the equivalent index of the total amount of labor saved in the project.

[0117] The equivalence metric compares the total amount of human labor saved by the project with a pre-defined unit of human labor, resulting in a standardized measurement that reflects how many units of standard human labor are equivalent to the work saved by AI. This metric makes the contribution of AI comparable across different projects and teams.

[0118] S706. Calculate the project efficiency multiplication factor based on the total saved labor workload and the actual total labor workload of the project.

[0119] The actual total manual workload of the project is the sum of the actual manual workload consumed by all tasks in the software delivery project. All tasks may include at least one of the following: fully automated tasks, auxiliary tasks, and purely manual tasks.

[0120] In some implementations, the actual total human workload of a project is obtained as follows: For fully automated tasks, the human workload includes the operation time consumed by human engineers performing code checks at review nodes; for assisted tasks, the human workload is the operation time consumed by human engineers collaborating with AI to complete the task; for purely manual tasks, the human workload is the operation time consumed by human engineers independently completing the task.

[0121] The manual workload for each of the above types of tasks can be obtained from the operation time data or work hour reporting data recorded in the project management system. The total manual workload of the project is obtained by adding up the above manual workloads.

[0122] The total workload of the project is determined by summing the actual total manual workload and the total saved manual workload. The total workload of the project is then divided by the actual total manual workload to obtain the efficiency multiplication factor, which is used to characterize the improvement factor of the overall efficiency of the project after the introduction of AI.

[0123] In some embodiments of this disclosure, after generating project-level contribution indicators in step S112, the AI-based automatic contribution measurement method M100 of this disclosure may further include: One or more of the following can be displayed to the client through a data visualization dashboard: total labor savings, equivalent indicators of total labor savings, and project efficiency multiplication factor.

[0124] Data visualization dashboards are user interface components that present measurement results in visual formats such as charts and numerical panels. The client can be the project management application interface on the terminal device used by the project client. Through data visualization dashboards, project clients can view the contribution of AI to the current project in real time, including but not limited to: key indicators such as the amount of manual work saved by AI and the efficiency improvement factor.

[0125] The executing entity of the automatic measurement method for artificial intelligence contribution in the specific embodiments of this disclosure can be an electronic device such as a mobile phone or computer.

[0126] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the automatic measurement method for artificial intelligence contribution of any of the embodiments described above.

[0127] Figure 8 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0128] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0129] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0130] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0131] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0132] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0133] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0138] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0139] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. An automatic measurement method for the contribution of artificial intelligence, characterized in that, The method includes: Obtain the task set in the software delivery project; Based on a preset classification standard for the degree of task automation, the execution type of each task in the task set is determined, and the execution type includes at least the fully automated type that is executed independently by an artificial intelligence agent; For tasks of the fully automated type, the corresponding code generation agent is driven to execute, and process data generated by the code generation agent during task execution is collected. The process data includes at least operation records and code change information. The collected process data is measured in multiple dimensions to generate a task-level contribution index for the fully automated type of task; wherein, the multi-dimensional measurement includes a basic contribution calculated based on the code change information; Based on the execution complexity analyzed from the operation records, a contribution adjustment factor is calculated, and the equivalent manual workload corresponding to the task is determined according to the base contribution and the contribution adjustment factor; and Summarize the task-level contribution metrics of all fully automated tasks within the software delivery project to generate project-level contribution metrics.

2. The method according to claim 1, characterized in that, The code generation agent corresponding to the driver is executed, including: The fully automated tasks are dispatched to the background task queue by the main project management process to drive the code generation agent to execute in an isolated workspace environment, so that the background task execution process of the code generation agent and the foreground operation process of the main project management process can be carried out in parallel.

3. The method according to claim 1, characterized in that, The method further includes: For the fully automated type of task, lifecycle state management is performed based on preset state transition rules and state set, wherein the state set includes at least one of the following: pending state, in progress state, pending inspection state, and completed state. Upon receiving a preset trigger event, the task is driven to transition from the current state to the next state according to the state transition rules.

4. The method according to claim 3, characterized in that, After the code-generated agent corresponding to the driver is executed, the method further includes: The execution results output by the code generation agent are subjected to structured parsing to extract the generated code blocks; The extracted code blocks are marked as pending inspection and pushed to the manual inspection node; In response to receiving a pass inspection instruction, the code block is added to the project deliverables, and the task status is transitioned from pending inspection to completed. Alternatively, in response to receiving an inspection rejection instruction, the task status can be switched from pending inspection back to pending.

5. The method according to claim 1, characterized in that, Optionally, before performing multi-dimensional measurements on the collected process data, the method further includes: validating the process data; if the verification result shows that the code generation agent has not produced effective code changes, then the artificial intelligence contribution index of the task is directly set to zero. Optionally, the validity verification of the process data includes: determining whether there is a preset write or edit operation in the operation record; if not, it is determined that no valid code change has occurred.

6. The method according to claim 1, characterized in that, Optionally, the basic contribution calculated based on the code change information includes: extracting the number of newly added lines and the number of deleted lines of code from the code change information, and determining the difference between the number of newly added lines and the number of deleted lines as the net number of newly added lines of code; calculating the basic equivalent manual workload based on the net number of newly added lines of code and a preset baseline code productivity, as the basic contribution. Optionally, the step of calculating the contribution adjustment factor based on the execution complexity analyzed from the operation records includes: performing pattern recognition on the operation records to extract complexity features, wherein the complexity features include iteration depth characterizing the number of "modify-verify" loops and context breadth characterizing the number of different code files involved; and calculating a complexity weight coefficient based on the iteration depth and the context breadth as the contribution adjustment factor. Optionally, determining the equivalent manual workload corresponding to the task based on the basic contribution and the contribution adjustment factor includes: multiplying the basic contribution by the complexity weight coefficient to obtain the weighted equivalent manual workload; comparing the weighted equivalent manual workload with the preset upper limit of the estimated manual workload for the task, and taking the smaller one as the final task-level equivalent manual workload for the task.

7. The method according to claim 1, characterized in that, The process of summarizing the task-level contribution metrics of all fully automated tasks within the software delivery project to generate project-level contribution metrics includes: The total amount of manual labor saved in the project is obtained by summing up the equivalent manual workload of each fully automated type of task. Based on the total amount of labor saved in the project and the preset unit amount of labor, calculate the equivalent index of the total amount of labor saved in the project; The project efficiency multiplication factor is calculated based on the total saved labor workload and the actual total labor workload of the project.

8. The method according to claim 7, characterized in that, Optionally, after generating the project-level contribution index, the method further includes: displaying one or more of the following to the client through a data visualization dashboard: the total amount of labor saved by the project, the equivalent index of the total amount of labor saved by the project, and the project efficiency multiplication factor. Optionally, the tasks in the task set may also include at least one of the following: assisted tasks completed by AI in collaboration with the user, and purely manual tasks completed independently by the user.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images