Performance evaluation method and device for man-machine cooperation development system

Abstract

The invention provides a performance evaluation method and device for a man-machine cooperation development system, and relates to the technical field of data processing, and the method comprises the steps: monitoring the operation of the man-machine cooperation development system, and obtaining a plurality of pieces of monitoring data in a current evaluation period; obtaining a code productivity in a current evaluation period, and correcting the code productivity in multiple dimensions according to the monitoring data to obtain evaluation parameters in each dimension; wherein the dimensions comprise task execution effectiveness, organization effectiveness and decision effectiveness, and each dimension comprises a plurality of sub-dimensions; and step-by-step correction is carried out on the evaluation parameters of each dimension to generate a target evaluation result, so that the problem that the existing evaluation of the man-machine cooperation development system depends on artificial experience and is low in efficiency and accuracy is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a method and device for evaluating the performance of a human-machine collaborative development system. Background Art

[0002] In recent years, artificial neural networks, especially large language models based on Transformer, have achieved booming development and applications in various industries.

[0003] In the field of software development, software development agents mainly based on large language models + retrieval-augmented generation + chain of thought have gradually emerged, supplemented by some traditional practical tools. These software development intelligent tools, software development agents and human programmers together constitute a human-machine collaborative team, which has multiplied the software development efficiency.

[0004] This new development mode has changed the working mode of human programmers, from the original code writing mode to the mode of operating intelligent tools or commanding agents, which puts forward new ability requirements for human programmers. This makes the original method for evaluating the capabilities of individual programmers and the method for evaluating teams composed of pure human programmers no longer applicable. Most of the existing methods rely on manual experience for improvement, lacking a data-based evaluation method for such systems, and unable to achieve efficient and accurate supervision and improvement of such systems. Summary of the Invention

[0005] In order to overcome the above technical defects, the purpose of the present invention is to provide a method and device for evaluating the performance of a human-machine collaborative development system, so as to solve the problems of low efficiency and accuracy in the existing evaluation of human-machine collaborative development systems relying on manual experience.

[0006] The present invention discloses a method for evaluating the performance of a human-machine collaborative development system, including:

[0007] Monitoring the operation of the human-machine collaborative development system to obtain a number of monitoring data in the current evaluation period;

[0008] Obtaining the code productivity in the current evaluation period, and correcting the code productivity in multiple dimensions according to the monitoring data to obtain evaluation parameters in each dimension; where the dimensions include task execution effectiveness, organizational effectiveness, and decision-making effectiveness, and each dimension includes multiple sub-dimensions;

[0009] Gradually correcting the evaluation parameters of each dimension to generate a target evaluation result.

[0010] Preferably, for any dimension, metric parameters corresponding to each sub-dimension are determined according to the monitoring data by a pre-designed calculation function;

[0011] Taking the code productivity as the input, the code productivity is gradually corrected by using the metric parameters corresponding to each sub-dimension to obtain the evaluation parameter corresponding to the dimension.

[0012] Preferably, the priorities of each sub-dimension under each dimension are preset for gradual correction;

[0013] Among them, the priorities under the task execution effectiveness dimension, from first to last, include the document writing sub-dimension, the testing sub-dimension, and the code writing sub-dimension;

[0014] The priorities under the organizational effectiveness dimension, from first to last, include the organizational effectiveness sub-dimension, the organizational operation specification sub-dimension, and the organizational form sub-dimension;

[0015] The priorities under the decision-making effectiveness dimension, from first to last, include the decision-making process compliance sub-dimension, the execution compliance sub-dimension, and the direction compliance sub-dimension.

[0016] Preferably, the correction includes cumulative correction, logarithmic weighted correction, and / or control point correction.

[0017] Preferably, the gradual correction according to the evaluation parameters under each dimension includes:

[0018] Preset the priorities of each dimension. For any dimension:

[0019] Establish a correction function, where the correction function includes N w ' = N w *r M , N w is the input evaluation parameter, r is the evaluation parameter corresponding to the dimension; M is a preset correction coefficient;

[0020] Obtain the evaluation parameter output by the last-level dimension of the priority as the target correction result according to the correction function.

[0021] Preferably, the target evaluation result includes:

[0022] Generate the target evaluation result through logarithmic mapping according to the target correction result obtained after gradual correction.

[0023] Preferably, establish a logarithmic mapping function, where the logarithmic mapping function includes where N F is the target correction result and L is the target evaluation result.

[0024] Preferably, when the metric parameter corresponding to any sub-dimension is abnormal in the current evaluation period, obtain the target evaluation result of the previous evaluation period to calculate the metric parameter corresponding to the sub-dimension.

[0025] Preferably, provide the correlation relationship between the metric parameters and the target evaluation results of each sub-dimension in advance;

[0026] Perform performance analysis according to the metric parameters of each sub-dimension and the target evaluation results.

[0027] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the performance evaluation methods are implemented.

[0028] After adopting the above technical solutions, compared with the prior art, the following beneficial effects are achieved:

[0029] A performance evaluation method and device for a human-machine collaborative development system provided by the present application, by obtaining monitoring data under the human-machine collaboration system, and making step-by-step corrections based on the code generation rate in multiple dimensions such as decision-making effectiveness, task organization effectiveness of the intelligent agent, and business execution effectiveness of project promotion during the software project development process under the system, so as to achieve data processing in multiple dimensions and realize an accurate and quantitative performance evaluation method for the human-machine collaboration system, so as to solve the problems of low efficiency and accuracy in the existing evaluation of the human-machine collaborative development system relying on manual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of Embodiment 1 of a performance evaluation method and device for a human-machine collaborative development system of the present invention;

[0031] Figure 2 It is a schematic diagram of the relationship between each dimension in Embodiment 1 of a performance evaluation method and device for a human-machine collaborative development system of the present invention;

[0032] Figure 3 It is a schematic diagram of step-by-step correction of an example in Embodiment 1 of a performance evaluation method and device for a human-machine collaborative development system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The advantages of the present invention will be further elaborated below in conjunction with the accompanying drawings and specific embodiments.

[0034] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] The terms used in the present disclosure are for the purpose of describing particular embodiments only and are not intended to limit the present disclosure. The singular forms "a", "the", and "said" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0037] In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "installed", "connected", and "coupled" should be understood in a broad sense. For example, it may be a mechanical connection or an electrical connection, or it may be the communication inside two elements. It may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms may be understood according to specific circumstances.

[0038] In the following description, the suffixes such as "module", "component", or "unit" used to represent elements are only for the convenience of describing the present invention, and they have no specific meaning in themselves. Therefore, "module" and "component" may be used interchangeably.

[0039] Embodiment: This embodiment provides a method for evaluating the performance of a human-machine collaborative development system. Specifically, this solution measures the effectiveness of decision-making during the software project development process by the programmer (manual operation), the effectiveness of task organization for the software development agent, and the effectiveness of business execution for the overall project progress by collecting the collaboration data and project development status data between the programmer and the software development agent, and finally obtains a grade assessment representing the operation efficiency of the human-machine collaboration system; refer to Figures 1 - 3 , specifically including the following steps:

[0040] S10: Monitor the operation of the human-machine collaborative development system and obtain a number of monitoring data under the current evaluation period;

[0041] In this embodiment, as described above, data during the operation of the human-machine collaborative development system is collected for performance evaluation. The human-machine collaborative development system realizes human-machine collaboration through agents. Therefore, the above monitoring data is obtained through agents. It can be understood that the monitoring data is objective data collected during the operation of the system.

[0042] Illustratively, the monitoring data (i.e., objective data) includes but is not limited to: when the user (programmer) activates the agent, the working environment specified for the agent, what questions are asked of the agent at what time, whether the results are accepted by people, at what time the agent proactively proposes a certain modification, whether the modification is accepted by people, whether the person makes a modification himself after being accepted, and when the working environment is closed, etc., which are human-computer interaction information, and how many functional modules exist in the project itself, when these functional modules are modified, how the code is modified at a certain time, and project information such as the number of documents and test cases.

[0043] It can be understood that the above monitoring data can be numerically represented by parameters such as the number of functional modules, the number of changes, the number of tasks of the coding robot, and the number of tasks of the document robot. They can all be used to generate measurement parameters in the following various dimensions, so as to realize the performance evaluation from multiple dimensions.

[0044] In this embodiment, it can be understood that the evaluation period can be preset. The evaluation period can be set to a fixed value (absolute interval), such as one week / day; it can also be conditionally triggered, such as after a preset time period of a certain project / program runs (relative interval). Specifically, it can be determined according to the application scenario. This embodiment takes the preset evaluation period as an example.

[0045] S20: Obtain the code productivity under the current evaluation period, and correct the code productivity in multiple dimensions according to the monitoring data to obtain evaluation parameters in each dimension; where the dimensions include task execution effectiveness, organizational effectiveness, and decision-making effectiveness, and each dimension includes multiple sub-dimensions;

[0046] In this embodiment, compared with the traditional evaluation scheme, the influence of various factors related to the user, such as the usage habits, usage methods, and proficiency of programmers (humans) in using the programming assistance system, is additionally considered. And these influences can be numerically presented through code productivity. Therefore, this embodiment uses the code productivity N (N 0 / N_ori) for iterative update for evaluation.

[0047] Specifically, code productivity is an indicator that measures the amount of code completed by a software development team or individual within a unit of time. It is usually used to evaluate development efficiency and quality. In this embodiment, productivity is evaluated by counting the number of lines of code in a project (excluding blank lines and comment lines) and combining it with the development time. The calculation formula is: Productivity = Number of lines of code / Development time (person-hours or person-months); the number of lines of code, development time, etc. can all be obtained from the above monitoring data.

[0048] It should be noted that the performance evaluation in this embodiment is carried out from multiple dimensions. Figure 2 As mentioned before, measure the effectiveness of programmers' decisions during the software project development process, the effectiveness of task organization for agents, and the effectiveness of business execution for the overall project progress. Therefore, the set dimensions include but are not limited to task execution effectiveness, organizational effectiveness, and decision-making effectiveness. Further, each dimension also includes several sub-dimensions. Among them, the task execution effectiveness dimension includes but is not limited to the document writing sub-dimension, the testing sub-dimension, and the code writing sub-dimension; the organizational effectiveness dimension includes but is not limited to the organizational efficiency sub-dimension, the organizational operation specification sub-dimension, and the organizational form sub-dimension; the decision-making effectiveness dimension includes but is not limited to the decision-making process compliance sub-dimension, the execution compliance sub-dimension, and the direction compliance sub-dimension.

[0049] Specifically, as an illustration, each of the above sub-dimensions can be numerically characterized by measurement parameters. In this embodiment, each sub-dimension / dimension is represented hierarchically from top to bottom. The upper-level indicators can be calculated from the lower-level indicators. For example, the dimension is the upper-level indicator, and the sub-dimension is the lower-level indicator. Each sub-dimension also has a hierarchical relationship, that is, the lower-level indicators have a higher processing priority, and the upper-level indicators have a lower priority. It should be noted that in this application, "priority" refers to the order of correction execution of each dimension / sub-dimension, rather than the importance of the corresponding characterization indicators / abilities of that dimension.

[0050] Based on the above, as Figure 3 shown, preset the priorities of each sub-dimension under each dimension for hierarchical correction; among them, the priorities of the sub-nodes under the task execution effectiveness dimension are, in order from first to last, the document writing sub-dimension, the testing sub-dimension, and the code writing sub-dimension; the priorities of the sub-nodes under the organizational effectiveness dimension are, in order from first to last, the organizational efficiency sub-dimension, the organizational operation specification sub-dimension, and the organizational form sub-dimension; the priorities of the sub-nodes under the decision-making effectiveness dimension are, in order from first to last, the decision-making process compliance sub-dimension, the execution compliance sub-dimension, and the direction compliance sub-dimension. Further, the priorities of each dimension are also preset, where the priorities are, in order from first to last, task execution effectiveness, organizational effectiveness, and decision-making effectiveness (mutual sibling nodes). It can be understood that, in fact, the priorities can be adjusted or dynamically updated according to the actual scenario.

[0051] In this embodiment formula, as described above, the upper-level indicators can be calculated from the lower-level indicators. For any dimension, according to the monitoring data, the measurement parameters corresponding to each sub-dimension are determined by a preset calculation function; taking the code productivity as the input, the code productivity is gradually corrected by using the measurement parameters corresponding to each sub-dimension to obtain the evaluation parameter corresponding to the dimension.

[0052] Based on the above, in this embodiment, not only is the gradual correction performed at the dimension level, but also at each sub-dimension of each dimension. When correcting, the previous-level dimension / sub-dimension is used as the input, and the correction is performed considering the measurement parameters under this dimension, so as to perform data processing in multiple dimensions through gradual iteration, thereby comprehensively evaluating the performance and improving the accuracy and comprehensiveness of the evaluation results.

[0053] As a special note, the correction in this embodiment (in the above S20 or the following S30) includes but is not limited to cumulative correction, logarithmic weighted correction, and / or control point correction. The corresponding correction method is selected according to the data characteristics / scenario requirements under each dimension / sub-dimension. Specifically, the cumulative correction is simple addition and subtraction, which is applicable when the input data and output data are in the same dimension under each dimension / sub-dimension; the logarithmic correction is to perform weighted average after transforming into the logarithmic space and then transform back into the linear space. Its applicable range is relatively wide, and its limitation on the sequential data processing under each dimension / sub-dimension is small. The control point correction takes the weighted average of multiple control point positions (i.e., multiple inputs) as the correction. Its limitation on the sequential data processing under each dimension / sub-dimension is large, and it is a non-linear correction method. It can be understood that in addition to the above correction methods, other data processing methods that can be used to evaluate the system operation performance can also be used here.

[0054] As an illustration, the above measurement parameter is the degree of influence of the code productivity on the performance under this dimension in the evaluation period. For each sub-dimension, it is calculated by using a preset calculation function, and it is different for each sub-dimension. Different calculations (which can be updated) can be performed according to the characteristics of each sub-dimension or the performance it represents. In this embodiment, as an example:

[0055] The decision-making effectiveness dimension (which is used to characterize the performance evaluation / assessment of the manual part):

[0056] The direction compliance sub-dimension:

[0057] Measurement parameter: The modification rate at intervals of the same function module Where is the number of function modules, alter_count iis the number of changes to the i-th functional module, which can all be obtained from the monitoring data. For the definition of the same functional module: 1. It is a group of function / class nodes; 2. It belongs to the same non-top-level folder; 3. It has a completely direct call relationship; The interval means that after the completion of this functional module, more than three functional modules have been completed, and it is considered that the subsequent modifications to this functional module have an interval; As an explanation, within a relatively long period of time, this functional module has not been modified, and it is considered that this module has been completed, and the subsequent modifications are caused by requirement changes (direction changes), reflecting the requirement change rate.

[0058] The correction function under this sub-dimension: ; where N is the input of this sub-dimension and N' is the output of this sub-dimension; γ alter_rate is the correction coefficient (weight) of this sub-dimension;

[0059] Execution compliance sub-dimension:

[0060] Measurement parameter: Probability of correctly issuing commands Among them is the number of correctly issued commands; command_count i is the number of commands; both can be obtained from the monitoring data.

[0061] The correction function under this sub-dimension: where N is the input of this sub-dimension and N' is the output of this sub-dimension; k is the correction coefficient of this sub-dimension;

[0062] Decision process compliance sub-dimension: Default compliance;

[0063] In the human-machine interaction system of this embodiment, one operator corresponds to interacting with multiple devices. Therefore, the decision process compliance is defaulted to 1, that is, this indicator is used to characterize the rationality of manual decision-making. Since there is a single operator and no other decision-making aids, the default compliance is adopted.

[0064] Based on this default compliance, the correction coefficients of the direction compliance sub-dimension and the execution compliance sub-dimension can be set to 0.5 respectively. It can be understood that different (non-uniform) correction coefficients can also be set according to the scenario.

[0065] Organization effectiveness dimension (characterizing the evaluation of machine utilization rate):

[0066] Organization form effectiveness sub-dimension:

[0067] Measurement parameter: Ratio of the number of effective tasks of business robots Among them, the robots include a coding robot coder, a testing robot tester, and a documentation robot docer; the number of tasks received by these three robots are task_count_coder, task_count_taster, and task_count_doder respectively; all of which can be obtained from the monitoring data.

[0068] Correction function under this sub-dimension: ; where N is the input of this sub-dimension and N' is the output of this sub-dimension; γ task_count_rate is the correction coefficient of this sub-dimension.

[0069] Sub-dimension of organizational operation specifications:

[0070] Measurement parameter: Human coding participation

[0071] Ω is the correction parameter for the number of automatically triggered tasks, representing the ratio of the number of tasks completed by only relying on auto-completion and only relying on dialogue when completing the same task. In this embodiment, Ω = 10; auto_task_count is the number of automatically triggered tasks; manual_task_count is the number of manually triggered tasks; both can be obtained from the monitoring data.

[0072] Correction function under this sub-dimension: ; where N is the input of this sub-dimension and N' is the output of this sub-dimension; γ human_rate is the correction coefficient of this sub-dimension.

[0073] Sub-dimension of organizational effectiveness:

[0074] Measurement parameters: Tendency of effectiveness improvement, effectiveness stability (priority can also be set)

[0075] Tendency of effectiveness improvement growth_probablity where t is time, and k and x are intermediate parameters; all can be calculated from the monitoring data;

[0076] Correction function under this sub-dimension: where N is the input of this sub-dimension and N' is the output of this sub-dimension; γ growth_probablity is the correction coefficient of this sub-dimension.

[0077] Effectiveness stability stable_rate = σ(N); that is, take the standard deviation of N;

[0078] Correction function under this sub-dimension: N' = N - k s *stable_rate; where N is the input of this sub-dimension and N' is the output of this sub-dimension; k sis the correction coefficient for this sub-dimension. In this embodiment, twice the standard deviation is taken as the lower limit, i.e., k s = 2.

[0079] For each sub-dimension under this dimension, the correction coefficient of equal division can also be adopted as described above (such as Other preset correction coefficients can also be adopted.

[0080] Business execution effectiveness dimension (representing the work efficiency of the human-machine interaction team):

[0081] Code writing (human-machine team for code writing) sub-dimension:

[0082] Metric parameter: Code productivity;

[0083] Code productivity LoC is the number of lines of code, and time is the time, both of which can be calculated from the monitoring data; it can be understood that S = N 0 , N 0 That is, input the initial N of each dimension / sub-dimension, that is, the N that has not been corrected by any sub-dimension / dimension.

[0084] The above correction under this sub-dimension can be understood as the code productivity N in the current evaluation period 0 Output N' through the document writing sub-dimension and the test sub-dimension correction, and then correct it again using the code productivity N in the current evaluation period under the code writing sub-dimension 0 for correction.

[0085] Document writing (human-machine team for document writing) sub-dimension:

[0086] Metric parameter: Document coverage factor

[0087] The total number of lines of code is the total number of lines of code with modules such as functions and classes under non-testing;

[0088] The number of functions can be obtained from the monitoring data;

[0089] The amount of documents is the number of bytes of files without code modules in the project / program;

[0090] Document coverage factor

[0091]

[0092] , where byte is the total number of bytes of all documents, and func_count is the number of functions (quantity) in the project / program;

[0093] As an example, assume that 50% of the code can be maintained without documentation, 30% requires low-level documentation, and 20% requires high-level documentation; high-level documentation requires 100-1000 bytes of documentation for a function, with an average of 550; low-level documentation requires 10-100 bytes of documentation for a function, with an average of 55; high-level documentation is prioritized.

[0094] Testing (testing human-machine team) sub-dimension:

[0095] Metric: Test Coverage Factor

[0096] The number of test cases is the number of functions whose names contain the word "test";

[0097] Average cyclomatic complexity of code Cyclomatic Complexity is a software metric for measuring code complexity (optionally implemented with existing software), where func_count is the number of functions;

[0098] Test Coverage Factor Here, test_case_count is the number of test cases, and 0.99 is the preset value, which is intended to set a maximum preset value close to 1 to prevent zero errors from occurring in the calculation.

[0099] It is understandable that the correction functions of each sub-dimension under the above-mentioned business execution effectiveness dimension can also refer to the above-mentioned correction functions, and no specific examples are given here, and any correction method in the above-mentioned correction can also be adopted. Moreover, each variable in the above-mentioned calculation formula can be obtained from the monitoring data, and some specific values ​​are preset values, which can be adjusted according to the actual scenario (projects / programs running in the system).

[0100] Based on the above, in this implementation, under any dimension, taking the decision effectiveness dimension as an example, the priority under the decision effectiveness dimension includes the decision process compliance sub-dimension, the execution compliance sub-dimension, and the direction compliance sub-dimension in descending order. Therefore, the code productivity N under the current evaluation cycle is obtained, and N' is output through the decision process compliance sub-dimension; then the decision process compliance sub-dimension output N' is used as input N to the compliance sub-dimension, and N' is output; then the execution compliance sub-dimension direction output N' is used as input to the N compliance sub-dimension, that is, it is corrected step by step according to the priority. Similar processing is also performed on the code productivity N under the current evaluation cycle of S20 in other dimensions to obtain the final output N' of each dimension, that is, the evaluation parameters of each dimension.

[0101] S30: Correct the evaluation parameters of each dimension step by step to generate a target evaluation result.

[0102] In this embodiment, the correction processing for each sub-dimension in each dimension is described above. After obtaining the evaluation parameters for each dimension, a process similar to the correction for each of the above sub-dimensions is performed.

[0103] Specifically, hierarchical correction is performed according to the evaluation parameters for each dimension, including:

[0104] Pre-set the priorities for each dimension. For any dimension: establish a correction function, where the correction function includes N w ' = N w *r M where N w is the input evaluation parameter, r is the evaluation parameter corresponding to the dimension; M is a pre-set correction coefficient; obtain the evaluation parameter output by the last-level dimension of the priority as the target correction result according to the correction function.

[0105] In this embodiment, as described above, the priorities from first to last are the task execution effectiveness dimension, the organizational effectiveness dimension, and the decision-making effectiveness dimension.

[0106] Specifically, the correction function under the task execution effectiveness dimension:

[0107] where N_from_children is the N' output from the child node, that is, the evaluation parameter N' finally output by each sub-dimension under the task execution effectiveness dimension,

[0108] N_from_cousin is the N' output from the sibling node, that is, the evaluation parameter N' finally output by the dimension prior to the task execution effectiveness dimension. In this example, the task execution effectiveness dimension is the dimension with the highest priority. Therefore, N_from_cousin = 1; γexe_rate is the correction coefficient for this dimension. In this example, an equal division coefficient is used for each dimension, that is, If the number of sub-dimensions increases or the application scenario requires it, it can also be adjusted autonomously / passively.

[0109] The correction function under the organizational effectiveness dimension:

[0110]

[0111] where N_from_children is the N' output from the child node, that is, the evaluation parameter N' finally output by each sub-dimension of the organizational effectiveness dimension; N_from_cousin is the N' output from the sibling node, that is, the evaluation parameter N' finally output by the above task execution effectiveness dimension, γ organization_rate is the correction coefficient for this dimension. In this example, an equal division coefficient is used for each dimension, that is, If the sub-dimension increases or the requirements of the application scenario change, it can also be adjusted actively / passively.

[0112] Correction function under the decision-making effectiveness dimension:

[0113] Among them, N_from_children is N' output from the child node, that is, the evaluation parameter N' finally output by each sub-dimension of the decision-making effectiveness dimension; N_from_cousin is N' output from the sibling node, that is, the evaluation parameter N' finally output by the above-mentioned organizational effectiveness dimension, γ decision_rate is the correction coefficient under this dimension. In this example, an equal division coefficient is adopted for each dimension, that is If the sub-dimension increases or the requirements of the application scenario change, it can also be adjusted actively / passively.

[0114] The N' finally output by the decision-making effectiveness dimension is the target correction result.

[0115] As an explanation, the above N_ori = N 0 = N, that is, the code productivity in the current evaluation period obtained in S20, that is, the initial N, is the N input to the first sub-dimension of each priority. The purpose of using this N to process the N' (evaluation parameter) output by each dimension is to eliminate the dimension of the data.

[0116] Specifically, the target evaluation result includes: generating the target evaluation result through logarithmic mapping based on the target correction result obtained after successive corrections.

[0117] As an illustration, in this embodiment, the target evaluation result is a grade number, which is preset to include 0 to 10 grades. Each grade corresponds to a certain range of productivity (that is, performance quality). The larger the grade, the higher the production efficiency of the human-machine collaboration system. The above logarithmic mapping is to adjust the distribution or range of the data by mapping the data to the logarithmic space, and finally obtain a grade.

[0118] Specifically, establish a logarithmic mapping function, where the logarithmic mapping function includes where N F is the above-mentioned target correction result, and L is the target evaluation result. Thus, the grade L is obtained. This logarithmic mapping function is a function representation with relatively high accuracy in the current application system and can also be adjusted in different systems / subsequent optimization processes.

[0119] Based on the above, the evaluation scheme of this embodiment realizes an accurate and quantitative evaluation method by considering the influence of various factors related to users, such as the usage habits, usage methods, proficiency, etc. of humans on the programming assistance system, through the correction of the code generation rate in each dimension.

[0120] Secondly, the dataset of this solution is different from those commonly used in traditional evaluation solutions. After the programming assistance system has been working in the actual environment for a period of time, the objective records during this period can be used to statistically analyze the ability levels of the user and the assistance system (i.e., the preset evaluation period). The advantage of doing this is that it does not involve common problems such as the integrity, correctness, and data distribution of the dataset, and at the same time ensures that the results obtained are objective and reliable, with sufficient reference value.

[0121] Thirdly, the method of gradually upward correction such as logarithmic space weighting and control point correction adopted in this solution ( Figure 3 ) can effectively distinguish from the influence of extreme data caused by the existing direct weighted average based on data, improve the accuracy of the evaluation results, and independently / passively select the correction method according to the data characteristics.

[0122] Finally, the target evaluation result finally obtained by this solution is a grade, which can be preset to be associated with the corresponding productivity or performance description / quantification range. After comprehensively considering the data processing of each dimension, the grade is calculated. The system capabilities are similar at the same grade, and there are obvious differences between systems at different grades, which is more valuable in horizontal comparison and is conducive to providing data support for the subsequent supervision and optimization of the system.

[0123] Furthermore, in this embodiment, it can be found that there is a certain correlation (which can be a linear relationship) between the measurement parameters corresponding to each sub-dimension and the target evaluation result. Therefore, as an optional embodiment, the correlation between the measurement parameters of each sub-dimension and the target evaluation result can be provided in advance; performance analysis is performed according to the measurement parameters of each sub-dimension and the target evaluation result.

[0124] Specifically, as an example, the correlation between the measurement parameters of each sub-dimension and the target evaluation result includes but is not limited to the following:

[0125] Under the dimension of decision-making effectiveness:

[0126] Under the dimension of organizational form effectiveness:

[0127]

[0128] N is its input;

[0129] Under the dimension of business execution effectiveness:

[0130]

[0131] Therefore, based on the above relationships, analysis can be carried out according to the measurement parameters and target evaluation results of each sub-dimension actually obtained in the current evaluation period. For example, when the measurement parameters of a certain sub-dimension deviate significantly from the above expression compared with the target evaluation result, the problems / status of a certain type of performance in the system can be determined according to the sub-dimension, so as to provide data support for subsequent optimization.

[0132] In a preferred embodiment, based on the above various association relationships, when the measurement parameter corresponding to any sub-dimension is abnormal in the current evaluation period, the target evaluation result of the previous evaluation period is obtained to calculate the measurement parameter corresponding to the sub-dimension. As an illustration, the measurement parameter corresponding to a sub-dimension being abnormal in the current evaluation period means that the measurement parameter corresponding to the sub-dimension is 0, that is, the system has not produced any changes in this sub-dimension for a relatively long period of time / the performance corresponding to the sub-dimension is abnormal. Then, the current measurement parameter can be directly determined according to the target evaluation result of the previous evaluation period, that is, applying historical data. Further, network prediction can also be added in the future to further improve the accuracy of the data processing result in abnormal situations.

[0133] Therefore, based on the above, it can be understood that in the above evaluation method, parameters such as the measurement parameters and target evaluation results of each sub-dimension can be used for further data support for subsequent supervision or optimization, thereby making the evaluation method more widely applicable.

[0134] Embodiment 2: This embodiment also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the performance evaluation method described in any one of the above Embodiment 1.

[0135] The computer device can be a notebook computer, a desktop computer, a rack-mounted server, or a cabinet server (including an independent server or a server cluster composed of multiple servers) that executes the program. The memory can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store various types of data; in some embodiments, the processor can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips to execute the method of the above Embodiment 1.

[0136] It can be understood that the method provided in the above-mentioned first embodiment can be implemented by means of available hardware, software or a combination thereof, such as modules, and can be implemented and executed in combination with various illustrative steps, devices, processors, memories, integrated circuits or other combinations described in the present invention. When implemented in software, in combination with the illustrative steps described above, the module can also be transmitted as one or more instructions or codes stored on a computer-readable storage medium. Whether these methods and functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0137] This embodiment also provides a computer-readable storage medium, which includes a plurality of storage media, such as flash memory, hard disk, multimedia card, etc., on which a computer program is stored, and when the program is executed by a processor, corresponding functions are realized.

[0138] It should be noted that the embodiments of the present invention have better implementability and are not any form of limitation to the present invention. Any person skilled in the art may use the technical content disclosed above to modify or transform it into an equivalent effective embodiment. However, as long as it does not depart from the technical solution of the present invention, any modification, equivalent change or modification made to the above embodiments based on the technical essence of the present invention still belongs to the scope of the technical solution of the present invention.

Claims

1. A performance evaluation method for a human-machine collaborative development system, characterized in that: include: Monitor the operation of the human-machine collaborative development system and obtain a number of monitoring data in the current evaluation cycle; Obtaining the code productivity in the current evaluation cycle, and correcting the code productivity in multiple dimensions according to the monitoring data to obtain evaluation parameters in each dimension; wherein the dimensions include task execution effectiveness, organizational effectiveness, and decision effectiveness, and each dimension includes multiple sub-dimensions; The evaluation parameters of each dimension are modified step by step to generate the target evaluation result.

2. The performance evaluation method according to claim 1, characterized in that: For any dimension, the metric parameters corresponding to each sub-dimension are determined according to the monitoring data using a preset calculation function; with code productivity as input, the metric parameters corresponding to each sub-dimension are used to correct the code productivity step by step to obtain the evaluation parameters corresponding to the dimension.

3. The performance evaluation method according to claim 2, characterized in that: Preset the priority of each sub-dimension under each dimension to make corrections step by step; The priorities under the task execution effectiveness dimension include, in descending order, the document writing sub-dimension, the test sub-dimension, and the code writing sub-dimension; The priorities under the organizational effectiveness dimension include, in descending order, the organizational effectiveness sub-dimension, the organizational operation norms sub-dimension, and the organizational form sub-dimension; The priorities under the decision effectiveness dimension include, from first to last, the decision process compliance sub-dimension, the execution compliance sub-dimension, and the direction compliance sub-dimension.

4. The performance evaluation method according to claim 1, characterized in that: The correction includes cumulative correction, logarithmic weighted correction and / or control point correction.

5. The performance evaluation method according to claim 1, characterized in that: The evaluation parameters under each dimension are revised step by step, including: Preset the priority of each dimension. For any dimension: Establish a correction function, wherein the correction function includes N w '=N w *r M , N w is the input evaluation parameter, r is the evaluation parameter corresponding to the dimension; M is the preset correction coefficient; The evaluation parameters output by the last level dimension of the priority level are obtained according to the correction function as the target correction result.

6. The performance evaluation method according to claim 1, characterized in that: The target assessment results include: The target evaluation result is generated through logarithmic mapping based on the target correction result obtained after step-by-step correction.

7. The performance evaluation method according to claim 6, characterized in that: Establish a logarithmic mapping function, wherein the logarithmic mapping function includes Where N F is the target correction result, and L is the target evaluation result.

8. The performance evaluation method according to claim 1, characterized in that: When the metric parameter corresponding to any sub-dimension is abnormal in the current evaluation cycle, the target evaluation result of the previous evaluation cycle is obtained to calculate the metric parameter corresponding to the sub-dimension.

9. The performance evaluation method according to claim 1, characterized in that: Provide the correlation between the measurement parameters of each sub-dimension and the target evaluation results in advance; A performance analysis is performed based on the measurement parameters of each sub-dimension and the target evaluation results.

10. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the performance evaluation method according to any one of claims 1 to 9 are implemented.

Images