AI programming assistant application evaluation method and system, electronic equipment and storage medium

Abstract

The invention discloses an AI programming assistant application evaluation method and system, electronic equipment and a storage medium. The method comprises the steps that an evaluation index system is constructed; wherein the evaluation index system is constructed through preset evaluation indexes according to a preset hierarchical relationship; performing index data acquisition on a to-be-evaluated programming assistant according to the preset evaluation index in the evaluation index system to obtain evaluation index data; and performing data analysis according to the evaluation index data and the preset hierarchical relationship to obtain an evaluation analysis result. According to the AI programming assistant application effect evaluation method and device, the accuracy of AI programming assistant application effect evaluation can be effectively improved, and accurate support is provided for selection and introduction of AI programming assistants. The method can be widely applied to the technical field of information processing.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an AI programming assistant application evaluation method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, big language models have also made important progress in the field of code generation. This technology can help developers automatically generate code, improve efficiency and reduce errors. Among them, through the combination with the natural language interface, the big language model can also provide developers with more intelligent code generation services through dialogue. Such tools based on the capabilities of big language models are collectively referred to as AI programming assistants. In related technologies, AI programming assistants have functional homogeneity and price convergence, with no obvious differences between them, and the evaluation indicators provided by various AI programming assistants themselves are different, making it difficult to evaluate the application effect of AI programming assistants. When it is necessary to introduce or apply AI programming assistants, it is difficult to choose a suitable AI programming assistant.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an AI programming assistant application evaluation method, system, electronic device and storage medium, which can effectively improve the accuracy of the evaluation of the application effectiveness of the AI ​​programming assistant and provide more accurate support for the selection and introduction of the AI ​​programming assistant.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes an AI programming assistant application evaluation method, the method comprising the following steps:

[0006] Constructing an evaluation index system; wherein the evaluation index system is constructed by pre-setting evaluation indexes according to a preset hierarchical relationship;

[0007] Collecting indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data;

[0008] Data analysis is performed based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results.

[0009] In some embodiments, constructing an evaluation index system includes:

[0010] The code generation ratio indicator is constructed based on the new code line number indicator and the adopted code line number indicator;

[0011] A model effect index is constructed based on a preset logic correlation index and a preset style correlation index; wherein the preset logic correlation index includes a completion code and context logic correlation index, and the preset style correlation index includes a completion code and context style correlation index;

[0012] An objective indicator is obtained according to the code generation ratio indicator and the model effect indicator;

[0013] A subjective index is constructed based on a customer satisfaction index; wherein the customer satisfaction index is constructed from a number of customer evaluation indicators;

[0014] An overall evaluation index is constructed based on the objective index and the subjective index.

[0015] In some embodiments, collecting indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data includes:

[0016] Reading the newly added code line data and the adopted code line data of the programming assistant to be evaluated through a preset statistical module;

[0017] Calculate the text similarity between the completed code and the context through a preset similarity algorithm to obtain preset logical correlation data;

[0018] The text style relevance between the completed code and the context is calculated according to the code format consistency parameter and the naming specification consistency parameter to obtain preset style relevance data;

[0019] A number of customer evaluation data are obtained through a preset survey questionnaire; wherein the customer evaluation data corresponds to the customer evaluation index.

[0020] In some embodiments, the customer evaluation data obtained through the preset survey questionnaire includes:

[0021] The preset survey questionnaire is constructed by using a scoring system according to the customer evaluation indicators;

[0022] The customer evaluation data is collected through the preset survey questionnaire.

[0023] In some embodiments, performing data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results includes:

[0024] Compare and calculate the newly added code line number data and the adopted code line number data to obtain code generation ratio data;

[0025] Performing weighted average calculation according to the preset logic correlation data and the preset style correlation data to obtain model effect data;

[0026] Calculate customer satisfaction data based on the customer evaluation data;

[0027] Perform weighted average calculation based on the code generation ratio data and the model effect data to obtain objective parameter data;

[0028] The subjective parameter data is determined according to the customer satisfaction data, so as to obtain the evaluation analysis result by performing weighted calculation on the objective parameter data and the subjective parameter data.

[0029] In some embodiments, the calculating the text similarity between the completion code and the context by a preset similarity algorithm to obtain preset logical correlation data includes:

[0030] The text similarity between the completion code and the context is calculated using the cosine similarity algorithm to obtain the preset logical correlation data.

[0031] In some embodiments, the calculating the customer satisfaction data according to the customer evaluation data includes:

[0032] Determining a first number of target survey questionnaires that meet a preset scoring threshold condition according to the customer evaluation data;

[0033] The customer satisfaction data is calculated based on the first number and the second number; wherein the second number includes the number of distributed preset survey questionnaires.

[0034] To achieve the above objectives, another aspect of the embodiments of the present application provides an AI programming assistant application evaluation system, the system comprising:

[0035] The first module is used to construct an evaluation index system; wherein the evaluation index system is constructed by pre-setting evaluation indicators according to a preset hierarchical relationship;

[0036] The second module is used to collect indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data;

[0037] The third module is used to perform data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation and analysis results.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0043] The embodiments of the present application include at least the following beneficial effects: The present application provides an AI programming assistant application evaluation method, system, electronic device and storage medium, which is obtained by constructing an evaluation index system, and is constructed by presetting evaluation indicators according to a preset hierarchical relationship. Next, the embodiment of the present invention collects index data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation index system, obtains evaluation index data, and then performs data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results, thereby effectively improving the accuracy of the evaluation of the application effectiveness of the AI ​​programming assistant, and providing more accurate support for the selection and introduction of the AI ​​programming assistant. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the steps of the AI ​​programming assistant application evaluation method provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of an evaluation index system provided by an embodiment of the present invention;

[0046] Figure 3 This is a flow chart of the AI ​​programming assistant application evaluation provided by an embodiment of the present invention;

[0047] Figure 4 It is a structural diagram of the AI ​​programming assistant application evaluation system provided in an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0050] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0051] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] With the rapid development of artificial intelligence (AI) technology, big language models have also made important progress in the field of code generation. This technology can help developers automatically generate code, improve programming efficiency and reduce errors. Among them, by combining with natural language interfaces, big language models can also provide developers with more intelligent code generation services through dialogue. Such tools based on big language model capabilities are collectively referred to as AI programming assistants. In related technologies, AI programming assistants have functional homogeneity and price convergence, and there is no obvious difference between them. In addition, the evaluation indicators provided by various AI programming assistants themselves are different, making it difficult to evaluate the application effect of AI programming assistants. When it is necessary to introduce or apply AI programming assistants, it is difficult to choose a suitable AI programming assistant. For example, when selecting an AI programming assistant, it is necessary to consider whether the AI ​​programming assistant is compatible with the company's existing technology stack. If the company mainly uses languages ​​such as Java and JavaScript, the AI ​​programming assistant must support these programming languages ​​in order to maximize the effectiveness of the tool. At the same time, it is necessary to evaluate the procurement cost and maintenance cost of the AI ​​programming assistant in combination with the budget and scale of the enterprise, and compare it with its potential efficiency improvement and productivity increment to calculate a reasonable return on investment.

[0054] In view of this, an embodiment of the present application provides an AI programming assistant application evaluation method, system, electronic device and storage medium, which is obtained by constructing an evaluation index system, and is constructed by presetting evaluation indicators according to a preset hierarchical relationship. Then, the embodiment of the present invention collects index data for the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation index system, obtains evaluation index data, and then performs data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results, thereby effectively improving the accuracy of the AI ​​programming assistant application effectiveness evaluation and providing more accurate support for the selection and introduction of AI programming assistants.

[0055] Figure 1 is an optional flowchart of the AI ​​programming assistant application evaluation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S130.

[0056] Step S110, constructing an evaluation index system, wherein the evaluation index system is constructed by pre-setting evaluation indexes according to a pre-setting hierarchical relationship.

[0057] Step S120, collecting indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data.

[0058] Step S130, performing data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results.

[0059] During the working process of this specific embodiment, the embodiment of the present invention first constructs an evaluation index system. Specifically, the evaluation index system in the embodiment of the present invention refers to an index system for evaluating and analyzing an AI programming assistant. Among them, the embodiment of the present invention obtains an evaluation index system by presetting evaluation indicators according to a preset hierarchical relationship. Correspondingly, the preset evaluation indicators in the embodiment of the present invention refer to the evaluation indicators of the AI ​​programming assistant, and the preset hierarchical relationship refers to the association relationship between each preset evaluation indicator in the evaluation index system. For example, in the embodiment of the present invention, the evaluation index system is divided into four levels, where different levels correspond to different preset evaluation indicators, and the preset evaluation indicators in different levels are associated by the preset hierarchical relationship, so as to construct an evaluation index system. Next, the embodiment of the present invention collects index data for the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation index system to obtain evaluation index data. Specifically, in the embodiment of the present invention, the programming assistant to be evaluated refers to an AI programming assistant that needs to be evaluated for application effectiveness. Among them, after constructing the evaluation index system, the embodiment of the present invention collects various index data (preset evaluation indicators) of the programming assistant to be evaluated to obtain evaluation index data. Finally, the embodiment of the present invention performs data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results. Specifically, after collecting the evaluation index data of the programming assistant to be evaluated, the embodiment of the present invention combines the preset hierarchical relationship, that is, the relationship between the various evaluation index data, to perform data analysis, thereby obtaining the evaluation analysis results. It is easy to understand that the embodiment of the present invention obtains a relatively complete evaluation index system by combining the preset evaluation index and the preset hierarchical relationship, and at the same time obtains the corresponding evaluation index data of the programming assistant to be evaluated, and then performs data analysis through the evaluation index data and the preset hierarchical relationship, which can effectively improve the accuracy of the evaluation of the application effectiveness of the AI ​​programming assistant, and provide more accurate support for the selection and introduction of AI programming assistants.

[0060] In some embodiments of the present invention, constructing an evaluation index system includes but is not limited to the following steps:

[0061] The code generation ratio indicator is constructed based on the new code line number indicator and the adopted code line number indicator.

[0062] The model effect index is constructed based on the preset logic correlation index and the preset style correlation index. The preset logic correlation index includes the completion code and context logic correlation index, and the preset style correlation index includes the completion code and context style correlation index.

[0063] Objective indicators are constructed based on the code generation ratio indicator and the model effect indicator.

[0064] The subjective index is constructed based on the customer satisfaction index, which is constructed from several customer evaluation indicators.

[0065] The overall evaluation index is constructed based on objective indicators and subjective indicators.

[0066] In this specific embodiment, the embodiment of the present invention first constructs a code generation ratio index based on the newly added code line number index and the adopted code line number index. Specifically, the newly added code line number index in the embodiment of the present invention refers to the code line number parameter added to the project or system within a certain period of time, and the adopted code line number index refers to the code line number parameter added to the project or system by the code generated by the AI ​​programming assistant. Correspondingly, the code generation ratio index in the embodiment of the present invention refers to the proportion parameter of the adopted code to the newly added code. The embodiment of the present invention constructs the code generation ratio index by combining the newly added code line number index and the adopted code line number index. Next, the embodiment of the present invention constructs a model effect index based on the preset logic correlation index and the preset style correlation index. Specifically, the preset logic correlation index in the embodiment of the present invention includes the completion code and context logic correlation index, that is, the logic correlation parameter between the code completed or generated by the AI ​​programming assistant and the context. Correspondingly, the preset style correlation index in the embodiment of the present invention includes the completion code and context style correlation index, that is, the correlation parameter between the code writing style between the code completed or generated by the AI ​​programming assistant and the context. The embodiment of the present invention judges the model effect by the completion code and context logic correlation and style correlation, and constructs the model effect index. Furthermore, the embodiment of the present invention constructs objective indicators based on the code generation ratio index and the model effect index. Specifically, in the embodiment of the present invention, the code generation ratio and the model effect are used as objective evaluation parameters for the evaluation of the application effect of the AI ​​programming assistant, and the corresponding objective indicators are constructed in combination with the code generation ratio index and the model effect index. Finally, the embodiment of the present invention constructs subjective indicators based on the customer satisfaction index, and then constructs an overall evaluation index based on the objective and subjective indicators, thereby forming an evaluation index system. Specifically, the customer satisfaction index in the embodiment of the present invention is constructed from a number of customer evaluation indicators. For example, Figure 2As shown, the customer satisfaction index in the embodiment of the present invention is decomposed into eight customer evaluation indicators, namely, visual comfort, difficulty in getting started, willingness to adopt the generated suggested code, whether the coding problem is effectively solved, whether the coding work efficiency is higher, whether the coding flow can be maintained, whether repetitive code writing can be reduced, and whether the number of browser searches can be reduced. Accordingly, the embodiment of the present invention first constructs a number of customer evaluation indicators to evaluate customer satisfaction and obtain a customer satisfaction index, and then uses customer satisfaction as the subjective judgment standard of the AI ​​programming assistant to construct a subjective indicator. Then, the embodiment of the present invention combines subjective indicators and objective indicators to evaluate the AI ​​programming assistant as a whole and constructs an overall evaluation index.

[0067] In some embodiments of the present invention, index data collection is performed on the programming assistant to be evaluated according to the preset evaluation index in the evaluation index system to obtain the evaluation index data, including but not limited to the following steps:

[0068] The newly added code line data and the adopted code line data of the programming assistant to be evaluated are read through the preset statistical module.

[0069] The text similarity between the completion code and the context is calculated using a preset similarity algorithm to obtain preset logical correlation data.

[0070] The text style relevance between the completed code and the context is calculated according to the code format consistency parameter and the naming convention consistency parameter to obtain preset style relevance data.

[0071] A number of customer evaluation data are obtained through a preset survey questionnaire, wherein the customer evaluation data corresponds to the customer evaluation index.

[0072] In this specific embodiment, the embodiment of the present invention first reads the newly added code line number data and the adopted code line number data of the programming assistant to be evaluated through a preset statistical module. Specifically, the preset statistical module in the embodiment of the present invention includes a statistical function module of the background management system of the programming assistant to be evaluated. Exemplarily, the embodiment of the present invention reads the newly added code line number and the adopted code line number data through the statistical function module of the background management system of the programming assistant to be evaluated, and obtains the corresponding newly added code line number data and the adopted code line number data. At the same time, the embodiment of the present invention calculates the text similarity between the completion code and the context through a preset similarity algorithm to obtain preset logical correlation data. Specifically, the preset similarity algorithm in the embodiment of the present invention refers to an algorithm for calculating text similarity, including Jaccard similarity algorithm, SimHash algorithm and cosine similarity algorithm. Among them, the embodiment of the present invention calculates the similarity between the completion code generated by the programming assistant to be evaluated and the text of the original context, and then determines the logical correlation between the completion code and the context, that is, obtains the preset logical correlation data. Next, the embodiment of the present invention calculates the text style correlation between the completion code and the context according to the code format consistency parameter and the naming specification consistency parameter to obtain the preset style correlation data. Specifically, the code format consistency parameter in the embodiment of the present invention refers to an indicator of whether the writing format between the completion code generated by the programming assistant to be evaluated and the original context text is consistent. Correspondingly, the naming specification consistency parameter in the embodiment of the present invention refers to an indicator of whether the naming format between the completion code generated by the programming assistant to be evaluated and the original context text is standardized, such as parameter definition or naming. Among them, the embodiment of the present invention obtains the completion code and context style correlation index data, that is, the preset style correlation data, by weighted average calculation of the code format consistency parameter (accounting for 0.5 weight) and the naming specification consistency parameter (accounting for 0.5 weight). In addition, the embodiment of the present invention obtains a number of customer evaluation data through a preset survey questionnaire. Specifically, the customer evaluation data in the embodiment of the present invention corresponds to the customer evaluation index. Among them, the embodiment of the present invention distributes a preset survey questionnaire to relevant developers to collect customer evaluation data of each developer through a preset survey questionnaire, thereby forming subjective evaluation information on the programming assistant to be evaluated.

[0073] In some embodiments of the present invention, obtaining a number of customer evaluation data through a preset survey questionnaire includes but is not limited to the following steps:

[0074] A preset survey questionnaire is constructed through a scoring system based on customer evaluation indicators.

[0075] Customer evaluation data is collected through preset survey questionnaires.

[0076] In this specific embodiment, the embodiment of the present invention first constructs a preset survey questionnaire by a scoring system based on the customer evaluation indicators, and then collects customer evaluation data through the preset survey questionnaire. Specifically, the embodiment of the present invention adopts a scoring system to define the scoring criteria of each customer evaluation indicator, such as each indicator has a full score of 5 points, and the higher the score, the more satisfied the user is with the content of the indicator, thereby constructing a preset survey questionnaire. Next, the embodiment of the present invention distributes the preset survey questionnaire to several developers, thereby obtaining the customer evaluation data of these developers on the programming assistant to be evaluated. Among them, the customer evaluation data in the embodiment of the present invention includes visual comfort data (i.e., the visual comfort of the application interface of the programming assistant to be evaluated), getting started data (i.e., the difficulty of applying the programming assistant to be evaluated), whether the generated suggested code is willing to be adopted (i.e., whether the relevant developers are willing to adopt the code generated by the programming assistant to be evaluated), whether the coding problem is effectively solved (i.e., whether the coding problem of the relevant developers can be effectively solved after applying the programming assistant to be evaluated), whether the coding work efficiency is higher (i.e., whether the application of the programming assistant to be evaluated is helpful to the coding efficiency of the relevant developers), whether the coding flow can be maintained (i.e., whether it conforms to the daily usage habits of the developer and has no negative impact), whether repetitive code writing can be reduced (reflecting whether the use of the tool can effectively improve work efficiency) and whether the number of browser searches can be reduced (reflecting whether the use of the tool can effectively improve concentration).

[0077] In some embodiments of the present invention, data analysis is performed based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results, including but not limited to the following steps:

[0078] By comparing the number of new code lines and the number of adopted code lines, we can get the code generation ratio data.

[0079] The model effect data is obtained by performing a weighted average calculation based on the preset logical correlation data and the preset style correlation data.

[0080] The customer satisfaction data is calculated based on the customer evaluation data.

[0081] Objective parameter data is obtained by performing weighted average calculation based on code generation ratio data and model effect data.

[0082] The subjective parameter data is determined based on the customer satisfaction data, and the evaluation and analysis results are obtained by weighted calculation of the objective parameter data and the subjective parameter data.

[0083] In this specific embodiment, the embodiment of the present invention first compares and calculates the data of the number of new code lines and the number of adopted code lines to obtain the code generation proportion data. Specifically, the embodiment of the present invention calculates the ratio of the number of adopted code lines to the number of new code lines, thereby calculating the code generation proportion data. Then, the embodiment of this aspect performs weighted average calculation according to the preset logic correlation data and the preset style correlation data to obtain the model effect data. Specifically, in the embodiment of the present invention, the model effect data is obtained by weighted average calculation of the correlation between the completed code and the context logic (i.e., the preset logic correlation, accounting for 0.5 weight) and the correlation between the completed code and the context style (i.e., the preset style correlation, accounting for 0.5 weight). Further, the embodiment of the present invention calculates the customer satisfaction data according to the customer evaluation data. Specifically, the embodiment of the present invention performs customer satisfaction analysis by combining the collected customer evaluation data to obtain the customer satisfaction data. Then, the embodiment of the present invention performs weighted average calculation according to the code generation proportion data and the model effect data to obtain the objective parameter data. Specifically, the embodiment of the present invention combines the code generation proportion data and the model effect data for weighted average calculation to obtain the objective parameter data. For example, the calculation formula of the objective parameter data in the embodiment of the present invention is: objective parameter data = 0.5 * code generation ratio data + 0.5 * model effect data. Next, the embodiment of the present invention determines the subjective parameter data based on the customer satisfaction data, and then performs weighted calculation on the objective parameter data and the subjective parameter data to generate an evaluation and analysis result. Specifically, the weight of the customer satisfaction data in the embodiment of the present invention is 1, that is, the subjective parameter data = 1 * customer satisfaction data. Accordingly, the embodiment of the present invention combines subjective indicators and objective indicators to analyze the application effectiveness of the programming assistant to be evaluated, generates overall evaluation data, and obtains evaluation and analysis results. Among them, the embodiment of the present invention reflects the evaluation and analysis results through overall evaluation data. Accordingly, the calculation formula of the overall evaluation data in the embodiment of the present invention is: overall evaluation data = 0.8 * objective parameter data + 0.2 * subjective parameter data.

[0084] In some embodiments of the present invention, calculating the text similarity between the completion code and the context by a preset similarity algorithm to obtain preset logical correlation data includes but is not limited to the following steps:

[0085] The text similarity between the completion code and the context is calculated using the cosine similarity algorithm to obtain the preset logical correlation data.

[0086] In this specific embodiment, the embodiment of the present invention uses a preselected similarity algorithm to calculate the text similarity between the completion code and the context to obtain preset logical correlation data. Specifically, the cosine similarity algorithm in the embodiment of the present invention is an algorithm that evaluates the similarity by measuring the cosine value of the angle between two vectors. The embodiment of the present invention uses the cosine similarity algorithm to calculate the similarity between the completion code and the context text, which can effectively improve the calculation accuracy of the preset logical correlation data and can more accurately reflect the similarity between texts.

[0087] In some embodiments of the present invention, calculating the customer satisfaction data based on the customer evaluation data includes but is not limited to the following steps:

[0088] A first number of target survey questionnaires that meet a preset scoring threshold condition is determined based on the customer evaluation data.

[0089] The customer satisfaction data is calculated based on the first quantity and the second quantity, wherein the second quantity includes the number of distribution of the preset survey questionnaires.

[0090] In this specific embodiment, the embodiment of the present invention first determines the first number of target survey questionnaires that meet the preset evaluation threshold condition based on the customer evaluation data, and then calculates the customer satisfaction data based on the first number and the second number. Specifically, the preset score threshold condition in the embodiment of the present invention refers to the preset survey questionnaire score being greater than the preset score threshold. Correspondingly, the target survey questionnaire refers to the survey questionnaire with a score greater than the preset score threshold in the preset survey questionnaire, such as the preset survey questionnaire with a total score greater than 32 points. The embodiment of the present invention first determines the number of target survey questionnaires in the collected preset survey questionnaires, that is, the first number. Then, the embodiment of the present invention calculates the customer satisfaction data in combination with the first number and the second number. Among them, the second number in the embodiment of the present invention refers to the number of valid preset questionnaires issued and collected. For example, the calculation formula for customer satisfaction data in the embodiment of the present invention is: customer satisfaction data = (number of questionnaires with 32 points or above / total number of questionnaires issued) * 100%.

[0091] Below, in conjunction with a specific AI programming assistant application evaluation scenario, the solution of the embodiment of the present invention is described in detail and explained:

[0092] For example, refer to Figure 3 , Figure 3A flowchart of the AI ​​programming assistant application evaluation provided in the embodiment of the present invention. Specifically, the embodiment of the present invention first designs an evaluation index system. For example, the evaluation index system in the embodiment of the present invention is designed as a pyramid type, and the evaluation index is decomposed layer by layer from top to bottom, including 4 levels: the first level index is the overall evaluation index; the second level index is decomposed from the first level index, including subjective indicators and objective indicators; the third level index is decomposed from the second level index, and the objective index is decomposed into two indicators, code generation ratio and model effect, and the subjective index is decomposed into customer satisfaction index; the fourth level index is decomposed from the third level index, and the code generation ratio index is decomposed into two indicators, namely the number of new code lines and the number of adopted code lines, and the model effect index is decomposed into two indicators, namely the correlation between the completed code and the context logic and the correlation between the completed code and the context style. The customer satisfaction index is decomposed into 8 indicators, namely, visual comfort, difficulty of getting started, willingness to adopt the generated suggested code, whether the coding problem is effectively solved, whether the coding work efficiency is higher, whether the coding flow can be maintained, whether the repeated code writing can be reduced, and whether the number of browser searches can be reduced. Next, the embodiment of the present invention collects evaluation index data. For example, the collection object of the embodiment of the present invention is the fourth-level indicator in the evaluation index system. Among them, for the number of newly added code lines and the number of adopted code lines, the embodiment of the present invention directly reads from the statistical function module of the AI ​​programming assistant background management system. At the same time, for the index of the correlation between the completed code and the context logic, the embodiment of the present invention calculates the text similarity between the completed code and the context through the cosine similarity algorithm. In addition, for the index of the correlation between the completed code and the context style, the embodiment of the present invention obtains it by weighted average calculation of the code format consistency (0.5 weight) and the naming specification consistency (0.5 weight). Correspondingly, for the eight indicators of visual comfort, difficulty of getting started, willingness to adopt the generated suggested code, whether the coding problem is effectively solved, whether the coding work efficiency is higher, whether the coding flow can be maintained, whether repeated code writing can be reduced, and whether the number of browser searches can be reduced, the embodiment of the present invention first designs a survey questionnaire through a scoring system. Among them, each indicator has a full score of 5 points, and the higher the score, the more satisfied the user is with the evaluation content of the indicator. Then, the evaluation index data is obtained by distributing a survey questionnaire to developers. Finally, the embodiment of the present invention analyzes the collected evaluation index data. For example, the embodiment of the present invention gathers evaluation indicators layer by layer from bottom to top through a weighted average algorithm. First, the embodiment of the present invention assigns weights to evaluation indicators. For the third-layer indicators, the weights of code generation ratio and model effect are 0.5 and 0.5 respectively, and the weight of user satisfaction rate is 1. For the second-layer indicators, the weights of objective indicators and subjective indicators are 0.8 and 0.2 respectively. Secondly, the fourth-layer indicator data is processed to obtain the third-layer indicators. For the code generation ratio indicator, the embodiment of the present invention obtains it by calculating the ratio of the number of adopted code lines to the number of newly added code lines.At the same time, for the model effect index, the embodiment of the present invention obtains it by weighted average calculation of the correlation between the completion code and the context logic (0.5 weight) and the correlation between the completion code and the context style (0.5 weight). In addition, for customer satisfaction, the calculation formula is: customer satisfaction rate = (number of questionnaires with 32 points or above / total number of questionnaires issued) * 100%. Then, the third-level indicators are weighted to obtain the second-level indicators. For objective indicators, the calculation formula is: objective indicators = 0.5*code generation ratio + 0.5*model effect. For subjective indicators, the calculation formula is: subjective indicators = 1*user satisfaction. Finally, the second-level indicators are weighted to obtain the first-level evaluation indicators. The calculation formula for the overall evaluation index is: overall evaluation index = 0.8*objective indicators + 0.2*subjective indicators.

[0093] Exemplarily, the embodiment of the present invention verifies the actual application effects of three different AI coding assistants in improving programming efficiency through the above-mentioned AI programming assistant application evaluation method, so as to compare the advantages and limitations of different AI programming assistant products, and provide a basis for the subsequent formal introduction of AI programming assistants. Among them, the embodiment of the present invention uses three different AI coding assistants for the same length of time by several relevant developers, such as using them for ten days. Accordingly, the evaluation index of AI programming assistants in the embodiment of the present invention consists of two parts: objective index and subjective index. Among them, the objective index focuses on the proportion of code generation and model effect evaluation; the subjective index includes 10 subjective evaluation index items such as "visual comfort, operation in line with habits", each index item is 5 points, totaling 50 points, and the customer satisfaction rate can be calculated through these 10 index items, and the calculation formula is: customer satisfaction rate = (number of questionnaires with scores of 30 points or more / total number of questionnaires) * 100%. In addition, in the embodiment of the present invention, the code generation proportion index is automatically collected by these three AI coding assistants, which can be viewed by logging into the corresponding background management system, and the model effect evaluation and subjective index are obtained by issuing questionnaires.

[0094] Correspondingly, for the first AI programming assistant, during the use period, the total number of new code lines was 27,790 lines, the number of code lines recommended and adopted by the AI ​​programming assistant was 10,190 lines, and the code generation accounted for 37%. The 10,190 lines of adopted code were distributed by programming language as follows: 6,592 lines of Java, 1,245 lines of Vue, 600 lines of Python, 484 lines of JavaScript, 409 lines of TypeScript, and 860 lines of others (C#, Shell, etc.), accounting for 65%, 12%, 6%, 5%, 4%, and 8% respectively. In addition, regarding the relevance of the code supplemented by the model to the context, 45% of users believe that "the supplemented code is relevant to the context", 50% of users believe that "the supplemented code is relevant to the context, and the logic is more accurately connected to the context", and 5% of users believe that "the supplemented code is relevant to the context, and the logic is connected to the context and accurate". Regarding other effects of the model, 62% of users believe that "the completed code has no obvious logical errors and no obvious duplication", 29% of users believe that "the completed code style is consistent with the original text", 33% of users believe that "the completed code completely covers the answer code logic", 7% of users believe that "no code is generated", and 7% of users believe that "the code is generated, but it has no relevance to the context". At the same time, the subjective indicators include 10 indicators such as "visual comfort and operation in line with habits", each of which is 5 points. If the user checks each indicator and scores 3 points or more, it means that the user is satisfied with the indicator. According to the data analysis of the questionnaire and the user satisfaction calculation formula (the number of questionnaires with a score of 30 points or more / the total number of questionnaires)*100%, the user satisfaction is 37÷42*100%=88%. Therefore, after evaluation and analysis, the trial results of the first AI programming assistant are summarized as follows: the overall user satisfaction rate is 89%, 90% of users believe that "coding work is more efficient" and "it can reduce the writing of repetitive code", 50% of users believe that the assistant "the code supplemented is relevant to the context, and the logic and context are more accurately connected", 33% of users believe that "the code supplemented completely covers the answer code logic", code generation accounts for 37% (10,190 lines of adopted code / 27,790 lines of new code), and 21% of users reported that the assistant's automatic code generation speed needs to be further improved.

[0095] Correspondingly, for the second AI programming assistant, during the use period, the total number of recommended code lines was 12,877 lines, the number of adopted code lines was 5,134 lines, and the adoption rate was 40%. Among them, the 5,134 lines of adoption were distributed by function as follows: 2,386 lines of comment generation code, 2,398 lines of intelligent continuation, and 350 lines of other functions (code tuning suggestions, single test adoption, line comments, function comments), accounting for 47%, 47%, and 6% respectively. Correspondingly, the 5,134 lines of adoption were distributed by programming language as follows: 3,080 lines of Java, 874 lines of JavaScript, 306 lines of Python, and 874 lines of others (Shell, Vue, SQL, etc.), accounting for 60%, 17%, 6%, and 17% respectively. Among them, regarding the relevance between the code supplemented by the model and the context, 36% of users believe that "the supplemented code is relevant to the context", 62% of users believe that "the supplemented code is relevant to the context, and the logic is more accurately connected with the context", and 2% of users believe that "the supplemented code is relevant to the context, and the logic is connected and accurate with the context". As for other effects of the model, 58% of users believe that "the supplemented code has no obvious logical errors and no obvious repetitions", 40% of users believe that "the style of the supplemented code is consistent with the original text", 33% of users believe that "the supplemented code completely covers the logic of the answer code", 7% of users believe that "no code is generated", and 5% of users believe that "the code is generated, but it has no relevance to the context". Further, according to the data analysis of the questionnaire and the user satisfaction calculation formula (number of questionnaires with 30 points or more / total number of questionnaires)*100%, the user satisfaction is 49÷55*100%=95%. After evaluation and analysis, the trial results of Baidu Comate are summarized as follows: the user satisfaction rate (customer satisfaction rate) is 95%, 89% of users believe that "coding work efficiency is higher" and "it can reduce the writing of repeated code", 62% of users believe that the "code supplemented by the assistant is relevant to the context, and the logic and context are more accurately connected", 33% of users believe that "the code supplemented completely covers the logic of the answer code", the code adoption rate is 40% (5134 lines of adopted code / 12877 lines of recommended code), and 18% of users reported that the assistant "got stuck in the middle of generating code".

[0096] In addition, for the third AI programming assistant, during the use period, regarding the relevance of the code supplemented by the model to the context, 46% of users believed that "the supplemented code is relevant to the context", 50% of users believed that "the supplemented code is relevant to the context, and the logic is more accurately connected to the context", and 4% of users believed that "the supplemented code is relevant to the context, and the logic is connected and accurate to the context". Regarding other effects of the model, 62% of users believed that "the supplemented code has no obvious logical errors and no obvious repetitions", 38% of users believed that "the style of the supplemented code is consistent with the original text", 12% of users believed that "the supplemented code completely covers the answer code logic", 4% of users believed that "no code was generated", and 8% of users believed that "the code was generated, but it has no relevance to the context". Among them, user satisfaction is 20÷26*100%=77%. Therefore, after evaluation and analysis, the trial results of the third AI programming assistant are summarized as follows: the overall user satisfaction rate is 77%, 88% of users believe that "coding work is more efficient" and "it can reduce the writing of repeated code", 50% of users believe that the assistant "supplements the code with relevance to the context, and the logic and context are connected more accurately", and 15% of users reported that during use, "when the external network communication is actually normal, the product prompts "Network error" and cannot generate code".

[0097] Accordingly, the embodiment of the present invention performs data analysis on the evaluation data of the three AI programming assistants, thereby obtaining corresponding application effectiveness evaluation results, and further providing support for selecting a suitable AI programming assistant from among the many AI programming assistants that currently have no obvious differences among each other, and effectively improving the accuracy of the AI ​​programming assistant application effectiveness evaluation.

[0098] See also Figure 4 The embodiment of the present application also provides an AI programming assistant application evaluation system, which can implement the above-mentioned AI programming assistant application evaluation method, and the system includes:

[0099] The first module 210 is used to construct an evaluation index system, wherein the evaluation index system is constructed by pre-setting evaluation indexes according to a pre-setting hierarchical relationship.

[0100] The second module 220 is used to collect indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data.

[0101] The third module 230 is used to perform data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results.

[0102] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0103] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned AI programming assistant application evaluation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0104] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0105] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0106] The processor 310 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0107] The memory 320 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 320 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 320, and the processor 310 calls and executes the AI ​​programming assistant application evaluation method of the embodiment of this application;

[0108] Input / output interface 330, used to implement information input and output;

[0109] Communication interface 340, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0110] bus 350 , which transmits information between the various components of the device (e.g., processor 310 , memory 320 , input / output interface 330 , and communication interface 340 );

[0111] The processor 310 , the memory 320 , the input / output interface 330 and the communication interface 340 are connected to each other in communication within the device via the bus 350 .

[0112] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned AI programming assistant application evaluation method is implemented.

[0113] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0116] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0117] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0119] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0120] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0122] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0125] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for evaluating the application of an AI programming assistant, characterized in that: The method comprises the following steps: Constructing an evaluation index system; wherein the evaluation index system is constructed by pre-setting evaluation indexes according to a preset hierarchical relationship; Collecting indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data; Data analysis is performed based on the evaluation index data and the preset hierarchical relationship to obtain evaluation analysis results.

2. The method according to claim 1, characterized in that The construction of the evaluation index system includes: The code generation ratio indicator is constructed based on the new code line number indicator and the adopted code line number indicator; A model effect index is constructed based on a preset logic correlation index and a preset style correlation index; wherein the preset logic correlation index includes a completion code and context logic correlation index, and the preset style correlation index includes a completion code and context style correlation index; An objective indicator is obtained according to the code generation ratio indicator and the model effect indicator; A subjective index is constructed based on a customer satisfaction index; wherein the customer satisfaction index is constructed from a number of customer evaluation indicators; An overall evaluation index is constructed based on the objective index and the subjective index.

3. The method according to claim 2, characterized in that The collecting of indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain the evaluation indicator data includes: Reading the newly added code line data and the adopted code line data of the programming assistant to be evaluated through a preset statistical module; Calculate the text similarity between the completed code and the context through a preset similarity algorithm to obtain preset logical correlation data; The text style relevance between the completed code and the context is calculated according to the code format consistency parameter and the naming specification consistency parameter to obtain preset style relevance data; A number of customer evaluation data are obtained through a preset survey questionnaire; wherein the customer evaluation data corresponds to the customer evaluation index.

4. The method according to claim 3, characterized in that The customer evaluation data obtained through the preset survey questionnaire includes: The preset survey questionnaire is constructed by using a scoring system according to the customer evaluation indicators; The customer evaluation data is collected through the preset survey questionnaire.

5. The method according to claim 3, characterized in that: The data analysis is performed according to the evaluation index data and the preset hierarchical relationship to obtain the evaluation analysis results, including: Compare and calculate the newly added code line number data and the adopted code line number data to obtain code generation ratio data; Performing weighted average calculation according to the preset logic correlation data and the preset style correlation data to obtain model effect data; Calculate customer satisfaction data based on the customer evaluation data; Perform weighted average calculation based on the code generation ratio data and the model effect data to obtain objective parameter data; The subjective parameter data is determined according to the customer satisfaction data, so as to obtain the evaluation analysis result by performing weighted calculation on the objective parameter data and the subjective parameter data.

6. The method according to claim 3, characterized in that: The method of calculating the text similarity between the completion code and the context by using a preset similarity algorithm to obtain preset logical correlation data includes: The text similarity between the completion code and the context is calculated using the cosine similarity algorithm to obtain the preset logical correlation data.

7. The method according to claim 5, characterized in that The customer satisfaction data is calculated based on the customer evaluation data, including: Determining a first number of target survey questionnaires that meet a preset scoring threshold condition according to the customer evaluation data; The customer satisfaction data is calculated based on the first number and the second number; wherein the second number includes the number of distributed preset survey questionnaires.

8. An AI programming assistant application evaluation system, characterized in that: The system comprises: The first module is used to construct an evaluation index system; wherein the evaluation index system is constructed by pre-setting evaluation indicators according to a preset hierarchical relationship; The second module is used to collect indicator data of the programming assistant to be evaluated according to the preset evaluation indicators in the evaluation indicator system to obtain evaluation indicator data; The third module is used to perform data analysis based on the evaluation index data and the preset hierarchical relationship to obtain evaluation and analysis results.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images