Coding service quality evaluation method and device

Abstract

The invention provides a coding service quality evaluation method and device. The coding service quality evaluation method comprises the steps of obtaining interaction data of a user and an auxiliary coding service in a preset time period and a code merging request submitted by the user in a code warehouse; comparing first code data contained in the interaction data with second code data contained in the code merging request to obtain third code data; wherein the third code data comprises the same code data in the second code data and the first code data; according to the third code data, determining an evaluation parameter for performing service quality evaluation on the auxiliary coding service; therefore, the service quality of the auxiliary coding service can be quantified accurately and objectively, and the evaluation accuracy of the auxiliary coding service is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and computing device for evaluating the quality of coding services. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI-assisted coding has become an important tool in the software development field in recent years. AI-assisted coding tools, through deep learning and natural language processing technologies, can effectively improve development efficiency and lower the programming threshold. Currently, various AI-assisted coding tools exist, offering features such as code auto-completion, code generation, and error detection.

[0003] Typically, when using AI-assisted coding tools, their effectiveness can be evaluated based on the accuracy of code generation or runtime performance. However, this evaluation method only considers the performance of the AI-assisted coding tool itself and does not form an analytical framework that covers multiple levels such as team collaboration, user differences, and code quality, resulting in low accuracy in the evaluation of AI-assisted coding tools. Summary of the Invention

[0004] This application provides a method and computing device for evaluating the quality of coding services, which can quantify the quality of auxiliary coding services more accurately and objectively, thereby improving the evaluation accuracy of auxiliary coding services.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a method for evaluating the quality of coding services. The method includes: acquiring interaction data between a user and an auxiliary coding service within a preset time period, as well as code merge requests submitted by the user in a code repository; comparing first code data contained in the interaction data with second code data contained in the code merge requests to obtain third code data; wherein the third code data includes code data identical to that in the second code data and the first code data; and determining evaluation parameters for evaluating the quality of service of the auxiliary coding service based on the third code data.

[0007] Based on this solution, by comparing the first code data in the interaction data of the auxiliary coding service within a preset time period with the second code data in the code merging request, the overlapping third code data between the first and second code data can be obtained. Since the third code data is generated by the auxiliary coding service and applied in the code merging request, when determining the evaluation parameters for assessing the service quality of the auxiliary coding service based on the overlapping third code data, the application of the code data generated by the auxiliary coding service in the code merging request is considered. This allows for the quantification of the actual contribution of the auxiliary coding service to development efficiency, code quality, and team collaboration. Therefore, the solution in this application can more accurately and objectively quantify the service quality of the auxiliary coding service when evaluating it, thus improving the accuracy of the evaluation.

[0008] In another possible implementation, the evaluation parameters for assessing the service quality of the auxiliary coding service based on the third code data include: determining the code upload rate based on the third code data and the first code data; the code upload rate is used to characterize the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; determining the proportion of newly added code based on the third code data and the second code data; the proportion of newly added code is used to characterize the proportion of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; wherein the evaluation parameters include the code upload rate and / or the proportion of newly added code.

[0009] Based on this scheme, by determining the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository, and by determining the proportion of the first code data generated by the auxiliary coding service calling the target coding model in the code repository, the service quality of the auxiliary coding service can be evaluated from multiple dimensions, effectively quantifying the effectiveness of the quality of the code generated by the auxiliary coding service.

[0010] In another possible implementation, determining the code upload rate based on the third code data and the first code data includes: determining the code upload rate based on the ratio of the number of lines of code in the third code data to the number of lines of code in the first code data.

[0011] Based on this scheme, the ratio of the number of lines of code in the third code data to the number of lines of code in the first code data is used as the code upload rate. This can accurately assess the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository, and achieve an effective evaluation of the auxiliary coding service in terms of coding usage.

[0012] In another possible implementation, determining the proportion of newly added code based on the third code data and the second code data includes: determining the proportion of newly added code based on the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data.

[0013] Based on this scheme, the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data is used as the proportion of newly added code. This can accurately assess the proportion of the first code data generated by the auxiliary coding service calling the target coding model in the code repository, and achieve an effective evaluation of the auxiliary coding service in terms of coding proportion.

[0014] In another possible implementation, the above-mentioned acquisition of user interaction data with the auxiliary coding service within a preset time period includes: acquiring the coding request stored in the database of the auxiliary coding service, and the first code data obtained by the auxiliary coding service calling the target coding model to encode the coding request; wherein the interaction data includes the first code data and the coding request.

[0015] Based on this solution, by storing the encoding request and the first code data as interactive data in the database and transmitting data based on the database, the reliability and security of data transmission can be improved. Furthermore, when interactive data needs to be called later, it can be directly retrieved from the database, thus improving evaluation efficiency.

[0016] In another possible implementation, the above-mentioned acquisition of the user's interaction data with the auxiliary coding service and the code merge request submitted by the user in the code repository within the preset time period includes: when the timed service starts a timed task, acquiring the above-mentioned interaction data between the user and the auxiliary coding service and the code merge request submitted by the user in the code repository within the preset time period.

[0017] Based on this solution, the scheduled tasks of the scheduled service can automatically acquire the interaction data and code merging requests of the auxiliary coding service, so as to evaluate the service quality of the auxiliary coding service in different time periods and ensure the accuracy of time control.

[0018] In another possible implementation, comparing the first code data contained in the above-mentioned interaction data with the second code data contained in the above-mentioned code merging request to obtain the third code data includes: comparing the first code data and the second code data based on a text matching algorithm or an abstract syntax tree to obtain the third code data.

[0019] Based on this scheme, a text matching algorithm or abstract syntax tree is used in the process of comparing the first code data and the second code data, which can improve the accuracy of code comparison and thus provide reliable data basis for subsequent coding service quality assessment.

[0020] In another possible implementation, the method further includes: obtaining the user's parameter viewing request and determining the target parameter corresponding to the parameter viewing request; displaying the target parameter through the service dashboard of the auxiliary coding service; wherein the target parameter includes at least one of the evaluation parameters, code parameters, user parameters, and capability parameters of the auxiliary coding service.

[0021] Based on this solution, after determining the evaluation parameters, the target parameters, including the evaluation parameters, can be displayed through the service dashboard of the auxiliary coding service to show the usage status of the auxiliary coding service in real time, thus optimizing the user experience. In addition, the target parameters also include other parameters such as code parameters, user parameters, and the capability parameters of the auxiliary coding service. These parameters enable a more comprehensive evaluation of the auxiliary coding service, allowing users to have a more comprehensive understanding of the service quality of the auxiliary coding service.

[0022] Secondly, embodiments of this application provide a coding service quality assessment device, which includes: a data acquisition module, used to acquire interaction data between a user and an auxiliary coding service within a preset time period, as well as code merging requests submitted by the user in a code repository; a data comparison module, used to compare first code data contained in the interaction data with second code data contained in the code merging requests to obtain third code data; wherein the third code data includes code data that is identical in the second code data and the first code data; and a parameter determination module, used to determine assessment parameters for evaluating the service quality of the auxiliary coding service based on the third code data.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program for executing the coding service quality assessment method provided in the first aspect above.

[0024] Fourthly, this application also provides a computing device, including: one or more processors; and a memory configured to store one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the coding service quality assessment method as described in any one of the first aspects above.

[0025] Fifthly, embodiments of this application provide a computer program product that, when instructions in the computer program product are executed by a processor, performs the coding service quality assessment method provided in the first aspect above. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the architecture of an encoding service quality assessment system provided in an embodiment of this application.

[0027] Figure 2 This is a flowchart illustrating a coding service quality assessment method provided in an embodiment of this application.

[0028] Figure 3 This is a flowchart for determining evaluation parameters provided in an embodiment of this application.

[0029] Figure 4 This is a schematic diagram of the first page of a service dashboard provided in an embodiment of this application.

[0030] Figure 5 This is a schematic diagram of the second page of a service dashboard provided in an embodiment of this application.

[0031] Figure 6 This is a schematic diagram of the third page of a service dashboard provided in an embodiment of this application.

[0032] Figure 7 This is a schematic diagram of an encoding service quality assessment device provided in an embodiment of this application.

[0033] Figure 8 This is a schematic diagram of a coding service quality assessment device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. To facilitate a clear description of the technical solutions of the embodiments of this application, the use of terms such as "first," "second," etc., in the embodiments of this application is for illustrative purposes and to distinguish the objects being described. There is no particular order between them, nor does it indicate a specific limitation on the number of devices in the embodiments of this application, and they do not constitute any limitation on the embodiments of this application.

[0035] First, the terms used in the embodiments of this application will be explained.

[0036] Merge Request (MR): A merge request is used to request the merging of code developed or modified in one code branch into another code branch. For example, when a developer finishes developing or modifying code on their assigned code branch and needs to merge the code from that branch into a target code branch managed by other team members, the developer can initiate a merge request to merge the modified code with the code from the target code branch.

[0037] Code Repository (CR): A code repository is a digital system used for storing, version controlling, managing, and tracking source code. A code repository records every modification to the code, supports multi-branch parallel development, and provides team collaboration features, aiming to ensure the integrity, traceability, and collaborative efficiency of code during the software development process.

[0038] In the field of Natural Language Processing (NLP), a token refers to the smallest segmented unit of language, which can be a word, punctuation mark, or word fragment. In AI-assisted coding, a token typically represents a code snippet.

[0039] To address the issue of low accuracy in evaluating AI-assisted coding tools, this application provides a coding service quality evaluation method. This method determines evaluation parameters for the assisted coding service based on third-party code data generated by the assisted coding service and applied in the code merging request. Because this application considers the application of code data generated by the assisted coding service in the code merging request, it can quantify the actual contribution of the assisted coding service to development efficiency, code quality, and team collaboration. Therefore, the solution in this application can more accurately and objectively quantify the service quality of the assisted coding service, improving the accuracy of the evaluation.

[0040] Figure 1 An architecture diagram of an encoding service quality assessment system provided in this application embodiment is shown below. Figure 1 As shown, the coding service quality assessment system includes at least auxiliary coding service 101, scheduled service 102, code repository 103, database 104, and service dashboard 105.

[0041] The auxiliary coding service 101 generates code data based on user requests. This auxiliary coding service 101 can run on a computing device, which may be one or more servers, a server cluster consisting of several servers, or a cloud server on a cloud computing platform. For example, the auxiliary coding service 101 can call a coding model to process the user's coding request and obtain the corresponding coding result. If the coding service quality assessment system includes a scheduled service 102, the auxiliary coding service 101 can also obtain the assessment parameters sent by the scheduled service 102, and display these parameters on the service dashboard 105 corresponding to the auxiliary coding service 101.

[0042] The scheduled service 102 is used to acquire data based on a scheduled task. The scheduled service 102 can run on the same computing device as the auxiliary coding service 101, or it can run on different computing devices. This application embodiment does not limit this.

[0043] Code repository 103 is used to store and manage the source code of software development projects, as well as the code data corresponding to code merge requests submitted by users in the code repository. The code merge request is used to request the merging of the code data to be merged with the source code in code repository 103. The code data to be merged includes user-written code data and code data generated by users through assisted coding services.

[0044] Database 104 is used to store interaction data generated during interactions with the auxiliary coding service 101. Database 104 can also be used to store evaluation parameters for assessing the service quality of the auxiliary coding service.

[0045] like Figure 1 As shown, scheduled task 102 can retrieve interaction data of auxiliary coding service 101 from database 104 and code merge request from code repository 103 within a preset time period after the scheduled task starts; then, it compares the first code data in the interaction data with the second code data in the code merge request to obtain overlapping third code data. This third code data is the code data generated by the auxiliary coding service and applied in the code merge request. Finally, the evaluation parameters for service quality assessment of the auxiliary coding service are determined based on the overlapping third code data. Because this application considers the application of code data generated by the auxiliary coding service in the code merge request when evaluating the service quality of the auxiliary coding service, it can quantify the service quality of the auxiliary coding service more accurately and objectively, thus improving the accuracy of the evaluation of the auxiliary coding service.

[0046] Figure 2 This is a flowchart illustrating a coding service quality assessment method provided in an embodiment of this application, as shown below. Figure 2As shown, the method includes steps 201 to 203.

[0047] Step 201: Obtain the interaction data between the user and the auxiliary coding service within the preset time period, as well as the code merge requests submitted by the user in the code repository.

[0048] In some examples, interactive data generated by user calls to the auxiliary coding service within a preset time period can be retrieved from the auxiliary coding service's database. This interactive data includes the user-input coding request and the first code data obtained after the auxiliary coding service processes the coding request. This first code data may be a code snippet output by the auxiliary coding service when calling the target coding model. After the target coding model outputs the first code data, the coding request and the first code data can be stored in the database.

[0049] For example, the assisted coding service may provide at least one of the following services: intelligent question answering service, code completion service, code annotation service, unit testing service, and code interpretation service. This application embodiment does not limit the type of service provided by the assisted coding service.

[0050] The assisted encoding service can determine a target encoding model from multiple encoding models based on the encoding request, and call the target encoding model to encode the user-input encoding request to obtain first code data. These multiple encoding models may include at least one of intelligent question-answering models, code completion models, code annotation models, unit testing models, and code interpretation models. This application embodiment does not limit the type of encoding model.

[0051] For example, taking the encoding request as "Please help me complete this code", after the user enters the encoding request, the auxiliary encoding service can identify the encoding request, determine the target encoding model as the code completion model among multiple encoding models based on the identification result, and generate a prompt word based on the identification result; based on the prompt word, the code completion model is called to perform code completion processing and generate the first code data.

[0052] In some examples, user-submitted code merge requests within a preset time period can be retrieved from the code repository. These merge requests are submissions from users to the code repository to merge code data into a target branch within the repository. The code data to be merged includes at least one of the following: user-written code data and code data generated by the user through an assisted coding service.

[0053] Step S201 above includes: when the scheduled service starts a scheduled task, obtaining the interaction data between the user and the auxiliary coding service and the code merge request submitted by the user in the code repository within a preset time period.

[0054] In some examples, when a scheduled task is enabled by the scheduled service, the interaction data between the user and the auxiliary coding service, as well as the code merge requests submitted by the user in the code repository, are obtained within a preset time period according to the triggering rules of the scheduled task. The triggering rules of the scheduled task can be triggered at preset time intervals or at preset time points; this application embodiment does not limit the triggering rules of the scheduled task. By using the timing rules of the scheduled task, interaction data and code merge requests can be automatically obtained, thereby improving data acquisition efficiency.

[0055] The preset time period can be a time period determined by a scheduled task of a scheduled service. Taking a scheduled task that runs every 10 minutes as an example, it can retrieve user interaction data and code merging requests every 10 minutes. It should be noted that the number of times a user interacts with the auxiliary coding service within the preset time period may be once or multiple times.

[0056] For example, if a user submits three encoding requests to the auxiliary encoding service within 10 minutes after the scheduled task is started, and the auxiliary encoding service generates three sets of first code data respectively, then the obtained interaction data includes the three interaction data between the user and the auxiliary encoding service.

[0057] In some examples, within a preset time period, multiple user interaction data and code merging requests with the auxiliary coding service can be acquired. This interaction data and code merging requests may include user information, which can be used to distinguish between different users' interaction data and code merging requests. For example, user information may include at least one of the following: user identifier (e.g., ID), user role (e.g., software development engineer), and the user's affiliated organization or team.

[0058] For example, after the scheduled task is started, the interaction data A1 between user A and the auxiliary coding service and the code merge request A2 submitted by user A in the code repository can be retrieved from the database, as well as the interaction data B1 between user B and the auxiliary coding service and the code merge request B2 submitted by user B in the code repository.

[0059] Step 202: Compare the first code data in the interaction data with the second code data in the code merging request to obtain the third code data.

[0060] In some examples, first code data can be extracted from the interaction data. This first code data includes code snippets output by the auxiliary encoding service based on the user's encoding request and the target encoding model. After obtaining the interaction data, code data can be extracted using a code extraction algorithm, and the extracted code data can be preprocessed to obtain first code data in a standard format.

[0061] For example, the code extraction algorithm may be at least one of regular expression matching, abstract syntax tree, and machine learning-based intelligent extraction algorithms. This application does not limit the specific type of code recognition algorithm in its embodiments.

[0062] In some examples, second code data can be extracted from the code merge request. The second code data includes at least one of the user-written code data in the code merge request and the code data generated by the user through an auxiliary coding service. After obtaining the code merge request, a code extraction instruction can be triggered. In response to this instruction, code data is extracted from the code merge request, and the extracted code data is preprocessed to obtain second code data in a standard format.

[0063] For example, code extraction instructions may include at least one of the following: data acquisition interface call instructions, Git commands, and script call instructions. This application does not limit the specific type of code extraction instructions.

[0064] In some examples, after extracting the first code data from the interaction data and the second code data from the code merge request, the first and second code data can be compared to obtain the third code data. The third code data includes the same code data found in both the second and first code data; that is, the third code data is the overlapping code data between the second and first code data. Since a user's code merge request in the code repository may contain user-written code data or code data generated by a user-initiated auxiliary coding service, comparing the first and second code data reveals the third code data generated by the auxiliary coding service and applied in the code merge request. If there is significant overlap between the second and first code data in the third code data, it indicates that a large amount of code data generated by the auxiliary coding service is applied in the code merge request, suggesting that the quality of the auxiliary coding service's generated code data is high. Conversely, if there is limited overlap between the second and first code data in the third code data, it indicates that less code data generated by the auxiliary coding service is applied in the code merge request, suggesting that the quality of the auxiliary coding service's generated code data is low.

[0065] In some examples, step S202 above includes: comparing the first code data and the second code data based on a text matching algorithm or an abstract syntax tree to obtain the third code data.

[0066] For example, text matching algorithms may include string matching algorithms, such as the Knuth-Morris-Pratt (KMP) string matching algorithm or the Rabin-Karp (RK) hash function-based string matching algorithm. When comparing the first code data and the second code data using a text matching algorithm, the first code data and the second code data can be split into code fragments (Tokens) respectively, and the same code fragments can be found based on the string matching algorithm.

[0067] For example, when comparing the first code data and the second code data using an abstract syntax tree, the first code data and the second code data can be converted into structured representations (Abstract Syntax Tree, AST) respectively. Then, by recursively traversing the nodes of the two ASTs, the node matching results are determined layer by layer. Then, based on the node matching results, the similarity between the two AST segments is quantified, and the matching subtrees are located. Finally, the matching subtrees in the ASTs are mapped back to the original code to determine the identical code segments.

[0068] This application embodiment compares the first code data and the second code data using a text matching algorithm or an abstract syntax tree, which can achieve accurate code difference detection to obtain overlapping third code data. Then, when performing quality assessment based on the third code data, the service quality of the auxiliary coding service can be quantified more accurately and objectively.

[0069] In other examples, when comparing the first code data with the second code data, a first code feature can be extracted from the first code data, and a second code feature can be extracted from the second code data. Based on the first and second code features, the first code data is compared with the second code data to obtain the third code data. The first code feature may include at least one of the following: code generation location, code generation time, and code line number. The second code feature may include at least one of the following: code commit time, modified file list, and code line number.

[0070] If multiple users' interaction data and code merging requests are obtained within a preset time period, when comparing the first code data in the interaction data with the second code data in the code merging request, the first code data in the interaction data between the same user and the auxiliary coding service and the second code data in the code merging request submitted by the user can be compared based on the user information to determine the code data in the code data generated by the user calling the auxiliary coding service that is applied in the code merging request submitted by the user.

[0071] In some examples, when acquiring interaction data and code merging requests from multiple users with the auxiliary coding service, during the comparison of the interaction data and code merging requests from multiple users with the auxiliary coding service, the interaction data and code merging requests of different users can be distinguished based on user information, and the first code data in the interaction data and the second code data in the code merging request can be compared on a user-by-user basis to obtain the third code data for each user.

[0072] For example, taking step 201 as obtaining user A's interaction data A1 with the auxiliary coding service and user A's code merge request A2 submitted in the code repository, and user B's interaction data B1 with the auxiliary coding service and user B's code merge request B2 submitted in the code repository as examples, the first code data A11 in interaction data A1 can be compared with the second code data A21 in code merge request A2 to obtain a third code data A3 that overlaps with the first code data A11 and the second code data A21. This third code data A3 represents the code data used in the code merge request submitted by user A in the code data generated by user A calling the auxiliary coding service. Similarly, the first code data B11 in interaction data B1 can be compared with the second code data B21 in code merge request B2 to obtain a third code data B3 that overlaps with the first code data B11 and the second code data B21. This third code data B3 represents the code data used in the code merge request submitted by user B in the code data generated by user B calling the auxiliary coding service.

[0073] Step 203: Based on the third code data, determine the evaluation parameters for assessing the service quality of the auxiliary coding service.

[0074] In some instances, after comparing the first code data in the interaction data with the second code data in the code merging request to obtain the third code data, since the third code data includes the same code data in the second code data and the first code data, the evaluation parameters for the service quality assessment of the auxiliary coding service can be determined based on the third code data and the first code data, or based on the third code data and the second code data.

[0075] For example, evaluation parameters include code upload rate and / or new code percentage. The code upload rate characterizes the usage rate of the first code data generated by the auxiliary coding service calling the target coding model within the code repository; the new code percentage characterizes the proportion of the first code data generated by the auxiliary coding service calling the target coding model that is applied in the code repository. By determining the code upload rate and / or new code percentage, the service quality of the auxiliary coding service can be effectively evaluated.

[0076] A high code integration rate indicates that the initial code data generated by the auxiliary coding service from the target coding model is used extensively in the code repository. This means that a large portion of the initial code data generated by the auxiliary coding service can be applied to the code merge request; in other words, a high percentage of the initial code data generated by the auxiliary coding service is used in the code repository. A higher code integration rate also indicates a higher percentage of the initial code data generated by the auxiliary coding service being used in the code repository, meaning that most of the code data generated by the auxiliary coding service can be used in the code merge request. This means a higher percentage of the coding results generated by the auxiliary coding service based on the user's coding needs are ultimately adopted by the user. Conversely, a low code integration rate indicates that a large portion of the initial code data generated by the auxiliary coding service is not used in the code merge request. This means a lower percentage of the initial code data generated by the auxiliary coding service from the target coding model is used in the code repository, meaning only a small portion of the code data generated by the auxiliary coding service can be used in the code merge request. In other words, a lower percentage of the coding results generated by the auxiliary coding service based on the user's coding needs are ultimately adopted by the user. Therefore, the code upload rate can be used to assess the proportion of code data generated by the assisted coding service that is actually used in the code repository, and thus evaluate the service quality of the assisted coding service.

[0077] A high percentage of newly added code indicates that most of the code data in the merge requests submitted to the code repository is generated by the auxiliary coding service. A higher percentage of newly added code suggests a higher proportion of auxiliary coding service data in the merge requests submitted by users. Therefore, a high percentage of newly added code indicates high service quality from the auxiliary coding service, which effectively identifies user coding requests and accurately provides corresponding coding results. Conversely, a low percentage of newly added code indicates that most of the code data in the merge requests is not generated by the auxiliary coding service; for example, most of the code data in the merge requests is user-written code. Therefore, a low percentage of newly added code suggests poor service quality from the auxiliary coding service, which cannot generate the desired code data, resulting in a lower proportion of auxiliary coding service data in the merge requests submitted to the code repository. Thus, the percentage of newly added code can be used to assess the proportion of code data generated by the auxiliary coding service in the code data submitted to the code repository, thereby evaluating the service quality of the auxiliary coding service.

[0078] Figure 3 A flowchart for determining evaluation parameters is provided for an embodiment of this application, such as... Figure 3As shown, step 203 above includes steps 301 to 302.

[0079] Step 301: Determine the code upload rate based on the third code data and the first code data.

[0080] In some examples, step 301 includes determining the code upload rate based on the ratio of the number of lines of code in the third code data to the number of lines of code in the first code data. By calculating the ratio of the number of lines of code in the third code data to the number of lines of code in the first code data, the proportion of the third code data used in the code merge request within the first code data generated by the auxiliary coding service can be obtained. This proportion indicates the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository.

[0081] The code inclusion rate can be calculated in the following way:

[0082]

[0083] For example, if the third code data has 8 lines of code and the first code data has 16 lines of code data, the code upload rate can be calculated to be 50%, which means that half of the 16 lines of first code data generated by the auxiliary coding service can be used in the code merging request.

[0084] Step 302: Determine the proportion of newly added codes based on the third code data and the second code data.

[0085] In some examples, step 302 includes determining the proportion of newly added code based on the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data. By calculating the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data, the proportion of the third code data used in the code merge request in the first code data generated by the auxiliary coding service can be obtained, which can determine the proportion of the code data generated by the auxiliary coding service.

[0086] The percentage of newly added code can be calculated as follows:

[0087]

[0088] For example, taking a third code data set with 8 lines of code and a third code data set with 10 lines of code as an example, we can calculate that the newly added code accounts for 80%, which means that 80% of the code data in the code merge requests submitted to the code repository is code data generated by the auxiliary coding service.

[0089] In some examples, before determining the code inclusion rate and the percentage of newly added code, the number of lines of code in the third, first, and second code data can be determined first. For instance, code statistics tools or command-line tools can be used to count the lines of code in the first, second, and third code data.

[0090] In some instances, when multiple users' interaction data with the auxiliary coding service and coding merging requests are obtained, it is possible to calculate only the evaluation parameters corresponding to each user, or to combine the evaluation parameters corresponding to each user with the number of users to determine the full set of evaluation parameters for service quality assessment of the auxiliary coding service.

[0091] For example, full-scale evaluation parameters include the full code upload rate and / or the percentage of newly added code. The full code upload rate characterizes the total usage rate of the first code data generated by the auxiliary coding service calling the target coding model within the code repository; the percentage of newly added code characterizes the total percentage of the first code data generated by the auxiliary coding service calling the target coding model within the code repository. By determining the full code upload rate and / or the percentage of newly added code, the overall service quality of the auxiliary coding service can be effectively evaluated.

[0092] For example, the full code upload rate can be calculated as follows:

[0093]

[0094] The percentage of all newly added code can be calculated as follows:

[0095]

[0096] In some examples, after determining the evaluation parameters for assisting the quality assessment of the assistive coding service based on third-party code data, the evaluation parameters can be sent to the assistive coding service. For instance, the assistive coding service can also store the evaluation parameters in a database to improve data security and ensure efficient data retrieval.

[0097] The coding service quality assessment method provided in this application compares first code data in the interaction data of the auxiliary coding service within a preset time period with second code data in the code merging request to obtain third code data that overlaps between the first and second code data. Since the third code data is generated by the auxiliary coding service and applied in the code merging request, when determining the assessment parameters for evaluating the service quality of the auxiliary coding service based on the overlapping third code data, the application of the code data generated by the auxiliary coding service in the code merging request is considered. This allows for the quantification of the actual contribution of the auxiliary coding service to development efficiency, code quality, and team collaboration. If the code data generated by the auxiliary coding service is frequently used in the code merging request, it indicates a significant improvement in development efficiency and high code quality. Since the code merging request merges branch code from team members into the overall code, calculating the assessment parameters based on the application of the code data generated by the auxiliary coding service in the code merging request quantifies the actual contribution of the auxiliary coding service to development efficiency, code quality, and team collaboration, and accurately and objectively determines the service quality of the auxiliary coding service, thus improving the accuracy of the auxiliary coding service assessment.

[0098] In some examples, after the evaluation parameters are determined, the auxiliary coding service can receive parameter viewing requests from administrators; it can then search the database for the target parameters corresponding to the parameter viewing requests and display them through the service dashboard of the auxiliary coding service. The target parameters include at least one of the following: evaluation parameters, code parameters, user parameters, and capability parameters of the auxiliary coding service.

[0099] For example, the auxiliary coding service can receive a first parameter viewing request sent by an administrator, search for the corresponding evaluation parameter in the database, and display the evaluation parameter through the service dashboard corresponding to the auxiliary coding service. For instance, the first parameter viewing request could be a request to view the code inclusion rate, or it could be a request to view the percentage of newly added code.

[0100] For example, after logging in, an administrator can trigger a first parameter viewing request by clicking on the parameter control configured on the service dashboard of the auxiliary coding service. In response to the first parameter viewing request, the auxiliary coding service identifies the first parameter viewing request, obtains the target parameter identifier, and queries the database for the evaluation parameter corresponding to the target parameter identifier. After finding the evaluation parameter, the auxiliary coding service returns the evaluation parameter to the service dashboard for display in the corresponding location.

[0101] For example, such as Figure 4As shown, when a user triggers the first parameter viewing request, the evaluation parameters can be displayed on the first page of the service dashboard. These evaluation parameters may include each user's user information and each user's code upload rate. Administrators can view each user's code upload rate in the service dashboard of the auxiliary coding service.

[0102] In some examples, the auxiliary coding service can receive second parameter viewing requests sent by administrators, search the database for the code parameters, user parameters, and / or capability parameters corresponding to the second parameter viewing request, and display the code parameters, user parameters, and / or capability parameters through a service dashboard.

[0103] Since the code parameters, user parameters, and / or capability parameters are generated in real time based on the parameter generation data corresponding to the second parameter viewing request stored in the database after receiving the second parameter viewing request sent by the administrator, the specific values ​​of the code parameters, user parameters, and / or capability parameters are related to the parameter viewing conditions included in the second parameter viewing request. For example, the parameter viewing conditions in the second parameter viewing request may include time conditions.

[0104] For example, after logging in, an administrator can click on a parameter control configured on the service dashboard of the assisted coding service to trigger a parameter viewing request. This request requests viewing code parameters, user parameters, and capability parameters for the past week. In response to this request, the assisted coding service retrieves parameter generation data corresponding to the code parameters, user parameters, and capability parameters from the database. Based on this parameter generation data, it calculates and obtains the code parameters, user parameters, and capability parameters. The assisted coding service can then display these parameters in the corresponding locations on the service dashboard. For example, ... Figure 5 As shown in the diagram on the second page of the service dashboard, the code parameters, user parameters, and capability parameters for the most recent week can be displayed.

[0105] The code parameters can be at least one of the following: generated lines of code, adopted lines of code, average adopted lines of code per user, and uploaded lines of code. For example, generated lines of code can be the total number of lines of code generated by all users calling the auxiliary coding service within a unit of time, and the growth rate compared to the previous unit of time. Adopted lines of code can be the total number of lines of code generated by all users adopting the auxiliary coding service within a unit of time, and the growth rate compared to the previous unit of time. Average adopted lines of code per user can be the average number of lines of code generated by users adopting the auxiliary coding service within a unit of time, and the growth rate compared to the previous unit of time. Uploaded lines of code can be the number of lines of code generated by all users calling the auxiliary coding service within a unit of time and included in the code merge request, and the growth rate compared to the previous unit of time.

[0106] User parameters can be one or more of the following: total number of service users, number of user requests, average number of requests per user, number of new users, and user retention rate. For example, the total number of service users can be the number of users calling the auxiliary coding service per unit time, and the growth rate compared to the previous unit time; the number of user requests can be the total number of times all users call the auxiliary coding service per unit time, and the growth rate compared to the previous unit time; the average number of requests per user can be the average number of times a user calls the auxiliary coding service per unit time, and the growth rate compared to the previous unit time; the number of new users can be the number of new users calling the auxiliary coding service per unit time, and the growth rate compared to the previous unit time; and the user retention rate can be the proportion of users who have called the auxiliary coding service to the total number of users per unit time, and the growth rate compared to the previous unit time.

[0107] Capability parameters can be one or more of the following: output adoption rate, service duration, and output character length. For example, data adoption rate includes unit test case generation adoption rate, intelligent question answering code generation adoption rate, block-level or line-level code completion adoption rate, and line-count adoption rate. Specifically, the unit test case generation adoption rate can be the total number of test cases generated by all users calling the auxiliary coding service within a unit testing scenario based on auxiliary coding services, and the growth rate compared to the previous unit time. The intelligent question answering code generation adoption rate can be the number of answers adopted by all users from the auxiliary coding service within a unit time in an intelligent question answering scenario based on auxiliary coding services, and the growth rate compared to the previous unit time. The block-level or line-level code completion adoption rate and line-count adoption rate can be the number of code completions adopted by all users from the auxiliary coding service within a unit time in a code completion scenario based on auxiliary coding services, and the growth rate compared to the previous unit time.

[0108] For example, service duration refers to the average response time of the service when all users call the auxiliary coding service in all scenarios using the auxiliary coding service within a unit of time, and the growth rate compared to the previous unit of time. Output character length refers to the proportion of the amount of code generated and submitted to the merge request by all users calling the auxiliary coding service in all scenarios using the auxiliary coding service within a unit of time, to the total amount of code in the merge request, and the growth rate compared to the previous unit of time.

[0109] In some examples, after displaying the target parameters through the service dashboard corresponding to the assisted coding service, it is also possible to obtain the evaluation metrics of the assisted coding service from administrators. Evaluation parameters are then determined and displayed based on these metrics. These evaluation metrics can be either positive or negative. For example, evaluation parameters could be the total number of positive or negative evaluation metrics submitted by all users when calling the assisted coding service in all scenarios using the service within a unit of time, the proportion of positive or negative evaluation metrics to the total number of calls, and the growth rate compared to the previous unit of time. These evaluation parameters allow for the assessment of user evaluation of the quality of the assisted coding service.

[0110] In some examples, the first code data, second code data, and / or third code data can also be displayed through a service dashboard corresponding to the auxiliary coding service. For example... Figure 6 As shown in the diagram of the third page of the service dashboard, section 601 on the left side of the page displays the second code data included in the code merge request submitted by the user in the code repository, while section 601 on the right side of the page displays the first code data generated by the auxiliary coding service. Since the overlapping code data between the first and second code data are "Aaabbbccc", "Aabbcc", and "Aaa", the third code data generated by the auxiliary coding service and applied in the code merge request can be displayed on the third page using a different style (such as highlighting). Furthermore, Figure 6 In this context, "Xxx" can be code data written by the user, and "Fffff" can be code data generated by the auxiliary coding service that is not applied in the code merge request.

[0111] In some examples, the target parameters can also be visualized using visualization tools. For instance, for target parameters including evaluation parameters, code parameters, user parameters, and auxiliary coding service capability parameters, trend analysis charts (line charts, bar charts, etc.) can be used to visually display how each parameter changes over time, and data can be filtered by date range to meet flexible analysis needs.

[0112] The coding service quality assessment method provided in this application can evaluate the service quality of auxiliary coding services from multiple perspectives. It quantifies the service quality and its impact on overall development efficiency and code quality through evaluation parameters and code parameters, quantifies user usage of the code data generated by the auxiliary coding service through user parameters, and quantifies the service quality of the auxiliary coding service in various service scenarios (e.g., code completion service) through capability parameters. This disclosure, by evaluating auxiliary coding services from multiple perspectives including business, users, capabilities, and feedback, can accurately and objectively quantify the service quality of auxiliary coding services, improving the accuracy of the evaluation and providing a reliable basis for optimizing auxiliary coding services.

[0113] Figure 7 An encoding service quality assessment device provided in the embodiments of this application, such as Figure 7 As shown, the coding service quality assessment device 700 includes a data acquisition module 701, a data comparison module 702, and a parameter determination module 703.

[0114] Data acquisition module 701 is used to acquire interaction data between the user and the auxiliary coding service within a preset time period, as well as code merging requests submitted by the user in the code repository;

[0115] The data comparison module 702 is used to compare the first code data contained in the interaction data with the second code data contained in the code merging request to obtain the third code data; wherein, the third code data includes the same code data in the second code data and the first code data;

[0116] The parameter determination module 703 is used to determine the evaluation parameters for evaluating the service quality of the auxiliary coding service based on the third code data.

[0117] In another possible implementation, the parameter determination module 703 is used to determine the code upload rate based on the third code data and the first code data; the code upload rate is used to characterize the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; and to determine the proportion of newly added code based on the third code data and the second code data; the proportion of newly added code is used to characterize the proportion of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; wherein, the evaluation parameters include the code upload rate and / or the proportion of newly added code.

[0118] In another possible implementation, the parameter determination module 703 is used to determine the code upload rate based on the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data.

[0119] In another possible implementation, the parameter determination module 703 is used to determine the proportion of newly added code based on the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data.

[0120] In another possible implementation, the data acquisition module 701 is used to acquire the user's encoding requests within the preset time period stored in the database of the auxiliary encoding service, and the first code data obtained by the auxiliary encoding service calling the target encoding model to encode the encoding requests; wherein, the interaction data includes the first code data and the encoding requests.

[0121] In another possible implementation, the data acquisition module 701 is used to acquire the interaction data between the user and the auxiliary coding service and the code merging request submitted by the user in the code repository within the preset time period when the timed service starts a timed task.

[0122] In another possible implementation, the data comparison module 702 is used to compare the first code data and the second code data based on a text matching algorithm or an abstract syntax tree to obtain the third code data.

[0123] In another possible implementation, such as Figure 7 As shown, the coding service quality assessment device 700 also includes a display module 704.

[0124] The display module 704 is used to receive parameter viewing requests sent by administrators through the auxiliary coding service; search for the target parameter corresponding to the parameter viewing request in the database; and display the target parameter through the service dashboard corresponding to the auxiliary coding service; wherein, the target parameter includes at least one of the evaluation parameters, code parameters, user parameters, and capability parameters of the auxiliary coding service.

[0125] The beneficial technical effects corresponding to the exemplary embodiment of the above-described encoding service quality assessment device 700 can be found in the corresponding beneficial technical effects in the above-described method embodiment section, and will not be repeated here.

[0126] Figure 8 This is a schematic diagram of a coding service quality assessment device provided in some embodiments of this application. In some embodiments, the computing device can be a server, terminal device, etc. The computing device includes a multi-core processor and a memory. The multi-core processor includes multiple processor cores; the memory is configured to store one or more programs. When the one or more programs are executed by the multi-core processor, the multi-core processor implements the master station communication method or the master station module deployment method in the above embodiments.

[0127] like Figure 8 As shown, the computing device 800 includes a multi-core processor 801 and a memory 802. Exemplarily, the computing device 800 may also include a communications interface 803 and a communications bus 804.

[0128] The multi-core processor 801, memory 802, and communication interface 803 communicate with each other via communication bus 804. Communication interface 803 is used to communicate with other network elements such as clients or other servers.

[0129] In some examples, the multi-core processor 801 is used to execute program 805, specifically performing the relevant steps in the above-described master station communication method or master station module deployment method embodiments. Specifically, program 805 may include program code, which includes computer-executable instructions.

[0130] For example, the multi-core processor 801 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of this application. The computing device 800 may include multi-core processors, which may be of the same type, such as one or more CPUs; or they may be multi-core processors of different types, such as one or more CPUs and one or more ASICs.

[0131] In some examples, memory 802 is used to store program 805. Memory 802 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0132] Specifically, program 805 can be called by multi-core processor 801 to enable computing device 800 to execute master station communication methods or master station module deployment methods.

[0133] Some embodiments of this application provide a computer-readable storage medium storing at least one executable instruction that, when executed on a computing device 800, causes the computing device 800 to perform the master station communication method or the master station module deployment method described above.

[0134] Specifically, the executable instructions can be used to enable the computing device 800 to perform master station communication methods or master station module deployment methods.

[0135] For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device.

[0136] The beneficial effects that the readable storage medium provided in some embodiments of this application can achieve can be referred to the beneficial effects in the corresponding master station communication method or master station module deployment method provided above, and will not be repeated here.

[0137] In addition to the methods, apparatus, and devices described above, embodiments of this application may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the coding service quality assessment methods of various embodiments of this application described in the above method embodiment section.

[0138] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the coding quality of service evaluation methods of various embodiments of this application described in the above-described method embodiment section.

[0140] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] The basic principles of this application have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details of the above embodiments are for illustrative and facilitative purposes only, and are not limitations. These details do not restrict this application from being implemented using the aforementioned specific details.

[0142] Those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0143] Furthermore, the embodiments described above are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A method for evaluating the quality of coding services, characterized in that, The method includes: Obtain user interaction data with the assisted coding service within a preset time period, as well as code merge requests submitted by the user in the code repository; The first code data in the interaction data is compared with the second code data in the code merging request to obtain the third code data; wherein, the third code data includes the same code data in the second code data and the first code data; Based on the third code data, evaluation parameters for assessing the service quality of the auxiliary coding service are determined.

2. The coding service quality assessment method according to claim 1, characterized in that, The step of determining the evaluation parameters for service quality assessment of the auxiliary coding service based on the third code data includes: Based on the third code data and the first code data, the code upload rate is determined; the code upload rate is used to characterize the usage rate of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; The newly added code percentage is determined based on the third code data and the second code data; the newly added code percentage is used to characterize the proportion of the first code data generated by the auxiliary coding service calling the target coding model in the code repository; The evaluation parameters include the code upload rate and / or the proportion of newly added code.

3. The coding service quality assessment method according to claim 2, characterized in that, The step of determining the code upload rate based on the third code data and the first code data includes: The code upload rate is determined based on the ratio of the number of lines of code in the third code data to the number of lines of code in the first code data.

4. The coding service quality assessment method according to claim 2, characterized in that, The step of determining the proportion of newly added code based on the third code data and the second code data includes: The proportion of newly added code is determined based on the ratio of the number of lines of code in the third code data to the number of lines of code in the second code data.

5. The coding service quality assessment method according to any one of claims 1-4, characterized in that, The step of obtaining user interaction data with the assisted coding service within a preset time period includes: The system retrieves user encoding requests within the preset time period stored in the database of the auxiliary encoding service, as well as the first code data obtained by the auxiliary encoding service calling the target encoding model to encode the encoding requests; wherein, the interaction data includes the first code data and the encoding requests.

6. The coding service quality assessment method according to any one of claims 1-5, characterized in that, The acquisition of user interaction data with the assisted coding service within a preset time period and the code merge requests submitted by the user in the code repository includes: When the scheduled service starts a scheduled task, it acquires the interaction data between the user and the auxiliary coding service and the code merge request submitted by the user in the code repository within the preset time period.

7. The coding service quality assessment method according to any one of claims 1-6, characterized in that, The step of comparing the first code data contained in the interaction data with the second code data contained in the code merging request to obtain the third code data includes: The first code data and the second code data are compared based on a text matching algorithm or an abstract syntax tree to obtain the third code data.

8. The coding service quality assessment method according to any one of claims 1-7, characterized in that, The method further includes: The auxiliary coding service receives parameter viewing requests sent by administrators. The database is searched for the target parameter corresponding to the parameter viewing request, and the target parameter is displayed through the service dashboard corresponding to the auxiliary coding service; The target parameters include at least one of the evaluation parameters, code parameters, user parameters, and capability parameters of the auxiliary coding service.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the coded quality of service assessment method according to any one of claims 1-8.

10. A coding service quality assessment device, characterized in that, include: One or more processors; and The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the coding service quality assessment method according to any one of claims 1-8.

Images