Code evaluation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a code evaluation method and device, equipment and a storage medium. The method comprises the following steps: processing a code file by utilizing a first model to determine that a first code snippet in the code file has a first problem; constructing a first cue word associated with the first question, wherein the first cue word comprises question description about the first question and position information of the first code snippet; providing a first cue word for the second model to instruct the second model to generate an evaluation of the first question, the evaluation indicating whether the first question is valid; and based on the evaluation of the first question, generating an evaluation result of the code file. Therefore, the accuracy and efficiency of code evaluation can be effectively improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to a code evaluation method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the development of computer technology, code evaluation is the key to improving code quality. It can find and fix code errors and ensure that the code complies with the specifications, thereby improving code evaluation efficiency and code quality. How to accurately conduct code evaluation is a focus of attention. Summary of the invention

[0003] In a first aspect of the present disclosure, a code evaluation method is provided. The method includes: processing a code file using a first model to determine that a first code snippet in the code file has a first problem; constructing a first prompt word associated with the first problem, the first prompt word including a problem description of the first problem and location information of the first code snippet; providing the first prompt word to a second model to instruct the second model to generate an evaluation of the first problem, the evaluation indicating whether the first problem is valid; and generating an evaluation result of the code file based on the evaluation of the first problem.

[0004] In a second aspect of the present disclosure, a device for code evaluation is provided. The device includes: a processing module configured to process a code file using a first model to determine that a first code snippet in the code file has a first problem; a construction module configured to construct a first prompt word associated with the first problem, the first prompt word including a problem description about the first problem and location information of the first code snippet; a providing module configured to provide the first prompt word to a second model to instruct the second model to generate an evaluation of the first problem, the evaluation indicating whether the first problem is valid; and a generating module configured to generate an evaluation result of the code file based on the evaluation of the first problem.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer executable instructions, and when the instructions are executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart showing a code evaluation process according to some embodiments of the present disclosure is shown;

[0012] Figure 3 shows an example flow chart of code evaluation according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic structural block diagram of a code evaluation device according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0018] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0019] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0020] Traditionally, a language model can be used for code evaluation. Specifically, this method mainly utilizes the capabilities of the native language model and based on a unified prompt item, directly inputs the code file to be evaluated as a context into the language model to obtain the evaluation result corresponding to the code file to be evaluated generated by the language model. Although this method can ensure that the language model can perform code evaluation without restrictions, and thus can correspond to a stronger flexibility, it is difficult to avoid the hallucination problem of the language model, which leads to a low accuracy rate of code evaluation.

[0021] In order to solve the problem of low accuracy when performing code evaluation based on native language models, a large amount of internal data / open source data can be used to perform supervised fine-tuning (SFT) on the language model to perform code evaluation based on the fine-tuned language model. Although this method can improve the accuracy to a certain extent compared to using the capabilities of the native language model for code evaluation, due to the lack of high-quality code evaluation data, the output content of this method is relatively single and homogeneous, affecting the user experience. In addition, since this method is mainly fine-tuned based on limited domain data, it is extremely easy to produce overfitting, which makes the output results of the language model converge and the diversity is poor.

[0022] Traditionally, it is also possible to evaluate code based on complex agents, divided into multiple roles and functions, and finally output the evaluation results. However, the overall evaluation link of this method is relatively complex. Generally, the evaluation of a piece of code often requires calling the language model many times, resulting in low efficiency of code evaluation. In addition, this method also requires high costs.

[0023] The embodiment of the present disclosure proposes a code evaluation scheme. According to the scheme, a code file is processed using a first model to determine that a first code snippet in the code file has a first problem; a first prompt word associated with the first problem is constructed, the first prompt word includes a problem description about the first problem and location information of the first code snippet; the first prompt word is provided to a second model to instruct the second model to generate an evaluation of the first problem, the evaluation indicating whether the first problem is valid; and based on the evaluation of the first problem, an evaluation result of the code file is generated.

[0024] Based on this approach, the embodiments of the present disclosure can first accurately locate the problem based on the first model, effectively narrowing the scope of problem investigation, and further judge the effectiveness of small-scale problems based on the second model. Compared with the one-step code evaluation method of a single model, the phased evaluation method can avoid the low efficiency caused by a single model processing multiple complex tasks at the same time, and can effectively improve the accuracy of code evaluation.

[0025] Example Environment

[0026] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0027] In the example environment 100 , the electronic device 110 may support evaluating a code to be evaluated, etc., and the embodiments of the present disclosure are not limited in this regard.

[0028] Specifically, the electronic device 110 may receive a code file to be evaluated, and output an evaluation result corresponding to the code file to be evaluated, and the evaluation result may at least indicate whether there are errors or problems in a code snippet in the code file to be evaluated. In response to determining that there are errors or problems in a code snippet, the evaluation result may also indicate what specific problems or errors exist in the code snippet with errors or problems. In some embodiments, the code snippet is at least part of the code in the code file to be estimated. The code snippet may have any appropriate problems, such as duplicate code, null pointer exception, and the like.

[0029] In some embodiments, the electronic device 110 can communicate with the server 120 to provide services for code evaluation. The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio receiver, an e-book device, a game device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the target user (such as a "wearable" circuit, etc.).

[0030] The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 120 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0031] A communication connection may be established between the electronic device 110 and the server 120. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In an embodiment of the present disclosure, two devices in a data transmission relationship may implement signaling interaction through a communication connection between the two devices.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0033] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0034] Example Process

[0035] Figure 2 FIG. 2 is a flowchart of a process 200 for code evaluation according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. Figure 1 Process 200 is described.

[0036] In block 210 , the electronic device 110 processes a code file using a first model to determine that a first code segment in the code file has a first problem.

[0037] In some embodiments, the first model may be any appropriate machine learning model, such as a first language model. In some embodiments, the first code snippet may be at least part of a code snippet in a code file, such as a code snippet corresponding to lines 3 to 10 in the code file. The first problem may be any appropriate code problem, such as a syntax error, a logic error, a code specification problem, code redundancy, etc., which will not be described in detail here.

[0038] In some embodiments, the electronic device may perform code evaluation on the entire code file.

[0039] In other embodiments, the electronic device may also evaluate a portion of the code in the code file. As an example, the electronic device may extract a code portion having a specific function or logic unit from a complete executable code to evaluate the extracted code portion. As another example, the electronic device may determine a first code portion that has changed in the code file in response to a change in the code file. Further, the electronic device may perform an evaluation based on the changed first code portion.

[0040] The first code portion may also be referred to as a changed code block. The changed code block is the code determined after a predetermined change operation is performed at a predetermined position in the code file. The predetermined change operation includes at least one of the following: a modification operation, an addition operation, and a deletion operation. The predetermined position may be any appropriate position in the code file, which will not be described in detail herein. In some embodiments, the electronic device 110 may present a code change identifier at a predetermined position in response to performing a predetermined change operation at a predetermined position in the code file to indicate that the code fragment has performed a predetermined change operation. As an example, the code change identifier corresponding to the deletion operation may be "-", and the code identifier corresponding to the addition operation may be "+". Furthermore, the electronic device 110 may determine the code fragment corresponding to the code change identifier as a changed code block based on the code change identifier presented in the code file.

[0041] In order to ensure the integrity of the code and improve the accuracy of code evaluation, the electronic device can evaluate the first code portion based on the changed first code portion and the second code portion associated with the first code portion. As an example, the electronic device can provide the changed portion and the second code portion associated with the first code portion to the first model to determine that the first code segment in the first code portion has a first problem.

[0042] As an example, the electronic device 110 may determine a predetermined number of lines of surrounding code portions located before and after the first code portion as the second code portion.

[0043] As another example, the electronic device 110 can determine at least one entity associated with the first code portion based on a syntax tree associated with the code file. The syntax tree can be an abstract syntax tree (AST). In some embodiments, the at least one entity includes but is not limited to a function entity and / or a variable entity. Further, the electronic device can determine a second code portion corresponding to the at least one entity from the code file. In some embodiments, the second code portion can also include definition information corresponding to the at least one entity, and the definition can include a declaration of a variable, a prototype of a function, etc.

[0044] In other embodiments, the electronic device may expand the range corresponding to the first code portion to determine the second code portion, wherein the first code portion and the second code portion correspond to the complete code of the target function. The target function may include one or more functional functions. Specifically, the electronic device may determine at least one functional function corresponding to the first code portion based on the AST corresponding to the code file. Further, the electronic device 110 may determine the second portion of code based on the first target code fragment corresponding to at least one first functional function, wherein the first target code fragment includes at least the second portion of code and the first portion of code.

[0045] In other embodiments, the electronic device 110 may not only expand the scope of the first code portion, but also determine the second code portion in combination with the code portion corresponding to at least one entity. Specifically, the electronic device 110 may assemble the first target code segment with the code portion corresponding to at least one entity to generate a semantically complete code representation. Further, the electronic device may provide the complete code representation to the first model to determine that the first code segment in the first code portion has a first problem.

[0046] In some embodiments, in order to accurately locate the problem, the first model can be obtained by fine-tuning the first candidate model, and the first candidate model can be any appropriate machine learning model, such as a language model. For ease of description, the electronic device 110 fine-tuning the first candidate model is used as an example for explanation. It should be noted that the first candidate model can be obtained by fine-tuning on any appropriate device, such as fine-tuning on the server 120, which is not repeated here.

[0047] The electronic device 110 may obtain the first sample code and the first annotation information. In some embodiments, the first annotation information indicates that the first sample code has a second problem. The first annotation information may be manually annotated real problem information corresponding to the first sample code, or may be real problem information generated based on a predetermined model and manually proofread, which will not be described in detail here. As an example, the first annotation information corresponding to the first sample code A may be that the first sample code A has problem 1. Further, the electronic device 110 may determine the first prediction information for the first sample code using the first candidate model, wherein the first prediction information indicates the first prediction problem corresponding to the first sample code. Further, the electronic device 110 may fine-tune the first candidate model based on the comparison between the first prediction information and the first annotation information to obtain the first model. Specifically, the electronic device 110 may determine that the fine-tuning of the first candidate model is completed in response to reaching the condition for the completion of fine-tuning, and determine the model obtained after the completion of fine-tuning as the first model. The condition for the completion of fine-tuning may be any appropriate condition, such as the accuracy reaching the target, the recall rate meeting the threshold, and the like.

[0048] In block 220 , the electronic device 110 constructs a first prompt word associated with the first question, where the first prompt word includes a question description about the first question and location information of the first code snippet.

[0049] In order to help the second model understand and process the first question, in some embodiments, the electronic device 110 may construct a first prompt word associated with the first question, where the prompt word may also be called a prompt item, a guide word, etc., which is used to guide the second model to understand and process the first question so as to accurately determine the validity of the first question.

[0050] In some embodiments, the problem description of the first problem can be a specific description of the first problem, accurately describing the type and nature of the problem in the first code snippet. The location information of the first code snippet (first location information) is any appropriate form of information used to characterize the location of the first code snippet in the code file, such as the corresponding line number.

[0051] In some embodiments, in order to ensure the integrity of the code, the first prompt word may further include second position information of a second code segment associated with the first code segment.

[0052] As an example, the electronic device 110 may determine a predetermined number of lines of surrounding code fragments located before and after the first code fragment as the second code fragment.

[0053] As another example, the electronic device may expand the scope corresponding to the first code snippet to determine the second code snippet, wherein the first code snippet and the second code snippet correspond to the complete code of the target function. Specifically, the electronic device 110 may determine at least one second function function corresponding to the first code snippet based on the syntax tree corresponding to the code file. In some embodiments, the first code snippet may correspond to one second function function, and may also correspond to multiple second function functions. It should be noted that since the first code snippet is a problematic code snippet obtained after further narrowing the scope on the basis of the first code portion, the function function corresponding to the first code portion includes the function function corresponding to the first code snippet. For example, if the function function corresponding to the first code portion is function function 1 and function function 2, the function function corresponding to the first code snippet may include function function 1. Further, the electronic device 110 may determine the second code snippet based on the second target code snippet corresponding to at least one second function function. The second target code snippet includes at least the first code snippet and the second code snippet.

[0054] In block 230 , the electronic device 110 provides the first prompt word to the second model to instruct the second model to generate an evaluation of the first question, the evaluation indicating whether the first question is valid.

[0055] In some embodiments, the second model may be any appropriate machine learning model, such as a second language model. Whether the first question is valid represents whether the first code snippet can be finally determined to have the first question. Specifically, if the first question is valid, it represents that the first code snippet is finally determined to have the first question. If the first question is invalid, it represents that the first code snippet is determined to have the first question. That is, the second model is used to further determine whether the result of the first model determining that the first code snippet has the first question is accurate.

[0056] In some embodiments, in order to accurately determine the effectiveness of the problem and improve the accuracy of code evaluation, the second model can be obtained by fine-tuning the second candidate model, and the second candidate model can be any appropriate machine learning model, such as a language model. For ease of description, the electronic device 110 fine-tuning the second candidate model is used as an example for explanation. It should be noted that the second candidate model can be obtained by fine-tuning on any appropriate device, such as fine-tuning on the server 120, which will not be repeated here.

[0057] The electronic device 110 may obtain the second sample code and the second annotation information, and the second annotation information indicates whether the third question corresponding to the second sample code is valid. The first annotation information may be manually annotated real evaluation information corresponding to the second sample code, or may be real evaluation information generated based on a predetermined model and manually proofread, which will not be described in detail here. In some embodiments, the second sample code may include a positive sample code and a negative sample code, wherein the second annotation information corresponding to the positive sample code indicates that the corresponding third question is valid, and the second annotation information corresponding to the negative sample code indicates that the corresponding third question is invalid. As an example, the second annotation information corresponding to the second sample code B may be that the problem 2 of the second sample code B is valid. As another example, the second annotation information corresponding to the second sample code C may be that the problem 3 of the second sample code C is invalid. Further, the electronic device 110 may determine the second prediction information based on the second sample code and the third question using the second candidate model. Further, the electronic device 110 may fine-tune the second candidate model based on the comparison of the second prediction information with the second annotation information to obtain the second model. Specifically, the electronic device 110 can determine that the fine-tuning of the second candidate model is completed in response to the condition that the fine-tuning is completed, and determine the model obtained after the fine-tuning is completed as the second model. The condition for completing the fine-tuning can be any appropriate condition, such as the accuracy reaching the target, the recall rate meeting the threshold, etc.

[0058] In block 240 , the electronic device 110 generates an evaluation result of the code file based on the evaluation of the first question.

[0059] In some embodiments, if the evaluation of the first question indicates that the first question is valid, the evaluation result of the code file may indicate that there is a problem with the code file, specifically, there is a first problem with the first code segment in the code file. If the evaluation of the first code portion that has been changed in the code file is performed, the evaluation result may indicate that there is a first problem with the first code in the first code portion.

[0060] In other embodiments, if the evaluation of the first problem indicates that the first problem is invalid, the evaluation result of the code file may indicate that there is no problem with the code file. If the evaluation of the first code portion that has been changed in the code file is performed, the evaluation result may indicate that there is no problem with the first code portion.

[0061] In some embodiments, in addition to indicating that the first code snippet has the first problem, the evaluation result may also indicate guidance information, guidance examples, etc. on solving the first problem, which are not elaborated here.

[0062] Figure 3 An example flow chart of a code evaluation provided in some embodiments of the present disclosure is now directed to Figure 3 Provide explanation.

[0063] As an example, code evaluation mainly includes three stages: code preprocessing 310, problem location and evaluation generation 320, and comment validity determination 330. Code preprocessing 310 mainly includes four steps corresponding to box 310-1, box 310-2, box 310-3, and box 310-4. Problem location and evaluation generation 320 mainly includes two steps corresponding to box 320-1 and box 320-2. Comment validity determination 330 mainly includes two steps corresponding to box 330-1 and box 330-2.

[0064] In block 310 - 1 , the electronic device 110 performs ATS parsing on the original changed code block.

[0065] As an example, the original changed code block may be a changed code snippet 1 existing in the code file. Further, the electronic device 110 may perform AST analysis on the code snippet 1 to determine a functional function corresponding to the code snippet 1.

[0066] In block 310 - 2 , the electronic device 110 performs line number expansion based on the parsing result of the ATS parsing.

[0067] As an example, the electronic device 110 may determine that the code snippet 2 corresponding to the functional function corresponding to the code snippet 1 is the code with line number expansion, wherein the code snippet 2 includes the code snippet 1 and other code snippets.

[0068] In block 310 - 3 , the electronic device 110 performs entity association based on the parsing result of the ATS parsing.

[0069] As an example, the electronic device 110 may obtain key entities and their definitions in the code snippet 1 .

[0070] In block 310 - 4 , the electronic device 110 obtains a code representation based on the result of the line number expansion and the result of the entity association.

[0071] As an example, the electronic device 110 may assemble the key entities and their definitions in the code snippet 2 and the code snippet 1 to generate a complete code representation.

[0072] In block 320 - 1 , the electronic device 110 locates a code problem based on the code representation.

[0073] As an example, the electronic device 110 may use the first model to determine which code segments in the code representation have problems.

[0074] In block 320 - 2 , the electronic device 110 may perform location-code matching.

[0075] As an example, the electronic device 110 can use the first model to determine suspected problem 1 and positioning information 1 corresponding to code 1, suspected problem 2 and positioning information 2 corresponding to code 2, and suspected problem 3 and positioning information 3 corresponding to code 3, where code 1, code 2 and code 3 are partial codes in code fragment 1.

[0076] In block 330 - 1 , the electronic device 110 assembles prompt items.

[0077] As an example, the electronic device 110 may determine the functional function 1 corresponding to the code 1, and determine the relevant code corresponding to the functional function 1 as the target code 1 after the code 1 is expanded. Further, the electronic device 110 may assemble the code snippet 2, the location information corresponding to the target code 1, and the suspected problem 1 into prompt item 1, and then perform the invalid comment filtering operation of box 330-2, that is, determine whether the code 1 actually has a suspected problem, that is, determine whether the suspected problem 1 existing in the code 1 is valid.

[0078] As an example, the electronic device 110 may determine the functional function 2 corresponding to the code 2, and determine the relevant code corresponding to the functional function 2 as the target code 2 after the code 2 is expanded. Further, the electronic device 110 may assemble the code snippet 2, the location information corresponding to the target code 2, and the suspected problem 2 into a prompt item 2, and then perform the invalid comment filtering operation of box 330-2, that is, determine whether the code 2 really has a suspected problem, that is, determine whether the suspected problem 2 existing in the code 2 is valid.

[0079] As an example, the electronic device 110 may determine the functional function 3 corresponding to the code 3, and determine the relevant code corresponding to the functional function 3 as the target code 3 after the code 3 is expanded. Further, the electronic device 110 may assemble the code snippet 2, the location information corresponding to the target code 3, and the suspected problem 3 into prompt item 3, and then perform the invalid comment filtering operation of box 330-2, that is, determine whether the code 3 really has a suspected problem, that is, determine whether the suspected problem 3 in the code 3 is valid.

[0080] In block 330 - 2 , the electronic device 110 performs an invalid comment filtering operation to obtain a final evaluation result for the original changed code block.

[0081] As an example, the electronic device 110 may obtain evaluation opinion 1 and evaluation opinion 2, wherein evaluation opinion 1 may indicate that suspected problem 1 corresponding to code 1 is valid, that is, it is determined that code 1 does have corresponding suspected problem 1. Evaluation opinion 2 may indicate that suspected problem 3 corresponding to code 3 is valid, that is, it is determined that code 3 does have corresponding suspected problem 3. Since code 2 is confirmed to have no corresponding suspected problem 2, it is filtered in the invalid comment filtering operation of box 330-2.

[0082] Based on this approach, the embodiments of the present disclosure can first accurately locate the problem based on the first model, effectively narrow the scope of problem investigation, and further judge the effectiveness of small-scale problems based on the second model. Compared with the one-step code evaluation method of a single model, the phased evaluation method can avoid the low efficiency caused by a single model processing multiple complex tasks at the same time, and can effectively improve the accuracy of code evaluation.

[0083] Example devices and equipment

[0084] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for code evaluation according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in the electronic device 110 discussed above. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0085] like Figure 4As shown, the apparatus 400 includes a processing module 410, configured to process a code file using a first model to determine that a first problem exists in a first code snippet in the code file; a construction module 420, configured to construct a first prompt word associated with the first problem, the first prompt word including a problem description about the first problem and location information of the first code snippet; a providing module 430, configured to provide the first prompt word to the second model to instruct the second model to generate an evaluation of the first problem, the evaluation indicating whether the first problem is valid; and a generating module 440, configured to generate an evaluation result of the code file based on the evaluation of the first problem.

[0086] In some embodiments, the processing module 410 is further configured to: in response to a change in the code file, determine a first code portion that has changed in the code file; and provide the changed portion and a second code portion associated with the first code portion to the first model to determine that a first problem exists in the first code snippet in the first code portion.

[0087] In some embodiments, the device 400 also includes a first determination module configured to: determine at least one entity associated with the first code portion based on a syntax tree associated with the file; and a second determination module configured to: determine a second code portion corresponding to the at least one entity from the code file.

[0088] In some embodiments, at least one entity includes a function entity and / or a variable entity.

[0089] In some embodiments, the apparatus 400 further comprises a range extension module configured to: extend the range corresponding to the first code portion to determine a second code portion, wherein the first code portion and the second code portion correspond to a complete code of the target function.

[0090] In some embodiments, the location information is first location information, and the first prompt word includes second location information of a second code snippet associated with the first code snippet.

[0091] In some embodiments, the first model is determined based on the following process: obtaining a first sample code and first annotation information, the first annotation information indicating that the first sample code has a second problem; using a first candidate model, determining first prediction information for the first sample code; and based on a comparison of the first prediction information and the first annotation information, fine-tuning the first candidate model to obtain the first model.

[0092] In some embodiments, the second model is determined based on the following process: obtaining a second sample code and second annotation information, the second annotation information indicating whether a third question corresponding to the second sample code is valid; using a second candidate model, determining second prediction information based on the second sample code and the third question; and based on a comparison of the second prediction information and the second annotation information, fine-tuning the second candidate model to determine the second model.

[0093] In some embodiments, the second sample code includes at least one of the following: a positive sample code, wherein the second annotation information corresponding to the positive sample code indicates that the corresponding third question is valid; a negative sample code, wherein the second annotation information corresponding to the negative sample code indicates that the corresponding third question is invalid.

[0094] The units included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0095] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 A receiving device 120 is shown.

[0096] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0097] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 500.

[0098] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0099] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0100] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0102] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0104] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0105] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0106] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A code evaluation method, comprising: Processing a code file using a first model to determine that a first code segment in the code file has a first problem; Constructing a first prompt word associated with the first problem, wherein the first prompt word includes a problem description of the first problem and location information of the first code snippet; providing the first prompt word to a second model to instruct the second model to generate an evaluation of the first question, the evaluation indicating whether the first question is valid; as well as Based on the evaluation of the first question, an evaluation result of the code file is generated.

2. The method according to claim 1, wherein processing the code file using the first model comprises: In response to a change in the code file, determining a first code portion in the code file that has been changed; as well as The changed portion and a second code portion associated with the first code portion are provided to the first model to determine that a first code segment in the first code portion has the first problem.

3. The method according to claim 2, further comprising: determining, based on a syntax tree associated with the file, at least one entity associated with the first code portion; as well as The second code portion corresponding to the at least one entity is determined from the code file. The method according to claim 3 , wherein the at least one entity comprises a function entity and / or a variable entity.

5. The method according to claim 1, further comprising: A range corresponding to the first code portion is expanded to determine the second code portion, wherein the first code portion and the second code portion correspond to a complete code of a target function. 6 . The method according to claim 1 , wherein the position information is first position information, and the first prompt word includes second position information of a second code snippet associated with the first code snippet.

7. The method of claim 1, wherein the first model is determined based on the following process: Acquire a first sample code and first annotation information, wherein the first annotation information indicates that the first sample code has a second problem; Determine first prediction information for the first sample code using a first candidate model; as well as Based on a comparison between the first prediction information and the first annotation information, fine-tune the first candidate model to obtain the first model.

8. The method of claim 1, wherein the second model is determined based on the following process: acquiring a second sample code and second annotation information, wherein the second annotation information indicates whether a third question corresponding to the second sample code is valid; Determining second prediction information based on the second sample code and the third question using a second candidate model; as well as Based on a comparison between the second prediction information and the second annotation information, the second candidate model is fine-tuned to determine the second model.

9. The method according to claim 8, wherein the second sample code comprises at least one of the following: A positive sample code, wherein the second annotation information corresponding to the positive sample code indicates that the corresponding third question is valid; Negative sample code, wherein the second annotation information corresponding to the negative sample code indicates that the corresponding third question is invalid.

10. An apparatus for code evaluation, comprising: A processing module is configured to process a code file using a first model to determine that a first code segment in the code file has a first problem; A construction module is configured to construct a first prompt word associated with the first problem, wherein the first prompt word includes a problem description of the first problem and location information of the first code snippet; A providing module, configured to provide the first prompt word to a second model to instruct the second model to generate an evaluation of the first question, wherein the evaluation indicates whether the first question is valid; as well as A generation module is configured to generate an evaluation result of the code file based on the evaluation of the first question.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images