Code processing method and electronic device

Abstract

The present application discloses a sample construction method, a code processing method and an electronic device, the sample construction method comprising: constructing the grammatical elements contained in the source code file into an abstract syntax tree according to the grammatical structure corresponding to the source code file; determining all the grammatical elements that the current main function depends on based on the association relationship between the various grammatical elements contained in all abstract syntax trees; extracting the code content corresponding to all the dependent grammatical elements from each source code file, reorganizing the code content corresponding to all the grammatical elements that the current main function depends on, and obtaining the reorganized code; obtaining the code comments corresponding to the reorganized code, and using the reorganized code and the code comments corresponding to the reorganized code as model training samples. By reconstructing each source code file, presenting the functional function and its related dependencies and implementations in the same file, it is convenient for the subsequent model to learn and understand the structure of the code, and improve the code processing ability of the code processing model obtained by subsequent training.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a code processing method and electronic equipment. Background Art

[0002] The intelligent programming method based on deep learning models is an innovative achievement that combines artificial intelligence and software development in recent years. By training the model through a massive open source code base, the model can understand the input context for reasoning and generate the required Python, Java and other code snippets, realizing automatic conversion from natural language to code and improving programming efficiency.

[0003] The intelligent programming method collects a large number of relevant code samples to build a training data set, and then inputs these samples into the model for model training, so that the model can understand and generate code that meets the specifications. In practical applications, only a simple description is needed, and the model can automatically generate actual executable code.

[0004] However, this approach also has some significant disadvantages. The first is the limitation of input expression. Since the model relies on text input, complex logic or specific industry knowledge may be difficult to express accurately, resulting in the generated code not fully meeting actual needs. The second is that the model only focuses on the code generation task and may ignore the readability and maintainability of the code, making it difficult to understand the code logic, especially complex business logic. Summary of the invention

[0005] In order to solve the above technical problems, the present application at least provides a sample construction method, a code processing method and an electronic device.

[0006] In a first aspect, the present application provides a sample construction method, which includes: parsing the syntax elements contained in each source code file, and constructing the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file; determining all syntax elements that the current main function depends on based on the association relationship between the various syntax elements contained in all abstract syntax trees; wherein the current main function is any main function contained in each source code file; extracting the code content corresponding to all the syntax elements that the current main function depends on from each source code file, and reorganizing the code content corresponding to all the syntax elements that the current main function depends on to obtain a reorganized code; obtaining the code comments corresponding to the reorganized code, and using the reorganized code and the code comments corresponding to the reorganized code as model training samples.

[0007] In one embodiment, code contents corresponding to all syntax elements that the current main function depends on are reorganized to obtain reorganized code, including: determining the programming language type corresponding to the current main function; obtaining a reorganized code format that matches the programming language type; and reorganizing code contents corresponding to all syntax elements that the current main function depends on according to the matching reorganized code format to obtain a reorganized code.

[0008] In one embodiment, obtaining code comments corresponding to the reorganized code includes: obtaining comment content corresponding to the code in the source code file to obtain the original comment, and obtaining a comment conversion template; using the comment conversion template to convert the original comment, and using the comment conversion result as the code comment corresponding to the code in the reorganized code.

[0009] In one embodiment, obtaining a comment conversion template includes: analyzing the comment styles corresponding to all comment contents corresponding to the code in each source code file to obtain a plurality of comment style types; generating a comment conversion template corresponding to each comment style type; using the comment conversion template to convert the original comment, and using the comment conversion result as the code comment of the corresponding code in the reorganized code, including: using the comment conversion templates corresponding to the plurality of comment style types to convert the original comment multiple times to obtain a plurality of comment conversion results; using the plurality of comment conversion results as the code comment of the corresponding code in the reorganized code.

[0010] In one embodiment, obtaining code annotations corresponding to the reorganized code includes: obtaining business information and / or function information corresponding to the corresponding code in the reorganized code; and generating code annotations corresponding to the code in the reorganized code based on the business information and / or function information.

[0011] The second aspect of the present application provides a code processing method, which includes: using the model training sample obtained by the sample construction method to train the code processing model; inputting the code processing request into the trained code processing model to obtain the code processing result output by the trained code processing model.

[0012] In one embodiment, a code processing request includes one or more of a text description of a pregenerated code, a pregenerated interpreted code snippet, and a code retrieval prompt; the code processing request is input into a trained code processing model to obtain a code processing result output by the trained code processing model, including: inputting a text description of the pregenerated code into the trained code processing model to obtain a code generation result output by the trained code processing model; and / or, inputting a pregenerated interpreted code snippet into the trained code processing model to obtain a code interpretation result output by the trained code processing model; and / or, inputting a code retrieval prompt into the trained code processing model to obtain a retrieved code snippet output by the trained code processing model.

[0013] In one embodiment, the code processing model includes a semantic matching network and multiple code processing networks, and different code processing networks correspond to different business fields; a code processing request is input into the trained code processing model to obtain a code processing result output by the trained code processing model, including: inputting the code processing request into the semantic matching network for classification to obtain a business field output by the semantic matching network that matches the code processing request; inputting the code processing request into the code processing network corresponding to the matched business field to obtain a code processing result output by the code processing network.

[0014] In one embodiment, a code processing model includes a retrieval network and a code processing network, and a code knowledge base is pre-stored, wherein the code knowledge base contains multiple sample code snippets or sample annotation snippets; a code processing request is input into the trained code processing model to obtain a code processing result output by the trained code processing model, including: using the retrieval network to calculate the matching degree between the code processing request and each sample code snippet or sample annotation snippet in the code knowledge base, and selecting the sample code snippet or sample annotation snippet whose matching degree meets the preset conditions as the matching knowledge snippet; inputting the code processing request into the code processing network to obtain the code processing result output by the code processing network; or, inputting the code processing request and the matching knowledge snippet into the code processing network to obtain the code processing result output by the code processing network; and taking the code processing result and the matching knowledge snippet as the final result.

[0015] The third aspect of the present application provides a sample construction device, which includes: a code parsing module, which is used to parse the syntax elements contained in each source code file, and construct the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file; a dependency analysis module, which is used to determine all syntax elements that the current main function depends on based on the association relationship between the various syntax elements contained in all abstract syntax trees; wherein the current main function is any main function contained in each source code file; a code reorganization module, which is used to extract the code content corresponding to all syntax elements that the current main function depends on from each source code file, and reorganize the code content corresponding to all syntax elements that the current main function depends on to obtain a reorganized code; a sample generation module, which is used to obtain the code comments corresponding to the reorganized code, and use the reorganized code and the code comments corresponding to the reorganized code as model training samples.

[0016] In a fourth aspect, the present application provides a code processing device, which includes: a model training module, which is used to train a code processing model using a model training sample obtained by a sample construction method; a code processing module, which is used to input a code processing request into the trained code processing model to obtain a code processing result output by the trained code processing model.

[0017] A fifth aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned sample construction method or code processing method.

[0018] In a sixth aspect, the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned sample construction method or code processing method is implemented.

[0019] The above scheme parses the syntax elements contained in each source code file, and constructs the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file; based on the association relationship between the various syntax elements contained in all abstract syntax trees, all syntax elements that the current main function depends on are determined; the code content corresponding to all syntax elements that the current main function depends on is extracted from each source code file, and the code content corresponding to all syntax elements that the current main function depends on is reorganized to obtain the reorganized code; the code comments corresponding to the reorganized code are obtained, and the reorganized code and the code comments corresponding to the reorganized code are used as model training samples to reconstruct each source code file, and the functional functions and their related dependencies and implementations are presented in the same file, so as to facilitate the subsequent model to learn and understand the structure of the code, and improve the code processing ability of the code processing model obtained by subsequent training.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0022] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0023] Figure 2 is a flow chart of a sample construction method shown in an exemplary embodiment of the present application;

[0024] Figure 3 is a schematic diagram of variables and called functions associated with the current main function shown in an exemplary embodiment of the present application;

[0025] Figure 4 is a flowchart of a code processing method shown in an exemplary embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a code processing model shown in an exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of code processing shown in an exemplary embodiment of the present application;

[0028] Figure 7 is a schematic diagram of code processing shown in another exemplary embodiment of the present application;

[0029] Figure 8 is a block diagram of a sample construction device shown in an exemplary embodiment of the present application;

[0030] Fig. 9 is a block diagram of a code processing device shown in an exemplary embodiment of the present application;

[0031] Fig.10 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0032] Fig.11 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0035] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0036] The following is an explanation of the sample construction method and code processing method provided in the embodiments of the present application.

[0037] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other in communication.

[0038] The number of the terminal 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0039] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0040] In one example, the server 120 constructs a model training sample according to the source code file, and obtains a code processing model based on the model training sample training. The code processing model can be directly deployed on the server or sent to the terminal 110 for deployment. The terminal 110 inputs the code processing request into the trained code processing model to obtain the code processing result output by the trained code processing model.

[0041] It should be noted that, in the sample construction method or code processing method provided in the embodiments of the present application, the execution subject of each step can be the terminal 110 or the server 120, or the terminal 110 and the server 120 can interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other steps are executed by the server 120.

[0042] See also Figure 2 , Figure 2 is a flow chart of a sample construction method shown in an exemplary embodiment of the present application. The sample construction method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0043] like Figure 2 As shown, the sample construction method at least includes steps S210 to S240, which are described in detail as follows:

[0044] Step S210: parsing the syntax elements contained in each source code file, and constructing the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file.

[0045] By parsing the syntax elements contained in the source code file, the syntax elements contained in the source code file are constructed into an abstract syntax tree (Abstract Syntax Tree, AST) according to the syntax structure corresponding to the source code file. It should be noted that different programming languages ​​have different syntax structures, and the syntax elements contained in the source code file are constructed into an abstract syntax tree according to the syntax structure corresponding to the programming language to which the source code file belongs.

[0046] Among them, the abstract syntax tree is an abstract representation of the syntax structure of the source code, which displays the syntax structure of the programming language in a tree form. Each node represents a syntax element in the source code. The syntax elements include but are not limited to classes, functions, variables, etc. It should be noted that the type of programming language used in the source code file is different, and the corresponding types of syntax elements are also different.

[0047] It should be noted that the syntax elements considered in constructing the abstract syntax tree of the present application may be all the syntax elements contained in the source code file, or may be some key syntax elements contained in the source code file.

[0048] Step S220: based on the association relationship between the various syntax elements contained in all abstract syntax trees, determine all syntax elements that the current main function depends on; wherein the current main function is any main function contained in each source code file.

[0049] Among them, the main function is the starting point of the execution of the program corresponding to the source code file.

[0050] Taking Java code as an example, you can use depth-first traversal to obtain the main function and its related contents for the import part, package name, class name, etc. corresponding to the source code file. For example, syntax elements include classes, functions, and variables. Based on the abstract syntax tree, you can get all the syntax elements that the current main function of the current source code file depends on. Figure 3 , Figure 3 A schematic diagram of variables and called functions associated with the current main function is shown as an exemplary embodiment of the present application, such as Figure 3 As shown, the syntax elements that the current main function depends on include all called functions and variables associated with it.

[0051] Step S230: extracting code contents corresponding to all syntax elements that the current main function depends on from each source code file, and reorganizing the code contents corresponding to all syntax elements that the current main function depends on to obtain reorganized code.

[0052] It should be noted that since the specific implementation codes corresponding to some of the grammatical elements that the current main function depends on are not included in the specific implementation code corresponding to the current main function, it is necessary to extract the code content corresponding to all the grammatical elements that the current main function depends on, and reorganize the extracted code content to obtain the reorganized code, which contains all the code content of all the grammatical elements that the current main function depends on.

[0053] Exemplarily, the code content corresponding to each syntax element can be directly added to the corresponding position according to the original syntax structure of the current main function to obtain the reorganized code.

[0054] Exemplarily, the code contents corresponding to all the grammatical elements that the current main function depends on are reorganized to obtain the reorganized code, including: determining the programming language type corresponding to the current main function; obtaining the reorganized code format that matches the programming language type; and reorganizing the code contents corresponding to all the grammatical elements that the current main function depends on according to the matching reorganized code format to obtain the reorganized code.

[0055] Reorganized code formats corresponding to various types of programming languages ​​are pre-stored, and a matching reorganized code format is obtained according to the programming language type corresponding to the current main function, so that the code contents corresponding to all syntax elements that the current main function depends on are reorganized according to the matching reorganized code format to obtain a reorganized code.

[0056] Taking Java code as an example, the format of the reorganized code is: import part + class implementation, where the class implementation includes: variable definition + called function implementation + main function implementation. The structure of the reorganized code is expressed as:

[0057] / / Import part, limit the packages that need to be imported

[0058] import java.util.*;

[0059] / / Class implementation

[0060] public class xxClass {

[0061] / / Variable definition part

[0062] private int XXVariable;

[0063] / / The called function implementation part (the called method)

[0064] public void xXMethod(){

[0065] System.out.println("This is a method."),

[0066] }

[0067] / / Main function implementation part

[0068] @GenerateMain

[0069] public static void main(String[] args) {

[0070] XXClass obj = new XXClass (10,"Hello"); / / Create an object

[0071] obj.xxMethod(); / / Call method

[0072] }

[0073] }

[0074] It should be noted that the above reorganization code is only an illustrative example. For different types of programming languages, different reorganization code formats can be used to implement code reorganization, and this application does not limit this.

[0075] The code content corresponding to the function's functional implementation may be distributed in multiple different source code files. If the code is only cleaned and standardized according to the relevant technology, the subsequent model training will not be able to accurately understand the structural information of the code, which will lead to poor readability and maintainability of the model-generated code. Taking the above problems into consideration, this application reconstructs each source code file and presents the functional function and its related dependencies and implementations in the same file to facilitate subsequent model learning and understanding of the code structure.

[0076] Step S240: Obtain code annotations corresponding to the reorganized code, and use the reorganized code and the code annotations corresponding to the reorganized code as model training samples.

[0077] The comment content corresponding to the code in the source code file can be directly obtained and used as the code comment corresponding to the reorganized code.

[0078] Optionally, for code that lacks comments in the source code file, a code processing model with a code interpretation function can be used to generate comment content, wherein the code processing model used to generate comment content can be a model that already exists in the prior art, or it can be a model trained based on reorganized code with comment content, and this application does not limit this.

[0079] Next, some embodiments of the present application are described by way of example:

[0080] In some embodiments, obtaining code comments corresponding to the reorganized code in step S240 includes: obtaining comment content corresponding to the code in the source code file to obtain the original comment, and obtaining a comment conversion template; using the comment conversion template to convert the original comment, and using the comment conversion result as the code comment corresponding to the code in the reorganized code.

[0081] A pre-stored comment conversion template is obtained, the original comment is converted using the comment conversion template, and the comment conversion result is used as the code comment of the corresponding code in the reorganized code.

[0082] Exemplarily, obtaining a comment conversion template includes: analyzing the comment styles corresponding to all comment contents corresponding to the code in each source code file to obtain a plurality of comment style types; generating a comment conversion template corresponding to each comment style type; using the comment conversion template to convert the original comment, and using the comment conversion result as the code comment of the corresponding code in the reorganized code, including: using the comment conversion templates corresponding to the plurality of comment style types to convert the original comment multiple times to obtain a plurality of comment conversion results; using the plurality of comment conversion results as the code comment of the corresponding code in the reorganized code.

[0083] Optionally, after obtaining all the comment contents corresponding to the code in each source code file, the comment contents are filtered, such as filtering out erroneous comments or incomplete comments, such as filtering out noise data such as code, copyright statements, author information, website links, etc. contained in the comments through regular expressions.

[0084] The filtered annotation content is analyzed for annotation style to identify multiple different annotation style types, and a corresponding annotation conversion template is created for each identified annotation style analysis. The annotation content belonging to the same annotation style can be analyzed to generate an annotation conversion template corresponding to the annotation style based on the similarities of the annotation content of the same annotation style.

[0085] Then, the original comments are converted multiple times using comment conversion templates corresponding to various comment style types to obtain multiple comment conversion results, and the multiple comment conversion results are used as code comments for corresponding codes in the reorganized code.

[0086] For example, one of the comment styles is "descriptive comments starting with a verb", and the corresponding comment conversion template is used to add a verb before the original comment. For example, if the original comment is "calculate the sum of two numbers", the comment conversion template adds the verb "how" before the original comment, and the comment conversion result is "how to calculate the sum of two numbers".

[0087] For each code in the reorganized code, all comment conversion templates can be used to convert the corresponding original comments. The number of comments corresponding to the code can also be determined according to the importance of the code in the reorganized code, so as to select the corresponding number of comment conversion templates to convert the code. The importance can be determined according to the business module corresponding to the code and the number of associated functions. The higher the importance of the code, the larger the corresponding number of comments. Conversely, the lower the importance of the code, the smaller the corresponding number of comments. This ensures the ability to understand important codes in the subsequent model training process while reducing the amount of calculation.

[0088] Of course, the original comments can also be used as code comments for the corresponding code in the reorganized code.

[0089] In actual applications, since different developers may have different comment descriptions for the same code, the expression style of the comments in the source code is analyzed to create corresponding comment conversion templates based on the different comment style types obtained from the analysis, and then the original comment content of the comment conversion template is converted to expand the original comment description style, so that subsequent models can understand code descriptions of various styles.

[0090] In some implementations, obtaining code annotations corresponding to the reorganized code includes: obtaining business information and / or function information corresponding to the corresponding code in the reorganized code; and generating code annotations corresponding to the code in the reorganized code based on the business information and / or function information.

[0091] Optionally, for the code that originally has no comments, code comments for the code can be directly generated according to the business information and / or functional information corresponding to the code, so as to be used as code comments for the corresponding code in the reorganized code.

[0092] Optionally, for the code that originally has comments, the original comments can be modified according to the business information and / or functional information corresponding to the code to obtain new code comments, and the new code comments are used as code comments for the corresponding code in the reorganized code.

[0093] For example, for more professional industry code problems, more background information needs to be added. Therefore, more background information also needs to be added to the code comments used for model training to improve the model's understanding and accuracy of the input problems.

[0094] You can add both business information and function information to the original annotation, you can add only business information to the original annotation, or you can add only function information to the original annotation.

[0095] For example, according to the type of the code, an information modification strategy is determined, and the original comment is modified based on the information modification strategy and business information and / or functional information to obtain code comments corresponding to the code in the reorganized code.

[0096] For example, the code types may include business class codes and service call class codes, etc. For business class codes: add business information and function information to obtain a modified code annotation as "in the scenario corresponding to [business information], implement the [function information] function"; for service call class codes: add function information to obtain a modified code annotation as "implement the interface call of [function information]".

[0097] Through the above-mentioned embodiments, multiple types of code comments can be obtained for the same section of code. The converted code comments and the original comments can be combined to obtain a set of comments for the reorganized code corresponding to the code for subsequent model training.

[0098] See also Figure 4 , Figure 4 is a flowchart of a code processing method shown in an exemplary embodiment of the present application. The code processing method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0099] like Figure 4 As shown, the code processing method at least includes steps S410 to S420, which are described in detail as follows:

[0100] Step S410: Using the model training samples obtained by the sample construction method, the training code processes the model.

[0101] The model training samples obtained through the sample construction method are used to train the code processing model so that the model can learn programming patterns, function libraries, business implementation logic, etc., so that the trained code processing model can perform downstream tasks.

[0102] Step S420: input the code processing request into the trained code processing model to obtain the code processing result output by the trained code processing model.

[0103] Take the following example to illustrate that the downstream tasks include at least one of code generation, code interpretation, and code retrieval:

[0104] The code processing request includes one or more of a text description of a pre-generated code, a pre-generated interpreted code snippet, and a code retrieval prompt; the code processing request is input into a trained code processing model to obtain a code processing result output by the trained code processing model, including: inputting the text description of the pre-generated code into the trained code processing model to obtain a code generation result output by the trained code processing model; and / or, inputting the pre-generated interpreted code snippet into the trained code processing model to obtain a code interpretation result output by the trained code processing model; and / or, inputting the code retrieval prompt into the trained code processing model to obtain a retrieved code snippet output by the trained code processing model.

[0105] Code generation task: During the training process, the pre-trained code processing model receives the code comments in the model training samples as input, wherein a text description corresponding to the code generation problem can be generated based on the functional description and usage scenario in the code comments, and the text description is input into the pre-trained code processing model, and the code snippet corresponding to the code comment is used as the expected output to train the code processing model. In the application scenario, code generation prompts can be obtained based on the code processing request and the prompt template, that is, the text description of the pre-generated code is obtained. The text description is used to require the code processing model to generate a specific code snippet. For example, the text description of the pre-generated code is "Write a Java function to implement XX function". The code processing model will generate the corresponding code snippet based on the text description to improve the accuracy and readability of the generated code.

[0106] Code interpretation task: Similar to the model training process of the code generation task, the code snippet is used as the model input, and the code comment corresponding to the code snippet is used as the expected output to train the code processing model so that the model understands the function and logic of the input code, and then generates the corresponding comment. In the application scenario, the model input is a code snippet, and a prompt can be added to inquire about its function or logic. For example, the model input obtained based on the input code snippet and the prompt template is "Explain the function of the following code: [code snippet]", and the code processing model outputs the comment of the code function of the code snippet, which helps to improve the maintainability of the code and the efficiency of team collaboration.

[0107] Code retrieval task: The code processing model finds the most matching code snippet based on the key information (such as function name, parameter type, etc.) input, concatenates the key information input based on the prompt template to obtain the code retrieval prompt, inputs the code retrieval prompt into the code processing model, and obtains the relevant code snippet output by the model. For example, the code retrieval prompt is "According to the requirements, provide a Java code example, requirements: the main function name is addNumbers, and the input parameters are a and b". The code processing model returns a known function code snippet based on the input code retrieval prompt, which can improve the efficiency of code retrieval and code reuse.

[0108] Of course, in addition to the downstream tasks shown in the above embodiments, the code processing model can also be used to perform other code-related downstream tasks, which is not limited in this application.

[0109] In some implementations, a multi-level, multi-stage training strategy may be employed when training a code processing model based on model training samples.

[0110] Specifically, taking downstream tasks including code generation, code interpretation, and code retrieval as an example, given that the learning difficulty and requirements of tasks such as code generation, code interpretation, and code retrieval are different, a phased fine-tuning strategy is adopted to train the code processing model to ensure that the model can deepen the understanding of code tasks and improve task execution capabilities.

[0111] For example, in the initial stage of training, the model will focus on the two tasks of code generation and code retrieval, and use the model training samples corresponding to code generation and code retrieval to make preliminary fine-tuning of the code processing model to improve the model's code generation and code retrieval capabilities. After the accuracy of the model's code generation and code retrieval reaches the preset requirements, the training of the code interpretation task is introduced. By further fine-tuning the code processing model, the language comprehension ability, logical reasoning and abstract thinking ability of the code processing model are improved, so that the code processing model can perform the code interpretation task.

[0112] By introducing a multi-task and multi-stage learning mechanism, the code processing model will face mutual influence and promotion between different tasks during the entire training process, which helps to enhance the ability of the code processing model to handle complex tasks. For example, the combination of code generation tasks and code interpretation tasks can enable the model to pay more attention to logical clarity and readability when generating code; and the fusion of code retrieval tasks and code interpretation tasks enables it to provide more accurate results based on the understanding of the meaning of the code when retrieving the code. The above embodiments enable the model to better adapt to the needs of specific industries and improve the accuracy and practicality of code processing.

[0113] Furthermore, the code processing results output by the model are evaluated so as to continuously learn and optimize the parameters of the code processing model according to the evaluation results.

[0114] Exemplarily, the code processing result evaluation method includes:

[0115] Code correctness assessment: Static code analysis tools can be used to check whether the generated code has syntax errors, potential runtime errors, or non-compliance with programming specifications; test cases can be used to verify whether the functions of the generated code are consistent with business requirements, ensuring that the functions implemented by the code are consistent with user expectations.

[0116] Comment correctness assessment: Natural Language Processing (NLP) technology can be used to compare the semantic similarity between the annotation content in the model training samples and the model output annotations, which helps to ensure the validity of the annotations and prevent incorrect annotations. The readability and quality of the model output annotations can be evaluated to ensure that the model output annotations can clearly explain the functions and logic of the code.

[0117] Robustness and fault tolerance evaluation: Simulate different inputs to check whether the code processing model can distinguish between different and identical questions and give appropriate prompts. Specifically, the training model gives suggestions for question correction when receiving incorrect or incomplete question input. Test whether the model can give correction suggestions instead of generating incorrect code or comments when encountering incorrect or incomplete question input.

[0118] Based on the above-mentioned evaluation results, the model is continuously learned, model parameters are adjusted, and model structure is optimized to improve the model's ability to handle exceptions, errors, and complex tasks. Through this iterative process, the model will continue to evolve, better adapt to various programming scenarios, and provide higher-quality code generation and parsing services. At the same time, the prompt template can be further optimized through continuous usage feedback to make it closer to actual needs and improve the performance of the model.

[0119] In some embodiments, the code processing model includes a semantic matching network and multiple code processing networks, and different code processing networks correspond to different business fields; a code processing request is input into the trained code processing model to obtain a code processing result output by the trained code processing model, including: inputting the code processing request into the semantic matching network for classification to obtain a business field output by the semantic matching network that matches the code processing request; inputting the code processing request into the code processing network corresponding to the matched business field to obtain a code processing result output by the code processing network.

[0120] See also Figure 5 , Figure 5A schematic diagram of a code processing model is shown as an exemplary embodiment of the present application. Figure 5 As shown, the code processing model includes a semantic matching network and multiple code processing networks, such as a code processing network including a specified industry code processing network, a general code processing network, a language processing network, etc. The semantic matching network is used to divert the input code processing request to the matching code processing network, such as classifying the code processing request into a specified industry code type question, a general code type question, a chat type question, a knowledge question and answer type question, or a question without practical significance, etc., inputting the code processing request belonging to the specified industry code type question into the specified industry code processing network, inputting the code processing request belonging to the general code type question into the general code processing network, and inputting the code processing request belonging to the chat type question, the knowledge question and answer type question, and the question without practical significance into the language processing network.

[0121] Since the industry code model for a specific field usually contains a large amount of domain expertise and specific programming patterns during the actual use of the model, the model may over-learn these specific features and patterns during training, and there is a risk of overfitting. Although the model can solve specific programming problems in the training set well, it may not be effectively applied or adapted in scenarios of different industries. In order to solve this problem, the above-mentioned embodiment adopts a strategy of multi-code processing network collaboration, and automatically calls multiple code processing networks through a semantic matching network. It can utilize the advantages of each code processing network, reduce the risk of overfitting of a single code processing network, and improve the model's generalization ability and actual application effect.

[0122] In some embodiments, the code processing model includes a retrieval network and a code processing network, and a code knowledge base is pre-stored, wherein the code knowledge base contains multiple sample code snippets or sample annotation snippets; a code processing request is input into the trained code processing model to obtain a code processing result output by the trained code processing model, including: using the retrieval network to calculate the matching degree between the code processing request and each sample code snippet or sample annotation snippet in the code knowledge base, and selecting the sample code snippet or sample annotation snippet whose matching degree meets the preset conditions as the matching knowledge snippet; inputting the code processing request into the code processing network to obtain the code processing result output by the code processing network; or, inputting the code processing request and the matching knowledge snippet into the code processing network to obtain the code processing result output by the code processing network; and taking the code processing result and the matching knowledge snippet as the final result.

[0123] When the questions carried by the code processing request are vague or incomplete, relying solely on the code processing model may not accurately understand the code processing requirements, and the generated results may not meet expectations. Based on this, the present application constructs a code knowledge base for retrieval, which contains multiple sample code snippets or sample annotation snippets. Among them, the sample code snippets or sample annotation snippets can be obtained after collecting and analyzing historical data, or can be obtained by training samples of models built based on samples.

[0124] For example, the code of the model training sample is divided into multiple code segments, each of which has a corresponding code comment, thereby obtaining multiple knowledge fragments. A single knowledge fragment includes code comments, specific implementation code, and main function name. The specific form of the knowledge fragment is as follows:

[0125] {

[0126] "Function description":"Code comments"

[0127] "Implementation code":"

[0128] import java.util.*,

[0129] public class XXClass {

[0130] private int XXVariable,

[0131] public void XXMethod() {

[0132] System.out.println("This is a method.");

[0133] }

[0134] @GenerateMain

[0135] public static void main(String[] args) {

[0136] XXClass obj= new XXClass(10,"Hello");

[0137] obj.xxMethod();

[0138] }

[0139] }",

[0140] "Main function name": "main", ..}

[0141] In some embodiments, see Figure 6 , Figure 6 A schematic diagram of code processing is shown for an exemplary embodiment of the present application. Figure 6 As shown, a code processing request is obtained, and an embedding vector corresponding to the code processing request is extracted using a retrieval network, and the similarity between the embedding vector corresponding to the code processing request and the embedding vector of a sample code fragment or the embedding vector of a sample annotation fragment in a code knowledge base is calculated, and the sample code fragment and / or sample annotation fragment with the highest similarity is selected to obtain a matching knowledge fragment; and the code processing request is input into the code processing network to obtain a code processing result output by the code processing network. The code processing result and the matching knowledge fragment are displayed as the final result. After obtaining the matching knowledge fragment, the matching knowledge fragment can be converted into text to obtain a retrieval generation result that is easy to understand.

[0142] In some embodiments, see Figure 7 , Figure 7 A schematic diagram of code processing is shown for another exemplary embodiment of the present application. Figure 7 As shown, a code processing request is obtained, an embedding vector corresponding to the code processing request is extracted using a retrieval network, the similarity between the embedding vector corresponding to the code processing request and the embedding vector of a sample code fragment or the embedding vector of a sample annotation fragment in a code knowledge base is calculated, and the sample code fragment and / or the sample annotation fragment with the highest similarity is selected to obtain a matching knowledge fragment; and the code processing request and the matching knowledge fragment are input into a code processing network to obtain a code processing result output by the code processing network; or, the matching knowledge fragment is converted into text, and the converted text and the code processing request are input into the code processing network to obtain a code processing result output by the code processing network.

[0143] The prompt template can be used to combine the code processing request and the matching knowledge fragment, or to combine the text converted from the matching knowledge fragment and the code processing request. For example, the combined model input is "[knowledge fragment]\nrefer to the above example content, [user request]". In this way, when the code processing model performs the code processing task, it can not only rely on the prompt content given by the code processing request, but also refer to the actual code instance, and the generated result will be closer to the actual demand.

[0144] It should be noted that Figure 6 and Figure 7 The corresponding embodiments can be used alone or in combination to meet the needs of different scenarios and provide more comprehensive, efficient and accurate code generation, code interpretation and code retrieval services.

[0145] The sample construction method provided in the present application parses the syntax elements contained in each source code file, and constructs the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file; based on the association relationship between the various syntax elements contained in all abstract syntax trees, determines all syntax elements that the current main function depends on; extracts the code content corresponding to all syntax elements that the current main function depends on from each source code file, and reorganizes the code content corresponding to all syntax elements that the current main function depends on to obtain reorganized code; obtains the code comments corresponding to the reorganized code, and uses the reorganized code and the code comments corresponding to the reorganized code as model training samples to reconstruct each source code file, and presents the functional functions and their related dependencies and implementations in the same file, so as to facilitate the subsequent model to learn and understand the structure of the code, and improve the code processing capability of the code processing model obtained by subsequent training.

[0146] Figure 8 FIG. 1 is a block diagram of a sample construction device shown in an exemplary embodiment of the present application. Figure 8 As shown, the exemplary sample construction device 800 includes: a code parsing module 810, a dependency analysis module 820, a code reorganization module 830 and a sample generation module 840. Specifically:

[0147] A code parsing module 810 is used to parse the syntax elements contained in each source code file, and construct the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file;

[0148] A dependency analysis module 820, for determining all syntax elements that the current main function depends on based on the association relationship between the various syntax elements contained in all abstract syntax trees; wherein the current main function is any main function contained in each source code file;

[0149] A code reorganization module 830 is used to extract code contents corresponding to all syntax elements that the current main function depends on from each source code file, and reorganize the code contents corresponding to all syntax elements that the current main function depends on to obtain reorganized code;

[0150] The sample generation module 840 is used to obtain the code annotations corresponding to the reorganized code, and use the reorganized code and the code annotations corresponding to the reorganized code as model training samples.

[0151] Fig. 9 FIG. 1 is a block diagram of a code processing device shown in an exemplary embodiment of the present application. Fig. 9 As shown, the exemplary code processing device 900 includes: a model training module 910 and a code processing module 920. Specifically:

[0152] A model training module 910 is used to train a code processing model using a model training sample obtained by a sample construction method;

[0153] The code processing module 920 is used to input the code processing request into the trained code processing model to obtain the code processing result output by the trained code processing model.

[0154] It should be noted that the sample construction device and code processing device provided in the above embodiment belong to the same concept as the sample construction method and code processing method provided in the above embodiment, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, which will not be repeated here. In actual applications, the sample construction device and code processing device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, which is not limited here.

[0155] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 1000 includes a memory 1001 and a processor 1002, and the processor 1002 is used to execute program instructions stored in the memory 1001 to implement the steps in any of the above sample construction methods and code processing method embodiments. In a specific implementation scenario, the electronic device 1000 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 1000 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0156] Specifically, the processor 1002 is used to control itself and the memory 1001 to implement the steps in any of the above-mentioned sample construction methods and code processing method embodiments. The processor 1002 can also be called a central processing unit (CPU). The processor 1002 may be an integrated circuit chip with signal processing capabilities. The processor 1002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 1002 can be implemented by an integrated circuit chip.

[0157] See also Fig.11 , Fig.11 The computer-readable storage medium 1100 stores program instructions 1110 that can be executed by a processor, and the program instructions 1110 are used to implement the steps in any of the above sample construction methods and code processing method embodiments.

[0158] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0159] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0161] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A code processing method, characterized in that: The method comprises: Parsing the syntax elements contained in each source code file, and constructing the syntax elements contained in the source code file into an abstract syntax tree according to the syntax structure corresponding to the source code file; Based on the association relationship between the various syntax elements contained in all abstract syntax trees, determine all syntax elements that the current main function depends on; wherein the current main function is any main function contained in each source code file; Extracting code contents corresponding to all syntax elements that the current main function depends on from each source code file, and reorganizing the code contents corresponding to all syntax elements that the current main function depends on to obtain reorganized code; Analyze the comment styles corresponding to all the comment contents corresponding to the code in each source code file to obtain multiple comment style types, and generate comment conversion templates corresponding to each comment style type; Obtaining the comment content corresponding to the code in the source code file to obtain the original comment, and, based on the importance of the code, determining the number of comments corresponding to the code, and selecting the comment conversion templates with the number of comments from the comment conversion templates of multiple comment style types; The original annotations are converted respectively by using each selected annotation conversion template to obtain a plurality of annotation conversion results corresponding to a plurality of annotation style types, the plurality of annotation conversion results are used as code annotations corresponding to the code in the reorganized code, and the reorganized code and the code annotations corresponding to the reorganized code are used as model training samples; Using the model training samples, the training code processes the model; The code processing request is input into the trained code processing model to obtain the code processing result output by the trained code processing model.

2. The method according to claim 1, characterized in that The reorganizing the code contents corresponding to all the syntax elements that the current main function depends on to obtain the reorganized code includes: Determine the programming language type corresponding to the current main function; Acquire a reorganized code format matching the programming language type; The code contents corresponding to all the syntax elements that the current main function depends on are reorganized according to the matched reorganized code format to obtain the reorganized code.

3. The method according to claim 1, characterized in that Obtaining code comments corresponding to the reorganized code, including: Obtaining business information and / or function information corresponding to a corresponding code in the reorganized code; Generate code annotations for corresponding codes in the reorganized code based on the business information and / or function information.

4. The method according to claim 1, characterized in that The code processing request includes one or more of a text description of a pre-generated code, a code snippet of a pre-generated explanation, and a code search prompt; the code processing request is input into a trained code processing model to obtain a code processing result output by the trained code processing model, including: Inputting the text description of the pre-generated code into a trained code processing model to obtain a code generation result output by the trained code processing model; and / or, Inputting the pre-generated interpreted code snippet into a trained code processing model to obtain a code interpretation result output by the trained code processing model; and / or, The code retrieval prompt is input into the trained code processing model to obtain the retrieved code snippet output by the trained code processing model.

5. The method according to claim 1, characterized in that The code processing model includes a semantic matching network and multiple code processing networks, and different code processing networks correspond to different business fields; the code processing request is input into the trained code processing model to obtain the code processing result output by the trained code processing model, including: Inputting the code processing request into the semantic matching network for classification, and obtaining the business field output by the semantic matching network that matches the code processing request; The code processing request is input into the code processing network corresponding to the matched business field, and a code processing result output by the code processing network is obtained.

6. The method according to claim 1, characterized in that The code processing model includes a retrieval network and a code processing network, and a code knowledge base pre-stored therein, wherein the code knowledge base contains a plurality of sample code snippets or sample annotation snippets; the code processing request is input into the trained code processing model to obtain a code processing result output by the trained code processing model, including: Calculating the matching degree between the code processing request and each sample code segment or sample annotation segment in the code knowledge base by using the search network, and selecting the sample code segment or sample annotation segment whose matching degree meets the preset condition as the matching knowledge segment; Inputting the code processing request into the code processing network to obtain a code processing result output by the code processing network; or inputting the code processing request and the matching knowledge fragment into the code processing network to obtain a code processing result output by the code processing network; The code processing result and the matching knowledge fragment are taken as the final result.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the code processing method as described in any one of claims 1-6.

Images