Code completion large model data set construction method and device, equipment and medium

Abstract

The invention discloses a code completion large model data set construction method and device, equipment and a medium, and relates to the field of computers.The method comprises the steps that code data obtained in advance is analyzed and stored in a preset code data set; constructing a preset code completion scene set; the preset code completion scene set comprises a preset code completion scene; any code completion scene comprises a matching rule corresponding to the code completion scene and a completion node extraction strategy; for any target code in the preset code data set, the target code is matched with the code completion scene with the minimum historical matching number in the preset code completion scene set in sequence, and if the target code completion scene corresponding to the target code is matched, the target code is completed. Extracting a target completion node code from the target code based on a corresponding completion node extraction strategy; and obtaining a first data set according to the target completion node code, compressing the first data set to obtain a second data set, and training the code completion large model according to the second data set.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device, equipment and medium for constructing a large model data set for code completion. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, code completion has become an important tool for programmers to improve efficiency and reduce errors when writing code. The code completion model can help reduce the time of writing code and improve accuracy by predicting the code snippets that users will enter.

[0003] In the existing code completion model training, the code completion dataset in FIM (Fill-in-the-Middle) format is usually used for model training. Most of the existing technologies are based on the Abstract Syntax Tree (AST) method to construct the code completion dataset in FIM format. However, this method lacks consideration of the code completion requirements in actual development tasks, resulting in the generated data not being consistent with the actual completion scenario. On the other hand, the common grammatical structures in the code completion dataset obtained by the above method occupy a large amount of data, and there is less rare or complex data, resulting in insufficient generalization ability of the model in diversified completion scenarios. At the same time, sometimes the obtained code completion dataset is too long, resulting in reduced reasoning efficiency and accuracy of the model. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a method, device, equipment and medium for constructing a large model dataset for code completion, which can construct a set of completion scenarios according to the characteristics of different programming languages ​​and actual code completion requirements, so as to better simulate the actual completion requirements and obtain a high-quality code completion dataset. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a method for constructing a large model dataset for code completion, comprising:

[0006] Parsing the pre-acquired code data and saving the parsed code data into a preset code data set;

[0007] Constructing a preset code completion scenario set; the preset code completion scenario set includes different preset code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0008] For any target code in the preset code data set, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence; if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, a target completion node code is extracted from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario;

[0009] A first data set is obtained based on the target completion node code and the preceding code and succeeding code corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train the code completion model according to the second data set.

[0010] Optionally, parsing the pre-acquired code data and saving the parsed code data into a preset code data set includes:

[0011] Acquire code source data of various types of programming languages ​​based on preset code acquisition conditions, and perform syntax analysis on the code source data using an abstract syntax tree technology to convert the code source data into an abstract syntax tree;

[0012] Each node in the abstract syntax tree is traversed, a target node and a node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set.

[0013] Optionally, the code completion large model data set construction method further includes:

[0014] Based on the preset filtering criteria, target nodes and node contents corresponding to the preset filtering criteria are deleted from the preset code data set to filter the preset code data set.

[0015] Optionally, for any target code in the preset code data set, sequentially matching the target code with the code completion scenarios with the smallest number of historical matches in the preset code completion scenario set, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario, including:

[0016] For any target code in the preset code data set, sequentially matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set;

[0017] If there is a matching rule corresponding to the target code, taking the code completion scenario corresponding to the matching rule as the target code completion scenario, and extracting the target completion node code from the target code based on the completion node extraction strategy corresponding to the target code completion scenario;

[0018] If there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the process jumps to the step of matching the target code with the matching rules corresponding to the code completion scene with the smallest number of historical matches in the preset code completion scene set in sequence until the preset code data set is traversed.

[0019] Optionally, the code completion large model data set construction method further includes:

[0020] Obtaining the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and storing each code completion scenario and the corresponding number of historical matches;

[0021] Correspondingly, the taking the code completion scenario corresponding to the matching rule as the target code completion scenario further includes:

[0022] The code completion scenario corresponding to the matching rule is determined, and the number of historical matches corresponding to the code completion scenario is increased, and each stored code completion scenario and the corresponding number of historical matches are updated.

[0023] Optionally, compressing the first data set to obtain a second data set includes:

[0024] A preset post-context length is set, and based on the preset post-context length, the post-context code farthest from the target completion node code in the first data set is deleted to obtain a compressed second data set.

[0025] Optionally, the code completion large model data set construction method further includes:

[0026] For the preceding code and the succeeding code corresponding to each target completion node code in the second data set, data of the target file called in the preceding code and the succeeding code are obtained, and the data is saved in the second data set to obtain an updated second data set.

[0027] In a second aspect, the present application provides a device for constructing a large model data set for code completion, comprising:

[0028] A data parsing module is used to parse the pre-acquired code data and save the parsed code data into a preset code data set;

[0029] A set construction module, used to construct a preset code completion scenario set; the preset code completion scenario set includes different preset code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0030] a code acquisition module, for sequentially matching any target code in the preset code data set with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario;

[0031] The data set acquisition module is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

[0032] In a third aspect, the present application provides an electronic device, including:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned code completion large model data set construction method.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned code completion large model dataset construction method.

[0036] In the present application, the pre-acquired code data is parsed and the parsed code data is saved in a preset code data set; a preset code completion scenario set is constructed; the preset code completion scenario set includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; for any target code in the preset code data set, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the historical matching number is the number of target codes that have been successfully matched in any code completion scenario; a first data set is obtained based on the target completion node code and the preceding and succeeding codes corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train the code completion large model according to the second data set. From the above, it can be seen that this application pre-sets different code completion scenarios according to the characteristics of different programming languages ​​and the actual code completion needs, and sets corresponding matching rules and completion node extraction strategies for each code completion scenario, so as to solve the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn, ensuring balance to improve the generalization ability of the model; at the same time, by compressing the first data set to obtain the second data set, it is ensured that the obtained second data set meets the input context length limit of the model, thereby effectively improving the reasoning efficiency and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A flowchart of a method for constructing a large model data set for code completion disclosed in this application;

[0039] Figure 2 A schematic diagram of a specific code completion large model data set construction process disclosed in this application;

[0040] Figure 3 A schematic diagram of the structure of a device for constructing a large model data set for code completion disclosed in this application;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In the existing code completion model training, a code completion dataset in FIM format is usually used for model training. Most of the existing technologies use the abstract syntax tree method to construct a code completion dataset in FIM format. However, this method lacks consideration of the code completion requirements in actual development tasks, resulting in the generated data not being consistent with the actual completion scenario. On the other hand, the common grammatical structures in the code completion dataset obtained by the above method occupy a large amount of data, and there is less rare or complex data, resulting in insufficient generalization of the model in diversified completion scenarios. At the same time, sometimes the acquired code completion dataset is too long, resulting in reduced reasoning efficiency and accuracy of the model. To this end, the present application provides a method for constructing a large model dataset for code completion, which can construct a set of completion scenarios according to the characteristics of different programming languages ​​and actual code completion requirements, so as to better simulate the actual completion requirements and obtain a high-quality code completion dataset.

[0044] See also Figure 1 As shown, the embodiment of the present application discloses a method for constructing a large model data set for code completion, including:

[0045] Step S11, parsing the pre-acquired code data and saving the parsed code data into a preset code data set.

[0046] In this embodiment, it is first necessary to collect codes of various programming languages ​​as source data of the code completion large model data set, and then parse the source data and save the parsed source data into a preset code data set. Specifically, it includes: obtaining code source data of various types of programming languages ​​based on preset code acquisition conditions, and using abstract syntax tree technology to perform syntax analysis on the code source data to convert the code source data into an abstract syntax tree; traversing each node in the abstract syntax tree, extracting the target node and the node content corresponding to the target node from the node, and saving the target node and the node content into the preset code data set.

[0047] For example, high-star open source code repositories and high-quality business code repositories are collected from open source code platforms (such as GitHub and Gitee) to obtain the codes of various projects. In order to ensure the quality of the code, the following restrictions (i.e., preset code acquisition conditions) can be set: the time is within the past three years and the number of collections is at least greater than 100. Based on the above restrictions, and for each programming language, the projects are sorted according to the number of collections, and the first preset number of projects are selected from high to low; based on the selected project code, the project code files are parsed using the AST (Abstract SyntaxTree) parser (such as Tree-sitter) corresponding to each programming language in the project code to convert the code into the form of an abstract syntax tree; among them, the abstract syntax tree clearly displays the syntax elements and structural relationships of the code in a tree structure, abstracts various components in the code (such as variables, operators, control structures, etc.) as nodes, and accurately reflects the logical structure of the code through the hierarchical relationship between nodes. This parsing method facilitates in-depth analysis of the code, so that more sophisticated operations and information extraction can be performed based on the structure of the syntax tree in the future.

[0048] After obtaining the abstract syntax tree, all nodes in the syntax tree can be traversed. During the traversal process, the target nodes of the required type are screened out according to the pre-set rules, and the target nodes and the node contents they contain are saved in the preset code data set; wherein the preset code data set can be implemented by using classes or other suitable data structures. For example, taking the construction of the method node class of Java (Java programming language, an object-oriented programming language) as an example, multiple attributes are defined to store different information related to the method, such as the document comment node is used to store the annotation description of the method, which helps to understand the function and purpose of the method; the modifier node records the modifier of the method (such as public, private, etc.), which determines the access rights and other characteristics of the method; the method name node saves the name of the method, which is the key identifier for identifying and calling the method; the method body node contains the specific implementation logic of the method. In this way, the scattered key information in the code is integrated and stored according to a certain structure, which facilitates the rapid and accurate acquisition and utilization of this information in the subsequent construction of the data set and training model, laying the foundation for efficient code completion.

[0049] Furthermore, based on the preset filtering criteria, the target nodes and node contents corresponding to the preset filtering criteria may be deleted from the preset code data set to filter the preset code data set.

[0050] It is understandable that after obtaining the preset code data set, the meaningless information in the data set can be filtered. This type of information mainly includes comments that are not related to the actual code, such as license information, author information, timestamps, and "TODO", etc.; among them, TODO is used to mark to-do items; the specific filtering method can locate the comment node in the current file according to the results of code parsing, and judge whether it is a meaningless comment based on the content of the node. If it is confirmed that the comment is meaningless, the corresponding node is deleted in the code structure. It should be noted that the judgment rules can be defined according to the characteristics of the node. For example, the comment node of the license information is usually the first comment node in the current code syntax tree, and its content generally contains keywords such as "License". Through this filtering process, redundant information can be effectively removed, thereby providing a more accurate and clear code context for subsequent model reasoning.

[0051] Step S12: construct a preset code completion scenario set; the preset code completion scenario set includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario.

[0052] In this embodiment, according to the characteristics of different programming languages ​​and actual code completion requirements, a preset code completion scenario set is constructed so as to set specific code matching rules and completion node extraction strategies for each scenario.

[0053] Specifically, define a set of preset code completion scenarios :

[0054] ;

[0055] Among them, each code completion scenario Include matching rules and completion node extraction strategy .

[0056] The above code completion scenarios are abstractions and simulations of various code completion situations that may be encountered in actual programming. For example, in actual programming, you may encounter situations where you need to complete method documentation comments. In order to better simulate the actual completion needs, it is necessary to ensure that the nodes extracted from the code (i.e., completion nodes) can match the actual application scenarios, that is, the extracted information is what really needs to be completed in actual programming, rather than some irrelevant information or information that does not meet actual needs, so as to ensure the practicality and accuracy of code completion. For example, during the coding process, if you need to complete the document comments of the method based on the method content, the matching rule for the corresponding scenario is to determine whether the current method node contains document comments, and the completion node extraction strategy is to extract the method document comment node content as the completion content.

[0057] Step S13: for any target code in the preset code data set, the target code is matched with the code completion scene with the smallest number of historical matches in the preset code completion scene set in sequence; if a target code completion scene corresponding to the target code is matched from the preset code completion scene set, a target completion node code is extracted from the target code based on a completion node extraction strategy corresponding to the target code completion scene; the number of historical matches is the number of target codes that have been successfully matched by any code completion scene.

[0058] In this embodiment, scene matching is performed based on pre-set code completion scenarios, and corresponding matching rules and completion node extraction strategies, including: for any target code in a preset code data set, the target code is matched in sequence with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; if there is a matching rule corresponding to the target code, the code completion scenario corresponding to the matching rule is used as the target code completion scenario, and the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; if there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the step of matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence is jumped to, until the preset code data set is traversed.

[0059] Usually, the triggering scenario of code completion mostly occurs inside the method. Therefore, in this embodiment, method nodes are extracted from the preset code data set, scenario rule matching is performed on each method in the code file at the method level, and the corresponding completion node is extracted according to the matching result. The specific formula is as follows:

[0060] For method code and scenes , define the matching function :

[0061] ;

[0062] like , then according to the scene Corresponding completion node extraction strategy Extract the corresponding completion node:

[0063] ;

[0064] in, From the method code The completion node extracted from the function Used to extract completion nodes.

[0065] It should be noted that in order to ensure the balance of the acquired completion nodes, in this embodiment, for each target code in the preset code data set, it is not randomly matched with the scenes in the code completion scene set, but the matching rules corresponding to the code completion scene with the smallest number of historical matches are preferentially selected for matching, so that each scene can be used relatively evenly in the entire data set, avoiding over-matching of some common scenes and ignoring some uncommon but important scenes, so that the model can learn various code completion situations more comprehensively.

[0066] Specifically, the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set is obtained, and each code completion scenario and the corresponding number of historical matches are stored, the scene with the smallest number of historical matches is matched first as the target code completion scenario, and the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; then the number of historical matches corresponding to the above target code completion scenario is increased, and the stored code completion scenarios and the corresponding number of historical matches are updated.

[0067] For example, a small root heap can be maintained to store the number of samples for each scenario (i.e., the number of historical matches), and the scenarios with fewer samples are matched first. The specific process is as follows:

[0068] Defining a small root pile , used to store each scene and its corresponding number of samples:

[0069] ;

[0070] Small root pile Ensure that the number of samples at the root node is minimal:

[0071] ;

[0072] Scene matching prioritizes the root node as the matching scene:

[0073] ;

[0074] in, Indicates the selected code completion scenario.

[0075] If the match fails, the other scenes in the stack are matched in turn until a matching scene is found or all scenes are traversed.

[0076] If a matching scene is found, the number of samples of the matching scene is updated:

[0077] ;

[0078] in, The number of samples corresponding to the successfully matched code completion scenario.

[0079] Reinsert the updated number of samples into the heap :

[0080] ;

[0081] Among them, update_heap is the update function.

[0082] Through the above steps, the balance of sample quantity can be ensured, thereby optimizing the training effect of the model.

[0083] Step S14, obtaining a first data set based on the target completion node code and the preceding code and succeeding code corresponding to the target completion node code in the preset code data set, and compressing the first data set to obtain a second data set, so as to train the code completion large model according to the second data set.

[0084] In this embodiment, after obtaining the target completion node code, the target completion node code and the corresponding preceding code and following code are saved in the first data set, wherein the current code in the first data set can be divided into a preceding text (prefix), a completion content (infill) and a following text (suffix), from the beginning of the code to the completion node is the preceding text, the completion node itself is the completion content, and from the completion node to the end of the code is the following text.

[0085] In order to ensure that the data set does not exceed the input context length of the large model, content compression can be performed on ultra-long data. Since the preceding text information of the completion content is more important than the following text, the following text can be compressed first, including: setting a preset following text length, and based on the preset following text length, deleting the following text code in the first data set that is farthest from the target completion node code, to obtain a compressed second data set.

[0086] For example, first determine the default length of the following text To ensure that the length of the following text is less than Generally, the farther the method node is from the completion node in the following text, the less important it is. Therefore, the method body content can be removed from the back to the front, retaining the basic method signature and other information until the length of the following text is less than .

[0087] It should be noted that in code completion, sometimes it is necessary to complete the code of other called files. At this time, the relevant file information is crucial to the model. However, the AST method cannot capture relevant file information across files or modules. Therefore, the LSP (Language Server Protocol) technology can be used to find the relevant files referenced by the current file. Specifically, for the preceding code and following code corresponding to each target completion node code in the second data set, the data of the target file called in the preceding code and following code is obtained, and the data is saved in the second data set to obtain an updated second data set. At the same time, in order to avoid the content of the second data set being too long, the relevant file content in the second data set can also be compressed.

[0088] For example, obtain the compressed second data set, determine the post-context and pre-context corresponding to the target completion node code in the second data set, use LSP technology to obtain the paths and codes of other files in the project called in the pre-context and post-context, and add them to the target data set in the order of the pre-context being greater than the post-context and the closer the distance to the completion node, the higher the priority; set the model input context length limit to , delete the method body nodes and redundant information in the code from the above target data set, add the path and the code in the compressed target data set to the second data set, until the data length in the second data set reaches Through this extraction process, relevant file information across files can be effectively extracted, thereby improving the model's code completion capabilities in multi-file projects.

[0089] As can be seen from the above, after the present embodiment performs syntax parsing on the code files in the open source repository, the comment nodes in the syntax tree are screened and meaningless nodes are deleted to ensure that invalid information does not interfere with the model's understanding of the valid context, so as to improve the quality and relevance of the data; according to the characteristics of different programming languages ​​and the actual code completion requirements, different code completion scenarios are pre-set, and corresponding matching rules and completion node extraction strategies are set for each code completion scenario, so as to solve the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn, ensuring balance to improve the generalization ability of the model; a code compression method is also proposed in this embodiment to ensure that the data meets the input context length limit of the large model. This method dynamically evaluates the importance of method nodes in the subsequent text, removes redundant content from back to front, retains only basic method signatures and other information, and executes the compression steps cyclically until the length of the subsequent text meets the set target, thereby effectively improving the reasoning efficiency and accuracy of the model; at the same time, the LSP technology is used to make up for the shortcomings of the AST method in capturing cross-file information. According to the principle that previous text dependencies take precedence over subsequent text dependencies, the relevant file information called by the current file is added to the data set, thereby enhancing the completion ability of the model in multi-file projects.

[0090] See also Figure 2 As shown, the technical solution in this application is explained below by taking the specific code completion large model data set construction process as an example.

[0091] First, high-star open source code repositories and high-quality business code repositories are collected from the open source code platform, and the codes of each project are obtained from them to obtain code source data; the code source data is parsed using abstract syntax tree technology to convert the code source data into an abstract syntax tree; each node in the abstract syntax tree is traversed, the target node and the node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set; then, the code in the above preset code data set is filtered for redundant information based on the preset filtering criteria to obtain a filtered preset code data set; then a preset code completion scenario set is constructed to simulate a real completion scenario, and scenario matching is performed based on the preset code completion scenario set and the preset code data set to obtain a first data set consisting of the target completion node code and the corresponding preceding and succeeding codes; further, the first data set is compressed to obtain a second data set, and the LSP technology is used to extract cross-file information from the second data set to obtain the final required target data set. The target data set is used to train the large code completion model. In the code compression operation and cross-file information extraction operation, the data set can be compressed by setting a preset data length to perform data fine-tuning. By fine-tuning the data, the input data does not exceed the model input context length limit.

[0092] As can be seen from the above, this embodiment takes the specific code completion large model dataset construction process as an example to elaborate on the process of building a high-quality code completion dataset, and provides a systematic and scientific data construction method to provide high-quality data support for the training of the code completion large model, thereby improving the programming efficiency and accuracy of developers.

[0093] See also Figure 3 As shown, the embodiment of the present application also discloses a device for constructing a large model data set for code completion, including:

[0094] The data analysis module 11 is used to analyze the pre-acquired code data and save the analyzed code data into a preset code data set;

[0095] A set construction module 12, used to construct a preset code completion scenario set; the preset code completion scenario set includes different preset code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0096] a code acquisition module 13, for sequentially matching any target code in the preset code data set with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario;

[0097] The data set acquisition module 14 is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

[0098] From the above, it can be seen that this application pre-sets different code completion scenarios according to the characteristics of different programming languages ​​and the actual code completion needs, and sets corresponding matching rules and completion node extraction strategies for each code completion scenario, so as to solve the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn, ensuring balance to improve the generalization ability of the model; at the same time, by compressing the first data set to obtain the second data set, it is ensured that the obtained second data set meets the input context length limit of the model, thereby effectively improving the reasoning efficiency and accuracy of the model.

[0099] In some specific implementations, the data parsing module 11 includes:

[0100] A data parsing unit, configured to obtain code source data of various types of programming languages ​​based on a preset code acquisition condition, and perform syntax parsing on the code source data using an abstract syntax tree technique to convert the code source data into an abstract syntax tree;

[0101] The data set determination submodule is used to traverse each node in the abstract syntax tree, extract a target node and a node content corresponding to the target node from the node, and save the target node and the node content into a preset code data set.

[0102] In some specific implementations, the data set determination submodule further includes:

[0103] The filtering unit is used to delete the target node and the node content corresponding to the preset filtering standard from the preset code data set based on the preset filtering standard, so as to filter the preset code data set.

[0104] In some specific implementations, the code acquisition module 13 includes:

[0105] a matching unit, configured to sequentially match, for any target code in the preset code data set, the target code with a matching rule corresponding to a code completion scenario with the smallest number of historical matches in the preset code completion scenario set;

[0106] A first processing submodule is configured to, if there is a matching rule corresponding to the target code, use the code completion scenario corresponding to the matching rule as the target code completion scenario, and extract a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario;

[0107] The second processing unit is used to reselect a new target code from the preset code data set if there is no matching rule corresponding to the target code, and jump to the step of matching the target code with the matching rules corresponding to the code completion scene with the smallest number of historical matches in the preset code completion scene set based on the new target code, until the preset code data set is traversed.

[0108] In some specific implementations, the code completion large model data set construction device further includes:

[0109] A storage unit, used to obtain the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and store each code completion scenario and the corresponding number of historical matches;

[0110] Correspondingly, the first processing submodule further includes:

[0111] The data updating unit is used to determine the code completion scenario corresponding to the matching rule, increase the number of historical matches corresponding to the code completion scenario, and update each stored code completion scenario and the corresponding number of historical matches.

[0112] In some specific implementations, the data set acquisition module 14 includes:

[0113] The compression unit is used to set a preset post-context length, and based on the preset post-context length, delete the post-context code farthest from the target completion node code in the first data set to obtain a compressed second data set.

[0114] In some specific implementations, the code completion large model data set construction device further includes:

[0115] The data set acquisition unit is used to acquire the data of the target file called in the preceding code and the succeeding code corresponding to each target completion node code in the second data set, and save the data into the second data set to obtain an updated second data set.

[0116] Furthermore, the present application also discloses an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0117] Figure 4 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the code completion large model data set construction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0118] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0119] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0120] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the code completion large model data set construction method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0121] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed method for constructing a large model data set for code completion is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0122] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0123] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0124] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0125] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0126] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for constructing a large model dataset for code completion, characterized in that: include: Parsing the pre-acquired code data and saving the parsed code data into a preset code data set; Build a collection of preset code completion scenarios; The preset code completion scenario set includes different preset code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; For any target code in the preset code data set, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence; if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, a target completion node code is extracted from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario; A first data set is obtained based on the target completion node code and the preceding code and succeeding code corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train the code completion model according to the second data set.

2. The method for constructing a large model data set for code completion according to claim 1, characterized in that: The parsing of the pre-acquired code data and saving the parsed code data into a preset code data set includes: Acquire code source data of various types of programming languages ​​based on preset code acquisition conditions, and perform syntax analysis on the code source data using an abstract syntax tree technology to convert the code source data into an abstract syntax tree; Each node in the abstract syntax tree is traversed, a target node and a node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set.

3. The method for constructing a large model data set for code completion according to claim 2, characterized in that: Also includes: Based on the preset filtering criteria, target nodes and node contents corresponding to the preset filtering criteria are deleted from the preset code data set to filter the preset code data set.

4. The method for constructing a large model dataset for code completion according to claim 1, characterized in that: For any target code in the preset code data set, sequentially matching the target code with the code completion scenarios with the smallest number of historical matches in the preset code completion scenario set, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario, including: For any target code in the preset code data set, sequentially matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; If there is a matching rule corresponding to the target code, taking the code completion scenario corresponding to the matching rule as the target code completion scenario, and extracting the target completion node code from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; If there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the process jumps to the step of matching the target code with the matching rules corresponding to the code completion scene with the smallest number of historical matches in the preset code completion scene set in sequence until the preset code data set is traversed.

5. The method for constructing a large model data set for code completion according to claim 4, characterized in that: Also includes: Obtaining the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and storing each code completion scenario and the corresponding number of historical matches; Correspondingly, the taking the code completion scenario corresponding to the matching rule as the target code completion scenario further includes: The code completion scenario corresponding to the matching rule is determined, and the number of historical matches corresponding to the code completion scenario is increased, and each stored code completion scenario and the corresponding number of historical matches are updated.

6. The method for constructing a large model data set for code completion according to claim 1, characterized in that: The compressing the first data set to obtain a second data set includes: A preset post-context length is set, and based on the preset post-context length, the post-context code farthest from the target completion node code in the first data set is deleted to obtain a compressed second data set.

7. The method for constructing a large model dataset for code completion according to any one of claims 1 to 6, characterized in that: Also includes: For the preceding code and the succeeding code corresponding to each target completion node code in the second data set, data of the target file called in the preceding code and the succeeding code are obtained, and the data is saved in the second data set to obtain an updated second data set.

8. A device for constructing a large model data set for code completion, characterized in that: include: A data parsing module is used to parse the pre-acquired code data and save the parsed code data into a preset code data set; A set construction module, used to construct a preset code completion scenario set; the preset code completion scenario set includes different preset code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; a code acquisition module, for sequentially matching any target code in the preset code data set with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario; The data set acquisition module is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for constructing a large model data set with code completion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the method for constructing a large model data set with code completion as described in any one of claims 1 to 7.

Images