A method, device, equipment and medium for constructing a large model data set for code completion

Abstract

The present application discloses a method, apparatus, device and medium for constructing a code completion large model dataset, which relates to the field of computers, including: parsing pre-acquired code data and saving it into a preset code dataset; constructing a preset code completion scenario set; the preset code completion scenario set includes pre-set code completion scenarios; any code completion scenario includes its corresponding matching rules and completion node extraction strategy; for any target code in the preset code dataset, matching the target code with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence; if a target code completion scenario corresponding to the target code is matched, extracting the target completion node code from the target code based on the corresponding completion node extraction strategy; obtaining a first dataset based on the target completion node code and compressing the first dataset to obtain a second dataset, and training the code completion large model based on the second dataset.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device, equipment and medium for constructing a large model data set for code completion. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, code completion has become an important tool for programmers to improve efficiency and reduce errors when writing code. Code completion models can help reduce coding time and improve accuracy by predicting the code snippets that users are about to enter.

[0003] Existing code completion model training typically uses code completion datasets in the FIM (Fill-in-the-Middle) format. Existing techniques often construct FIM-formatted code completion datasets based on the Abstract Syntax Tree (AST) approach. However, this approach lacks consideration of the code completion requirements of actual development tasks, resulting in data that is inconsistent with real-world completion scenarios. Furthermore, the resulting code completion datasets are heavily populated with common grammatical structures, while containing relatively few rare or complex data. This results in insufficient generalization across diverse completion scenarios. Furthermore, the resulting code completion datasets are sometimes too long, reducing the model's inference efficiency and accuracy. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, device, and medium for constructing a large code completion model dataset. This method can construct a set of completion scenarios based on the characteristics of different programming languages ​​and actual code completion requirements, so as to better simulate real completion requirements and obtain a high-quality code completion dataset. The specific solution is as follows:

[0005] In a first aspect, the present application provides a method for constructing a large model dataset for code completion, comprising:

[0006] Parsing the pre-acquired code data and saving the parsed code data into a preset code data set;

[0007] Constructing a preset code completion scenario set; the preset code completion scenario set includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0008] For any target code in the preset code data set, the target code is sequentially matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set. If a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, a target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario;

[0009] A first data set is obtained based on the target completion node code and the preceding and following codes corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train the code completion model according to the second data set.

[0010] Optionally, parsing the pre-acquired code data and saving the parsed code data into a preset code data set includes:

[0011] Acquire code source data of various types of programming languages ​​based on preset code acquisition conditions, and perform syntax parsing on the code source data using abstract syntax tree technology to convert the code source data into an abstract syntax tree;

[0012] Each node in the abstract syntax tree is traversed, a target node and node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set.

[0013] Optionally, the code completion large model dataset construction method further includes:

[0014] Based on a preset filtering criterion, target nodes and node contents corresponding to the preset filtering criterion are deleted from the preset code data set to filter the preset code data set.

[0015] Optionally, for any target code in the preset code data set, sequentially matching the target code with the code completion scenarios with the smallest number of historical matches in the preset code completion scenario set; if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario includes:

[0016] For any target code in the preset code data set, sequentially matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set;

[0017] If there is a matching rule corresponding to the target code, taking the code completion scenario corresponding to the matching rule as the target code completion scenario, and extracting the target completion node code from the target code based on the completion node extraction strategy corresponding to the target code completion scenario;

[0018] If there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the process jumps to the step of matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set, until the preset code data set is traversed.

[0019] Optionally, the code completion large model dataset construction method further includes:

[0020] Obtaining the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and storing each code completion scenario and the corresponding number of historical matches;

[0021] Correspondingly, the method of using the code completion scenario corresponding to the matching rule as the target code completion scenario further includes:

[0022] The code completion scenario corresponding to the matching rule is determined, the number of historical matches corresponding to the code completion scenario is increased, and each stored code completion scenario and the corresponding number of historical matches are updated.

[0023] Optionally, compressing the first data set to obtain a second data set includes:

[0024] A preset post-context length is set, and based on the preset post-context length, the post-context code farthest from the target completion node code in the first data set is deleted to obtain a compressed second data set.

[0025] Optionally, the code completion large model dataset construction method further includes:

[0026] For the preceding and succeeding codes corresponding to each target completion node code in the second data set, data of the target file called in the preceding and succeeding codes are obtained, and the data are saved in the second data set to obtain an updated second data set.

[0027] In a second aspect, the present application provides a device for constructing a large model dataset for code completion, comprising:

[0028] A data parsing module is used to parse the pre-acquired code data and save the parsed code data into a preset code data set;

[0029] A set construction module, configured to construct a set of preset code completion scenarios; the set of preset code completion scenarios includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0030] a code acquisition module configured to, for any target code in the preset code data set, sequentially match the target code with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extract a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the historical match number being the number of target codes successfully matched by any code completion scenario;

[0031] The data set acquisition module is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

[0032] In a third aspect, the present application provides an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned code completion large model data set construction method.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned code completion large model dataset construction method.

[0036] In the present application, pre-acquired code data is parsed and the parsed code data is saved in a preset code data set; a preset code completion scenario set is constructed; the preset code completion scenario set includes different pre-set code completion scenarios; wherein, any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; for any target code in the preset code data set, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence, and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the historical matching number is the number of target codes that have been successfully matched in any code completion scenario; a first data set is obtained based on the target completion node code and the preceding and following codes corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train a large code completion model based on the second data set. From the above, it can be seen that this application pre-sets different code completion scenarios according to the characteristics of different programming languages ​​and the actual code completion requirements, and sets corresponding matching rules and completion node extraction strategies for each code completion scenario, thereby solving the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn to ensure balance and improve the generalization ability of the model; at the same time, the second data set is obtained by compressing the first data set to ensure that the obtained second data set meets the input context length limit of the model, thereby effectively improving the reasoning efficiency and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0038] Figure 1 This is a flow chart of a method for constructing a large model dataset for code completion disclosed in this application;

[0039] Figure 2 A schematic diagram of a specific code completion large model dataset construction process disclosed in this application;

[0040] Figure 3 A schematic diagram of the structure of a device for constructing a large model data set for code completion disclosed in this application;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] In the existing code completion model training, a code completion dataset in FIM format is usually used for model training. Most of the existing technologies use the abstract syntax tree method to construct a code completion dataset in FIM format. However, this method lacks consideration of the code completion requirements in actual development tasks, resulting in the generated data not being consistent with the actual completion scenario. On the other hand, the common grammatical structures in the code completion dataset obtained by the above method occupy a large amount of data, and there is less rare or complex data, resulting in insufficient generalization ability of the model in diversified completion scenarios. At the same time, sometimes the acquired code completion dataset is too long, resulting in reduced reasoning efficiency and accuracy of the model. To this end, the present application provides a method for constructing a large model dataset for code completion, which can construct a set of completion scenarios according to the characteristics of different programming languages ​​and actual code completion requirements, so as to better simulate the actual completion requirements and obtain a high-quality code completion dataset.

[0044] See also Figure 1 As shown, the embodiment of the present application discloses a method for constructing a large model dataset for code completion, including:

[0045] Step S11: parse the pre-acquired code data and save the parsed code data into a preset code data set.

[0046] In this embodiment, it is first necessary to collect the codes of various programming languages ​​as the source data of the code completion large model dataset, and then parse the source data and save the parsed source data into a preset code dataset. Specifically, it includes: obtaining the code source data of various types of programming languages ​​based on preset code acquisition conditions, and using abstract syntax tree technology to perform syntax analysis on the code source data to convert the code source data into an abstract syntax tree; traversing each node in the abstract syntax tree, extracting the target node and the node content corresponding to the target node from the node, and saving the target node and the node content into the preset code dataset.

[0047] For example, we collect high-star open source code repositories and high-quality business code repositories from open source platforms (such as GitHub and Gitee) to obtain project code. To ensure code quality, we can set the following restrictions (i.e., preset code acquisition conditions): the code must be from the past three years and have at least 100 favorites. Based on these restrictions and for each programming language, we sort the projects by the number of favorites and select a preset number of projects from highest to lowest. Based on the selected project code, we use the AST (Abstract Syntax Tree) parser (such as Tree-sitter) corresponding to the programming language in the project code to parse the project code files and convert them into an abstract syntax tree. The abstract syntax tree clearly displays the syntactic elements and structural relationships of the code in a tree-like structure. It abstracts various code components (such as variables, operators, control structures, etc.) into nodes, and accurately reflects the logical structure of the code through the hierarchical relationships between nodes. This parsing method facilitates in-depth code analysis, enabling more sophisticated operations and information extraction based on the syntax tree structure.

[0048] After obtaining the abstract syntax tree, all nodes in the syntax tree can be traversed. During the traversal process, target nodes of the required type are selected according to pre-defined rules, and the target nodes and their content are saved to a pre-set code dataset. This pre-set code dataset can be implemented as a class or other suitable data structure. For example, in constructing a method node class in Java (an object-oriented programming language), multiple attributes are defined to store different information related to the method. For example, the documentation node stores method annotations, which help understand the method's function and purpose; the modifier node records the method's modifiers (such as public and private), which determine the method's access permissions and other characteristics; the method name node stores the method's name, which is a key identifier for identifying and calling the method; and the method body node contains the method's specific implementation logic. In this way, key information scattered throughout the code is consolidated and stored according to a specific structure, making it easier to quickly and accurately retrieve and utilize this information when building datasets and training models, laying the foundation for efficient code completion.

[0049] Furthermore, based on the preset filtering criteria, the target nodes and node contents corresponding to the preset filtering criteria may be deleted from the preset code data set to filter the preset code data set.

[0050] It is understandable that after obtaining the preset code data set, the meaningless information in the data set can be filtered. This type of information mainly includes comments that are not related to the actual code, such as license information, author information, timestamps, and "TODO", etc.; among them, TODO is used to mark to-do items; the specific filtering method can locate the comment node in the current file based on the results of code parsing, and judge whether it is a meaningless comment based on the content of the node. If it is confirmed that the comment is meaningless, the corresponding node is deleted from the code structure. It should be noted that the judgment rules can be defined according to the characteristics of the node. For example, the comment node of the license information is usually the first comment node in the current code syntax tree, and its content generally contains keywords such as "License". Through this filtering process, redundant information can be effectively removed, thereby providing a more accurate and clear code context for subsequent model reasoning.

[0051] Step S12: Construct a preset code completion scenario set; the preset code completion scenario set includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario.

[0052] In this embodiment, a set of preset code completion scenarios is constructed based on the characteristics of different programming languages ​​and actual code completion requirements, so as to set specific code matching rules and completion node extraction strategies for each scenario.

[0053] Specifically, define a set of preset code completion scenarios :

[0054] ;

[0055] Among them, each code completion scenario Include matching rules and completion node extraction strategy .

[0056] The above code completion scenarios are abstractions and simulations of various code completion situations that may be encountered in actual programming. For example, in actual programming, you may encounter situations where you need to complete method documentation comments. In order to better simulate real completion needs, it is necessary to ensure that the nodes extracted from the code (i.e., completion nodes) can match the actual application scenarios, that is, the extracted information is what really needs to be completed in actual programming, rather than some irrelevant information or information that does not meet actual needs, so as to ensure the practicality and accuracy of code completion. For example, during the coding process, if it is necessary to complete the documentation comments of the method based on the method content, the matching rule for the corresponding scenario is to determine whether the current method node contains document comments, and the completion node extraction strategy is to extract the content of the method document comment node as the completion content.

[0057] Step S13: For any target code in the preset code data set, the target code is matched in sequence with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, a target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the historical matching number is the number of target codes that have been successfully matched by any code completion scenario.

[0058] In this embodiment, scene matching is performed based on pre-set code completion scenarios, as well as corresponding matching rules and completion node extraction strategies, including: for any target code in a preset code data set, matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence; if there is a matching rule corresponding to the target code, the code completion scenario corresponding to the matching rule is used as the target code completion scenario, and the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; if there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the process jumps to the step of matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in sequence, until the preset code data set is traversed.

[0059] Typically, code completion triggers occur within methods. Therefore, in this embodiment, method nodes are extracted from a preset code dataset, scenario rule matching is performed on each method in the code file at the method-level granularity, and the corresponding completion node is extracted based on the matching results. The specific formula is as follows:

[0060] For method code and scenes , define the matching function :

[0061] ;

[0062] like , then according to the scene Corresponding completion node extraction strategy Extract the corresponding completion node:

[0063] ;

[0064] in, From the method code The completion node extracted from the function Used to extract completion nodes.

[0065] It should be noted that in order to ensure the balance of the obtained completion nodes, in this embodiment, for each target code in the preset code data set, it is not randomly matched with the scenes in the code completion scene set, but the matching rules corresponding to the code completion scene with the smallest number of historical matches are preferentially selected for matching, so that each scene can be used relatively evenly in the entire data set, avoiding over-matching of certain common scenes and ignoring some uncommon but important scenes, so that the model can learn various code completion situations more comprehensively.

[0066] Specifically, the number of historical matches corresponding to each code completion scenario in the preset code completion scenario set is obtained, and each code completion scenario and the corresponding number of historical matches are stored, and the scene with the smallest number of historical matches is matched first as the target code completion scenario, and the target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; then the number of historical matches corresponding to the above target code completion scenario is increased, and the stored code completion scenarios and the corresponding number of historical matches are updated.

[0067] For example, a small root heap can be maintained to store the number of samples for each scenario (i.e., the number of historical matches), and scenarios with fewer samples can be matched first. The specific process is as follows:

[0068] Defining a small root heap , used to store each scene and its corresponding number of samples:

[0069] ;

[0070] Small root pile Ensure that the number of samples at the root node is minimal:

[0071] ;

[0072] Scene matching prioritizes the root node as the matching scene:

[0073] ;

[0074] in, Indicates the selected code completion scenario.

[0075] If the match fails, the other scenarios in the stack are matched in turn until a matching scenario is found or all scenarios are traversed.

[0076] If a matching scene is found, the number of samples of the matching scene is updated:

[0077] ;

[0078] in, The number of samples corresponding to the successfully matched code completion scenario.

[0079] Reinsert the updated number of samples into the heap :

[0080] ;

[0081] Among them, update_heap is the update function.

[0082] Through the above steps, the balance of sample size can be ensured, thereby optimizing the training effect of the model.

[0083] Step S14: obtaining a first data set based on the target completion node code and the preceding and succeeding codes corresponding to the target completion node code in the preset code data set, and compressing the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

[0084] In this embodiment, after obtaining the target completion node code, the target completion node code and the corresponding preceding code and following code are saved in the first data set, wherein the current code in the first data set can be divided into a preceding text (prefix), a completion content (infill) and a following text (suffix). From the beginning of the code to the completion node is the preceding text, the completion node itself is the completion content, and from the completion node to the end of the code is the following text.

[0085] In order to ensure that the data set does not exceed the input context length of the large model, content compression can be performed on ultra-long data. Since the preceding text information of the completion content is more important than the following text, the following text can be compressed first, including: setting a preset following text length, and based on the preset following text length, deleting the following text code in the first data set that is farthest away from the target completion node code to obtain a compressed second data set.

[0086] For example, first determine the default length of the post To ensure that the length of the following text is less than In general, the further away the method node is from the completion node in the text, the less important it is. Therefore, the method body content can be removed from the back to the front, retaining the basic method signature and other information until the length of the text is less than .

[0087] It should be noted that in code completion, it is sometimes necessary to complete the code of other files called. At this time, the relevant file information is crucial to the model. However, the AST method cannot capture relevant file information across files or modules. Therefore, LSP (Language Server Protocol) technology can be used to find the relevant files referenced by the current file. Specifically, for the preceding and following codes corresponding to each target completion node code in the second data set, the data of the target files called in the preceding and following codes are obtained, and the data is saved in the second data set to obtain an updated second data set. At the same time, in order to avoid the content of the second data set being too long, the relevant file content in the second data set can also be compressed.

[0088] For example, obtain the compressed second data set, and determine the post-context and pre-context corresponding to the target completion node code in the second data set, use LSP technology to obtain the paths and codes of other files in the project called in the pre-context and post-context, and add them to the target data set in the order of the pre-context being greater than the post-context and the closer to the completion node, the higher the priority; set the model input context length limit to , delete the method body nodes and redundant information in the code from the above target data set, add the path and the code in the compressed target data set to the second data set, until the data length in the second data set reaches Through this extraction process, relevant file information across files can be effectively extracted, thereby improving the model's code completion capabilities in multi-file projects.

[0089] As can be seen from the above, after this embodiment performs syntax parsing on the code files in the open source warehouse, it filters the comment nodes in the syntax tree and deletes meaningless nodes to ensure that invalid information does not interfere with the model's understanding of the valid context, so as to improve the quality and relevance of the data; according to the characteristics of different programming languages ​​​​and the actual code completion requirements, different code completion scenarios are pre-set, and corresponding matching rules and completion node extraction strategies are set for each code completion scenario, so as to solve the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn, ensuring balance to improve the generalization ability of the model; a code compression method is also proposed in this embodiment to ensure that the data meets the input context length limit of the large model. This method dynamically evaluates the importance of method nodes in the subsequent text, removes redundant content from back to front, retains only basic method signatures and other information, and loops through the compression steps until the subsequent text length meets the set target, thereby effectively improving the reasoning efficiency and accuracy of the model; at the same time, the LSP technology is used to make up for the shortcomings of the AST method in capturing cross-file information. According to the principle that previous text dependencies take precedence over subsequent text dependencies, the relevant file information called by the current file is added to the data set, thereby enhancing the model's completion ability in multi-file projects.

[0090] See also Figure 2 As shown, the technical solution in this application is explained below by taking the specific code completion large model data set construction process as an example.

[0091] First, high-star open source code repositories and high-quality business code repositories are collected from the open source code platform, and the codes of various projects are obtained from them to obtain code source data; the code source data is parsed using abstract syntax tree technology to convert the code source data into an abstract syntax tree; each node in the abstract syntax tree is traversed, the target node and the node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set; then, redundant information of the code in the above preset code data set is filtered based on the preset filtering criteria to obtain a filtered preset code data set; then, a preset code completion scenario set is constructed to simulate a real completion scenario, and scene matching is performed based on the preset code completion scenario set and the preset code data set to obtain a first data set consisting of the target completion node code and the corresponding previous code and subsequent code; further, the first data set is code compressed to obtain a second data set, and the LSP technology is used to extract cross-file information from the second data set to obtain the final required target data set. The target dataset is used to train the large code completion model. In the code compression operation and cross-file information extraction operation, the dataset can be compressed by setting a preset data length to perform data fine-tuning. By fine-tuning the data, the input data does not exceed the model input context length limit.

[0092] As can be seen from the above, this embodiment takes the specific code completion large model dataset construction process as an example to elaborate on the process of constructing a high-quality code completion dataset, and provides a systematic and scientific data construction method to provide high-quality data support for the training of the code completion large model, thereby improving the programming efficiency and accuracy of developers.

[0093] See also Figure 3 As shown, the embodiment of the present application also discloses a device for constructing a large model dataset for code completion, including:

[0094] The data parsing module 11 is used to parse the pre-acquired code data and save the parsed code data into a preset code data set;

[0095] A set construction module 12 is configured to construct a set of preset code completion scenarios; the set of preset code completion scenarios includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario;

[0096] a code acquisition module 13 configured to, for any target code in the preset code data set, sequentially match the target code with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; and, if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extract a target completion node code from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the historical matching number being the number of target codes successfully matched by any code completion scenario;

[0097] The data set acquisition module 14 is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

[0098] From the above, it can be seen that this application pre-sets different code completion scenarios according to the characteristics of different programming languages ​​and the actual code completion requirements, and sets corresponding matching rules and completion node extraction strategies for each code completion scenario, thereby solving the problem that the generated data does not match the actual completion scenario; in the scene matching process, the target code is matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set in turn to ensure balance and improve the generalization ability of the model; at the same time, the second data set is obtained by compressing the first data set to ensure that the obtained second data set meets the input context length limit of the model, thereby effectively improving the reasoning efficiency and accuracy of the model.

[0099] In some specific implementations, the data parsing module 11 includes:

[0100] a data parsing unit, configured to acquire code source data of various types of programming languages ​​based on preset code acquisition conditions, and perform syntax parsing on the code source data using an abstract syntax tree technique to convert the code source data into an abstract syntax tree;

[0101] The data set determination submodule is used to traverse each node in the abstract syntax tree, extract a target node and node content corresponding to the target node from the node, and save the target node and the node content into a preset code data set.

[0102] In some specific embodiments, the data set determination submodule further includes:

[0103] The filtering unit is configured to delete target nodes and node contents corresponding to a preset filtering criterion from the preset code data set based on the preset filtering criterion, so as to filter the preset code data set.

[0104] In some specific implementations, the code acquisition module 13 includes:

[0105] a matching unit, configured to sequentially match, for any target code in the preset code data set, the target code with matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set;

[0106] a first processing submodule configured to, if a matching rule corresponding to the target code exists, use the code completion scenario corresponding to the matching rule as the target code completion scenario, and extract a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario;

[0107] A second processing unit is configured to reselect a new target code from the preset code data set if no matching rule corresponding to the target code exists, and jump to the step of sequentially matching the target code with the matching rule corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set based on the new target code, until the preset code data set is traversed.

[0108] In some specific implementations, the apparatus for constructing a large model dataset for code completion further includes:

[0109] a storage unit, configured to obtain a number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and store each code completion scenario and the corresponding number of historical matches;

[0110] Accordingly, the first processing submodule further includes:

[0111] The data updating unit is configured to determine a code completion scenario corresponding to the matching rule, increase the number of historical matches corresponding to the code completion scenario, and update each stored code completion scenario and the corresponding number of historical matches.

[0112] In some specific implementations, the data set acquisition module 14 includes:

[0113] The compression unit is configured to set a preset post-context length and delete the post-context code farthest from the target completion node code in the first data set based on the preset post-context length to obtain a compressed second data set.

[0114] In some specific implementations, the apparatus for constructing a large model dataset for code completion further includes:

[0115] The data set acquisition unit is used to acquire the data of the target file called in the preceding code and the following code corresponding to each target completion node code in the second data set, and save the data into the second data set to obtain an updated second data set.

[0116] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0117] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the code completion large model dataset construction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0118] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0119] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0120] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of completing the code completion large model dataset construction method executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of completing other specific tasks.

[0121] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for constructing a large model dataset for code completion. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0123] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0125] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0126] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for constructing a large model dataset for code completion, characterized in that: include: Parsing the pre-acquired code data and saving the parsed code data into a preset code data set; Build a collection of preset code completion scenarios; The preset code completion scenario set includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; For any target code in the preset code data set, the target code is sequentially matched with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set. If a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, a target completion node code is extracted from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; the number of historical matches is the number of target codes that have been successfully matched by any code completion scenario; A first data set is obtained based on the target completion node code and the preceding and following codes corresponding to the target completion node code in the preset code data set, and the first data set is compressed to obtain a second data set, so as to train the code completion model according to the second data set.

2. The method for constructing a large code completion model dataset according to claim 1, wherein: The parsing of the pre-acquired code data and saving the parsed code data into a preset code data set includes: Acquire code source data of various types of programming languages ​​based on preset code acquisition conditions, and perform syntax parsing on the code source data using abstract syntax tree technology to convert the code source data into an abstract syntax tree; Each node in the abstract syntax tree is traversed, a target node and node content corresponding to the target node are extracted from the node, and the target node and the node content are saved in a preset code data set.

3. The method for constructing a large code completion model dataset according to claim 2, wherein: Also includes: Based on a preset filtering criterion, target nodes and node contents corresponding to the preset filtering criterion are deleted from the preset code data set to filter the preset code data set.

4. The method for constructing a large code completion model dataset according to claim 1, wherein: For any target code in the preset code data set, sequentially matching the target code with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extracting a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario, including: For any target code in the preset code data set, sequentially matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; If there is a matching rule corresponding to the target code, taking the code completion scenario corresponding to the matching rule as the target code completion scenario, and extracting the target completion node code from the target code based on the completion node extraction strategy corresponding to the target code completion scenario; If there is no matching rule corresponding to the target code, a new target code is reselected from the preset code data set, and based on the new target code, the process jumps to the step of matching the target code with the matching rules corresponding to the code completion scenario with the smallest number of historical matches in the preset code completion scenario set, until the preset code data set is traversed.

5. The method for constructing a large code completion model dataset according to claim 4, wherein: Also includes: Obtaining the number of historical matches corresponding to each code completion scenario in a preset code completion scenario set, and storing each code completion scenario and the corresponding number of historical matches; Correspondingly, the method of using the code completion scenario corresponding to the matching rule as the target code completion scenario further includes: The code completion scenario corresponding to the matching rule is determined, the number of historical matches corresponding to the code completion scenario is increased, and each stored code completion scenario and the corresponding number of historical matches are updated.

6. The method for constructing a large code completion model dataset according to claim 1, wherein: The compressing the first data set to obtain a second data set includes: A preset post-context length is set, and based on the preset post-context length, the post-context code farthest from the target completion node code in the first data set is deleted to obtain a compressed second data set.

7. The method for constructing a large code completion model dataset according to any one of claims 1 to 6, characterized in that: Also includes: For the preceding and succeeding codes corresponding to each target completion node code in the second data set, data of the target file called in the preceding and succeeding codes are obtained, and the data are saved in the second data set to obtain an updated second data set.

8. A device for constructing a large model data set for code completion, characterized in that: include: A data parsing module is used to parse the pre-acquired code data and save the parsed code data into a preset code data set; A set construction module, configured to construct a set of preset code completion scenarios; the set of preset code completion scenarios includes different pre-set code completion scenarios; wherein any code completion scenario includes a matching rule and a completion node extraction strategy corresponding to the code completion scenario; a code acquisition module configured to, for any target code in the preset code data set, sequentially match the target code with the code completion scenario with the smallest number of historical matches in the preset code completion scenario set; and if a target code completion scenario corresponding to the target code is matched from the preset code completion scenario set, extract a target completion node code from the target code based on a completion node extraction strategy corresponding to the target code completion scenario; the historical match number being the number of target codes successfully matched by any code completion scenario; The data set acquisition module is used to obtain a first data set based on the target completion node code and the preceding code and following code corresponding to the target completion node code in the preset code data set, and compress the first data set to obtain a second data set, so as to train the code completion model according to the second data set.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for constructing a large model dataset for code completion according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the code completion large model dataset construction method as described in any one of claims 1 to 7.

Images