Training data generation method and device, equipment and medium

Abstract

The invention provides a training data generation method and device, equipment and a medium, and can be applied to the technical field of artificial intelligence. The method comprises the steps of generating a to-be-tested code corresponding to a code generation request based on a pre-trained preset model, wherein the code generation request comprises programming information used for indicating the to-be-tested code; extracting target function information from the to-be-tested code based on a function extraction rule for programming information; inputting a case generation request generated by using the target function information into a preset model to obtain a plurality of test cases for the to-be-tested code; under the condition that the test passing rate of the to-be-tested code is determined to be greater than a preset threshold value by utilizing the respective test results of the plurality of test cases, taking a first question and answer pair formed by the code generation request and the to-be-tested code and a second question and answer pair formed by the case generation request and the test cases representing that the test passes as training data, the training data is used for performing fine-tuning training on the preset model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a training data generation method, apparatus, device, medium and program product. Background Art

[0002] With the rapid development of artificial intelligence and natural language processing technology, large language models have outstanding performance in many fields such as chat dialogue, logical reasoning, and code writing. In order to achieve these powerful functions, large language models usually need to rely on large-scale training data for training to improve performance and generalization ability.

[0003] However, due to the uneven quality of text data on the Internet and the limited amount of valid data, large language models obtained by using training data with small amounts or low data quality may have problems with poor model performance and low reliability. Summary of the invention

[0004] In view of the above problems, the present application provides a training data generation method, apparatus, device, medium and program product.

[0005] According to one aspect of the present application, a training data generation method is provided, including: generating a code to be tested corresponding to a code generation request based on a pre-trained preset model, the code generation request including programming information used to indicate the code to be tested; extracting target function information from the code to be tested based on a function extraction rule for the programming information; inputting a use case generation request generated using the target function information into the preset model to obtain a plurality of test cases for the code to be tested; when it is determined that the test pass rate of the code to be tested is greater than a preset threshold using the test results of each of the plurality of test cases, a first question-answer pair consisting of the code generation request and the code to be tested, and a second question-answer pair consisting of the use case generation request and the test case representing the test pass are both used as training data, and the training data are used to fine-tune the preset model.

[0006] According to an embodiment of the present application, target function information is extracted from the code to be tested based on function extraction rules for programming information, including: determining a function extraction instruction based on definition keywords corresponding to the programming information, wherein the definition keywords are used to define functions included in the code to be tested; extracting function information of at least one function from the code to be tested based on the function extraction instruction to obtain at least one candidate function information; determining target function information from at least one candidate function information based on the function position corresponding to the programming information in the code to be tested, wherein the target function information is function information of the main function.

[0007] According to an embodiment of the present application, programming information includes a programming language; at least one candidate function information is stored in the order of appearance of at least one function in the code to be tested; wherein, based on the function position corresponding to the programming information in the code to be tested, the target function information is determined from the at least one candidate function information, and the target function information is the function information of the main function, including: based on the function position, taking the latest candidate function information in at least one function as the selected function information; when the programming language is a first programming language, matching the function name included in the selected function information with the preset function name to obtain a first matching result; when the first matching result indicates that the function name is a preset function name, taking the candidate function information whose appearance order is before the selected function information as the target function information.

[0008] According to an embodiment of the present application, the training data generation method also includes: associating the code to be tested and multiple test cases according to the programming information to obtain the target test code; based on the target test code, testing the code to be tested in a test environment corresponding to the programming information to obtain the test results of each of the multiple test cases.

[0009] According to an embodiment of the present application, in the case where the programming language included in the programming information is a second programming language, based on the target test code, the code to be tested is tested in a test environment corresponding to the programming information to obtain test results of multiple test cases, including: matching the target test code with the test keyword to obtain multiple second matching results representing the match with the test keyword, the target test code is obtained by splicing the code to be tested and multiple test cases, each test case includes a predetermined number of test keywords; based on the multiple second matching results, the number of test cases is determined; based on a preset execution function, the target test code is executed in a temporary directory to obtain an execution result; in the case of determining that there is target error data related to the test keyword in the execution result, based on a space management tool, a result storage space for storing multiple test results is created; the code to be tested is tested by multiple test cases respectively to obtain multiple test results; and the multiple test results are stored in the result storage space.

[0010] According to an embodiment of the present application, in a case where the programming language included in the programming information is a first programming language, wherein, based on a target test code, in a test environment corresponding to the programming information, the code to be tested is tested to obtain test results of multiple test cases, including: generating the same identification identifier for the code to be tested and the target test code; storing the code to be tested and the target test code in a temporary storage space created based on the identification identifier; compiling the target test code in the temporary storage space based on preset compilation instructions to obtain compilation results; and in a case where the compilation result indicates that the compilation is successful, executing the target test code based on preset execution instructions to obtain test results of multiple test cases.

[0011] According to an embodiment of the present application, a code generation request is obtained in the following manner: matching a code generation problem with a plurality of preset request templates to obtain a target request template corresponding to the code generation problem; embedding the code generation problem into the target request template to obtain an intermediate generation request, the target request template including programming information and quantity information indicating the number of code generation requests to be generated; inputting the intermediate generation request into a preset model to obtain a code generation request having the same quantity as the quantity information.

[0012] Another aspect of the present application provides a training data generation device, including: a code generation module, used to generate a code to be tested corresponding to a code generation request based on a pre-trained preset model, the code generation request including programming information used to indicate the code to be tested; a function extraction module, used to extract target function information from the code to be tested based on function extraction rules for programming information; a use case determination module, used to input the use case generation request generated using the target function information into the preset model to obtain multiple test cases for the code to be tested; a data determination module, used to use a first question-and-answer pair consisting of a code generation request and the code to be tested, and a second question-and-answer pair consisting of a use case generation request and a test case representing a passed test as training data when it is determined that the test pass rate of the code to be tested is greater than a preset threshold using the test results of each of the multiple test cases, and the training data is used to fine-tune the preset model.

[0013] Another aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] Another aspect of the present application also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0015] Another aspect of the present application also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the computer program or instructions are executed by a processor.

[0016] According to the training data generation method of the present application, the code to be tested corresponding to the code generation request is generated by a pre-trained preset model, and multiple test cases corresponding to the code to be tested generated by the preset model are used to test the code to be tested, so that when the code to be tested passes the test, the question-answer pairs used in the test process are all used as fine-tuning training data of the preset model. Since the generation of the code to be tested is realized by the preset model, and the target function information extracted by the function extraction rule for programming information is determined to determine the case generation request for instructing the preset model to generate multiple test cases for the code to be tested, the fully automated generation from the code to be tested to the test case is realized, and the correctness of the code to be tested is verified by the test results of the code to be tested using multiple test cases, and then the validity of the question-answer pairs used in the whole process is verified. And the test cases can be determined for the code to be tested using different programming information, that is, the richness of the training data is guaranteed. Therefore, at least part of the problems existing in the related art of using training data with a small amount of data or low data quality, resulting in poor model performance and low reliability, are solved, and the technical effect of being able to efficiently generate diversified and effective training data and significantly improve the scale of training data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other purposes, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the training data generation method, apparatus, device, medium and program product according to the embodiments of the present application is schematically shown;

[0019] Figure 2 A flowchart of a method for generating training data according to an embodiment of the present application is schematically shown;

[0020] Figure 3 A flowchart of testing a code to be tested when the programming language is a first programming language according to an embodiment of the present application is schematically shown;

[0021] Figure 4 A flowchart of a method for generating training data according to another embodiment of the present application is schematically shown;

[0022] Figure 5 A structural block diagram of a training data generating device according to an embodiment of the present application is schematically shown; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a training data generating method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0024] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the technical solution of the present application, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided in the embodiments of the present application provide users with corresponding operation portals for users to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through a computer program, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0030] During the research process, it was found that code datasets play an important role in improving programming capabilities and code generation in large model training. The specific effects are: Strengthening basic programming skills: Through a large number of programming code samples, the model can better understand various programming languages. Improving problem solving and logical reasoning capabilities: It helps the model learn how to solve problems through step-by-step execution. Writing and understanding code requires a high degree of logical reasoning ability. By analyzing a large number of code samples, the model can learn the application of different programming patterns and infer the optimal solution or the simplest implementation method.

[0031] Improve code generation and optimization capabilities: By processing a large amount of actual code data, the model can improve its ability to generate efficient and easy-to-read code, optimize unnecessary repetitive parts, and reduce computational complexity. The large number of code data sets helps the model learn to identify and fix common programming errors, such as syntax errors, logic errors, runtime errors, etc., improving code quality. Code data sets help the model understand complex code logic and structure, such as recursion, asynchronous programming, concurrency and other technologies, and improve its ability to cope with complex programming tasks.

[0032] Therefore, it can be seen that the diversity and scale of the dataset are crucial for the training of large models. By expanding the dataset, the model can be exposed to more codes of different types, formats, styles, and complexity, thereby improving its adaptability and generalization ability in various development tasks.

[0033] An embodiment of the present application provides a training data generation method, including: generating a code to be tested corresponding to a code generation request based on a pre-trained preset model, the code generation request including programming information used to indicate the code to be tested; extracting target function information from the code to be tested based on a function extraction rule for the programming information; inputting a use case generation request generated using the target function information into the preset model to obtain multiple test cases for the code to be tested; when it is determined that the test pass rate of the code to be tested is greater than a preset threshold using the test results of each of the multiple test cases, a first question-and-answer pair consisting of the code generation request and the code to be tested, and a second question-and-answer pair consisting of the use case generation request and the test case representing the test pass are both used as training data, and the training data is used to fine-tune the preset model.

[0034] Figure 1 The application scenario diagram of the training data generation method, apparatus, device, medium and program product according to the embodiments of the present application is schematically shown.

[0035] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0036] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the training data generation method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training data generation device provided in the embodiment of the present application can generally be set in the server 105. The training data generation method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the training data generation device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The training data generation method of the embodiment of the present application is described in detail.

[0042] Figure 2 The flowchart of the training data generation method according to an embodiment of the present application is schematically shown.

[0043] like Figure 2 As shown, the method includes operations S210 to S240.

[0044] In operation S210 , a code to be tested corresponding to a code generation request is generated based on a pre-trained preset model, and the code generation request includes programming information for indicating the programming information adopted by the code to be tested.

[0045] In operation S220 , target function information is extracted from the code to be tested based on a function extraction rule for programming information.

[0046] In operation S230, a use case generation request generated by using the target function information is input into a preset model to obtain a plurality of test cases for the code to be tested.

[0047] In operation S240, when it is determined that the test pass rate of the code to be tested is greater than a preset threshold value using the test results of multiple test cases, a first question-and-answer pair consisting of a code generation request and the code to be tested and a second question-and-answer pair consisting of a use case generation request and a test case representing the test pass are used as training data, and the training data is used to fine-tune the preset model.

[0048] According to an embodiment of the present application, the preset model may be a large language model (LLM).

[0049] According to an embodiment of the present application, the code generation request may be a request for requesting a large language model to generate relevant code, and the code generation request may include: target code generation problems, programming information, and quantity information indicating the number of codes to be tested, etc.

[0050] According to an embodiment of the present application, the programming information may include a programming language and a programming type. The programming language is not limited and may be Python, go language, Java, etc. The programming type may be a programming type included in each language.

[0051] According to an embodiment of the present application, the code generation request may be generated by a preset model, and the code to be tested may be output again by the preset model based on the code generation request.

[0052] According to an embodiment of the present application, both the code generation request and the use case generation request may be prompt statements for input into a preset model, so that the preset model can understand the input.

[0053] According to an embodiment of the present application, the number of codes to be tested is related to the number of codes to be tested generated for a target code generation problem indicated in the code generation request. Therefore, the output of multiple codes to be tested can be achieved simultaneously, and the above-mentioned training data generation method can be executed in parallel for multiple codes to be tested, thereby quickly obtaining a large amount of training data.

[0054] According to an embodiment of the present application, different programming information may correspond to different function extraction rules.

[0055] According to the embodiments of the present application, the target function information is not limited and may be information about a function that enables the test case to effectively perform unit testing on the code to be tested, for example, function information of the main function.

[0056] According to an embodiment of the present application, the target function information can be embedded into a preset use case request generation template to obtain a use case generation request for the code to be tested. In some embodiments, the use case generation request can be obtained by embedding both the code generation question and the target function information used to generate the code generation request into a preset use case request generation template.

[0057] According to an embodiment of the present application, for example, a use case request generation template may be: for a "code generation request", please generate "I" test cases using "test keyword statements" to complete the unit test, without generating code implementation, only outputting unit test content about the code, and the target function information is "{}", where the "" part may be an embedded part. Or the parts other than the target function information may be provided in the request generation template.

[0058] According to the embodiment of the present application, the above-mentioned use case request generation template is only illustrative, and different use case request generation templates can be determined according to actual needs.

[0059] According to an embodiment of the present application, a test case is a basic unit for verifying the function, performance or other characteristics of the code to be tested, which is also implemented through code.

[0060] According to the embodiments of the present application, the number of test cases that passed the test can be characterized by the total number of test cases and the test results, and the test pass rate can be calculated.

[0061] According to the embodiments of the present application, the preset threshold is not limited and can be set according to actual conditions, for example, 90%.

[0062] According to an embodiment of the present application, both the first question-answer pair and the second question-answer pair may be stored in a training set to perform fine-tuning training on a preset model or to perform pre-training or fine-tuning training on other large language models.

[0063] According to an embodiment of the present application, when the test pass rate of the code to be tested is less than or equal to a preset threshold, the code generation request can be regenerated using the code generation problem, and the operations included in the above-mentioned training data generation method can be repeatedly executed until the target amount of training data is obtained.

[0064] According to an embodiment of the present application, when it is determined that the test pass rate of the code to be tested is greater than a preset threshold, synonym replacement can be performed on the code generation request or the use case generation request, so as to obtain a new code generation request or a new use case generation request. Specifically, taking the code generation request as an example, the code generation request can be segmented, and the stop words determined after segmentation can be filtered out, so as to reduce redundant information. The remaining words are matched with multiple preset words in terms of word meaning, so as to perform synonym replacement on the keywords included in the code generation request, generate a new code generation request with similar semantics but different expressions, and use the new code generation request and the original corresponding code to be tested as a new question-and-answer pair. Stop words refer to words that frequently appear in the text but have little meaning, such as "de", "le", "zai", "he", etc.

[0065] According to an embodiment of the present application, a preset function library can also be constructed, and functions in the code to be tested or the test case can be replaced with functions with the same function, so as to obtain a new code to be tested or a new test case. Specifically, taking the code to be tested as an example, the function in the code to be tested is matched with the preset function in the function library, and the function in the code to be tested is replaced with a replacement function with the same function in the preset function library.

[0066] According to an embodiment of the present application, based on the above-mentioned synonym replacement for requests or function replacement for code, by permuting and combining the newly generated requests and the newly generated answers, the expansion of the question-and-answer pairs can be realized, that is, the re-expansion of the training data can be realized, and the amount of training data can be increased.

[0067] According to the training data generation method of the present application, a code to be tested corresponding to the code generation request is generated by a pre-trained preset model, and a plurality of test cases corresponding to the code to be tested generated by the preset model are used to test the code to be tested. Therefore, when the code to be tested passes the test, all the question-and-answer pairs used in the test process are used as the fine-tuning training data of the preset model. Since the generation of the code to be tested is realized through the preset model, and the target function information is determined by the function extraction rule for programming information, a use case generation request for instructing the preset model to generate a plurality of test cases for the code to be tested is determined, so as to realize the full-automatic generation from the code to be tested to the test cases to be tested, and use a plurality of test cases to verify the correctness of the code to be tested based on the test results of the code to be tested, and then verify the effectiveness of the question-and-answer pairs used in the whole process. And test cases can be determined for the code to be tested using different programming information, that is, the richness of the training data is ensured. Therefore, at least partially solve the problems in the related art that the model performance is poor and the reliability is low due to the use of training data with a small amount or low quality, and realize the technical effect of being able to efficiently generate diversified and effective training data, and significantly improve the scale of the training data.

[0068] According to an embodiment of the present application, extracting target function information from the code to be tested based on a function extraction rule for programming information may include the following operations.

[0069] Based on definition keywords corresponding to the programming information, a function extraction instruction is determined, where the definition keywords are used to define the functions included in the code to be tested; based on the function extraction instruction, function information of at least one function is extracted from the code to be tested to obtain at least one candidate function information; based on the function position corresponding to the programming information in the code to be tested, target function information is determined from at least one candidate function information, where the target function information is function information of the main function.

[0070] According to an embodiment of the present application, the definition keyword is a word that defines a function in the code to be tested, for example: def in Python or func in Go.

[0071] According to an embodiment of the present application, a function extraction instruction can be obtained by embedding a definition keyword into a preset instruction generation template.

[0072] According to the embodiment of the present application, the preset instruction generation templates corresponding to different definition keywords may be the same or different. For example, when the preset instruction generation templates corresponding to different definition keywords are different, the preset instruction generation template corresponding to def may be: The preset instruction generation template corresponding to func can be: .

[0073] According to an embodiment of the present application, when the programming language is Python, since Python is an interpreted language, the function definition will be read and processed by the interpreter before the program is executed. Therefore, generating a template through the above-mentioned preset instructions can make all functions must be defined before being called, and in this way, all function information contained in the code to be tested will be matched and returned.

[0074] According to an embodiment of the present application, the function information may be a function header, which may specifically include a function name, a function list, and the like.

[0075] According to an embodiment of the present application, after the function information of all functions in the code to be tested is extracted by a function extraction instruction, the target function information of the main function can be determined from the extracted function information based on a preset function position.

[0076] According to an embodiment of the present application, the main function is usually located in the last position or the second to last position of the entire code in the function included in the code to be tested. Therefore, the final function position can be determined based on the storage order of at least one function, and the target function information can be extracted according to the function position. According to an embodiment of the present application, when the programming language of the code to be tested is Python, the code to be tested can be as follows. Among them, def another_function(a, b, c) is the function information of the main function.

[0077] def func1(arg1, arg2):

[0078] pass

[0079] def func2(x):

[0080] return x

[0081] def another_function(a, b, c):

[0082] pass

[0083] According to an embodiment of the present application, when the programming language of the code to be tested is the Go language, the code to be tested may be as follows: wherein func func1(arg1, arg2 int) is the function information of the main function.

[0084] func func1(arg1, arg2 int) {

[0085] / / Do something

[0086] func main() {

[0087] / / Main function

[0088] }

[0089] According to an embodiment of the present application, after obtaining the target function information, the use case generation request can be obtained by embedding the target function information into the use case request generation template. Specifically, the use case request generation templates corresponding to different programming information may be the same or different. In different situations, for example, when the programming language is python, the English version of the use case generation request obtained can be f'\nFor this problem, please generate 5 unit tests with assert without generating code, just output the unit tests and the functionheader is {head_def}'. The Chinese version can be: For the above code generation request, please generate 5 unit tests using assert statements, without generating code implementation, only output these unit test contents, and the target function information is {head_def}.

[0090] According to an embodiment of the present application, when the programming language is Go language, the obtained use case generation request can be: f'\n\nBased on the above problem description and the function header {target_function_header}, do not generate code and explanation, just generate 5 assert test cases in Go language, or it can be based on the above code generation request and target function information {target_function_header}, do not generate code and explanation, and only generate 5 assert test cases in Go language.

[0091] According to an embodiment of the present application, by using the above-mentioned type of use case generation request, the programming information targeted by the test case can be limited, and the preset model can be limited to only generate test cases for unit testing the code to be tested, without generating the code to be tested or unnecessary explanations.

[0092] According to an embodiment of the present application, both the code to be tested and the test cases can be stored in a file in the form of a dictionary.

[0093] According to the embodiment of the present application, different definition keywords are determined for different programming information, and the automatic generation of function extraction instructions is realized, and the function extraction instructions are quickly determined. Based on the function extraction position corresponding to the programming information, the function information of the main function is determined, and the function information of the main function is quickly determined, and the use case generation request is generated using less but most accurate information.

[0094] According to an embodiment of the present application, it is characterized in that the programming information includes a programming language; at least one candidate function information is stored in the order of appearance of at least one function in the code to be tested; based on the function position corresponding to the programming information in the code to be tested, the target function information is determined from the at least one candidate function information, and the target function information is the function information of the main function, which may include the following operations.

[0095] Based on the function position, the latest candidate function information in at least one function is used as the selected function information; when the programming language is the first programming language, the function name included in the selected function information is matched with the preset function name to obtain a first matching result; when the first matching result represents that the function name is the preset function name, the candidate function information that appears in an order before the selected function information is used as the target function information.

[0096] According to an embodiment of the present application, the candidate function information may be stored in a list, an array, or the like to implement orderly storage of the candidate function information.

[0097] According to an embodiment of the present application, the function position corresponds to the arrangement order of the function information of the main function when it is stored. For example, if at least one function is arranged from first to last in the order of appearance of each function, the function position can be the position of the last stored function information in at least one function information.

[0098] According to an embodiment of the present application, since the main function is usually the last function in the code to be tested, its extraction order is usually the last when it is extracted, so the main function can also be considered to be the latest one.

[0099] According to an embodiment of the present application, python may be the second programming language. When the programming language of the code to be tested is python, it may be considered that the selected function information is the target function information.

[0100] According to an embodiment of the present application, the first programming language may be the Go language. When the programming language of the code to be tested is the Go language, since the main function may be the last function in the code to be tested in the Go language, it is possible to determine whether the selected function information is the function information of the main function by comparing the function names.

[0101] According to an embodiment of the present application, the preset function name may be the function name of the main function.

[0102] According to an embodiment of the present application, when it is determined that the selected function information is the function information of the main function, the candidate function information that appears before the selected function information, i.e., the second to last in the code to be tested, can be selected as the target function information. When determining the candidate function information that is located before the selected function information, it can also be determined based on the positions of each function determined when storing the candidate function information.

[0103] According to an embodiment of the present application, candidate function information is stored based on the order of appearance of target function information in the code to be tested, so that the function information of the main function is determined from two angles of function position and function name corresponding to programming information, thereby achieving fast and accurate determination of the main function.

[0104] According to an embodiment of the present application, the above training data generating method may further include the following operations.

[0105] The code to be tested and multiple test cases are associated with each other according to the programming information to obtain a target test code; based on the target test code, the code to be tested is tested in a test environment corresponding to the programming information to obtain test results of each of the multiple test cases.

[0106] According to the embodiments of the present application, different association processing methods can be used for different programming information. For example, when the programming language in the programming information is Python, the association processing can be achieved by splicing the code to be tested with multiple test cases, so that the obtained target test code includes the code to be tested and all test cases. When the language is Go, the target test code can be obtained by determining the package to which the code to be tested belongs from the code to be tested, and splicing the definition code related to the package to which it belongs with multiple test cases.

[0107] According to an embodiment of the present application, specifically, the definition code related to the package to which it belongs can be obtained through a preset definition template, which includes a definition code, a test library, a test package definition code, etc. The relevant definition code is, for example:

[0108] package main

[0109] import (

[0110] "testing"

[0111] "github.com / stretchr / testify / assert" )

[0113] If the package to which the code to be tested belongs is the main package, package main can be used to declare that the current file belongs to the main package. To support the definition and running of test cases, the Go language provides a standard library testing package. In addition, the Go language itself does not have a built-in assertion function, so a third-party test library testify can be introduced to implement the assertion function.

[0114] According to an embodiment of the present application, by building a test environment, the code to be tested is tested through the target test code, and the test results of each of the multiple test cases are obtained.

[0115] According to the embodiments of the present application, different target test codes are constructed for the code to be tested with different programming information, and a test environment matching the target test code is constructed to test the code to be tested, thereby achieving training data generation for different programming information, improving the versatility of the overall solution, and making the programming information of the training data finally obtained diverse.

[0116] According to an embodiment of the present application, the code to be tested may be obtained by preprocessing the initial code output by the model, and the specific preprocessing steps may include: since the initial code output by the preset model may be wrapped in a preset format and carry certain redundant text information around it, a preset function may be used to extract the initial code from the preset format to obtain the extracted code. For example, when the programming language is python, re.findall is used to extract the extracted code surrounded by ```python and ``` from the initial code. During the extraction process, the line breaks in the initial code may be replaced with preset characters to convert the initial code into one line, which is convenient for extracting the initial code. The preset characters are not limited and can be <n>.

[0117] According to an embodiment of the present application, due to the output format of the preset model, the initial code may be divided into code blocks, so the extracted code may be in the form of code blocks. Therefore, the code blocks can be spliced ​​to obtain a complete extracted code. Specifically, if the extracted code block is not empty, check whether the first code block ends with a preset character. If there is no preset character at the end, a preset character at the end can be added. All the extracted code blocks are thus connected into a string to obtain a spliced ​​code, and the preset characters are inserted between the code blocks, so that the code blocks can also be separated normally when the preset characters are subsequently converted into line breaks.

[0118] According to an embodiment of the present application, code lines containing assertion statements in the concatenated code may be deleted. Specifically, these code lines are assertion statements surrounded by preset characters, and the assertion statements may be assert statements.

[0119] According to an embodiment of the present application, the preset characters in the splicing code can be replaced back to the line break character \n, and the line breaks at the beginning and the end can be removed, and it is determined whether the last line of the splicing code starts with a preset symbol. If it starts with a preset symbol such as: #, it can be considered as a comment and can be deleted to obtain the final code to be tested.

[0120] According to an embodiment of the present application, when the programming language included in the programming information is a second programming language, based on the target test code, the code to be tested is tested in a test environment corresponding to the programming information to obtain test results of multiple test cases, which may include the following operations.

[0121] Match the target test code with the test keywords to obtain multiple second matching results representing the match with the test keywords, the target test code is obtained by splicing the code to be tested and multiple test cases, and each test case includes a predetermined number of test keywords; based on the multiple second matching results, determine the number of test cases; based on a preset execution function, execute the target test code in a temporary directory to obtain an execution result; when it is determined that there is target error data related to the test keyword in the execution result, create a result storage space for storing multiple test results based on a space management tool; test the code to be tested by multiple test cases respectively to obtain multiple test results; store the multiple test results in the result storage space.

[0122] According to an embodiment of the present application, the test keyword may be a keyword in an assertion statement, or other test words, specifically: the assertion statement may be an assert statement.

[0123] According to an embodiment of the present application, by matching the code lines of the test keywords included in the target test code, the total number of test cases included in the target test code can be determined by the number of second matching results and the number of test keywords included in each test case.

[0124] According to an embodiment of the present application, the preset execution function is not limited and may be an unsafe_execute function.

[0125] According to an embodiment of the present application, the target error data may be AssertionError, that is, an error generated by an assertion statement. The target error data may be used to determine that the relevant definition information of the target code to be tested itself may have passed the test during the testing of some test cases.

[0126] According to an embodiment of the present application, if the execution result does not include the target error data, but only includes: FileNotFoundError, ModuleNotFoundError, TimeoutException, AttributeError, SyntaxError, TypeError, NameError, OSError, IndexError and other error data, it is considered that there may be problems with the code to be tested itself, such as: the relevant definition may not be correct, which results in the code to be tested failing to implement the function normally, and in this case it can be considered that the test results of multiple test cases are all failed, that is, the final test pass rate is 0.

[0127] According to an embodiment of the present application, the result storage space may also be created before the execution result is obtained, and after the execution result is obtained, the test results of each of the multiple test cases may be stored therein.

[0128] According to an embodiment of the present application, during the execution of the target test code, a preset error data acquisition tool can be used to acquire error data to obtain the final execution result, for example: using a try-except block to acquire error data.

[0129] According to the embodiment of the present application, the specific implementation of the space management tool is not limited, and may be: multiprocessing.Manager. When creating the result storage space, a test case storage space for storing test cases representing the test results passing the test may also be created.

[0130] According to the embodiments of the present application, there is no limitation on the specific forms of the result storage space and the use case storage space, and they may be any form capable of storing data, such as a list, a queue, and the like.

[0131] According to an embodiment of the present application, when it is determined that there is target error data related to the test keyword in the execution result, the code to be tested can be spliced ​​and executed separately with each test case, so that each test case tests the code to be tested one by one, and obtains the test result of each test case. In the process of testing, a multi-threaded parallel method can be used to implement the testing of the code to be tested by multiple test cases. Specifically, a new process can be created to execute the unsafe_execute function. In some embodiments, a timeout can be set to increase the test speed. If the process is not completed within the timeout, the process is terminated.

[0132] According to an embodiment of the present application, each process may store the test result in a result storage space after obtaining the test result, and the results of passing the test and failing the test may be represented by different values.

[0133] According to an embodiment of the present application, each process may also determine the test cases whose test results represent the test passing, and store them in the test case storage space.

[0134] According to an embodiment of the present application, when the programming information includes that the programming language is a second programming language, the test keyword is matched with the target test code obtained by splicing the code to be tested and multiple test cases to achieve accurate statistics on the number of test cases. And by executing the target test code in a temporary directory, that is, executing it in an environment corresponding to the coding information, and when it is determined through the execution result that there is error data related to the test keyword, that is, when it is determined that there may be no problem with the definition of the code to be tested itself, one by one is tested, thereby improving the test speed, achieving targeted testing of the code to be tested in different programming languages, and taking into account the technical effects of test speed and accuracy.

[0135] Figure 3 A flowchart of testing a code to be tested when the programming language is a first programming language according to an embodiment of the present application is schematically shown.

[0136] like Figure 3 As shown, when the programming language is the first programming language, testing the code to be tested includes operations S301 to S309.

[0137] In operation S301, the code to be tested and a plurality of test cases are concatenated to obtain a target test code.

[0138] In operation S302, the target test code is matched with the test keyword to obtain a plurality of second matching results representing a match with the test keyword.

[0139] In operation S303 , the number of test cases is determined based on the plurality of second matching results.

[0140] In operation S304, based on the preset execution function, the target test code is executed in the temporary directory to obtain an execution result.

[0141] In operation S305, it is determined whether the execution result includes target error data related to the test keyword. In the case that the target error data does not exist, operation S306 is performed. In the case that the target error data exists, operation S309 is performed.

[0142] In operation S306, it is determined whether there is other error reporting data except the target error reporting data, and if there is other error reporting data, operation S307 is performed. If there is no other error reporting data, operation S308 is performed.

[0143] In operation S307 , the test pass rate is determined to be a first preset value.

[0144] In operation S308 , the test pass rate is determined to be a second preset value.

[0145] According to an embodiment of the present application, the first preset value may be 0, and the second preset value may be 100%.

[0146] In operation S309, the codes to be tested are tested respectively by multiple test cases to obtain multiple test results.

[0147] According to an embodiment of the present application, when the programming language included in the programming information is a first programming language, based on the target test code, the code to be tested is tested in a test environment corresponding to the programming information to obtain test results of multiple test cases, which may include the following operations.

[0148] Generate the same identification mark for the code to be tested and the target test code; store the code to be tested and the target test code in a temporary storage space created based on the identification mark; compile the target test code in the temporary storage space based on preset compilation instructions to obtain a compilation result; when the compilation result indicates that the compilation is successful, execute the target test code based on preset execution instructions to obtain test results of multiple test cases.

[0149] According to an embodiment of the present application, the code to be tested and the target test code may be stored in respective target files, and the same identification mark may be generated for the target files where the code to be tested and the target test code are located.

[0150] According to an embodiment of the present application, the identification identifier may be a Universally Unique Identifier (UUID).

[0151] According to the embodiments of the present application, there is no limitation on the specific implementation form of the temporary storage space, for example, any temporary file storage space such as a temporary folder.

[0152] According to an embodiment of the present application, the preset compilation instruction is not limited and can be any instruction that can compile code, such as: go build command, etc.

[0153] According to an embodiment of the present application, when it is determined through the compilation results that the target test code cannot be compiled, it is considered that there is a problem with the target test code. At this time, it can be considered that the test results of each of the multiple test cases have failed the test.

[0154] According to the embodiments of the present application, the preset execution instruction is not limited, and can be an instruction that can control the target test code to execute, and during the execution process, the preset instruction can be executed using related functions, for example: the command timeout-k 5 5s go test instruction is executed through the subprocess.run function to implement unit testing of the target test code. This instruction will run the test within 5 seconds, and will be terminated forcibly if it times out.

[0155] According to an embodiment of the present application, the test results of multiple test cases can be obtained through standard error output and return code. For example, when the standard error output is empty and the return code is 0, it can be considered that the test results of multiple test cases of the code to be tested are all passed, thereby obtaining a test pass rate of 100%.

[0156] According to an embodiment of the present application, when the programming language is a second programming language, the target test code is constructed by the package to which the code to be tested belongs and multiple test cases, and by compiling first and then testing after successful compilation, the option of compiling first and then testing is achieved. In the case of compilation failure, testing is not performed first, thereby avoiding waste of resources caused by invalid testing and improving the testing efficiency of the code to be tested.

[0157] According to an embodiment of the present application, the code generation request is obtained in the following manner.

[0158] Match the code generation problem with multiple preset request templates to obtain a target request template corresponding to the code generation problem; embed the code generation problem into the target request template to obtain an intermediate generation request, the target request template includes programming information and quantity information for indicating the number of code generation requests to be generated; input the intermediate generation request into the preset model to obtain a code generation request with the same quantity and quantity information.

[0159] According to an embodiment of the present application, the code generation problem may be preset, and there may be multiple code generation problems. By executing the above training data generation method for multiple code generation problems, a large amount of training data may be obtained.

[0160] According to an embodiment of the present application, for example, a code generation problem may be: writing a function to connect two given tuples into a nested tuple. <n>Example: <n>```python <n>Assert that the function concatenate_nested returns (3, 4) and (5, 6) and that the return value is (3, 4, 5, 6).

[0161] According to an embodiment of the present application, the code generation problem may be stored in the original training data set, and the original training data set may be expanded by obtaining the code generation problem therefrom.

[0162] According to an embodiment of the present application, different code generation problems may correspond to different preset request templates, and an intermediate generation request input into a preset model may be obtained by embedding the code generation problem into the target request template.

[0163] According to an embodiment of the present application, the target request template may be: please imitate the "code generation problem" to generate "n" similar problems to be solved with "programming information", and there is no need to answer the questions. The information in "" may be information embedded later. Similarly, n and programming information may also be preset in the target request target in advance. The programming information in a target request template may not be limited to one type. For example, the programming information part in a target request template may be embedded in Python or go language.

[0164] According to an embodiment of the present application, if the preset model outputs a plurality of programming information of the code to be tested at one time, the specific programming information can be determined by analyzing the code to be tested according to programming rules for different languages, such as function definition, programming format, etc. Alternatively, the specific programming information can be determined by the keywords used to prompt the programming information when the preset model is output.

[0165] According to an embodiment of the present application, through the intermediate generation request generated by the target request target, it is possible to guide the model to generate n different programming questions, namely code generation requests, and make each programming question different in terms of content structure or data type. In this way, it can be ensured that the generated programming question is logically similar to the original question, and when answering, only the generated question is answered instead of the original programming question, thus saving model calling time.

[0166] According to an embodiment of the present application, when a preset model generates multiple code generation requests at a time, they may be separated by line breaks and numbers. Therefore, a preset calling method, such as text.split('\n'), etc., can be used. According to the number of line breaks, these questions can be regarded as several similar questions generated based on the original question. If the quantity does not match the quantity information in the intermediate generation request, the data is deleted and regenerated again. The generated text may contain serial numbers or irregular symbols (such as numerical labels, special characters, etc.). The above content can be cleared to extract the generated complete question, thereby achieving complete consistency of the question and answer pairs.

[0167] Figure 4 The following schematically shows a flow chart of a method for generating training data according to another embodiment of the present application.

[0168] like Figure 4 As shown, the method includes operations S401 to S409.

[0169] In operation S401, a code generation problem is matched with a plurality of preset request templates to obtain a target request template corresponding to the code generation problem.

[0170] In operation S402, the code generation problem is embedded into the target request template to obtain an intermediate generation request.

[0171] In operation S403, the intermediate generation request is input into a preset model to obtain a code generation request.

[0172] In operation S404 , a code to be tested corresponding to a code generation request is generated based on a pre-trained preset model, and the code generation request includes programming information used to indicate the programming information adopted by the code to be tested.

[0173] In operation S405 , target function information is extracted from the code to be tested based on a function extraction rule for programming information.

[0174] In operation S406, the use case generation request generated by using the target function information is input into a preset model to obtain a plurality of test cases for the code to be tested.

[0175] In operation S407, the test pass rate is determined using the test results of each of the plurality of test cases.

[0176] In operation S408, it is determined whether the test pass rate is greater than a preset threshold. If it is determined that the test pass rate is greater than the preset threshold, operation S409 is performed. If it is determined that the test pass rate is less than or equal to the preset threshold, the process returns to operation S401.

[0177] In operation S409, a first question-answer pair consisting of a code generation request and a code to be tested, and a second question-answer pair consisting of a use case generation request and a test case representing a passed test are used as training data.

[0178] Based on the above training data generation method, the present application also provides a training data generation device. Figure 5 The device is described in detail.

[0179] Figure 5 The structural block diagram of the training data generating device according to an embodiment of the present application is schematically shown.

[0180] like Figure 5 As shown, the training data generating device 500 of this embodiment includes a code generating module 510 , a function extracting module 520 , a use case determining module 530 and a data determining module 540 .

[0181] The code generation module 510 is used to generate a code to be tested corresponding to a code generation request based on a pre-trained preset model, and the code generation request includes programming information used to indicate the programming information adopted by the code to be tested.

[0182] The function extraction module 520 is used to extract target function information from the code to be tested based on the function extraction rule for programming information.

[0183] The use case determination module 530 is used to input the use case generation request generated by using the target function information into a preset model to obtain multiple test cases for the code to be tested.

[0184] The data determination module 540 is used to use the first question-answer pair consisting of the code generation request and the code to be tested, and the second question-answer pair consisting of the use case generation request and the test case representing the test pass as training data when it is determined by using the test results of multiple test cases that the test pass rate of the code to be tested is greater than a preset threshold. The training data is used to fine-tune the preset model.

[0185] According to an embodiment of the present application, the function extraction module 520 includes: an instruction determination submodule, an extraction submodule and a determination submodule.

[0186] The instruction determination submodule is used to determine the function extraction instruction based on the definition keyword corresponding to the programming information, and the definition keyword is used to define the function included in the code to be tested.

[0187] The extraction submodule is used to extract function information of at least one function from the code to be tested based on the function extraction instruction, and obtain at least one candidate function information.

[0188] The determination submodule is used to determine target function information from at least one candidate function information based on the function position corresponding to the programming information in the code to be tested, and the target function information is function information of the main function.

[0189] According to an embodiment of the present application, the programming information includes a programming language. The at least one candidate function information is stored in the order in which the at least one function appears in the code to be tested. The determination submodule includes: a first determination unit, a matching unit and a second determination unit.

[0190] The first determining unit is used to use the latest candidate function information in at least one function as the selected function information based on the function position.

[0191] The matching unit is used to match the function name included in the selected function information with the preset function name to obtain a first matching result when the programming language is the first programming language.

[0192] The second determining unit is configured to, when the function name represented by the first matching result is a preset function name, use the candidate function information that appears before the selected function information as the target function information.

[0193] According to an embodiment of the present application, the training data generating device 500 further includes: an association processing module and a testing module.

[0194] The association processing module is used to associate the code to be tested with multiple test cases according to the programming information to obtain the target test code.

[0195] The test module is used to test the code to be tested based on the target test code in a test environment corresponding to the programming information, and obtain the test results of each of the multiple test cases.

[0196] According to an embodiment of the present application, when the programming language included in the programming information is a second programming language, the test module includes: a word matching submodule, a quantity determination submodule, an execution submodule, a creation submodule, a test submodule and a storage submodule.

[0197] The word matching submodule is used to match the target test code with the test keywords to obtain multiple second matching results representing the match with the test keywords. The target test code is obtained by splicing the code to be tested and multiple test cases, and each test case includes a predetermined number of test keywords.

[0198] The quantity determination submodule is used to determine the quantity of the test cases based on the multiple second matching results.

[0199] The execution submodule is used to execute the target test code in a temporary directory based on the preset execution function to obtain the execution result.

[0200] A creation submodule is used to create a result storage space for storing multiple test results based on a space management tool when it is determined that target error data related to the test keyword exists in the execution result.

[0201] The test submodule is used to test the code to be tested by multiple test cases and obtain multiple test results.

[0202] The storage submodule is used to store multiple test results in the result storage space.

[0203] According to an embodiment of the present application, when the programming language included in the programming information is a first programming language, the test module includes: an identification generation submodule, a space creation submodule, a compilation submodule and an instruction execution submodule.

[0204] The identification generation submodule is used to generate the same identification identification for the code to be tested and the target test code.

[0205] The space creation submodule is used to store the code to be tested and the target test code in a temporary storage space created based on the identification mark.

[0206] The compiling submodule is used to compile the target test code in the temporary storage space based on the preset compiling instructions to obtain the compiling result.

[0207] The instruction execution submodule is used to execute the target test code based on the preset execution instructions to obtain the test results of each of the multiple test cases when the compilation result indicates that the compilation is successful.

[0208] According to an embodiment of the present application, the training data generating device 500 further includes: a template determining module, an embedding module and a request generating module.

[0209] The template determination module is used to match the code generation problem with multiple preset request templates to obtain a target request template corresponding to the code generation problem.

[0210] The embedding module is used to embed the code generation problem into the target request template to obtain an intermediate generation request. The target request template includes programming information and quantity information for indicating the number of code generation requests to be generated.

[0211] The request generation module is used to input the intermediate generation request into the preset model to obtain the code generation request with the same quantity and quantity information.

[0212] According to an embodiment of the present application, any multiple modules of the code generation module 510, the function extraction module 520, the use case determination module 530, and the data determination module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present application, at least one of the code generation module 510, the function extraction module 520, the use case determination module 530, and the data determination module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the code generation module 510, the function extraction module 520, the use case determination module 530, and the data determination module 540 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0213] Figure 6 A block diagram of an electronic device suitable for implementing a training data generating method according to an embodiment of the present application is schematically shown.

[0214] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0215] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0216] According to an embodiment of the present application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0217] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0218] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0219] The embodiment of the present application also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the training data generation method provided in the embodiment of the present application.

[0220] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 601 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0221] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0222] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0223] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level processes and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0224] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0225] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application.

[0226] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although each embodiment is described above, this does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present application, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present application.< / n> < / n> < / n> < / n>

Claims

1. A training data generation method, characterized in that: The method comprises: Generate a code to be tested corresponding to a code generation request based on a pre-trained preset model, wherein the code generation request includes programming information used to indicate the programming information adopted by the code to be tested; extracting target function information from the code to be tested based on a function extraction rule for the programming information; Inputting a use case generation request generated by using the objective function information into the preset model to obtain a plurality of test cases for the code to be tested; When it is determined that the test pass rate of the code to be tested is greater than a preset threshold by using the test results of each of the multiple test cases, the first question-answer pair consisting of the code generation request and the code to be tested and the second question-answer pair consisting of the case generation request and the test case representing the test pass are both used as training data, and the training data are used to fine-tune the preset model.

2. The method according to claim 1, characterized in that The step of extracting target function information from the code to be tested based on a function extraction rule for the programming information includes: Determining a function extraction instruction based on a definition keyword corresponding to the programming information, wherein the definition keyword is used to define a function included in the code to be tested; Extracting function information of at least one function from the code to be tested based on the function extraction instruction to obtain at least one candidate function information; Based on the function position corresponding to the programming information in the code to be tested, target function information is determined from at least one of the candidate function information, and the target function information is function information of the main function.

3. The method according to claim 2, characterized in that The programming information includes a programming language; at least one of the candidate function information is stored according to the order in which at least one of the functions appears in the code to be tested; Wherein, based on the function position corresponding to the programming information in the code to be tested, determining the target function information from at least one of the candidate function information, the target function information being function information of the main function, includes: Based on the function position, taking the latest candidate function information in at least one of the functions as the selected function information; In a case where the programming language is a first programming language, matching the function name included in the selected function information with a preset function name to obtain a first matching result; In a case where the first matching result indicates that the function name is the preset function name, the candidate function information that appears before the selected function information is used as the target function information.

4. The method according to claim 1, characterized in that: The method further comprises: Associating the code to be tested with the plurality of test cases according to the programming information to obtain a target test code; Based on the target test code, the code to be tested is tested in a test environment corresponding to the programming information to obtain test results of each of the plurality of test cases.

5. The method according to claim 4, characterized in that In the case where the programming language included in the programming information is a second programming language, The step of testing the code to be tested in a test environment corresponding to the programming information based on the target test code to obtain test results of each of the plurality of test cases includes: Matching the target test code with a test keyword to obtain a plurality of second matching results representing a match with the test keyword, wherein the target test code is obtained by concatenating the code to be tested and a plurality of the test cases, each of the test cases including a predetermined number of the test keywords; Determining the number of the test cases based on the plurality of the second matching results; Based on the preset execution function, the target test code is executed in the temporary directory to obtain the execution result; In the case where it is determined that the execution result contains target error data related to the test keyword, creating a result storage space for storing a plurality of the test results based on a space management tool; The code to be tested is tested by a plurality of the test cases respectively to obtain a plurality of the test results; The plurality of test results are stored in the result storage space.

6. The method according to claim 4, characterized in that In the case where the programming language included in the programming information is a first programming language, The step of testing the code to be tested in a test environment corresponding to the programming information based on the target test code to obtain test results of each of the plurality of test cases includes: Generating the same identification mark for the code to be tested and the target test code; Storing the code to be tested and the target test code in a temporary storage space created based on the identification mark; Based on the preset compilation instruction, compile the target test code in the temporary storage space to obtain a compilation result; When the compilation result indicates that the compilation is successful, the target test code is executed based on the preset execution instruction to obtain the test results of each of the plurality of test cases.

7. The method according to claim 1, characterized in that The code generation request is obtained in the following manner: Matching the code generation problem with a plurality of preset request templates to obtain a target request template corresponding to the code generation problem; embedding the code generation problem into the target request template to obtain an intermediate generation request, wherein the target request template includes the programming information and quantity information indicating the generation quantity of the code generation request; The intermediate generation request is input into the preset model to obtain a code generation request with the same quantity as the quantity information.

8. A training data generating device, characterized in that: The method comprises: A code generation module, configured to generate a code to be tested corresponding to a code generation request based on a pre-trained preset model, wherein the code generation request includes programming information used to indicate the programming information adopted by the code to be tested; A function extraction module, used for extracting target function information from the code to be tested based on a function extraction rule for the programming information; A use case determination module, used to input a use case generation request generated by using the objective function information into the preset model to obtain a plurality of test cases for the code to be tested; A data determination module is used to use a first question-answer pair consisting of the code generation request and the code to be tested, and a second question-answer pair consisting of the case generation request and the test case representing the test pass as training data when it is determined that the test pass rate of the code to be tested is greater than a preset threshold using the test results of each of the multiple test cases. The training data is used to fine-tune the preset model.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images