Code testing method and related system

Abstract

The present application provides a code testing method, applied to a code testing system. When a test code is generated, context information of a code under test is extracted, and the test code is generated on the basis of the context information of the code under test. The context information often has a service meaning and can embody a test scenario, so that the prior knowledge of the service and the test scenario is achieved in the generation process of test codes, thereby generating test codes having higher quality.

Description

A code testing method and related system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 16, 2023, with application number 202311544456.X and application name “A data processing method and related equipment”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 13, 2023, with application number 202311713742.4 and application name “A code testing method and related system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a code testing method, system, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0003] Code testing ensures the quality of software projects and is a crucial part of software development. During the development process, time is primarily allocated to code writing, code repair, code refactoring, and code testing. Code testing accounts for over 15% of the development time, requiring developers to expend considerable time and effort writing test code. To save development time, the use of automated test code generation technology has gained increasing attention in recent years.

[0004] Large language models are machine learning models with large parameter sizes. Test generation techniques based on large language models primarily rely on code sets related to the code (including the code to be tested). The large language model is used to generate test code corresponding to the code to be tested, and the code sets related to the code are used to train the weight parameters of the large model. Although test generation techniques based on large natural language models are not highly interpretable, their ability to abstract code features, generate rich use cases, and integrate inputs from different modalities are unmatched by traditional algorithms.

[0005] However, in the prior art, during the test code generation process, the sample input into the language model is the entire code of the code file related to the code to be tested, which makes the language model's analysis of the code data very rough, resulting in poor quality of the generated test code.

[0006] Summary of the Invention

[0007] The present application provides a code testing method that improves the quality of the generated test code. The present application also provides a code testing system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0008] In a first aspect, the present application provides a code testing method. The method is applied to a code testing system. The code testing system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the code testing method of the present application. In some possible implementations, the code testing system can also be a hardware system, for example, a computing device cluster with an unknown test code generation function. When the hardware system is running, the code testing method of the present application is executed.

[0009] Specifically, the method includes: receiving a test instruction for a first code, the first code being a code in a program to be edited, and the test instruction instructing generation of a test code through a language model; obtaining information of the first code from a file associated with the first code, the information of the first code including context information of a function included in the first code, and one or more of the following information: context information of the file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code, wherein the information of the first code is used to construct a prompt word prompt as input to the language model, and the first test code is obtained through the language model according to the prompt word prompt; obtaining a first test code corresponding to the first code according to the information of the first code; and testing the first code according to the first test code.

[0010] Based on this, when generating test code, this application extracts contextual information related to the code to be tested. Since contextual information often has business meaning and can reflect the test scenario, it provides a priori information about the business and test scenarios during the test code generation process, thereby generating higher-quality test code. On the other hand, in addition to the function-level context, this application also extracts coarser-grained contextual information than the function level. By generating test code through multi-level contextual information, the priori information about the business and test scenarios can be enriched, thereby making the quality of the generated test code higher.

[0011] In a possible implementation, the context information of the function included in the first code includes at least one of the types of the input parameters of the included function, the type of the function return value, and the method call in the function body.

[0012] In a possible implementation, the context information of the file associated with the first code includes: at least one of: a variable name, structure information, an object name, an object definition, a class member variable, a class member function, a class construction method, and a class inheritance relationship of the file associated with the first code.

[0013] In a possible implementation, the context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language.

[0014] In a possible implementation, the context information of the test framework used by the first code includes: a category of the test framework used by the first code, or a category of the mock framework.

[0015] In one possible implementation, a first prompt word can be sent to a computing device on the cloud side to instruct a language model to generate a test code for the first code based on information of the first code. The computing device on the cloud side can obtain a first test code corresponding to the first code through the language model based on the first prompt word, and the end side can receive the first test code corresponding to the first code obtained by the computing device on the cloud side.

[0016] In a possible implementation, a computing device on the terminal side or the cloud side may obtain a first test code corresponding to the first code according to the first prompt word through a language model.

[0017] In one possible implementation, obtaining information about the first code includes: obtaining a first prompt word prompt containing information about the first code; the first prompt word is used to instruct a language model to generate a test code for the first code based on the information about the first code; and obtaining a first test code corresponding to the first code based on the information about the first code includes: sending the first prompt word and receiving a first test code corresponding to the first code obtained based on the first prompt word; or obtaining the first test code corresponding to the first code through a language model based on the first prompt word.

[0018] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code.

[0019] In a possible implementation, obtaining the information of the first code includes: upon receiving a test instruction for the first code in the code editing interface, obtaining the information of the first code, wherein the test instruction instructs to generate a language model-based test code for the first code.

[0020] In an embodiment of the present application, the large language model can have the ability to process multiple code languages, that is, the ability to generate test codes for multiple code languages, that is, to generate corresponding test codes for the codes to be tested of multiple different code language types. For example, the information of the second code can be obtained; the second code and the first code are different types of programming languages; according to the information of the second code, the second test code is obtained through the language model, wherein the first test code and the first code are the same type of programming language, and the second test code and the second code are the same type of programming language. Among them, similar to the first code, the information of the second code may include context information of the function included in the second code, and at least one of the following information: context information of the file associated with the second code, context information of the project where the second code is located, or context information of the test framework used by the second code.

[0021] In one possible implementation, test samples in different coding languages ​​can be constructed during training to enable the trained language model to have the ability to process multiple coding languages, that is, the ability to generate test codes for multiple coding languages.

[0022] In a possible implementation, the method further includes: updating the language model according to the first test code and a third test code corresponding to the first code, wherein the third test code is used as a test code true value of the first code.

[0023] In a second aspect, the present application provides a code testing method, the method comprising:

[0024] Receive a test instruction for a first code, where the first code is code in a program to be edited, and the test instruction instructs generation of test code through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code based on a file of the first code, and generate test code for the first code based on the information of the first code; the information of the first code includes context information of functions included in the first code, and one or more of the following information: context information of files associated with the first code, context information of files associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code;

[0025] A first test code corresponding to the first code is obtained according to the second prompt word, and the first code is tested according to the first test code.

[0026] Different from the embodiment corresponding to the first aspect, a prompt can be used to guide the language model to extract context information.

[0027] In a possible implementation, the context information of the function included in the first code includes: at least one of the types of the function's input parameters, the function's return value type, and method calls within the function body; or,

[0028] The context information of the file associated with the first code includes at least one of: a variable name, structure information, object name, object definition, class member variables, class member functions, class constructors, and class inheritance relationships of the file associated with the first code; or

[0029] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or

[0030] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0031] In a possible implementation, obtaining a first test code corresponding to the first code according to the second prompt word includes:

[0032] sending the second prompt word and receiving a first test code corresponding to the first code obtained according to the second prompt word; or

[0033] According to the second prompt word, a first test code corresponding to the first code is obtained through a language model.

[0034] In a third aspect, the present application provides a sample providing method, the method comprising: obtaining a code set; obtaining a training sample of a language model based on the code set, the training sample comprising a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework being used to indicate constraints on a test function that can be used as a function to be tested, the constraints comprising constraints on the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints on the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints on the calling relationship, and the third test code is used as the true value of the test code of the first code.

[0035] The training samples extracted from the code set may include the code to be tested (e.g., the first code) and the true value of the test code corresponding to the first code (e.g., the third test code). In order to extract high-quality training samples, the embodiment of the present application is based on a preset test framework and extracts codes that meet the constraints specified by the test framework from the code set as training samples.

[0036] The test framework is used to indicate the constraints of the test function that can be used as the function to be tested. The constraints include the constraints of the test function itself and the constraints of the calling relationship between the test function and the function to be tested.

[0037] Due to different code language types and even different code writing methods of the same language, different code sets may be suitable for different test frameworks. The so-called "test framework adapted to the code set" here can be understood as the constraints that need to be met by the function to be tested and the test function under the language and writing method of the code set.

[0038] After obtaining the code set, a test framework that is compatible with the code set can be determined from the preset multiple test frameworks (that is, the first test framework in the embodiment of the present application). The test framework feature library of the above-mentioned preset multiple test frameworks can be constructed offline. By constructing the test framework feature library offline, the scope of data collection is expanded, and the diversity of training data can be increased. And the versatility and scalability are enhanced, and it can be widely used in various test model training and reasoning in multiple languages ​​(such as Java, Python, Go, Cxx, JS, Ts). When extracting training samples, automatic analysis can be performed without manually defining templates and constraints, reducing the test cost, and being more practical and versatile.

[0039] In a possible implementation, the constraint of the calling relationship includes at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0040] In a possible implementation, obtaining a training sample of a language model according to the code set includes:

[0041] Acquire a third test code from the code set that satisfies the constraints of the test function itself;

[0042] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0043] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationships satisfy the constraints of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0044] In a possible implementation, obtaining a training sample of a language model according to the code set includes:

[0045] Acquire a third test code from the code set that satisfies the constraints of the test function itself; the third test code includes an assertion;

[0046] determining a target variable included in the assertion;

[0047] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code meets the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0048] In one possible implementation, the method further includes:

[0049] The language model is updated according to the first test code and the third test code.

[0050] In a fourth aspect, the present application provides a code testing device, comprising:

[0051] an acquisition module, configured to receive a test instruction for a first code, the first code being code in a program to be edited, the test instruction instructing generation of test code through a language model; and obtaining information about the first code from a file associated with the first code, the information about the first code including context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project in which the first code is located, or context information of a test framework used by the first code, wherein the information about the first code is used to construct a prompt word prompt as input to the language model, and the first test code is obtained through the language model based on the prompt word prompt;

[0052] A processing module is used to obtain a first test code corresponding to the first code according to information of the first code.

[0053] In a possible implementation, the context information of the function included in the first code includes: at least one of the types of the function's input parameters, the function's return value type, and method calls within the function body; or,

[0054] The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, class member variable, class member function, class construction device, and class inheritance relationship of the file associated with the first code; or

[0055] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or

[0056] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0057] In a possible implementation, the processing module is specifically configured to:

[0058] Sending information about the first code, and receiving a first test code corresponding to the first code obtained according to the information about the first code; or,

[0059] According to the information of the first code, a first test code corresponding to the first code is obtained through a language model.

[0060] In a possible implementation, obtaining the information of the first code includes: obtaining a first prompt word prompt containing the information of the first code; the first prompt word is used to instruct the language model to generate a test code for the first code based on the information of the first code;

[0061] The processing module is specifically used to:

[0062] sending the first prompt word and receiving a first test code corresponding to the first code obtained according to the first prompt word; or

[0063] According to the first prompt word, a first test code corresponding to the first code is obtained through a language model.

[0064] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code.

[0065] In a possible implementation, the acquisition module is specifically configured to:

[0066] When a test instruction for a first code in a code editing interface is received, information of the first code is acquired, wherein the test instruction instructs to generate a test code based on a language model for the first code.

[0067] In a possible implementation, the acquisition module is further configured to:

[0068] Obtaining information of a second code; wherein the second code and the first code are in different types of programming languages;

[0069] The processing module is further configured to obtain a second test code according to the second prompt word, wherein the first test code and the first code are in the same type of programming language, and the second test code and the second code are in the same type of programming language.

[0070] In a possible implementation, the processing module is further configured to:

[0071] The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

[0072] In a possible implementation, the acquisition module is further used to: acquire a code set;

[0073] The processing module is further used to obtain a training sample of a language model based on the code set, wherein the training sample includes the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints on a test function that can be used as a function to be tested, wherein the constraints include constraints on the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints on the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints on the calling relationship.

[0074] In a possible implementation, the processing module is specifically configured to: obtain, from the code set, a third test code that satisfies the constraints of the test function itself;

[0075] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0076] In a possible implementation, the processing module is specifically configured to: obtain, from the code set, a third test code that satisfies the constraints of the test function itself; the third test code includes an assertion;

[0077] determining a target variable included in the assertion;

[0078] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code meets the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0079] In a fifth aspect, the present application provides a code testing device, comprising:

[0080] an acquisition module, configured to receive a test instruction for a first code, where the first code is code in a program to be edited, and the test instruction instructs generation of test code through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code based on a file of the first code, and generate test code for the first code based on the information of the first code; the information of the first code includes context information of functions included in the first code, and one or more of the following information: context information of files associated with the first code, context information of files associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code;

[0081] A processing module is configured to obtain a first test code corresponding to the first code according to the second prompt word.

[0082] In a possible implementation, the context information of the function included in the first code includes: at least one of the types of the function's input parameters, the function's return value type, and method calls within the function body; or,

[0083] The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, class member variable, class member function, class construction device, and class inheritance relationship of the file associated with the first code; or

[0084] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or

[0085] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0086] In a possible implementation, the processing module is specifically configured to:

[0087] sending the second prompt word and receiving a first test code corresponding to the first code obtained according to the second prompt word; or

[0088] According to the second prompt word, a first test code corresponding to the first code is obtained through a language model.

[0089] In a sixth aspect, the present application provides a sample providing device, the device comprising:

[0090] Acquisition module, used to obtain code sets;

[0091] A processing module is used to obtain a training sample of a language model based on the code set, wherein the training sample includes a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, wherein the constraints include constraints of the test function itself and constraints of a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship, and the third test code is used as the true value of the test code of the first code.

[0092] In a possible implementation, the constraint of the calling relationship includes at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0093] In a possible implementation, the processing module is specifically configured to:

[0094] Acquire a third test code from the code set that satisfies the constraints of the test function itself;

[0095] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0096] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationships satisfy the constraints of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0097] In a possible implementation, the processing module is specifically configured to:

[0098] Acquire a third test code from the code set that satisfies the constraints of the test function itself; the third test code includes an assertion;

[0099] determining a target variable included in the assertion;

[0100] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code meets the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0101] In a possible implementation, the processing module is further configured to:

[0102] The language model is updated according to the first test code and the third test code.

[0103] In a seventh aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or computing device cluster performs the method described in the first aspect or any implementation of the first aspect, the method described in the second aspect or any implementation of the second aspect, or the method described in the third aspect or any implementation of the third aspect.

[0104] In an eighth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and the instructions instruct a computing device or a computing device cluster to execute the code testing method described in the first aspect or any one of the implementations of the first aspect, the method described in the second aspect or any one of the implementations of the second aspect, and the method described in the third aspect or any one of the implementations of the third aspect.

[0105] In the ninth aspect, the present application provides a computer program product comprising instructions, which, when run on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the code testing method described in the first aspect or any one of the implementations of the first aspect, the method described in the second aspect or any one of the implementations of the second aspect, and the method described in the third aspect or any one of the implementations of the third aspect.

[0106] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0108] FIG1 is a schematic diagram of the architecture of a code testing system provided in an embodiment of the present application;

[0109] FIG2a is a schematic diagram of the architecture of a code testing system provided in an embodiment of the present application;

[0110] FIG2 b is a schematic diagram of the architecture of a model training system provided in an embodiment of the present application;

[0111] FIG2c is a schematic diagram of the architecture of a model training system provided in an embodiment of the present application;

[0112] FIG3a is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0113] FIG3 b is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0114] FIG4 is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0115] FIG5 is a schematic diagram of a method for providing training samples according to an embodiment of the present application;

[0116] FIG6 is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0117] FIG7 is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0118] FIG8 is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0119] FIG9 is a schematic diagram of a code testing method provided in an embodiment of the present application;

[0120] FIG10 is a schematic diagram of a test tool interface provided in an embodiment of the present application;

[0121] 11 to 13 are schematic diagrams of a code testing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0122] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0123] First, some technical terms involved in the embodiments of this application are introduced.

[0124] Code testing ensures the quality of software projects and is a crucial part of software development. During the development process, time is primarily allocated to code writing, code repair, code refactoring, and code testing. Code testing accounts for over 15% of the development time, requiring developers to expend considerable time and effort writing test code. To save development time, the use of automated test code generation technology has gained increasing attention in recent years.

[0125] Currently, there are many tools for automatically generating test code. Different technologies require different inputs related to the code under test to output the corresponding test code. Common existing technologies include random test generation, symbolic execution-based test generation, search-based test generation, and large language model-based test generation.

[0126] Large language models are machine learning models with large parameter sizes. Test generation techniques based on large language models primarily rely on code sets related to the code (including the code to be tested). The large language model is used to generate test code corresponding to the code to be tested, and the code sets related to the code are used to train the weight parameters of the large model. Although test generation techniques based on large natural language models are not highly interpretable, their ability to abstract code features, generate rich use cases, and integrate inputs from different modalities are unmatched by traditional algorithms.

[0127] However, in the prior art, during the test code generation process, the sample input into the language model is the entire code of the code file related to the code to be tested, which makes the language model's analysis of the code data very rough, resulting in poor quality of the generated test code.

[0128] In view of this, the present application provides a code testing method. The method can be executed by a code testing system (or can be referred to as a test code generator). The code testing system is used to generate corresponding test code based on the code to be tested. The code testing system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the code testing method of the present application. The software system can also be integrated into software with a test code generation function, for example, it can be in the form of a functional module such as a plug-in in the software. Alternatively, the software system can be independent of the software with a test code generation function, and the software with a test code generation function calls the software system to implement the test code generation function. The software system can be deployed on a computing device, for example, on a terminal device (also referred to as a terminal) such as a desktop computer, a laptop computer, a tablet computer, or a smartphone. The computing device executes the program code of the software system, thereby executing the code testing method of the present application. In some possible implementations, the code testing system can also be a hardware system, for example, a computing device cluster with a test code generation function. When the hardware system is running, the code testing method of the present application is executed.

[0129] Specifically, when generating test code, the context information of the code to be tested is extracted. Since the context information often has business meaning and can reflect the test scenario, the test code generation process has prior knowledge of the business and test scenarios, thereby generating higher quality test code.

[0130] In particular, when context information is used as a prompt input to a large language model, the large language model can generate test method names and variable names that have business meanings and reflect the test scenarios, and can enable the large model to generate valid test input initial values, thereby improving the quality of the generated test code.

[0131] In addition, the present application provides a training sample providing method. The method can be executed by a training sample providing system. The training sample providing system is used to identify training samples that can be used as language models from a code set, and the training samples include source code and corresponding code to be tested. The training sample providing system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the training sample providing method of the present application. In some possible implementations, the training sample providing system can also be a hardware system, for example, a computing device cluster with a training sample providing function. When the hardware system is running, the training sample providing method of the present application is executed.

[0132] In addition, the present application provides a model training method. The method can be executed by a model training system. The model training system is used to identify training samples that can be used as language models from a code set, and the training samples include source code and corresponding code to be tested. The model training system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the model training method of the present application. In some possible implementations, the model training system can also be a hardware system, for example, a computing device cluster with model training capabilities. When the hardware system is running, the model training method of the present application is executed.

[0133] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the present application is introduced below with reference to the accompanying drawings.

[0134] Figure 1 is a diagram illustrating an embodiment 100 of system code development, such as the code development system 100 shown in Figure 1. Embodiment 100 is a simplified example of an apparatus that may be used to write, edit, test, and debug computer executable code.

[0135] The diagram of Fig. 1 illustrates each functional component of system.In some cases, component can be hardware component, software component, or the combination of hardware and software.Some components can be application layer software, and other components can be operating system layer components.In some cases, the connection of one component to another component can be a close connection, wherein two or more components operate on a single hardware platform.In other cases, the connection can be carried out by connecting to a network across a long distance.Each embodiment can use different hardware, software, and interconnection architecture to realize the described function.

[0136] Embodiment 100 is an example of a system that can be used to develop software, firmware, or other executable code. The final product developed on the system of embodiment 100 can be referred to as application code. As used in this specification and claims, application code is a software product that is tested by test code. As used in the embodiments of this application, test code is software used to test application code. Generally speaking, test code is not transmitted with the application code and is not used when executing the application.

[0137] A software development system can be used to generate test code, and the test code can then be automatically evaluated to determine quality metrics for the test code. In many cases, the test code can be manually authored and then evaluated to generate quality metrics. Analysis of the test code can determine various features or aspects of the test code so that quality metrics can be established.

[0138] Using various analyses that can be performed on the test code, the quality metric can be used to determine how 'good' the test code is.The quality metric can be used to identify portions of the test code that may have defects and give some measure of confidence in the test results.

[0139] For example, a high-quality metric can be given to test code that scores high on the analysis. When executed against application code, the results of the test code can be considered a good indication of the quality of the application code. Conversely, the results of test code with poor quality metrics may be problematic.

[0140] The modules in the code development system 100 shown in FIG1 may belong to an independent computing device or a separate computing device. For example, one part of the modules may belong to one computing device and another part may belong to another computing device. Furthermore, the modules may be further divided into different sub-modules, and multiple sub-modules may collaborate to implement the functions of the corresponding modules.

[0141] The computing device may be a general-purpose computer (or a cluster of computing devices on the cloud side) having hardware components 104 and software components 106. The computing device may have several tools that can be used to develop application code and that can be used by programmers or application developers.

[0142] In some implementations, a computing device may be similar to a device on which an application is intended to be executed. Thus, a computing device may represent a personal computer or other similar device.

[0143] In other embodiments, the computing device may be a development platform for software or firmware that can be executed on a different device. For example, the computing device may be used to develop and test software that can be executed on another type of computer device, such as a cell phone or other device with a computer processor. In such embodiments, the computing device may include an emulator or simulator that simulates the operation of the intended device. Such an emulator may allow the application code to be tested and executed before being installed and run on the intended device hardware.

[0144] The hardware component 104 may include a processor 108 that may use random access memory 110 and non-volatile storage 112. The hardware component 104 may have a network interface 114 and a user interface 116.

[0145] In the example of embodiment 100, hardware component 104 may represent a general-purpose personal computer. In other embodiments, hardware component 104 may be a desktop computer or a server computer. Some embodiments may be portable devices such as laptop computers, notebook computers, or even cell phones, personal digital assistants, or other devices.

[0146] The software components 106 may include an operating system 118 on which several different types of software may operate.

[0147] Executable code 120 may be any type of code that can be executed directly within an operating system. In some cases, executable code 120 may be machine code that may include instructions that can be directly executed by processor 108. In other cases, executable code 120 may be assembly code that can be executed after being processed by an assembler.

[0148] Some software components may be defined using intermediate code 122. Intermediate code 122 may be source code that is compiled into an intermediate representation and then may be further compiled using a just-in-time compiler 124 and executed by an execution engine 126. Intermediate code 122 is useful in managed code applications or applications where several different programming languages ​​may be used.

[0149] The application development system 128 may have many of the elements used to write, edit, test, debug, and publish applications. In the example of embodiment 100, the features of the application development system 128 may be shown as part of a larger application or programming environment in which a programmer can perform many different operations within a single application. In other embodiments, the various components described may be separate applications that execute independently.

[0150] In some embodiments, portions of the application development system 128 may be executed by other devices. For example, a server device may be used to compile the code into executable code. In another example, another device may be used to execute the application code and the test code. Such an example may be for a device with a dedicated processor or hardware on which the application code is intended to be used.

[0151] Many embodiments of the application development system or various components in the application development system 128 may have a graphical user interface. The graphical user interface can be used to browse code, write and edit code, and select from many different functions of each component.

[0152] In some implementations, some or all of the portions of application development system 128 may be executed using a command line or other interface.

[0153] Many application development systems 128 may have an editor 130, a compiler 132, and an execution environment 134. The editor 130 may allow a user to view, write, and edit code, including application code and test code. The compiler 132 may compile source code into executable code that may be executed using the execution environment 134.

[0154] The execution environment 134 may be similar to running compiled code in an operating system environment. The execution environment 134 may include some debugging and tracing functionality that may provide detailed information about executing code that is not available in an operating system environment.

[0155] A set of testing components 138 can be used to develop and test application code 140. Testing components 138 can include various components for creating and analyzing test code 142. Test code 142 can test application code 140 to reveal any defects in application code 140 and ensure that application code 140 performs as expected.

[0156] Some embodiments may include a test code generation module 144 that can create various test code elements from the application code 140. For example, the test code generation module 144 can create unit tests that can test commands or functions within the application code. A unit test can test a small portion of the application code 140, such as a short function, subroutine, or individual command.

[0157] In some embodiments, the test code generation module 144 can create portions of test code that can be modified or changed by a programmer to fully test that portion of the application code. In such embodiments, the test code generation module 144 can create a shell of a test, such as a unit test, and the programmer can perform some editing or write a small portion of the test code to complete the test.

[0158] The test code 142 may include unit tests that can test an isolated portion of the application 140. In large applications, hundreds or even thousands of unit tests may be created to test various parts of the application. Unit tests can ensure that each small portion of the application code 140 performs as expected and operates within a single class without the need for external components.

[0159] Test code 142 may include integration tests, where the various software modules or components within application code 140 are combined and tested as a group. Integration tests can test a large subset of application code 140 in a single test. In many cases, multiple integration tests can be created to test different usage scenarios or test cases. Integration tests may include testing communication between processes and other events.

[0160] Test code 142 may also include system tests, which test application code 140 as a single unit. System tests may test application code 140 at a high level in various usage scenarios. Many such tests may treat application code 140 as a 'black box,' providing input and expecting specific outputs. System tests may include performance testing, functional testing, error handling testing, load testing, stress testing, reliability testing, recovery and failover testing, and many other types of testing.

[0161] Test code 142 can be tailored to suit the type of application code 140. One example might be software that operates in an aircraft's aviation computer. Such software might operate when human life is at risk, and test code 142 can be crafted with great detail and rigor. In another example, a simple gaming application might have specific quality standards that the game manufacturer might want to maintain before shipping. In yet another example, a software application that processes health records or other personal information might be rigorously tested for security reasons.

[0162] The test code evaluator 146 can analyze the test code 142 using the test strategy 148 to determine a quality assessment of the test code 142. The test code evaluator 146 can analyze the test code 142 to find assertions, descriptions within assertions, rank complexity, dependencies, and other factors. From these and other analyses, quality test code health metrics can be created for each individual test and for the entire test code 142.

[0163] Quality test code health metrics can be displayed with each of the individual tests within the test code 142. The health metrics can be used to indicate which tests meet and which tests do not meet the criteria defined in the test strategy 148. Those tests that do not meet these criteria can be flagged for improvement or further development.

[0164] Test policy 148 may define best practices or testing standards that may be used during the development of application code 140. Test policy 148 may define rules that are applied by test code evaluator 146 to determine whether test code 142 complies with best practices.

[0165] In some embodiments, test strategy 148 may include several sets of rules for various test criteria. These rules can be selected by the programmer, allowing the programmer to focus on specific aspects of test code 142. For example, the programmer may select assertion-related tests and may not select other metrics. In this case, test code evaluator 146 may analyze test code 142 to find a subset of tests that are suitable for assertions, without analyzing other types of potential problems.

[0166] A set of test policies 148 may be created for different types of application software being developed and for internal management by a programming team or company. The test policies 148 may define various characteristics of the test code that the test code evaluator 146 may use to verify compliance. In some cases, different types of application code 140 may determine the test policies 148.

[0167] Application code evaluator 152 can evaluate application code 140 in a similar manner as test code 142. Code design policy 154 can describe the analysis that application code evaluator 152 performs on application code 140.

[0168] Application code evaluator 152 can examine application code 140 to determine compliance with code design policies 154. Code design policies 154 can include policies related to how application code 140 is structured, such as defining modularity policies, commenting and documentation policies, checking of input and output parameters of functions, and many other types of analysis.

[0169] In some implementations, test code evaluator 146 and application code evaluator 152 may be the same application or executable code, but may use test policies 148 and code design policies 154 to perform different types of inspection and analysis on test code 142 and application code 140 , respectively.

[0170] The outputs of test code evaluator 146 and application code evaluator 152 can be combined to produce a quality metric for the test results. For example, the completeness or coverage of a set of test code can be determined by analyzing the functions in application code 140 and matching the functions with test code 142 to determine whether all application functions have matching test routines.

[0171] In another example, the test results generated by the test code 142 can be qualified by the quality of the test code 142. For example, test code with a poor quality test code health metric can adversely affect the overall evaluation of the application code. In this example, poorly constructed test code 142 that executes on the application code 140 can generate a test result. The test result can be displayed next to the quality test code health metric, which can indicate that the test code 142 does not meet the basic standards defined in the test strategy 148. Such an indication can negate the test result. Conversely, a positive quality test code health metric can give a high degree of confidence to the test result.

[0172] The test code executor 150 can execute the test code 142 against the application code 140 to generate test results. Some embodiments can execute the test code in a manner that includes various debugging features that can collect various monitoring information during test execution. The debugging features can be turned on and off within the application development system 128 and can be present or absent when the application is executed outside of the application development system 128.

[0173] The application development system 128 may include a user interface 136 that may be used to display quality test code health metrics along with other quality metrics.

[0174] The program code 140 shown in FIG1 may include code to be tested, such as the first code in the embodiment of the present application. The program in which the first code is located may be a newly developed program or an updated program provided by an engineer or programmer. Before the program in which the first code is located is incorporated into an actual application (e.g., deployed in a product, released to the public, etc.), it may be necessary to verify the program in which the first code is located in order to discover whether there are any defects in the program in which the first code is located. The test component 138 may perform program testing on the program in which the first code is located in order to verify the program in which the first code is located, for example, the first code may be tested.

[0175] In the process of testing the first code, generating the test code is a very important step. The test code generation module 144 can generate the corresponding test code based on the information related to the code to be tested. In the process of generating the test code based on the language model, the test code generation module 144 needs to generate the test code by calling the language model. As shown in Figure 1, the training sample generation module 161 can construct the training samples of the language model, and the model training module can train the language model based on the training samples obtained by the training sample generation module 161 (for example, pre-training). Then, the test code generation module 144 can generate the test code based on the trained language model.

[0176] This application focuses on introducing the test code generation module 144 (or it can be called the test code generation module), the training sample generation module 161 (or it can be called the training sample generation module) and the model training module 162 (or it can be called the model training module).

[0177] Figure 1 introduces the architecture of an embodiment of the present application. The following example illustrates the deployment form of the test code generation module 144. In some examples, the above-mentioned test code generation module 144 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the test code generation module can also be deployed in different environments, for example, it can be deployed in the cloud and edge in an edge-cloud collaborative manner. The following example illustrates the deployment of the test code generation module in an edge-cloud collaborative manner.

[0178] Refer to FIG2a for a schematic diagram of the deployment form of a test code generation module. As shown in FIG2a, the test code generation module can be subdivided into a context extraction module, a prompt construction module, a large language model processing module, an instruction forwarding module, etc. according to different functions. The deployment of the test code generation module may involve hardware devices in the cloud and hardware devices at the edge. The hardware devices in the cloud usually have rich computing power and huge storage capacity, such as servers and databases. The hardware devices at the edge have certain reasoning capabilities and storage capabilities, such as industrial computers and artificial intelligence edge computing devices. In some possible implementations, the deployment of the test code generation module also involves terminals, which may include mobile phones, self-driving cars, industrial quality inspection systems and other IoT devices.

[0179] In Example 1 of Figure 2a, the test code generation module 144 can be deployed on the end side or the edge side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (first code) is located. The prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, edge side or cloud side) according to the prompt obtained by the prompt construction module, and present the test code, for example, through the user interface 136. The interaction module can be used to receive the test instructions (including the instruction information of the first code) input by the user and to visually present the test code. For example, the interaction module can provide an interactive interface to the user. The interactive interface can be a graphical user interface (GUI) or a command user interface (CUI). The user can enter the test instructions through an interactive interface such as a GUI or CUI. For example, the interactive interface can display the test code.

[0180] In Example 2 of FIG2a , the context extraction module and prompt construction module in the test code generation module 144 can be deployed on the device side or the edge side, and the large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (first code) is located. The prompt construction module can construct a prompt based on the context information of the first code as input to the language model. The constructed prompt can be transmitted to the large language model processing module via the network 154. The large language model processing module can generate test code based on the prompt obtained by the prompt construction module by calling the language model (which can be deployed on the device side, the edge side, or the cloud side), and return the test code to the interaction module on the device side or the edge side to present the test code.

[0181] In Example 3 of FIG2a , the context extraction module in the test code generation module 144 can be deployed on the end side or the edge side, and the prompt construction module and the large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (first code) is located, and can transmit the context information of the first code to the prompt construction module via the network 154. The prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side, or the cloud side) based on the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side to present the test code.

[0182] In Example 4 of FIG2a , the prompt construction module in the test code generation module 144 can be deployed on the end side or the edge side, and the prompt obtained by the prompt construction module can be used to instruct the extraction of context information of the first code. The prompt construction module in the test code generation module 144 (different from the prompt construction module located on the end side) and the large language model processing module can be deployed on the edge side or the cloud side. The large language model processing module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located through the prompt (equivalent to implementing the function of the context extraction module). Then, the prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate test code by calling the language model (which can be deployed on the end side, the edge side, or the cloud side) based on the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side to present the test code.

[0183] In Example 5 of Figure 2a, the instruction forwarding module in the test code generation module 144 can be deployed on the end side or the edge side, and the context extraction module, prompt construction module, and large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The instruction forwarding module can transmit the test instruction to the context extraction module via the network 154. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (first code) is located. The prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side, or the cloud side) based on the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side to present the test code.

[0184] It should be noted that Figure 2a is only an example of the deployment form of the test code generation module. In other possible implementations of this application, the test code generation module may also adopt other deployment methods, and this application does not limit this.

[0185] The following is an example of the deployment form of the training sample generation module 161. In some examples, the above-mentioned training sample generation module 161 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the training sample generation module 161 can also be deployed in different environments, for example, it can be deployed in the cloud and edge in an edge-cloud collaborative manner. The following is an example of the deployment of the training sample generation module 161 on the cloud side.

[0186] Figure 2b shows a schematic diagram of a training sample generation module deployment. As shown in Figure 2b, the deployment of the training sample generation module can involve both cloud-side hardware devices and edge-side hardware devices. Cloud-side hardware devices typically have extensive computing power and storage capacity, such as servers and databases. Edge-side hardware devices have certain inference and storage capabilities, such as industrial computers and artificial intelligence edge computing devices.

[0187] In the example of Figure 2b, the training sample generation module 161 can be deployed on the cloud side. The end side can pass the code set to the test code generation module 144 through the network 154. The test code generation module 144 can extract training samples that can be used as language models from the code set. The training samples can include the true value of the code to be tested and the corresponding test code. In addition, the test code generation module 144 can also obtain the context information of the code to be tested and the context information of the test code from the code set to enhance the training samples. In addition, other post-processing can be performed on the training samples to improve the quality of the training samples. The test code generation module 144 can pass the obtained training samples to the end side through the network 154, or directly train the language model on the cloud side.

[0188] The following examples illustrate the deployment form of the training module 162. In some examples, the above-mentioned training module 162 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the training module 162 can also be deployed in different environments, for example, it can be deployed in the cloud and edge in an edge-cloud collaborative manner. The following example illustrates the deployment of the training module 162 on the cloud side.

[0189] Figure 2c shows a schematic diagram of a training module deployment. As shown in Figure 2c, the deployment of a training module can involve both cloud-based hardware and edge-side hardware. Cloud-based hardware typically has extensive computing power and storage capabilities, such as servers and databases. Edge-side hardware has certain inference and storage capabilities, such as industrial computers and AI edge computing devices.

[0190] In the example of Figure 2c, the training module 162 can be deployed on the cloud side. The client side can transmit the training samples to the test code generation module 144 through the network 154. The test code generation module 144 can train the language model based on the training samples to obtain a trained language model.

[0191] From a product perspective, the test code testing module can be provided to users as a plug-in. During the developer testing phase, software developers can select the business project to be tested in the code editor, and the code editor plug-in automatically generates test code containing semi-structured data. Specifically, the code editor plug-in's workflow includes: front-end interaction, program analysis, data pre-processing, test code generation, test code post-processing, result return, and user feedback.

[0192] From the product form, the test code testing module, training sample generation module and model training module can provide calling services in the form of cloud service capabilities through API structure and provide them to other testing tools, so as to automatically generate unit test code, execute unit test code and output test results.

[0193] From the product form, the test code test module can be executed in the form of command line. By executing the corresponding commands in the command line, it can automatically analyze the business code, extract context information, and automatically call the test code generation model to obtain executable unit test code and automatically execute to obtain test output.

[0194] Based on the test code generation module shown in Figures 1 and 2a, the present application also provides a code testing method, which is introduced below.

[0195] Referring to the flowchart of the code testing method shown in FIG3a, the method can be executed by the test code generation module. The method includes:

[0196] 301. Receive a test instruction for a first code, where the first code is code in a program to be edited, and the test instruction indicates generation of test code through a language model; obtain information about the first code from a file associated with the first code, where the information about the first code includes context information of functions included in the first code, and one or more of the following information: context information of the file associated with the first code, context information of the project where the first code is located, or context information of a test framework used by the first code.

[0197] In a possible implementation, the interaction module may receive a test instruction for the first code. At this time, the context extraction module in the test code generation module may obtain context information of the first code.

[0198] In a possible implementation, when a test instruction for a first code in a code editing interface is received, information of the first code may be obtained.

[0199] In a possible implementation, the test instruction may instruct to generate test code using a language model.

[0200] In a possible implementation, the context information of the first code may be obtained from the file where the first code is located.

[0201] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code. In other words, the first code can be unit tested.

[0202] Next, the context information of the first code is introduced.

[0203] The context information of the first code may be context information at multiple levels, including but not limited to function level, file level, function level, and framework level.

[0204] In a possible implementation, the context information of the function included in the first code includes at least one of the types of the input parameters of the included function, the type of the function return value, and the method call in the function body.

[0205] In a possible implementation, the context information of the file associated with the first code includes: at least one of: a variable name, structure information, an object name, an object definition, a class member variable, a class member function, a class construction method, and a class inheritance relationship of the file associated with the first code.

[0206] In a possible implementation, the context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language.

[0207] In a possible implementation, the context information of the test framework used by the first code includes: a category of the test framework used by the first code, or a category of the mock framework.

[0208] 302. Obtain a first test code corresponding to the first code based on information about the first code, wherein the information about the first code is used to construct a prompt word prompt as an input of a language model, and the first test code is obtained through the language model based on the prompt word prompt.

[0209] In one possible implementation, context information of the first code may be sent to a cloud-side computing device, and a first test code corresponding to the first code, obtained by the cloud-side computing device based on the context information of the first code, may be received. The cloud-side computing device may obtain the first test code corresponding to the first code using a language model based on the information of the first code.

[0210] In a possible implementation, a first test code corresponding to the first code may be obtained through a language model according to information of the first code.

[0211] In one possible implementation, as shown in Figure 3b, a first prompt word prompt containing information about the first code can be obtained. For example, a prompt generation module in a test code generation module can construct a first prompt based on context information of the first code. The first prompt word is used to instruct the language model to generate a test code for the first code based on the information of the first code. Then, the large language model processing module can obtain a first test code corresponding to the first code based on the first prompt word.

[0212] In one possible implementation, the first prompt word can be sent to a computing device on the cloud side. The computing device on the cloud side can obtain a first test code corresponding to the first code based on the first prompt word through a language model, and the terminal side can receive the first test code corresponding to the first code obtained by the computing device on the cloud side.

[0213] In a possible implementation, the terminal side obtains a first test code corresponding to the first code through a language model according to the first prompt word.

[0214] In embodiments of the present application, the large language model can be capable of processing multiple coding languages, that is, generating test code for multiple coding languages, that is, generating corresponding test code for the code to be tested in multiple different coding language types. For example, information about a second code can be obtained; the second code and the first code are in different programming languages; based on the second prompt word, a second test code is obtained, wherein the first test code and the first code are in the same programming language, and the second test code and the second code are in the same programming language.

[0215] In one possible implementation, test samples in different coding languages ​​can be constructed during training to enable the trained language model to have the ability to process multiple coding languages, that is, the ability to generate test codes for multiple coding languages.

[0216] In an embodiment of the present application, the method of the embodiment corresponding to FIG. 3a may also be a feedforward process in the language model training process. In a reverse update process, the language model may be updated based on the first test code and a third test code corresponding to the first code, with the third test code serving as the test code truth value of the first code. For example, a prompt containing the first test code and the third test code corresponding to the first code may be constructed to guide the language model update.

[0217] Based on the test code generation module shown in Example 4 in Figure 1 and Figure 2a, the present application also provides a code testing method, which is introduced below.

[0218] Referring to the flowchart of the code testing method shown in FIG4 , the method can be executed by the test code generation module. The method includes:

[0219] 401. Receive a test instruction for a first code, where the first code is code in a program to be edited, and the test instruction instructs generation of test code through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code based on a file of the first code, and generate test code for the first code based on the information of the first code; the information of the first code includes context information of functions included in the first code, and one or more of the following information: context information of files associated with the first code, context information of files associated with the first code, context information of the project where the first code is located, or context information of a test framework used by the first code.

[0220] Different from the embodiment corresponding to FIG. 3 a , a prompt can be used to guide the language model to extract context information.

[0221] In one possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameters of the included function, the type of the function return value, and the method call in the function body; or, the context information of the file associated with the first code includes: at least one of the variable name, structure information, object name, object definition, class member variables, class member functions, class constructor, and class inheritance relationship of the file associated with the first code; or, the context information of the project where the first code is located includes: statistical information or the type of programming language of the project where the first code is located; or, the context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0222] 402. Obtain a first test code corresponding to the first code according to the second prompt word.

[0223] In one possible implementation, the second prompt word can be sent to a computing device on the cloud side. The computing device on the cloud side can obtain a first test code corresponding to the first code based on the second prompt word through a language model, and return the first test code to the end side. The end side can then receive the first test code corresponding to the first code obtained based on the second prompt word.

[0224] In a possible implementation, a first test code corresponding to the first code may be obtained according to the second prompt word through a language model.

[0225] In an embodiment of the present application, the method of the embodiment corresponding to FIG. 4 may also be a feedforward process of the language model training process. In a reverse update process, the language model may be updated based on the first test code and a third test code corresponding to the first code, with the third test code serving as the test code truth value of the first code. For example, a prompt containing the first test code and the third test code corresponding to the first code may be constructed to guide the language model update.

[0226] Based on the training sample generation module shown in FIG1 and FIG2a, the present application also provides a training sample generation method, and the training sample generation method of the present application is introduced below.

[0227] Referring to the flowchart of the training sample generation method shown in FIG5 , the method can be executed by the training sample generation module. The method includes:

[0228] 501. Get code set;

[0229] For example, the code set may be a code file, and embodiments of the present application may extract training samples of a language model from the code set.

[0230] 502. According to the code set, a training sample of a language model is obtained, wherein the training sample includes a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, wherein the constraints include constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship, and the third test code is used as the true value of the test code of the first code.

[0231] The training samples extracted from the code set may include the code to be tested (e.g., the first code) and the true value of the test code corresponding to the first code (e.g., the third test code). In order to extract high-quality training samples, the embodiment of the present application is based on a preset test framework and extracts codes that meet the constraints specified by the test framework from the code set as training samples.

[0232] The test framework is used to indicate the constraints of the test function that can be used as the function to be tested. The constraints include the constraints of the test function itself and the constraints of the calling relationship between the test function and the function to be tested.

[0233] Due to different code language types and even different code writing methods of the same language, different code sets may be suitable for different test frameworks. The so-called "test framework adapted to the code set" here can be understood as the constraints that need to be met by the function to be tested and the test function under the language and writing method of the code set.

[0234] After the code set is obtained in step 501, a test framework that is compatible with the code set (that is, the first test framework in the embodiment of the present application) can be determined from the preset multiple test frameworks. The test framework feature library of the above-mentioned preset multiple test frameworks can be constructed offline. By constructing the test framework feature library offline, the scope of data collection is expanded, and the diversity of training data can be increased. And the versatility and scalability are enhanced, and it can be widely used in various test model training and reasoning in multiple languages ​​(such as Java, Python, Go, Cxx, JS, Ts). When extracting training samples, automatic analysis can be performed without the need to manually define templates and constraints, reducing the test cost and making it more practical and versatile.

[0235] Next, we will introduce the test framework and how to extract training samples based on the test framework:

[0236] In one possible implementation, the test framework may include constraints on the test code itself, as well as constraints on the calling relationship between the code to be tested and the test code.

[0237] Optionally, the constraint on the test code itself may be that the function type needs to be the type of the test function. For example, whether a function is a test function may be determined by testing the macro name.

[0238] Optionally, the constraints on the test code itself can be constraints on the function test suite name, such as whether the test suite is unique, for example, it can be determined by whether the first parameter in text() is unique.

[0239] Optionally, the constraints of the calling relationship may include at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0240] In one possible implementation, a third test code that satisfies the constraints of the test function itself can be obtained from the code set; when the first code is called by the third test code in the code set and the calling relationship satisfies the constraints of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0241] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationships satisfy the constraints of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0242] In one possible implementation, a third test code that satisfies the constraints of the test function itself can be obtained from the code set; the third test code includes an assertion; a target variable included in the assertion is determined; and when the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraints of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0243] Assertions are Boolean expressions used to debug programs and verify that certain tested functions are functioning correctly (i.e., comparing actual values ​​to expected values). Assertions are judgments about execution structure, not business processes. Assertions are the most fundamental component of unit testing, acting like an if() statement: if the assertion is met, the program is executed; if it is not, an error is thrown. Each type of assertion has two forms: one that accepts a message parameter, such as "static public void assertTrue(String message, boolean condition)," which contains a warning message when an error occurs; the other type does not. For example, a programmer can trust that an expression will evaluate to true at a specific point in the program. Assertions can be enabled and disabled at any time, enabling them during testing and disabling them during deployment. Once the program is running, users can re-enable assertions if they encounter problems. Using assertions can create more stable, high-quality, and less error-prone code. When you need to interrupt the current operation when a value is FALSE, use an assertion. Unit tests must use assertions (Junit / JunitX).

[0244] Exemplarily, referring to FIG7 , FIG7 is a schematic diagram of a construction process of a training sample, in which the feature rules corresponding to the framework can be queried and matched based on the call chain plus rules. However, due to the diversity of the code, various situations will be encountered during the matching process, such as multiple matches (i.e., a test function contains calls to multiple functions), which cannot be solved by only heuristic methods. Therefore, for this situation, a method based on matching features + scoring sorting is superimposed. For the heuristic rule matching algorithm in the test-source mapping model, it combines the forward matching from the source function to the test function, and the reverse matching from the test function to the source function. As shown in FIG6 , in the forward matching, the source function is first filtered by the general rule, and the function containing the test code is retained; then the function call stack is obtained, and the test code is judged and matched by the call type; in the reverse matching, the test assertion is first filtered and the call relationship is captured. If it is a one-to-many call, the rule judgment based on the call relationship classification is also entered to filter the source functions that meet the conditions.

[0245] Referring to FIG8 , FIG8 is a schematic diagram of a training sample construction process, wherein, first, a general project-level parsing regardless of language can be performed by constructing an abstract syntax tree or code graph, and then the call chain within the project code is further obtained. By defining filtering rules for source functions containing test code, all source functions and test functions that meet the rules are captured. The mapping of source / test functions is obtained through associations such as call relationships and naming methods. A forward and reverse matching method based on heuristic rules is adopted. The forward matching method uses the function call stack relationship as the main rule, supplemented by path matching rules and file type rules to match functions and test code. The reverse matching method extracts function return values ​​from the assertions in the test code, and then extracts the source function to be tested corresponding to the function return value from the test code function body. In addition, a multi-factor weighted priority calculation method and sorting mechanism are introduced. Taking C / C++ collection as an example, due to the characteristics of the GTest collection framework, the test code of functions in the same source file has the same test suite name. A model is used to convert the unit test code function and source function into vectors, and then the source function vectors are sorted. The function with the highest ranking after reordering is the function to be tested. The sorting strategy is to sort the unit test code according to the likelihood of the tested functions; after converting the unit test code functions and source functions into vectors, the similarity between them is directly calculated, and the function with the highest similarity is the function to be tested; after traversing and assigning values, the common information regardless of the language, namely the multi-level definition results of functions, files, projects, adaptation frameworks, etc., is stored in the database for training large models.

[0246] Refer to Figure 9, which illustrates the overall process of training sample construction, model training, and inference. This process includes a candidate code pre-processing module based on software analysis. After the user selects a function to be tested, the pre-processing module performs dynamic context analysis and data enhancement on the current code. This process then constructs rich prompt information. The data enhancement pre-processing method includes: extracting contextual information from the test class of the function under test, enabling the large model to recognize and utilize information such as variable naming in the context, thus avoiding incorrect variable naming; extracting the test framework and mock framework required for test code generation; extracting information such as member variables and member functions in the parameter and return classes of the function under test; performing program branch analysis and usage; and adding corresponding test code of similar source functions as reference. The training data optimization module then combines software analysis techniques to filter, statically check, and pre-process the training data to obtain the final code set. Specific data filtering rules include: 1) filtering data whose code lines for the method under test fall within the closed interval [4, 100]; and 2) ensuring that the proportion of English letters in all characters in the method under test is at least a given threshold. For example, 25%; 3) The percentage of numeric characters in the method under test does not exceed a given threshold, such as 90%. Next, the unit test code generation module based on the large model can be entered: for the candidate code selected by the user, prompts are constructed based on the information obtained after pre-processing, and then fed into the test generation model as input. The test code generated using the large model is then sent to the post-processing module for sorting and screening to obtain the final test code.

[0247] 10 , FIG10 illustrates a front-end interface when the test code generation module is used as an extended function or plug-in of a code editor or IDE.

[0248] In the specific implementation, the test code generation and test-assisted generation result screening are completed by interacting with the user and the test generation plug-in. The opportunities for human-computer interaction can include: (1) actively triggering project-level context analysis and test code generation by selecting the code content of the method to be tested and then using the right-click menu or shortcut keys; (2) triggering project-level context analysis and test code generation during the question-and-answer dialogue.

[0249] After triggering, you can further modify the setting options, including confirming the tested class and method. After the user confirms the setting information, the test code generation is triggered. The front-end constructs the information parsed by the tested code into a prompt and sends it as a request to the back-end. The back-end sends the prompt to the large model for inference, and then returns the candidate results returned by multiple large models to the front-end. When the test code is generated, it will be automatically opened and parallel to the current code editor. The test generation results will be presented to the user in the form of multiple code snippets in the sidebar.

[0250] From the model perspective, human-computer interaction is carried out in the prompt format. From the user perspective, after the use case generation results are presented, the user can directly accept the use case by copying the use case to an existing test file or creating a new test file and importing the use case. The user can also click the post-processing module to repair the generated test code snippet based on whether it meets expectations, and then determine whether to adopt the use case. The plug-in can have multiple possible implementation forms in terms of interaction.

[0251] Based on the aforementioned code testing method, the present application also provides a code testing device. As shown in FIG11 , the code testing device 1100 includes:

[0252] An interactive module, used for receiving test instructions input by a user;

[0253] An acquisition module 1101 (for example, which may include the context extraction module and prompt generation module described in the above embodiment) is configured to acquire information about the first code, where the information about the first code includes context information about functions included in the first code, and one or more of the following information: context information about files associated with the first code, context information about files associated with the first code, context information about a project in which the first code is located, or context information about a test framework used by the first code.

[0254] The processing module 1102 (for example, it may include the prompt generation module and the large language model processing module introduced in the above embodiment) is used to obtain the first test code corresponding to the first code according to the information of the first code.

[0255] Exemplarily, the acquisition module 1101 and the processing module 1102 may be implemented by hardware or software.

[0256] When implemented through software, the interaction module, acquisition module 1101, and processing module 1102 may be applications running on a computing device (such as a server). For example, processing module 1102 may be a computing engine running on a computing device. The application may be provided to users as a virtualized service. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. A VM service may be a service that uses virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) to provide users with VMs on demand. A BMS service is a service that creates a virtual BMS resource pool on multiple physical hosts to provide users with BMSs on demand. A container service is a service that creates a virtual container resource pool on multiple physical hosts to provide users with containers on demand. A VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance, elastically scalable computing service with computing performance comparable to that of a traditional physical machine and featuring secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user spaces, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0257] When implemented via hardware, the interaction module and acquisition module 1101 can be implemented via a transceiver or other transceiver module. The processing module 1102 can include at least one computing device, such as a server. Alternatively, the processing module 1102 can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0258] For a detailed introduction to the acquisition module 1101 , reference may be made to the detailed introduction to steps 301 , 401 , and 501 in the above embodiment, and similarities will not be repeated here.

[0259] For a detailed description of the processing module 1102 , reference may be made to the detailed description of steps 302 , 402 , and 502 in the above embodiment, and similarities will not be repeated here.

[0260] This application also provides a computing device 1200. As shown in Figure 12, computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. Processor 1204, memory 1206, and communication interface 1208 communicate with each other via bus 1202. Computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1200.

[0261] Bus 1202 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG12 shows a single bus line, but this does not imply a single bus or type of bus. Bus 1202 may include a path for transmitting information between various components of computing device 1200 (e.g., memory 1206, processor 1204, and communication interface 1208).

[0262] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0263] The memory 1206 may include a volatile memory, such as a random access memory (RAM). The memory 1206 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned code testing method. Specifically, the memory 1206 stores a test code generation module for executing instructions of the code testing method.

[0264] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or a communication network.

[0265] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0266] As shown in Figure 13, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same test code generation module for executing instructions of the code testing method.

[0267] In some possible implementations, one or more computing devices 1200 in the computing device cluster may also be used to execute some of the instructions of the test code generation module for executing the code testing method. In other words, the combination of one or more computing devices 1200 may jointly execute the instructions of the test code generation module for executing the code testing method.

[0268] It should be noted that the memories 1206 in different computing devices 1200 in the computing device cluster may store different instructions for executing part of the functions of the test code generation module.

[0269] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned code generation module for executing the code testing method, the training sample generation method, and the model training method.

[0270] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned code testing method, training sample generation method, and model training method.

[0271] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A code testing method, characterized in that: The method comprises: receiving a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction instructs generation of a test code by using a language model; Acquire information of the first code from a file associated with the first code, where the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; Obtaining a first test code corresponding to the first code according to information of the first code, wherein the information of the first code is used to construct a prompt word (prompt) as an input of a language model, and the first test code is obtained by the language model according to the prompt word (prompt); The first code is tested according to the first test code.

2. The method according to claim 1, characterized in that The context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or, The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variables of a class, member functions of a class, a constructor of a class, and an inheritance relationship of a class of the file associated with the first code; or The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or, The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

3. The method according to claim 1 or 2, characterized in that: The obtaining, according to the information of the first code, a first test code corresponding to the first code includes: sending information of the first code, and receiving a first test code corresponding to the first code obtained according to the information of the first code; or, According to the information of the first code, a first test code corresponding to the first code is obtained through a language model.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the information of the first code includes: obtaining a first prompt containing the information of the first code; the first prompt is used to instruct the language model to generate a test code of the first code according to the information of the first code; The obtaining, according to the information of the first code, a first test code corresponding to the first code includes: sending the first prompt word and receiving a first test code corresponding to the first code obtained according to the first prompt word; or, According to the first prompt word, a first test code corresponding to the first code is obtained through a language model.

5. The method according to any one of claims 1 to 4, characterized in that: The first code is a function-level code, a class-level code or a file-level code.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: A test instruction for the first code is received on a code editing interface, wherein the test instruction instructs to generate a language model-based test code for the first code.

7. The method according to any one of claims 1 to 6, characterized in that: The first test code is obtained through a language model according to information of the first code, and the method further includes: Acquire information of a second code; the second code and the first code are in different types of programming languages; According to the information of the second code, a second test code is obtained through the language model, wherein the first test code and the The first code is in the same type of programming language, and the second test code and the second code are in the same type of programming language.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

9. The method according to claim 8, characterized in that The method further comprises: Get code set; According to the code set, a training sample of the language model is obtained, the training sample includes the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, the constraints include constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship.

10. The method according to claim 9, characterized in that The step of obtaining a training sample of a language model according to the code set includes: Acquire a third test code from the code set that satisfies the constraint of the test function itself; When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

11. The method according to claim 9, characterized in that The step of obtaining a training sample of a language model according to the code set includes: Acquire a third test code satisfying the constraints of the test function itself from the code set; the third test code includes assertions; determining a target variable included in the assertion; When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

12. A code testing method, characterized in that: The method comprises: receiving a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction instructs generation of a test code by using a language model; Obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to the file of the first code, and generate test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; According to the second prompt word, obtaining a first test code corresponding to the first code; The first code is tested according to the first test code.

13. A code testing device, characterized in that: The device comprises: an acquisition module, configured to receive a test instruction for a first code, wherein the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; and to acquire information about the first code from a file associated with the first code, wherein the information about the first code includes context information about a function included in the first code, and one or more of the following information: context information about a file associated with the first code, context information about a project where the first code is located, or context information about a test framework used by the first code, wherein the information about the first code is used to construct a prompt word prompt as an input of the language model, and the first test code is obtained through the language model according to the prompt word prompt; The processing module is used to obtain a first test code corresponding to the first code according to information of the first code; and test the first code according to the first test code.

14. The device according to claim 13, characterized in that The context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or, The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variable of a class, member function of a class, construction device of a class, and inheritance relationship of a class of the file associated with the first code; or, The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or, The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

15. The device according to claim 13 or 14, characterized in that The acquisition module is specifically used for: When a test instruction for a first code in a code editing interface is received, information of the first code is acquired, wherein the test instruction instructs to generate a test code based on a language model for the first code.

16. The device according to any one of claims 13 to 15, characterized in that The first test code is obtained through a language model according to information of the first code, and the acquisition module is further used to: Acquire information of a second code; the second code and the first code are in different types of programming languages; The processing module is further used to obtain a second test code through the language model according to information of the second code, wherein the first test code and the first code are in the same type of programming language, and the second test code and the second code are in the same type of programming language.

17. The device according to any one of claims 13 to 16, characterized in that The processing module is further used for: The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

18. The device according to claim 17, characterized in that The acquisition module is further used to: acquire a code set; The processing module is further used to: obtain a training sample of a language model according to the code set, the training sample including the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, the constraints including constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship.

19. A code testing device, characterized in that: The device comprises: An acquisition module is used to receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to a file of the first code, and generate a test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; The processing module is used to obtain a first test code corresponding to the first code according to the second prompt word; and test the first code according to the first test code.

20. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 12.

21. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 12.

Images