Code testing method and related system

Abstract

The invention provides a code testing method, which is applied to a code testing system, when a test code is generated, context information of a to-be-tested code is extracted, the test code is generated based on the context information of the to-be-tested code, and since the context information often has a business meaning, a test scene can be reflected, and the test efficiency is improved. The prior of business and test scenes is realized in the test code generation process, so that the test code with higher quality can be generated.

Description

[0001] This application is required to be submitted to the State Intellectual Property Office of China on November 16, 2023, with application number 202311544456.X and invention name

[0002] The priority of the Chinese patent application "A data processing method and related equipment" is incorporated into the present application in its entirety by reference. Technical Field

[0003] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a code testing method, system, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0004] Code testing can ensure the quality of software engineering, and code testing is a very important part of software development. During the development process, time is mainly allocated to four parts: code writing, code repair, code refactoring, and code testing. Code testing takes up more than 15% of the time. Developers need to spend a lot of time and energy to write test code. In order to save development time, the application of automatic test code generation technology has been increasingly valued in recent years.

[0005] A large language model is a machine learning model with a large parameter scale. The test generation technology based on the large language model is mainly based on the code set related to the code (including the code to be tested). The large language model is used to generate the test code corresponding to the code to be tested, and the code set related to the code is used to train the weight parameters of the large model. Although the test generation technology based on the natural language large model is not very interpretable, the technology's ability to abstract code features, generate rich use cases, and integrate inputs of different modalities are incomparable to traditional algorithms.

[0006] However, in the prior art, during the test code generation process, the sample input into the language model is the entire code of the code file related to the code to be tested, which makes the language model's analysis of the code data very rough, resulting in poor quality of the generated test code. Summary of the invention

[0007] The present application provides a code testing method, which makes the quality of the generated test code higher. The present application also provides a code testing system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0008] In a first aspect, the present application provides a code testing method. The method is applied to a code testing system. The code testing system may be a software system, which may be deployed in a computing device cluster, such as a cloud computing cluster provided by a cloud service provider, or an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the code testing method of the present application. In some possible implementations, the code testing system may also be a hardware system, such as a computing device cluster with an unknown test code generation function. When the hardware system is running, the code testing method of the present application is executed.

[0009] Specifically, the method includes: receiving a test instruction for a first code, the first code is a code in a program to be edited, and the test instruction indicates the generation of a test code through a language model; obtaining information about the first code from a file associated with the first code, the information about the first code including context information of a function included in the first code, and one or more of the following information: context information of the file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code, wherein the information of the first code is used to construct a prompt word prompt as an input of a language model, and the first test code is obtained through the language model according to the prompt word prompt; obtaining a first test code corresponding to the first code according to the information of the first code; and testing the first code according to the first test code.

[0010] Based on this, when generating test code, this application extracts context information related to the code to be tested. Since context information often has business meanings and can reflect test scenarios, the priors of business and test scenarios are available during the generation of test code, thereby generating higher quality test code. On the other hand, in addition to the function-level context, this application also extracts coarser-grained context information than the function level. By generating test code through multi-level context information, the priors of business and test scenarios can be richer, thereby making the quality of the generated test code higher.

[0011] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameters of the included function, the function return value type, and the method call in the function body.

[0012] In a possible implementation, the context information of the file associated with the first code includes: at least one of: a variable name, structure information, an object name, an object definition, a member variable of a class, a member function of a class, a construction method of a class, and an inheritance relationship of a class of the file associated with the first code.

[0013] In a possible implementation, the context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language.

[0014] In a possible implementation, the context information of the test framework used by the first code includes: a category of the test framework used by the first code, or a category of the mock framework.

[0015] In a possible implementation, a first prompt word for instructing a language model to generate a test code for the first code based on information of the first code can be sent to a computing device on the cloud side. The computing device on the cloud side can obtain a first test code corresponding to the first code through the language model based on the first prompt word, and the terminal side can receive the first test code corresponding to the first code obtained by the computing device on the cloud side.

[0016] In a possible implementation, a computing device on the terminal side or the cloud side may obtain a first test code corresponding to the first code according to the first prompt word through a language model.

[0017] In a possible implementation, obtaining the information of the first code includes: obtaining a first prompt word prompt containing the information of the first code; the first prompt word is used to instruct a language model to generate a test code for the first code according to the information of the first code; obtaining a first test code corresponding to the first code according to the information of the first code includes: sending the first prompt word, receiving a first test code corresponding to the first code obtained according to the first prompt word; or obtaining the first test code corresponding to the first code through a language model according to the first prompt word.

[0018] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code.

[0019] In a possible implementation, obtaining the information of the first code includes: upon receiving a test instruction for the first code in the code editing interface, obtaining the information of the first code, wherein the test instruction indicates to generate a language model-based test code for the first code.

[0020] In an embodiment of the present application, the large language model may have the ability to process multiple code languages, that is, the ability to generate test codes for multiple code languages, that is, to generate corresponding test codes for the codes to be tested of multiple different code language types. For example, the information of the second code can be obtained; the second code and the first code are different types of programming languages; according to the information of the second code, a second test code is obtained through a language model, wherein the first test code and the first code are the same type of programming language, and the second test code and the second code are the same type of programming language. Among them, similar to the first code, the information of the second code may include context information of the function included in the second code, and at least one of the following information: context information of the file associated with the second code, context information of the project where the second code is located, or context information of the test framework used by the second code.

[0021] In a possible implementation, test samples of different code languages ​​may be constructed during training to enable the trained language model to have the ability to process multiple code languages, that is, the ability to generate test codes for multiple code languages.

[0022] In a possible implementation, the method further includes: updating the language model according to the first test code and a third test code corresponding to the first code, the third test code being used as a test code true value of the first code.

[0023] In a second aspect, the present application provides a code testing method, the method comprising:

[0024] Receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to a file of the first code, and generate a test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code;

[0025] A first test code corresponding to the first code is obtained according to the second prompt word, and the first code is tested according to the first test code.

[0026] Different from the embodiment corresponding to the first aspect, the language model can be guided to extract context information through prompt.

[0027] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or,

[0028] The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variables of a class, member functions of a class, a constructor of a class, and an inheritance relationship of a class of the file associated with the first code; or

[0029] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or,

[0030] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0031] In a possible implementation, obtaining, according to the second prompt word, a first test code corresponding to the first code includes:

[0032] sending the second prompt word, and receiving a first test code corresponding to the first code obtained according to the second prompt word; or,

[0033] According to the second prompt word, a first test code corresponding to the first code is obtained through a language model.

[0034] In a third aspect, the present application provides a sample providing method, the method comprising: acquiring a code set; obtaining a training sample of a language model based on the code set, the training sample comprising a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, the constraints comprising constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship, and the third test code is used as the true value of the test code of the first code.

[0035] The training samples extracted from the code set may include the code to be tested (e.g., the first code) and the true value of the test code corresponding to the first code (e.g., the third test code). In order to extract high-quality training samples, the embodiment of the present application is based on a preset test framework, and extracts codes that meet the constraints specified by the test framework from the code set as training samples.

[0036] The test framework is used to indicate constraints on a test function that can be used as a function to be tested, wherein the constraints include constraints on the test function itself and constraints on the calling relationship between the test function and the function to be tested.

[0037] Due to different code language types and even different code writing methods of the same language, different code sets may be suitable for different test frameworks. The so-called "test framework adapted to the code set" here can be understood as the constraints that need to be met by the function to be tested and the test function under the language and writing method of the code set.

[0038] After obtaining the code set, a test framework that is compatible with the code set can be determined from the preset multiple test frameworks (that is, the first test framework in the embodiment of the present application). The test framework feature library that the above-mentioned preset multiple test frameworks can construct offline can expand the scope of data collection by constructing the test framework feature library offline, thereby increasing the diversity of training data. And the versatility and ductility are enhanced, and it can be widely used in various test model training and reasoning in multiple languages ​​(such as Java, Python, Go, Cxx, JS, Ts). When extracting training samples, automatic analysis can be performed without the need to manually define templates and constraints, reducing the test cost, and being more practical and versatile.

[0039] In a possible implementation, the constraint of the calling relationship includes at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0040] In a possible implementation, obtaining a training sample of a language model according to the code set includes:

[0041] Acquire a third test code from the code set that satisfies the constraint of the test function itself;

[0042] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0043] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationship satisfies the constraint of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0044] In a possible implementation, obtaining a training sample of a language model according to the code set includes:

[0045] Acquire a third test code satisfying the constraints of the test function itself from the code set; the third test code includes assertions;

[0046] determining a target variable included in the assertion;

[0047] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0048] In a possible implementation, the method further includes:

[0049] The language model is updated according to the first test code and the third test code.

[0050] In a fourth aspect, the present application provides a code testing device, the device comprising:

[0051] an acquisition module, configured to receive a test instruction for a first code, wherein the first code is a code in a program to be edited, and the test instruction indicates generation of a test code through a language model; and to acquire information of the first code from a file associated with the first code, wherein the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project in which the first code is located, or context information of a test framework used by the first code, wherein the information of the first code is used to construct a prompt word prompt as an input of the language model, and the first test code is obtained through the language model according to the prompt word prompt;

[0052] The processing module is used to obtain a first test code corresponding to the first code according to information of the first code.

[0053] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or,

[0054] The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variable of a class, member function of a class, construction device of a class, and inheritance relationship of a class of the file associated with the first code; or,

[0055] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or,

[0056] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0057] In a possible implementation, the processing module is specifically configured to:

[0058] sending information of the first code, and receiving a first test code corresponding to the first code obtained according to the information of the first code; or,

[0059] According to the information of the first code, a first test code corresponding to the first code is obtained through a language model.

[0060] In a possible implementation, the obtaining the information of the first code includes: obtaining a first prompt word prompt containing the information of the first code; the first prompt word is used to instruct the language model to generate a test code of the first code according to the information of the first code;

[0061] The processing module is specifically used for:

[0062] sending the first prompt word and receiving a first test code corresponding to the first code obtained according to the first prompt word; or,

[0063] According to the first prompt word, a first test code corresponding to the first code is obtained through a language model.

[0064] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code.

[0065] In a possible implementation, the acquisition module is specifically used to:

[0066] When a test instruction for a first code in a code editing interface is received, information of the first code is acquired, wherein the test instruction instructs to generate a test code based on a language model for the first code.

[0067] In a possible implementation, the acquisition module is further used to:

[0068] Acquire information of a second code; the second code and the first code are in different types of programming languages;

[0069] The processing module is further configured to obtain a second test code according to the second prompt word, wherein the first test code and the first code are in the same type of programming language, and the second test code and the second code are in the same type of programming language.

[0070] In a possible implementation, the processing module is further configured to:

[0071] The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

[0072] In a possible implementation, the acquisition module is further used to: acquire a code set;

[0073] The processing module is further used to obtain a training sample of a language model based on the code set, wherein the training sample includes the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, wherein the constraints include constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship.

[0074] In a possible implementation, the processing module is specifically used to: obtain a third test code that satisfies the constraint of the test function itself from the code set;

[0075] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0076] In a possible implementation, the processing module is specifically used to: obtain a third test code that satisfies the constraints of the test function itself from the code set; the third test code includes an assertion;

[0077] determining a target variable included in the assertion;

[0078] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0079] In a fifth aspect, the present application provides a code testing device, the device comprising:

[0080] An acquisition module is used to receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to a file of the first code, and generate a test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code;

[0081] A processing module is used to obtain a first test code corresponding to the first code according to the second prompt word.

[0082] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or,

[0083] The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variable of a class, member function of a class, construction device of a class, and inheritance relationship of a class of the file associated with the first code; or,

[0084] The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or,

[0085] The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0086] In a possible implementation, the processing module is specifically configured to:

[0087] sending the second prompt word, and receiving a first test code corresponding to the first code obtained according to the second prompt word; or,

[0088] According to the second prompt word, a first test code corresponding to the first code is obtained through a language model.

[0089] In a sixth aspect, the present application provides a sample providing device, the device comprising:

[0090] An acquisition module, used to acquire a code set;

[0091] A processing module is used to obtain a training sample of a language model according to the code set, wherein the training sample includes a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, wherein the constraints include constraints of the test function itself and constraints of a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship, and the third test code is used as the true value of the test code of the first code.

[0092] In a possible implementation, the constraint of the calling relationship includes at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0093] In a possible implementation, the processing module is specifically configured to:

[0094] Acquire a third test code from the code set that satisfies the constraint of the test function itself;

[0095] When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0096] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationship satisfies the constraint of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0097] In a possible implementation, the processing module is specifically configured to:

[0098] Acquire a third test code satisfying the constraints of the test function itself from the code set; the third test code includes assertions;

[0099] determining a target variable included in the assertion;

[0100] When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0101] In a possible implementation, the processing module is further configured to:

[0102] The language model is updated according to the first test code and the third test code.

[0103] In a seventh aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the method as described in the first aspect or any implementation of the first aspect, the method as described in the second aspect or any implementation of the second aspect, and the method as described in the third aspect or any implementation of the third aspect.

[0104] In an eighth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the code testing method described in the first aspect or any one of the implementations of the first aspect, the method described in the second aspect or any one of the implementations of the second aspect, or the method described in the third aspect or any one of the implementations of the third aspect.

[0105] In the ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or a computing device cluster to execute the code testing method described in the first aspect or any one of the implementations of the first aspect, the method described in the second aspect or any one of the implementations of the second aspect, or the method described in the third aspect or any one of the implementations of the third aspect.

[0106] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0108] Figure 1 A schematic diagram of the architecture of a code testing system provided in an embodiment of the present application;

[0109] Figure 2a A schematic diagram of the architecture of a code testing system provided in an embodiment of the present application;

[0110] Figure 2b A schematic diagram of the architecture of a model training system provided in an embodiment of the present application;

[0111] Figure 2c A schematic diagram of the architecture of a model training system provided in an embodiment of the present application;

[0112] Figure 3aA schematic diagram of a code testing method provided in an embodiment of the present application;

[0113] Figure 3b A schematic diagram of a code testing method provided in an embodiment of the present application;

[0114] Figure 4 A schematic diagram of a code testing method provided in an embodiment of the present application;

[0115] Figure 5 A schematic diagram of a method for providing a training sample provided in an embodiment of the present application;

[0116] Figure 6 A schematic diagram of a code testing method provided in an embodiment of the present application;

[0117] Figure 7 A schematic diagram of a code testing method provided in an embodiment of the present application;

[0118] Figure 8 A schematic diagram of a code testing method provided in an embodiment of the present application;

[0119] Fig. 9 A schematic diagram of a code testing method provided in an embodiment of the present application;

[0120] Fig.10 A schematic diagram of a test tool interface provided in an embodiment of the present application;

[0121] Figures 11 to 13 A schematic diagram of a code testing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0122] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0123] First, some technical terms involved in the embodiments of the present application are introduced.

[0124] Code testing can ensure the quality of software engineering, and code testing is a very important part of software development. During the development process, time is mainly allocated to four parts: code writing, code repair, code refactoring, and code testing. Code testing takes up more than 15% of the time. Developers need to spend a lot of time and energy to write test code. In order to save development time, the application of automatic test code generation technology has been increasingly valued in recent years.

[0125] There are many tools for automatically generating test code. Different technologies require different information related to the code to be tested to output the corresponding test code. The more commonly used existing technologies include: random test generation, symbolic execution-based test generation, search-based test generation, and large language model-based test generation.

[0126] A large language model is a machine learning model with a large parameter scale. The test generation technology based on the large language model is mainly based on the code set related to the code (including the code to be tested). The large language model is used to generate the test code corresponding to the code to be tested, and the code set related to the code is used to train the weight parameters of the large model. Although the test generation technology based on the natural language large model is not very interpretable, the technology's ability to abstract code features, generate rich use cases, and integrate inputs of different modalities are incomparable to traditional algorithms.

[0127] However, in the prior art, during the test code generation process, the sample input into the language model is the entire code of the code file related to the code to be tested, which makes the language model's analysis of the code data very rough, resulting in poor quality of the generated test code.

[0128] In view of this, the present application provides a code testing method. The method can be executed by a code testing system (or can be referred to as a test code generator). The code testing system is used to generate corresponding test codes based on the code to be tested. The code testing system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system, thereby executing the code testing method of the present application. The software system can also be integrated into software with a test code generation function, for example, it can be in the form of functional modules such as plug-ins in the software. Alternatively, the software system can also be independent of the software with the test code generation function, and the software with the test code generation function calls the software system to implement the test code generation function. The software system can be deployed on a computing device, for example, it is deployed on a terminal device such as a desktop, a laptop, a tablet computer, a smart phone (also referred to as a terminal). The computing device executes the program code of the software system, thereby executing the code testing method of the present application. In some possible implementations, the code testing system can also be a hardware system, for example, a computing device cluster with a test code generation function. When the hardware system is running, the code testing method of the present application is executed.

[0129] Specifically, when generating test code, the context information of the code to be tested is extracted. Since the context information often has business meaning and can reflect the test scenario, the test code generation process has prior knowledge of the business and test scenarios, thereby generating higher quality test code.

[0130] In particular, when context information is used as a prompt for the input of a large language model, the test method names and variable names generated by the large language model can have business meanings, reflect the test scenarios, and enable the large model to generate valid test input initial values, thereby improving the quality of the generated test code.

[0131] In addition, the present application provides a training sample providing method. The method can be performed by a training sample providing system. The training sample providing system is used to identify training samples that can be used as language models from a code set, and the training samples include source code and corresponding code to be tested. The training sample providing system can be a software system, and the software system can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system to execute the training sample providing method of the present application. In some possible implementations, the training sample providing system can also be a hardware system, for example, a computing device cluster with a training sample providing function. When the hardware system is running, the training sample providing method of the present application is executed.

[0132] In addition, the present application provides a model training method. The method can be performed by a model training system. The model training system is used to identify training samples that can be used as language models from a code set, and the training samples include source code and corresponding code to be tested. The model training system can be a software system, which can be deployed in a computing device cluster, for example, in a cloud computing cluster provided by a cloud service provider, or in an edge computing cluster. The computing device cluster executes the program code of the software system to execute the model training method of the present application. In some possible implementations, the model training system can also be a hardware system, for example, a computing device cluster with a model training function. When the hardware system is running, the model training method of the present application is executed.

[0133] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the present application is introduced below with reference to the accompanying drawings.

[0134] Figure 1 is an embodiment 100 showing system code development (e.g. Figure 1 Implementation 100 is a simplified example of a device that can be used to write, edit, test, and debug computer executable code.

[0135] Figure 1 The diagrams of the various functional components of the system are shown. In some cases, the components can be hardware components, software components, or a combination of hardware and software. Some components can be application layer software, while other components can be operating system layer components. In some cases, the connection from one component to another component can be a close connection, where two or more components operate on a single hardware platform. In other cases, the connection can be made through a network connection across long distances. Each embodiment can use different hardware, software, and interconnection architectures to implement the described functions.

[0136] Embodiment 100 is an example of a system that can be used to develop software, firmware, or other executable code. The final product developed on the system of embodiment 100 can be referred to as application code. As used in this specification and claims, application code is a software product tested by test code. As used in the embodiments of the present application, test code is software for testing application code. Generally speaking, test code is not transmitted with application code and is not used when executing the application.

[0137] A software development system may be used to generate test code, and the test code may then be automatically evaluated to determine quality metrics for the test code. In many cases, the test code may be manually authored and then evaluated to generate quality metrics. Analysis of the test code may determine various features or aspects of the test code so that quality metrics may be established.

[0138] Using various analyses that can be performed on the test code, the quality metrics can be used to determine how 'good' the test code is. The quality metrics can be used to identify parts of the test code that may have defects and give some measure of confidence in the test results.

[0139] For example, high-quality metrics may be given to test code that scores high on the analysis. When executed against application code, the results for the test code may be considered a good representation of the quality of the application code. Conversely, results for test code with poor quality metrics may be problematic.

[0140] Figure 1 The modules in the code development system 100 shown may belong to an independent computing device or a separate computing device, for example, one part belongs to one computing device and the other part belongs to another computing device. Furthermore, the modules may be further divided into different sub-modules, and multiple sub-modules may collaborate to implement the functions of the corresponding modules.

[0141] The computing device may be a general purpose computer (or a cluster of computing devices on the cloud side) having hardware components 104 and software components 106. The computing device may have several tools that may be used to develop application code and may be used by a programmer or application developer.

[0142] In some implementations, a computing device may be similar to a device on which an application is intended to be executed. Thus, a computing device may represent a personal computer or other similar device.

[0143] In other embodiments, the computing device may be a development platform for software or firmware that can be executed on different devices. For example, the computing device may be used to develop and test software that can be executed on another type of computer device, such as a cell phone or other device with a computer processor. In such embodiments, the computing device may include an emulator or simulator that simulates the operation of the intended device. Such an emulator may allow the application code to be tested and executed before being installed and run on the intended device hardware.

[0144] The hardware component 104 may include a processor 108 that may use a random access memory 110 and a non-volatile storage 112. The hardware component 104 may have a network interface 114 and a user interface 116.

[0145] In the example of implementation 100, hardware component 104 may represent a general-purpose personal computer. In other implementations, hardware component 104 may be a desktop computer or a server computer. Some implementations may be portable devices such as laptop computers, notebook computers, or even cell phones, personal digital assistants, or other devices.

[0146] The software components 106 may include an operating system 118 upon which a number of different types of software may operate.

[0147] Executable code 120 may be any type of code that can be directly executed within an operating system. In some cases, executable code 120 may be machine code that may include instructions that can be directly executed by processor 108. In other cases, executable code 120 may be assembly code that can be executed after being processed by an assembler.

[0148] Some software components may be defined using intermediate code 122. Intermediate code 122 may be source code that is compiled into an intermediate representation and then may be further compiled using a just-in-time compiler 124 and executed by an execution engine 126. Intermediate code 122 is useful in managed code applications or in applications where several different programming languages ​​may be used.

[0149] The application development system 128 may have many of the elements used to write, edit, test, debug, and publish applications. In the example of implementation 100, the features of the application development system 128 may be shown as part of a larger application or programming environment where a programmer can perform many different operations in a single application. In other implementations, the components described may be separate applications that execute independently.

[0150] In some embodiments, portions of the application development system 128 may be performed by other devices. For example, a server device may be used to compile the code into executable code. In another example, another device may be used to execute the application code and the test code. Such an example may be directed to a device having a dedicated processor or hardware on which the application code is intended to be used.

[0151] Many embodiments of the application development system or various components in the application development system 128 may have a graphical user interface. The graphical user interface can be used to browse code, write and edit code, and select from many different functions of each component.

[0152] In some implementations, some or all of the portions of application development system 128 may be executed using a command line or other interface.

[0153] Many application development systems 128 may have an editor 130, a compiler 132, and an execution environment 134. The editor 130 may allow a user to view, write, and edit code, including application code and test code. The compiler 132 may compile source code into executable code that may be executed using the execution environment 134.

[0154] The execution environment 134 may be similar to running compiled code in an operating system environment. The execution environment 134 may include some debugging and tracing capabilities that may provide detailed information about executing code that is not available in an operating system environment.

[0155] A set of test components 138 may be used to develop and test application code 140. Test components 138 may include various components for creating and analyzing test code 142. Test code 142 may test application code 140 to reveal any defects in application code 140 and ensure that application code 140 performs as expected.

[0156] Some embodiments may have a test code generation module 144 that can create various test code elements from the application code 140. For example, the test code generation module 144 can create unit tests that can test commands or functions within the application code. A unit test can test a small portion of the application code 140, such as a short function, a subroutine, or an individual command.

[0157] In some embodiments, the test code generation module 144 can create portions of test code that can be modified or changed by a programmer to fully test that portion of the application code. In such embodiments, the test code generation module 144 can create a shell of a test such as a unit test, and the programmer can perform some editing or write a small portion of the test code to complete the test.

[0158] The test code 142 may include unit tests that can test an isolated portion of the application 140. In large applications, hundreds or even thousands of unit tests may be created to test various portions of the application. Unit tests can ensure that small portions of the application code 140 perform as expected and operate within a single class and without the need for external components.

[0159] Test code 142 may include integration tests, where various software modules or components within application code 140 may be combined and tested as a group. Integration tests may test a large subset of application code 140 in a single test. In many cases, multiple integration tests may be created to test different usage scenarios or test cases. Integration tests may include testing communications between processes as well as other events.

[0160] Test code 142 may also include system tests, which may test application code 140 as a single unit. System tests may test application code 140 at a high level in various usage scenarios. Many such tests may treat application code 140 as a 'black box', providing inputs and expecting specific outputs. System tests may include performance tests, functional tests, error handling tests, load tests, stress tests, reliability tests, recovery and failover tests, and many other types of tests.

[0161] The test code 142 may be tailored to suit the type of application code 140. One example may be software that operates in an aviation computer in an airplane. Such software may operate when human life is at risk, and the test code 142 may be very detailed and rigorously crafted. In another example, a simple gaming application may have specific quality standards that the game manufacturer may wish to maintain before shipping. In yet another example, a software application that processes health records or other personal information may be rigorously tested for security reasons.

[0162] The test code evaluator 146 can analyze the test code 142 using the test strategy 148 to determine a quality assessment of the test code 142. The test code evaluator 146 can analyze the test code 142 to find assertions, descriptions within assertions, rank complexity, dependencies, and other factors. From these and other analyses, quality test code health metrics can be created for each individual test as well as for the test code 142 as a whole.

[0163] Quality test code health metrics may be displayed with each of the individual tests within the test code 142. The health metrics may be used to indicate which tests meet and which tests do not meet the criteria defined in the test strategy 148. Those tests that do not meet these criteria may be flagged for improvement or further development.

[0164] Test policy 148 may define best practices or testing standards that may be used during development of application code 140. Test policy 148 may define some rules applied by test code evaluator 146 to determine whether test code 142 complies with best practices.

[0165] In some embodiments, the test strategy 148 may include several sets of rules for various test criteria. These rules may be selected by the programmer so that the programmer can focus on specific aspects of the test code 142. For example, the programmer may select assertion-related tests and may not select other metrics. In this case, the test code evaluator 146 may analyze the test code 142 to find a subset of tests that are suitable for assertions without analyzing other types of potential problems.

[0166] A set of test strategies 148 may be created for different types of application software being developed and for internal management of a programming team or company. Test strategies 148 may define various characteristics of test code that test code evaluators 146 may use to verify compliance. In some cases, different types of application code 140 may determine test strategies 148.

[0167] Application code evaluator 152 can evaluate application code 140 in a similar manner as test code 142. Code design strategy 154 can describe the analysis that application code evaluator 152 performs on application code 140.

[0168] Application code evaluator 152 can examine application code 140 to determine compliance with code design policies 154. Code design policies 154 can include policies related to how application code 140 is structured, such as defining modularity policies, commenting and documentation policies, checking of input and output parameters of functions, and many other types of analysis.

[0169] In some implementations, test code evaluator 146 and application code evaluator 152 may be the same application or executable code, but test strategies 148 and code design strategies 154 may be used to perform different types of checks and analyses on test code 142 and application code 140 , respectively.

[0170] The outputs of test code evaluator 146 and application code evaluator 152 may be combined to produce a quality metric for the test results. For example, the completeness or coverage of a set of test code may be determined by analyzing the functions in application code 140 and matching the functions with test code 142 to determine whether all application functions have matching test routines.

[0171] In another example, the test results generated by the test code 142 can be qualified by the quality of the test code 142. For example, test code with a poor quality test code health metric can adversely affect the overall assessment of the application code. In this example, a poorly constructed test code 142 that executes on the application code 140 can generate a test result. The test result can be displayed next to the quality test code health metric, which can indicate that the test code 142 does not meet the basic standards defined in the test strategy 148. Such an indication can negate the test result. In contrast, a positive quality test code health metric can give a high degree of confidence to the test result.

[0172] The test code executor 150 can execute the test code 142 on the application code 140 to generate test results. Some embodiments can execute the test code in a manner that includes various debugging features that can collect various monitoring information during test execution. The debugging features can be turned on and off within the application development system 128, and can be present or absent when the application is executed outside the application development system 128.

[0173] The application development system 128 may include a user interface 136 that may be used to display quality test code health metrics along with other quality metrics.

[0174] in, Figure 1 The program code 140 shown may include code to be tested, such as the first code in the embodiment of the present application. The program where the first code is located may be a newly developed program or an updated program provided by an engineer or programmer. Before the program where the first code is located is incorporated into an actual application (e.g., deployed in a product, released to the public, etc.), it may be necessary to verify the program where the first code is located in order to find out whether there are any defects in the program where the first code is located. The test component 138 may perform a program test on the program where the first code is located in order to verify the program where the first code is located, for example, the first code may be tested.

[0175] In the process of testing the first code, generating test code is an important step. The test code generation module 144 can generate corresponding test code based on the information related to the code to be tested. In the process of generating test code based on the language model, the test code generation module 144 needs to generate test code by calling the language model, such as Figure 1 As shown, the training sample generation module 161 can construct training samples of the language model, and the model training module can train the language model (for example, pre-training) based on the training samples obtained by the training sample generation module 161. Furthermore, the test code generation module 144 can generate test code based on the trained language model.

[0176] This application focuses on introducing the test code generation module 144 (or can be called the test code generation module), the training sample generation module 161 (or can be called the training sample generation module) and the model training module 162 (or can be called the model training module).

[0177] Figure 1 The architecture of the embodiment of the present application is introduced. The deployment form of the test code generation module 144 is illustrated below. In some examples, the above-mentioned test code generation module 144 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the test code generation module can also be deployed in different environments, for example, it can be deployed in the cloud and the edge in an edge-cloud collaborative manner. The following is an example of the deployment of the test code generation module in an edge-cloud collaborative manner.

[0178] See also Figure 2a A schematic diagram of a deployment form of a test code generation module is shown, such as Figure 2aAs shown, the test code generation module can be subdivided into context extraction module, prompt building module, large language model processing module, instruction forwarding module, etc. according to different functions. The deployment of the test code generation module may involve hardware devices in the cloud and hardware devices at the edge. The hardware devices in the cloud usually have rich computing power and huge storage capacity, such as servers and databases. The hardware devices at the edge have certain reasoning and storage capabilities, such as industrial computers and artificial intelligence edge computing devices. In some possible implementations, the deployment of the test code generation module also involves terminals, which may include mobile phones, self-driving cars, industrial quality inspection systems and other IoT devices.

[0179] exist Figure 2a In Example 1, the test code generation module 144 can be deployed on the end side or the edge side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located, and the prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side or the cloud side) according to the prompt obtained by the prompt construction module, and present the test code, for example, through the user interface 136. The interaction module can be used to receive the test instructions (including the indication information of the first code) input by the user and to perform a visual presentation of the test code. For example, the interaction module can provide an interactive interface to the user. The interactive interface can be a graphical user interface (GUI) or a command user interface (CUI). The user can enter the test instruction through an interactive interface such as a GUI or a CUI. For example, the interactive interface can display the test code.

[0180] exist Figure 2aIn Example 2, the context extraction module and the prompt construction module in the test code generation module 144 can be deployed on the end side or the edge side, and the large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located, and the prompt construction module can construct a prompt as the input of the language model based on the context information of the first code. The constructed prompt can be transmitted to the large language model processing module through the network 154. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side or the cloud side) according to the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side, and present the test code.

[0181] exist Figure 2a In Example 3, the context extraction module in the test code generation module 144 can be deployed on the end side or the edge side, and the prompt construction module and the large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The context extraction module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located, and the context information of the first code can be transmitted to the prompt construction module through the network 154. The prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side or the cloud side) according to the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side, and present the test code.

[0182] exist Figure 2aIn Example 4, the prompt construction module in the test code generation module 144 can be deployed on the end side or the edge side, and the prompt obtained by the prompt construction module can be used to indicate the extraction of context information of the first code. The prompt construction module in the test code generation module 144 (different from the prompt construction module located on the end side) and the large language model processing module can be deployed on the edge side or the cloud side. The large language model processing module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located through prompt (equivalent to realizing the function of the context extraction module), and then the prompt construction module can construct prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side or the cloud side) according to the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side, and present the test code.

[0183] exist Figure 2a In Example 5, the instruction forwarding module in the test code generation module 144 can be deployed on the end side or the edge side, and the context extraction module, prompt construction module, and large language model processing module in the test code generation module 144 can be deployed on the edge side or the cloud side. The instruction forwarding module can pass the test instruction to the context extraction module through the network 154, and the context extraction module can extract the context information of the first code from the file 140 where the code to be tested (the first code) is located. The prompt construction module can construct a prompt based on the context information of the first code as the input of the language model. The large language model processing module can generate the test code by calling the language model (which can be deployed on the end side, the edge side, or the cloud side) according to the prompt obtained by the prompt construction module, and return the test code to the interaction module on the end side or the edge side, and present the test code.

[0184] It should be noted that Figure 2a This is only an example of the deployment form of the test code generation module. In other possible implementations of the present application, the test code generation module may also adopt other deployment methods, which is not limited in the present application.

[0185] The following is an example of the deployment form of the training sample generation module 161. In some examples, the above-mentioned training sample generation module 161 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the training sample generation module 161 can also be deployed in different environments, for example, it can be deployed in the cloud and the edge in an edge-cloud collaborative manner. The following is an example of the deployment of the training sample generation module 161 on the cloud side.

[0186] See also Figure 2b A schematic diagram of a deployment form of a training sample generation module is shown, such as Figure 2b As shown, the deployment of the training sample generation module may involve cloud hardware devices and edge hardware devices. Cloud hardware devices usually have rich computing power and huge storage capacity, such as servers and databases, while edge hardware devices have certain reasoning and storage capabilities, such as industrial computers and artificial intelligence edge computing devices.

[0187] exist Figure 2b In the example, the training sample generation module 161 can be deployed on the cloud side. The end side can pass the code set to the test code generation module 144 through the network 154. The test code generation module 144 can extract training samples that can be used as language models from the code set. The training samples may include the true value of the code to be tested and the corresponding test code. In addition, the test code generation module 144 can also obtain the context information of the code to be tested and the context information of the test code from the code set to enhance the training samples. In addition, other post-processing can be performed on the training samples to improve the quality of the training samples. The test code generation module 144 can pass the obtained training samples to the end side through the network 154, or directly train the language model on the cloud side.

[0188] The following is an example of the deployment form of the training module 162. In some examples, the above-mentioned training module 162 can be deployed entirely in the cloud (cloud environment), or entirely on the edge (edge ​​environment), or on the terminal. In other examples, the training module 162 can also be deployed in different environments, for example, it can be deployed in the cloud and the edge in an edge-cloud collaborative manner. The following is an example of the deployment of the training module 162 on the cloud side.

[0189] See also Figure 2c A schematic diagram of a deployment form of a training module is shown, such as Figure 2cAs shown, the deployment of training modules can involve cloud hardware devices and edge hardware devices. Cloud hardware devices usually have rich computing power and huge storage capacity, such as servers and databases, while edge hardware devices have certain reasoning and storage capabilities, such as industrial computers and artificial intelligence edge computing devices.

[0190] exist Figure 2c In the example of , the training module 162 can be deployed on the cloud side. The terminal side can transmit the training sample to the test code generation module 144 through the network 154, and the test code generation module 144 can train the language model according to the training sample to obtain a trained language model.

[0191] From the product form, the test code test module can serve users in the form of a plug-in. In the developer testing phase, software developers can select the business project to be tested in the code editor, and the code editor plug-in automatically generates test code containing semi-structured data. Specifically, the workflow of the code editor plug-in includes: front-end interaction phase, program analysis phase, data pre-processing phase, test code generation phase, test code post-processing phase, result return phase, and user feedback phase.

[0192] From the product form, the test code testing module, training sample generation module and model training module can provide calling services in the form of cloud service capabilities through API structures and provide them to other testing tools, thereby automatically generating unit test code, executing unit test code, and outputting test results.

[0193] From the product form, the test code test module can be executed in the form of command line. By executing the corresponding commands in the command line, it can automatically analyze the business code, extract context information, and automatically call the test code generation model to obtain executable unit test code and automatically execute to obtain test output.

[0194] based on Figure 1 and Figure 2a The test code generation module shown, the present application also provides a code testing method, the code testing method of the present application is introduced below.

[0195] See also Figure 3a The flowchart of the code testing method shown in FIG. 1 can be executed by a test code generation module. The method includes:

[0196] 301. Receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates generation of a test code through a language model; obtain information about the first code from a file associated with the first code, where the information about the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code.

[0197] In a possible implementation, the interaction module may receive a test instruction for the first code. At this time, the context extraction module in the test code generation module may obtain context information of the first code.

[0198] In a possible implementation, when a test instruction for a first code in a code editing interface is received, information of the first code may be acquired.

[0199] In a possible implementation, the test instruction may instruct to generate the test code by using a language model.

[0200] In a possible implementation, the context information of the first code may be acquired from the file where the first code is located.

[0201] In a possible implementation, the first code is a function-level code, a class-level code, or a file-level code. That is, the first code can be unit tested.

[0202] Next, the context information of the first code is introduced.

[0203] The context information of the first code may be context information at multiple levels, which may include but are not limited to: function level, file level, function level, and framework level.

[0204] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameters of the included function, the function return value type, and the method call in the function body.

[0205] In a possible implementation, the context information of the file associated with the first code includes: at least one of: a variable name, structure information, an object name, an object definition, a member variable of a class, a member function of a class, a construction method of a class, and an inheritance relationship of a class of the file associated with the first code.

[0206] In a possible implementation, the context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language.

[0207] In a possible implementation, the context information of the test framework used by the first code includes: a category of the test framework used by the first code, or a category of the mock framework.

[0208] 302. Obtain a first test code corresponding to the first code according to information of the first code, wherein the information of the first code is used to construct a prompt word prompt as an input of a language model, and the first test code is obtained through the language model according to the prompt word prompt.

[0209] In a possible implementation, the context information of the first code may be sent to a computing device on the cloud side, and a first test code corresponding to the first code obtained by the computing device on the cloud side according to the context information of the first code may be received. The computing device on the cloud side may obtain the first test code corresponding to the first code through a language model according to the information of the first code.

[0210] In a possible implementation, a first test code corresponding to the first code may be obtained through a language model according to information of the first code.

[0211] In one possible implementation, refer to Figure 3b As shown, a first prompt word prompt containing information of the first code can be obtained. For example, a prompt generation module in a test code generation module can construct a first prompt through context information of the first code; the first prompt word is used to instruct a language model to generate a test code for the first code according to information of the first code; and then a large language model processing module can obtain a first test code corresponding to the first code according to the first prompt word.

[0212] In a possible implementation, the first prompt word can be sent to a computing device on the cloud side. The computing device on the cloud side can obtain a first test code corresponding to the first code based on the first prompt word through a language model, and the terminal side can receive the first test code corresponding to the first code obtained by the computing device on the cloud side.

[0213] In a possible implementation, the terminal side obtains a first test code corresponding to the first code through a language model according to the first prompt word.

[0214] In the embodiment of the present application, the large language model can have the ability to process multiple code languages, that is, the ability to generate test codes for multiple code languages, that is, to generate corresponding test codes for the codes to be tested of multiple different code language types. For example, the information of the second code can be obtained; the second code and the first code are different types of programming languages; according to the second prompt word, the second test code is obtained, wherein the first test code and the first code are the same type of programming language, and the second test code and the second code are the same type of programming language.

[0215] In a possible implementation, test samples of different code languages ​​may be constructed during training to enable the trained language model to have the ability to process multiple code languages, that is, the ability to generate test codes for multiple code languages.

[0216] In the embodiment of the present application, Figure 3a The method of the corresponding embodiment may also be a feedforward process of the training process of the language model. In the reverse update process, the language model may be updated according to the first test code and the third test code corresponding to the first code, and the third test code is used as the test code true value of the first code. For example, a prompt including the first test code and the third test code corresponding to the first code may be constructed to guide the language model to be updated.

[0217] based on Figure 1 and Figure 2a The test code generation module shown in Example 4, the present application also provides a code testing method, the code testing method of the present application is introduced below.

[0218] See also Figure 4 The flowchart of the code testing method shown in FIG. 1 can be executed by a test code generation module. The method includes:

[0219] 401. Receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates generation of a test code through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to a file of the first code, and generate test code for the first code according to the information of the first code; the information of the first code includes context information of functions included in the first code, and one or more of the following information: context information of files associated with the first code, context information of files associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code.

[0220] and Figure 3aThe difference in the corresponding embodiments is that the language model can be guided to extract context information through prompt.

[0221] In a possible implementation, the context information of the function included in the first code includes: at least one of the type of the input parameters of the included function, the type of the function return value, and the method call in the function body; or, the context information of the file associated with the first code includes: at least one of the variable name, structure information, object name, object definition, class member variables, class member functions, class construction method, and class inheritance relationship of the file associated with the first code; or, the context information of the project where the first code is located includes: statistical information or the type of programming language of the project where the first code is located; or, the context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

[0222] 402. Obtain a first test code corresponding to the first code according to the second prompt word.

[0223] In one possible implementation, the second prompt word can be sent to a computing device on the cloud side. The computing device on the cloud side can obtain a first test code corresponding to the first code based on the second prompt word through a language model, and return the first test code to the end side. The end side can then receive the first test code corresponding to the first code obtained based on the second prompt word.

[0224] In a possible implementation, a first test code corresponding to the first code may be obtained according to the second prompt word through a language model.

[0225] In the embodiment of the present application, Figure 4 The method of the corresponding embodiment may also be a feedforward process of the training process of the language model. In the reverse update process, the language model may be updated according to the first test code and the third test code corresponding to the first code, and the third test code is used as the test code true value of the first code. For example, a prompt including the first test code and the third test code corresponding to the first code may be constructed to guide the language model to be updated.

[0226] based on Figure 1 and Figure 2a The training sample generation module shown in the figure, the present application also provides a training sample generation method, and the training sample generation method of the present application is introduced below.

[0227] See also Figure 5 The flowchart of the training sample generation method shown in FIG. 1 can be performed by a training sample generation module. The method includes:

[0228] 501. Get code set;

[0229] For example, the code set may be a code file, and the embodiments of the present application may extract training samples of the language model from the code set.

[0230] 502. According to the code set, a training sample of a language model is obtained, wherein the training sample includes a first code and a third test code extracted from the code set, wherein the code set is adapted to a first test framework, and the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, wherein the constraints include constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and a calling relationship between the third test code and the first code satisfies the constraints of the calling relationship, and the third test code is used as the truth value of the test code of the first code.

[0231] The training samples extracted from the code set may include the code to be tested (e.g., the first code) and the true value of the test code corresponding to the first code (e.g., the third test code). In order to extract high-quality training samples, the embodiment of the present application is based on a preset test framework, and extracts codes that meet the constraints specified by the test framework from the code set as training samples.

[0232] The test framework is used to indicate constraints on a test function that can be used as a function to be tested, wherein the constraints include constraints on the test function itself and constraints on the calling relationship between the test function and the function to be tested.

[0233] Due to different code language types and even different code writing methods of the same language, different code sets may be suitable for different test frameworks. The so-called "test framework adapted to the code set" here can be understood as the constraints that need to be met by the function to be tested and the test function under the language and writing method of the code set.

[0234] After the code set is obtained in step 501, a test framework (that is, the first test framework in the embodiment of the present application) that is compatible with the code set can be determined from the preset multiple test frameworks. The test framework feature library that can be constructed offline by the preset multiple test frameworks can expand the scope of data collection by constructing the test framework feature library offline, thereby increasing the diversity of training data. And the versatility and ductility are enhanced, and it can be widely used in various test model training and reasoning in multiple languages ​​(such as Java, Python, Go, Cxx, JS, Ts). When extracting training samples, automatic analysis can be performed without the need to manually define templates and constraints, reducing the test cost, and being more practical and versatile.

[0235] Next, we will introduce the test framework and how to extract training samples based on the test framework:

[0236] In a possible implementation, the test framework may include constraints on the test code itself, as well as constraints on the calling relationship between the code to be tested and the test code.

[0237] Optionally, the constraint on the test code itself may be that the function type needs to be the type of the test function. For example, whether a function is a test function may be determined by testing a macro name.

[0238] Optionally, the constraint on the test code itself may be a constraint on the function test suite name, such as whether the test suite is unique, which can be determined by, for example, whether the first parameter in text() is unique.

[0239] Optionally, the constraint of the calling relationship may include at least one of the following: the existence of a direct calling relationship, the existence of a multi-level calling relationship, or the existence of an anonymous calling relationship.

[0240] In a possible implementation, a third test code that satisfies the constraints of the test function itself can be obtained from the code set; when the first code is called by the third test code in the code set and the calling relationship satisfies the constraints of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0241] In a possible implementation, the code set includes multiple codes called by the third test code and whose calling relationship satisfies the constraint of the calling relationship; the first code is the code with the highest similarity to the third test code among the multiple codes.

[0242] In a possible implementation, a third test code that satisfies the constraints of the test function itself can be obtained from the code set; the third test code includes an assertion; a target variable included in the assertion is determined; and when the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraints of the calling relationship, the third test code is used as the true value of the test code of the first code.

[0243] Among them, assertions refer to some Boolean expressions, which are used to debug the program and determine whether some tested functions work properly (i.e., compare whether the actual value is the same as the expected value). Assertions are judgments on the execution structure, not on the business process. Assertions are the most basic component of unit testing. Assertions are equivalent to an if() statement. If the assertion is met, the program is executed, and if it is not met, an error is thrown. Each type of assertion has two forms, one includes receiving a message parameter, such as "static public void assertTrue(Stringmessage,boolean condition)", where message represents the prompt word information when an error occurs, and the other does not have a message parameter. For example, the programmer believes that the expression value is true at a certain point in the program. Assertions can be enabled and disabled at any time, assertions can be enabled during testing, and assertions can be disabled during deployment. After the program is put into operation, users can re-enable assertions when encountering problems. Using assertions can create more stable, better quality, and less error-prone code. When you need to interrupt the current operation when a value is FALSE, use an assertion. Unit testing must use assertions (Junit / JunitX).

[0244] For example, refer to Figure 7 , Figure 7 The figure is a diagram of the construction process of a training sample, in which the feature rules corresponding to the framework can be queried to perform matching based on the call chain plus rules. However, due to the diversity of the code, various situations may occur during the matching process, such as multiple matches (i.e., a test function contains calls to multiple functions), which cannot be solved by heuristic methods alone. Therefore, for this situation, a method based on matching features + scoring and sorting is superimposed. For the heuristic rule matching algorithm in the test-source mapping model, it combines forward matching from the source function to the test function, and reverse matching from the test function to the source function. As Figure 6 As shown, in the forward match, the source functions are first filtered through the general rules to retain the functions containing the test code; then the function call stack is obtained, and the test code is judged and matched by the call type; in the reverse match, the test assertions are first filtered and the call relationship is captured. If it is a one-to-many call, the rule judgment based on the call relationship classification is also entered to filter the source functions that meet the conditions.

[0245] Reference Figure 8 , Figure 8The present invention is a schematic diagram of the construction process of a training sample, wherein, first, a general project-level analysis irrelevant to a language can be performed, which is constructed by constructing an abstract syntax tree or a code graph, and then the call chain inside the project code is further obtained; by defining the filtering rules of the source function containing the test code, all the source functions and test functions that meet the rules are captured; the mapping of the source / test function is obtained through the association relationships such as the call relationship and the naming method, and a forward and reverse matching method based on heuristic rules is adopted. The forward method uses the function call stack relationship as the main rule, and is assisted by the path matching rule and the file type rule to match the function and the test code. The reverse method extracts the function return value from the assertion of the test code, and then extracts the source function to be tested corresponding to the function return value from the test code function body. In addition, a multi-factor weighted priority calculation method and sorting mechanism are introduced. Taking C / C++ collection as an example, due to the characteristics of the GTest collection framework, the test code of the function in the same source file has the same test suite name. The unit test code function and the source function are converted into vectors by a model, and then the source function vectors are sorted. The highest ranking after rearrangement is the function to be tested. The sorting strategy is to sort the unit test code according to the possibility of the tested functions; after converting the unit test code functions and source functions into vectors, the similarity between them is directly calculated, and the function with the highest similarity is the function to be tested; after traversing and assigning values, the common information regardless of the language, namely the multi-level definition results of functions, files, projects, adaptation frameworks, etc., is stored in the database for training large models.

[0246] Reference Fig. 9 , Fig. 9The figure shows the overall process of training sample construction, model training and reasoning, which can include a candidate code pre-processing module based on software analysis. After the user selects the function to be tested, the pre-processing part is entered to perform dynamic context analysis and data enhancement on the current code. Then, rich prompt information is constructed. The pre-processing method of data enhancement includes: extracting context information in the test class of the tested function, so that the large model can recognize and use information such as variable naming in the context to avoid incorrect variable naming; extracting the test framework and mock framework required for generating test code; member variables, member functions and other information of parameter classes and return classes in the tested function; performing program branch analysis and use; adding the corresponding test code of similar source functions as a reference, etc. Then enter the training data optimization module, combine software analysis technology to filter, statically check and pre-process the training data to obtain the final code set. The specific data filtering rules include: 1) filter the data whose code lines of the tested method are within the closed interval of [4,100]; 2) the proportion of English letters in all characters of the tested method is not less than the given threshold. For example, 25%; 3) Among all the characters of the method to be tested, the proportion of numeric characters does not exceed the given threshold. For example, 90%. After that, you can enter the unit test code generation module based on the big model: for the candidate code selected by the user, according to the information obtained after pre-processing, a prompt is constructed and given as input to the test generation model. The test code is generated using the big model, and the test code generation result is entered into the post-processing module for a round of sorting and screening to obtain the final test code.

[0247] Reference Fig.10 , Fig.10 A front-end interface diagram of a test code generation module as an extended function or plug-in of a code editor or IDE.

[0248] In the specific implementation, the test code generation and the test-assisted generation result screening are completed by interacting with the user and the test generation plug-in. The opportunities for human-computer interaction can include: (1) actively triggering project-level context analysis and test code generation by selecting the code content of the method to be tested and then using the right-click menu or shortcut keys; (2) triggering project-level context analysis and test code generation in a question-and-answer dialogue.

[0249] After being triggered, the setting options can be further modified, including the confirmation of the tested class and method. After the user confirms the setting information, the test code generation is triggered. The front end constructs the information parsed from the tested code into a prompt and sends it to the back end as a request. The back end sends the prompt to the large model for reasoning, and then returns the candidate results returned by multiple large models to the front end. When the test code is generated, it will be automatically opened and parallel to the current code editor. The test generation results will be presented to the user in the form of multiple code snippets in the sidebar.

[0250] From the model perspective, human-computer interaction is carried out in the prompt format; from the user perspective, after the use case generation results are presented, the user can directly accept the use case, that is, by copying the use case to an existing test file or creating a new test file and importing the use case. The user can also click the post-processing module to repair the generated test code snippet based on whether it meets expectations, and then determine whether to adopt the use case. The plug-in can have multiple possible implementation forms in terms of interaction.

[0251] Based on the aforementioned code testing method, the present application also provides a code testing device. Fig.11 As shown, the code testing device 1100 includes:

[0252] An interactive module, used for receiving a test instruction input by a user;

[0253] An acquisition module 1101 (for example, it may include the context extraction module and prompt generation module described in the above embodiment) is used to acquire information of the first code, where the information of the first code includes context information of functions included in the first code, and one or more of the following information: context information of files associated with the first code, context information of files associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code;

[0254] The processing module 1102 (for example, may include the prompt generation module and the large language model processing module introduced in the above embodiment, etc.) is used to obtain the first test code corresponding to the first code according to the information of the first code.

[0255] Exemplarily, the acquisition module 1101 and the processing module 1102 may be implemented by hardware or software.

[0256] When implemented by software, the interaction module, the acquisition module 1101, and the processing module 1102 are applications running on a computing device (such as a server). For example, the processing module 1102 may be a computing engine running on a computing device, etc. The application may be provided to users in the form of a virtualized service. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service may be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only used as specific examples. In actual applications, the virtualization service can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0257] When implemented by hardware, the interaction module and the acquisition module 1101 can be implemented by transceiver modules such as transceivers. The processing module 1102 may include at least one computing device, such as a server, etc. Alternatively, the processing module 1102 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0258] For a detailed description of the acquisition module 1101 , reference may be made to the detailed description of steps 301 , 401 , and 501 in the above embodiment, and similarities will not be repeated here.

[0259] For a detailed description of the processing module 1102 , reference may be made to the detailed description of steps 302 , 402 , and 502 in the above embodiment, and similarities will not be repeated here.

[0260] The present application also provides a computing device 1200. Fig.12 As shown, the computing device 1200 includes: a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate through the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1200.

[0261] The bus 1202 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 The bus 1202 may include a path for transmitting information between various components of the computing device 1200 (eg, the memory 1206, the processor 1204, and the communication interface 1208).

[0262] The processor 1204 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0263] The memory 1206 may include a volatile memory, such as a random access memory (RAM). The memory 1206 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned code testing method. Specifically, the memory 1206 stores a test code generation module for executing instructions of the code testing method.

[0264] The communication interface 1208 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1200 and other devices or communication networks.

[0265] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0266] like Fig.13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster may store the same test code generation module for executing instructions of the code testing method.

[0267] In some possible implementations, one or more computing devices 1200 in the computing device cluster may also be used to execute part of the instructions of the test code generation module for executing the code testing method. In other words, the combination of one or more computing devices 1200 may jointly execute the instructions of the test code generation module for executing the code testing method.

[0268] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster may store different instructions for executing part of the functions of the test code generation module.

[0269] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned code generation module for executing the code testing method, the training sample generation method, and the model training method.

[0270] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned code testing method, training sample generation method, and model training method.

[0271] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A code testing method, It is characterized in that The method comprises: receiving a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction instructs generation of a test code by using a language model; Acquire information of the first code from a file associated with the first code, where the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; Obtaining a first test code corresponding to the first code according to information of the first code, wherein the information of the first code is used to construct a prompt word (prompt) as an input of a language model, and the first test code is obtained by the language model according to the prompt word (prompt); The first code is tested according to the first test code.

2. The method according to claim 1, It is characterized in that The context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or, The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variables of a class, member functions of a class, a constructor of a class, and an inheritance relationship of a class of the file associated with the first code; or The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or, The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

3. The method according to claim 1 or 2, It is characterized in that The obtaining, according to the information of the first code, a first test code corresponding to the first code includes: sending information of the first code, and receiving a first test code corresponding to the first code obtained according to the information of the first code; or, According to the information of the first code, a first test code corresponding to the first code is obtained through a language model.

4. The method according to any one of claims 1 to 3, It is characterized in that The obtaining of the information of the first code includes: obtaining a first prompt containing the information of the first code; the first prompt is used to instruct the language model to generate a test code of the first code according to the information of the first code; The obtaining, according to the information of the first code, a first test code corresponding to the first code includes: sending the first prompt word and receiving a first test code corresponding to the first code obtained according to the first prompt word; or, According to the first prompt word, a first test code corresponding to the first code is obtained through a language model.

5. The method according to any one of claims 1 to 4, It is characterized in that The first code is a function-level code, a class-level code or a file-level code.

6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: A test instruction for the first code is received on a code editing interface, wherein the test instruction instructs to generate a language model-based test code for the first code.

7. The method according to any one of claims 1 to 6, It is characterized in that The first test code is obtained through a language model according to information of the first code, and the method further includes: Acquire information of a second code; the second code and the first code are in different types of programming languages; According to the information of the second code, a second test code is obtained through the language model, wherein the first test code and the first code are in the same type of programming language, and the second test code and the second code are in the same type of programming language.

8. The method according to any one of claims 1 to 7, It is characterized in that The method further comprises: The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

9. The method according to claim 8, It is characterized in that The method further comprises: Get code set; According to the code set, a training sample of the language model is obtained, the training sample includes the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, the constraints include constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship.

10. The method according to claim 9, It is characterized in that The step of obtaining a training sample of a language model according to the code set includes: Acquire a third test code from the code set that satisfies the constraint of the test function itself; When the first code is called by the third test code in the code set and the calling relationship satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

11. The method according to claim 9, It is characterized in that The step of obtaining a training sample of a language model according to the code set includes: Acquire a third test code satisfying the constraints of the test function itself from the code set; the third test code includes assertions; determining a target variable included in the assertion; When the target variable calls the value assigned by the first code and the calling relationship between the first code and the third test code satisfies the constraint of the calling relationship, the third test code is used as the true value of the test code of the first code.

12. A code testing method, It is characterized in that The method comprises: receiving a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction instructs generation of a test code by using a language model; Obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to the file of the first code, and generate test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; According to the second prompt word, obtaining a first test code corresponding to the first code; The first code is tested according to the first test code.

13. A code testing device, It is characterized in that The device comprises: an acquisition module, configured to receive a test instruction for a first code, wherein the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; and to acquire information about the first code from a file associated with the first code, wherein the information about the first code includes context information about a function included in the first code, and one or more of the following information: context information about a file associated with the first code, context information about a project where the first code is located, or context information about a test framework used by the first code, wherein the information about the first code is used to construct a prompt word prompt as an input of the language model, and the first test code is obtained through the language model according to the prompt word prompt; The processing module is used to obtain a first test code corresponding to the first code according to information of the first code; and test the first code according to the first test code.

14. The device according to claim 13, It is characterized in that The context information of the function included in the first code includes: at least one of the type of the input parameter of the included function, the type of the function return value, and the method call in the function body; or, The context information of the file associated with the first code includes: at least one of a variable name, structure information, object name, object definition, member variable of a class, member function of a class, construction device of a class, and inheritance relationship of a class of the file associated with the first code; or, The context information of the project where the first code is located includes: statistical information of the project where the first code is located or the type of programming language; or, The context information of the test framework used by the first code includes: the category of the test framework used by the first code, or the category of the mock framework.

15. The device according to claim 13 or 14, It is characterized in that The acquisition module is specifically used for: When a test instruction for a first code in a code editing interface is received, information of the first code is acquired, wherein the test instruction instructs to generate a test code based on a language model for the first code.

16. The device according to any one of claims 13 to 15, It is characterized in that The first test code is obtained through a language model according to information of the first code, and the acquisition module is further used to: Acquire information of a second code; the second code and the first code are in different types of programming languages; The processing module is further used to obtain a second test code through the language model according to information of the second code, wherein the first test code and the first code are in the same type of programming language, and the second test code and the second code are in the same type of programming language.

17. The device according to any one of claims 13 to 16, It is characterized in that The processing module is further used for: The language model is updated according to the first test code and a third test code corresponding to the first code, and the third test code is used as a test code true value of the first code.

18. The device according to claim 17, It is characterized in that The acquisition module is further used to: acquire a code set; The processing module is further used to: obtain a training sample of a language model according to the code set, the training sample including the first code and the third test code extracted from the code set, wherein the code set is adapted to a first test framework, the first test framework is used to indicate constraints of a test function that can be used as a function to be tested, the constraints including constraints of the test function itself and constraints on a calling relationship between the test function and the function to be tested, wherein the third test code satisfies the constraints of the test function itself, and the calling relationship between the third test code and the first code satisfies the constraints of the calling relationship.

19. A code testing device, It is characterized in that The device comprises: An acquisition module is used to receive a test instruction for a first code, where the first code is a code in a program to be edited, and the test instruction indicates that a test code is generated through a language model; obtain a second prompt word, where the second prompt word is used to instruct the language model to extract information of the first code according to a file of the first code, and generate a test code for the first code according to the information of the first code; the information of the first code includes context information of a function included in the first code, and one or more of the following information: context information of a file associated with the first code, context information of a file associated with the first code, context information of a project where the first code is located, or context information of a test framework used by the first code; The processing module is used to obtain a first test code corresponding to the first code according to the second prompt word; and test the first code according to the first test code.

20. A computing device cluster, It is characterized in that The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 12.

21. A computer-readable storage medium, It is characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 12.

Images