Code testing method and device, electronic equipment and computer program product

Abstract

The invention discloses a code testing method and device, electronic equipment and a computer program product. The method relates to the technical field of code testing, and comprises the following steps: obtaining a code generated by a code generation service and used for realizing a target function, and obtaining a to-be-tested code; generating instruction information according to the judgment condition information, the to-be-tested code and the reference code, wherein the instruction information indicates whether processing logics of the to-be-tested code and the reference code are the same or not according to the judgment condition information; and inputting the instruction information into the large language model, and judging whether the to-be-detected code is accurate or not based on a returned result. The technical problem that in the related technology, whether the to-be-tested code generated by the code generation service is accurate or not is judged by comparing the execution result or the code text of the test code and the reference code, and the judgment result is inaccurate is solved.

Description

Technical Field

[0001] The present application relates to the field of code testing technology, and in particular to a code testing method, device, electronic device and computer program product. Background Art

[0002] In order to improve the efficiency of code generation, some code generation services that convert natural language text into code have emerged in the relevant technology. For code generation services, it is necessary to test the accuracy of the generated code to ensure the availability of the code generation service.

[0003] When testing the accuracy of the code generated by the code generation service, taking the test of SQL code as an example, the relevant technology judges the accuracy of the SQL code to be tested by comparing the query results of the SQL code to be tested with the reference SQL code in the question bank. If the query results of the two SQL codes are the same, the two codes are considered to be consistent. This method is accurate when there is sufficient data. However, the question bank from the outside has the problem of small amount of reference SQL code data, and the wrong SQL code may be judged as the correct code.

[0004] In addition, the related art also judges the accuracy of the SQL code to be tested by comparing the text of the SQL code to be tested with the reference SQL code in the question bank. If the query result of the SQL code to be tested is empty, but the text of the test SQL code and the reference SQL code are the same, the two codes are considered to be consistent. However, the same question may have multiple SQL implementation methods, and comparing the SQL text may result in the correct SQL code being judged as an incorrect code.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a code testing method, device, electronic device and computer program product to at least solve the technical problem in the related art of judging whether the code to be tested generated by the code generation service is accurate by comparing the execution results or code text of the test code and the reference code, and the judgment result is inaccurate.

[0007] According to one aspect of an embodiment of the present application, a code testing method is provided, including: obtaining a code generated by a code generation service for implementing a target function to obtain a code to be tested; generating instruction information according to judgment condition information, the code to be tested, and a reference code, wherein the reference code refers to a code that implements the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; inputting the instruction information into a large language model, and judging whether the code to be tested is accurate based on a return result of the large language model.

[0008] Optionally, generating instruction information based on judgment condition information, the code to be tested, and the reference code includes: obtaining target problem information and target prompt information used by the code generation service to generate the code to be tested, wherein the target problem information is used to request generation of code to implement the target function, and the target prompt information describes information required to generate the code to implement the target function; generating instruction information based on the judgment condition information, the code to be tested, the reference code, the target problem information, and the target prompt information.

[0009] Optionally, inputting instruction information into a large language model, and judging whether the code to be tested is accurate based on the return result of the large language model includes: inputting instruction information into N large language models to obtain N return results, where N is a positive integer greater than 2; when a return result greater than or equal to a first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; when a return result less than the first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is inaccurate.

[0010] Optionally, inputting instruction information into a large language model, and judging whether the code to be tested is accurate based on the return result of the large language model includes: inputting instruction information into the large language model to obtain a return result; when the return result indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; when the return result indicates that the processing logic of the code to be tested is different from that of the reference code, updating the instruction information by adding prompt information, and inputting the updated instruction information into the large language model until the number of updates of the instruction information does not reach a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate, or until the number of updates of the instruction information reaches a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is different from that of the reference code, determining that the code to be tested is inaccurate.

[0011] Optionally, before generating instruction information based on the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are both empty, executing the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code.

[0012] Optionally, before generating instruction information based on the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are the same, if the number of code lines of the first execution result and the second execution result are both less than a preset number of lines, then executing the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code.

[0013] Optionally, before generating instruction information based on the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are different, if a code accuracy verification instruction is received, the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code is executed; when the first execution result and the second execution result are the same, and the number of code lines of the first execution result and the second execution result are both greater than or equal to a preset number of lines, if a code accuracy verification instruction is received, the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code is executed.

[0014] Optionally, when the judgment condition information includes at least two judgment conditions, the instruction information is further used to indicate that the code to be tested does not meet the judgment condition when the processing logic of the code to be tested is different from that of the reference code.

[0015] According to another aspect of an embodiment of the present application, a code testing method is also provided, including: a cloud server receives a code generated by a code generation service for implementing a target function, and obtains a code to be tested; the cloud server receives instruction information generated by a client based on judgment condition information, the code to be tested, and a reference code, wherein the reference code refers to a code to implement the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same based on the judgment condition information; the cloud server processes the instruction information through a large language model, and determines whether the code to be tested is accurate based on a return result of the large language model; the cloud server sends the accuracy of the code to be tested to the client.

[0016] According to another aspect of an embodiment of the present application, a code testing device is also provided, including: an acquisition unit, used to acquire the code generated by a code generation service for realizing a target function, and obtain a code to be tested, wherein the target problem information is used to request generation of the code to realize the target function, and the target prompt information describes the information required to generate the code to realize the target function; a generation unit, used to generate instruction information according to judgment condition information, the code to be tested, and a reference code, wherein the reference code refers to the code to realize the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; a determination unit, used to input the instruction information into a large language model, and judge whether the code to be tested is accurate based on the return result of the large language model.

[0017] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0018] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0019] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0020] In an embodiment of the present application, a large language model is used to determine whether the processing logic of the code to be tested and the reference code are the same, and then the accuracy of the code to be tested in achieving the target function is determined based on the output result of the large language model. When constructing the instruction input into the large language model, not only the code to be tested and the reference code themselves are considered, but also the judgment condition information. By combining the judgment condition information, the code to be tested, and the reference code to generate instruction information, and inputting the generated instructions into the large language model, the large language model can accurately determine whether the processing logic of the code to be tested and the reference code are the same based on the input instructions, thereby achieving the technical effect of accurately determining the accuracy of the code to be tested in achieving the target function based on the output result of the large language model, thereby solving the technical problem of inaccurate judgment results in the related art by comparing the execution results or code texts of the test code and the reference code to determine whether the code to be tested generated by the code generation service is accurate.

[0021] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a code testing method according to an embodiment of the present application;

[0024] Figure 2 is a structural block diagram of a computing environment according to an embodiment of the present application;

[0025] Figure 3 is a flow chart of a code testing method according to an embodiment of the present application;

[0026] Figure 4 is a flowchart of an optional code testing method according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of an optional code testing method according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a code testing device according to an embodiment of the present application;

[0029] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0034] LLM, Large Language Model, is a natural language processing model based on deep learning. It learns the understanding and generation capabilities of human language through training with large amounts of text data. Large language models can have billions to trillions of parameters and can handle a variety of natural language processing tasks, such as text classification, text summarization, machine translation, speech recognition, etc.

[0035] SQL, Structured Query Language, is a special-purpose programming language. It is a database query and programming language used to access data and query, update, and manage relational database systems.

[0036] Example 1

[0037] According to an embodiment of the present application, a code testing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a code testing method according to an embodiment of the present application. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing the code testing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 102a, 102b, ..., 102n are used to illustrate) a processor 102 (the processor 102 may include but is not limited to a processing device such as a processor 102 (the processor 102 may include but is not limited to an MCU (Microcontroller Unit) or an FPGA (Field-Programmable Gate Array)), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the methods in the above embodiments. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] Figure 1 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Figure 2 The block diagram shows the use of the above Figure 1 The computer terminal 10 (or mobile device) is shown as an embodiment of a computing node in the computing environment 201 . Figure 2 is a structural block diagram of a computing environment according to an embodiment of the present application, such as Figure 2 As shown, the computing environment 201 includes multiple ( Figure 2210-1, 210-2, ..., are used to illustrate computing nodes (such as servers). The computing nodes all contain local processing and memory resources, and end users 202 can remotely run applications or store data in the computing environment 201. The applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in the computing environment 201, representing services "A", "D", "E" and "H" respectively.

[0044] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).

[0045] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0046] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

[0047] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of a service may require invoking one or more functions of another service. Figure 2As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.

[0048] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0049] Under the above operating environment, this application provides Figure 3 The code shown tests the method. Figure 3 It is a flowchart of a code testing method according to an embodiment of the present application.

[0050] Step S302: Obtain the code generated by the code generation service for implementing the target function to obtain the code to be tested.

[0051] Among them, the code generation service can be a service that converts input data into code. In the scenario of generating SQL statements, the code generation service can be a service that converts natural language text into SQL code. Exemplarily, the code generation service can be a large language model, such as the text2sql large model.

[0052] In the scenario of generating SQL code, the target function may be a data query function, a data insertion function, a data update function, and a data deletion function executed on the data in the database. Accordingly, the code to implement the target function may be an SQL query statement, an SQL insertion statement, an SQL update statement, an SQL deletion statement, etc. In the case where the code generation service is a text2sql large model, the code generation instructions are assembled, and the code generation instructions are input into the text2sql large model to obtain the return result.

[0053] Since the text2sql large model will add some explanation content in the process of generating code, the returned result contains not only SQL code, but also SQL code can be intercepted from the returned result. For example, the SQL code can be intercepted by writing a good code interception script, and the code interception instruction can be input to instruct the large language model to intercept the SQL code, so as to obtain the code to be tested.

[0054] Step S304, generating instruction information according to the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code for realizing the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information.

[0055] The reference code refers to the code that implements the target function and is the reference object used when testing whether the code to be tested accurately implements the target function.

[0056] In order to enable the large language model to understand and accurately determine whether the processing logic of the test code and the reference code is the same, this embodiment not only considers the test code and the reference code itself, but also considers the judgment condition information when constructing the instruction information input into the large language model, wherein the judgment condition information is used to determine whether the processing logic of the code to be tested is the same as that of the reference code, and the judgment condition information may include at least one judgment condition, for example, the judgment condition may include: whether the source table is the same, whether the filtering condition is the same, whether the fields extracted from the source table are the same, whether the number of fields returned is the same, etc. For example, if the code to be tested is a SQL query statement to be tested, the constructed instruction information may be "Please determine whether the processing logic of the SQL query statement to be tested and the reference SQL query statement is the same based on whether the source table is the same and whether the filtering condition is the same".

[0057] In addition, when constructing the instruction information input into the large language model, the target prompt information used when generating the code to be tested can also be used. For example, the target prompt information used when generating the code to be tested includes the table structure information of the source table, the query dimension information, and the industry information of the industry to which the data in the source table belongs. All or part of the target prompt information can be added to the instruction to further improve the judgment accuracy of the large language model. In this way, when the amount of true value data corresponding to the target problem information in the evaluation set is small, this embodiment can also use the understanding ability and reasoning ability of the large language model to analyze the constructed instruction information, and compare the code to be tested and the reference code from the internal processing logic.

[0058] Step S306: input the instruction information into the large language model, and determine whether the code to be tested is accurate based on the return result of the large language model.

[0059] Among them, the large language model can be a large language model for a general field. The return result of the large language model indicates whether the processing logic of the code to be tested and the reference code in the evaluation set is consistent. The return result can be yes or no. When the return result is yes, it indicates that the processing logic of the code to be tested and the reference code is the same, the code to be tested can accurately implement the target function, and the code to be tested can be judged to be accurate. When the return result is no, it indicates that the processing logic of the code to be tested and the reference code is different, and it is difficult for the code to be tested to accurately implement the target function, and the code to be tested can be judged to be inaccurate.

[0060] In the case where the judgment condition information includes multiple judgment conditions, the returned result may be similarity. For example, the judgment condition information includes 4 judgment conditions. When the code to be tested meets the 4 judgment conditions, the returned result is 1. When the code to be tested meets 3 judgment conditions, the returned result is 0.75. When the code to be tested meets 2 judgment conditions, the returned result is 0.5. When the code to be tested meets 1 judgment condition, the returned result is 0.25. When the code to be tested does not meet any judgment condition, the returned result is 0. After obtaining the similarity of the large language model, the accuracy of the code to be tested may be determined according to the similarity. For example, when the similarity is greater than 0.7, it indicates that the processing logic of the code to be tested and the reference code is the same, the code to be tested can accurately implement the target function, and the code to be tested can be determined to be accurate. When the similarity is equal to or less than 0.7, it indicates that the processing logic of the code to be tested and the reference code is different, the code to be tested cannot accurately implement the target function, and the code to be tested can be determined to be inaccurate.

[0061] In an embodiment of the present application, a large language model is used to determine whether the processing logic of the code to be tested and the reference code are the same, and then the accuracy of the code to be tested in achieving the target function is determined based on the output result of the large language model. When constructing the instruction input into the large language model, not only the code to be tested and the reference code themselves are considered, but also the judgment condition information. By combining the judgment condition information, the code to be tested, and the reference code to generate instruction information, and inputting the generated instructions into the large language model, the large language model can accurately determine whether the processing logic of the code to be tested and the reference code are the same based on the input instructions, thereby achieving the technical effect of accurately determining the accuracy of the code to be tested in achieving the target function based on the output result of the large language model, thereby solving the technical problem of inaccurate judgment results in the related art by comparing the execution results or code texts of the test code and the reference code to determine whether the code to be tested generated by the code generation service is accurate.

[0062] Optionally, in the code testing method of the present embodiment, generating instruction information based on judgment condition information, the code to be tested, and the reference code includes: obtaining target problem information and target prompt information used by the code generation service to generate the code to be tested, wherein the target problem information is used to request generation of code to implement the target function, and the target prompt information describes information required to generate the code to implement the target function; generating instruction information based on the judgment condition information, the code to be tested, the reference code, the target problem information, and the target prompt information.

[0063] It should be noted that the target problem information and target prompt information can be data in the evaluation set. The evaluation set is used to evaluate the quality of the code generated by the code generation service. The evaluation set includes multiple groups of data, each of which includes problem information, prompt information, and reference code. The reference code refers to the code that implements the target function indicated by the target problem information, that is, the true value corresponding to the target problem information obtained from the evaluation set.

[0064] The target question information is the question information used when generating the code to be tested, and the SQL code to realize the target function is generated by asking questions. The target prompt information may include the table structure information, query dimension information, industry information, etc. of the source table in the target database.

[0065] In the case where the code generation service is a text2sql large model, the target question information and the target prompt information are used to assemble the code generation instructions, and the code generation instructions are input into the text2sql large model to obtain the return result. For example, the target question information may be "whether it is possible to generate an SQL query code that executes the query function on the salary data of Zhang San in the salary table", and the target prompt information is "Zhang San's employee number is 111", and the text2sql large model generates a section of SQL query code to be tested. In order to enable the large language model to understand and accurately determine whether the processing logic of the test code and the reference code is the same, this embodiment, when constructing the instruction information input into the large language model (a large language model used to judge whether the processing logic of the code to be tested and the reference code is the same), not only the test code, the reference code itself, and the judgment condition information are considered, but also the target question information and the target prompt information used to generate the code to be tested can be considered. For example, the constructed instruction information may be "Please determine whether the processing logic of the SQL query code to be tested and the reference SQL query code is the same based on whether the source tables are the same and whether the filtering conditions are the same. The problem information used when generating the SQL query code to be tested is to instruct the generation of an SQL query code for executing a query function on the salary data of Zhang San in the salary table. The target prompt information indicates to execute the query function using Zhang San's employee number 111."

[0066] This embodiment generates instruction information by combining judgment condition information, code to be tested, reference code, target problem information, and target prompt information, and inputs the generated instructions into a large language model, so that the large language model can accurately determine whether the processing logic of the code to be tested and the reference code are the same based on the input instructions. In the case where the amount of true value data corresponding to the target problem information in the evaluation set is small, this embodiment can also use the understanding and reasoning capabilities of the large language model to analyze the constructed instruction information and compare the code to be tested and the reference code from the internal processing logic.

[0067] In order to improve the accuracy of code testing and avoid errors caused by a large language model during code testing, testing can be performed using multiple large language models. Optionally, in the code testing method of this embodiment, instruction information is input into the large language model, and whether the code to be tested is accurate is determined based on the return results of the large language model, including: inputting instruction information into N large language models to obtain N return results, where N is a positive integer greater than 2; when a return result greater than or equal to a first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; when a return result less than the first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is inaccurate.

[0068] Among them, the N large language models can be pre-trained models of various fields such as proprietary fields or general fields, and the training corpora of the N large language models are different. The first preset ratio is greater than 50%, for example, the first preset ratio can be 70%. When more than 70% of the return results indicate that the processing logic of the code to be tested is the same as that of the reference code, the code to be tested is determined to be accurate; when less than 70% of the return results indicate that the processing logic of the code to be tested is the same as that of the reference code, the code to be tested is determined to be inaccurate.

[0069] This embodiment can simultaneously call multiple different large language models as referee models and take the majority opinion as the final judgment result. Compared with using only one large language model as the referee model, the accuracy of the judgment of the accuracy of the code to be tested is improved.

[0070] In order to improve the accuracy of code testing and avoid errors caused by a single test, multiple rounds of testing can be performed using a large language model. Optionally, in the code testing method of this embodiment, the instruction information is input into the large language model, and whether the code to be tested is accurate is determined based on the return result of the large language model, including: inputting the instruction information into the large language model to obtain a return result; when the return result indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; when the return result indicates that the processing logic of the code to be tested is different from that of the reference code, updating the instruction information by adding prompt information, and inputting the updated instruction information into the large language model, until the number of updates to the instruction information does not reach a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate, or until the number of updates to the instruction information reaches a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is different from that of the reference code, determining that the code to be tested is inaccurate.

[0071] It should be noted that the prompt information in the instruction information input into the large language model may be part or all of the target prompt information used when generating the code to be tested, or may be information different from the target prompt information.

[0072] Exemplarily, in the scenario of testing SQL code, the target prompt information used when generating the SQL code to be tested includes the table structure information of the source table and the query dimension information. The prompt information in the instruction information constructed when calling the large language model for the first time can only include the table structure information of the source table in the target prompt information. If the return result obtained by the large language model for the first time indicates that the processing logic of the code to be tested is the same as that of the reference code, it means that the text similarity between the SQL code to be tested and the reference SQL code is very high, and it can be concluded that the internal processing logic of the two pieces of code is the same based on very little information, and it is determined that the code to be tested can accurately implement the target function and the code to be tested is accurate.

[0073] If the return result obtained by the large language model for the first time indicates that the processing logic of the code to be tested is different from that of the reference code, another judgment can be made. The query dimension information in the target prompt information can be used as new prompt information to update the instruction information. If the return result obtained by the large language model for the second time still indicates that the processing logic of the code to be tested is different from that of the reference code, it means that the text difference between the SQL code to be tested and the reference SQL code is large, and more information is needed to accurately determine whether the internal processing logic of the two pieces of code is the same. Another judgment can be made. For example, determine the industry information to which the data in the source table belongs. The industry information can be used as new prompt information to update the instruction information, and the large language model can be called again for judgment until the cutoff conditions for multiple rounds of judgment are met.

[0074] Among them, the cutoff condition of multiple rounds of judgment is related to the preset update times of the set instruction. In order to avoid resource consumption caused by too many judgment times when the SQL code to be tested and the reference SQL code are different, the value of the preset update times can be a value less than 10. For example, the preset update times can be 5 times. If the instruction information is continuously updated and the large language model is re-called for judgment until the update times of the instruction information reaches 5 times, and the large language model indicates that the processing logic of the code to be tested is different from that of the reference code, then it is determined that the code to be tested is inaccurate. Or, if the update times of the instruction information do not reach 5 times, but the large language model indicates that the processing logic of the code to be tested is the same as that of the reference code, then it is determined that the code to be tested is accurate.

[0075] In addition to simultaneously calling multiple different large language models to determine whether the processing logic of the code to be tested is the same as that of the reference code, this embodiment can also perform multiple rounds of calls to the same large language model to determine whether the processing logic of the code to be tested is the same as that of the reference code. In each round of calls, the instruction information is updated by adding prompt information, and the updated instruction information is input into the large language model to instruct the large language model to make a judgment again. In this way, even if the text difference between the code to be tested and the reference code is large, it is also possible to accurately determine whether the internal processing logic of the two is the same, thereby improving the accuracy of determining the accuracy of the code to be tested in achieving the target function.

[0076] It should be noted that in addition to directly calling the large language model to determine whether the code to be tested and the reference code are the same, you can also first make a judgment through other methods. If other methods cannot produce a result or the result is inaccurate, the large language model can be used for auxiliary judgment.

[0077] When the result cannot be determined by other means, the judgment can be performed through the large language model. Optionally, in the code testing method of this embodiment, before generating instruction information according to the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are both empty, executing the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code.

[0078] Exemplarily, in the SQL code test scenario, the SQL code to be tested and the reference SQL code can be executed separately based on the database. If the return data is obtained after the execution, it can be judged whether the code to be tested and the reference code are the same according to the return data. If no data is returned after the execution, that is, the first execution result and the second execution result are both empty, there may be a problem with the source table, or there is too little data in the source table, and it is impossible to judge whether the code to be tested and the reference code are the same through the execution result. In this case, a large language model can be used for judgment, and instruction information is generated according to the judgment condition information, the code to be tested, and the reference code, and the instruction is input into the large language model for judgment.

[0079] Through this embodiment, when the code execution conditions are met, the code to be tested and the reference code are first executed, and whether the code to be tested and the reference code are the same is determined by the execution results. If no result is returned after the code is executed, the large language model is called to determine whether the processing logic of the two is the same. The efficiency of code testing is improved while ensuring the accuracy of code testing.

[0080] When the judgment results of other methods are inaccurate, judgment can be performed through the large language model. Optionally, in the code testing method of this embodiment, before generating instruction information according to the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are the same, if the number of code lines of the first execution result and the second execution result are both less than a preset number of lines, then executing the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code.

[0081] Exemplarily, in a SQL code testing scenario, the code to be tested and the reference code can be executed separately based on the database. If there is returned data after execution, the first execution result and the second execution result are compared to determine whether they are the same to determine whether the code to be tested and the reference code are the same.

[0082] If the first execution result is the same as the second execution result, and the number of code lines of both is greater than or equal to the preset number of lines, it means that the returned content is large and consistent, and there is sufficient reason to determine that the internal processing logic of the two is the same, which means that the code to be tested can accurately implement the target function with a high probability, and the code to be tested can be determined to be accurate.

[0083] If the first execution result and the second execution result are the same, but the number of lines of code for both is less than the preset number of lines, a misjudgment may occur. For example, if both return a piece of data, the code to be tested and the reference code are likely to be different, that is, the internal processing logic of the two is different, but the results found are just the same by chance. In this case, a large language model can be used for judgment, and instruction information is generated based on the judgment condition information, the code to be tested, and the reference code, and the instruction is input into the large language model for judgment.

[0084] This embodiment takes into account that the reference code only represents one implementation method of the target function, but the target function indicated by the target problem information may have multiple code implementation methods. The code to be tested and the reference code may be executed first, and whether the processing logics of the two are the same may be determined based on the execution results. If the returned data is the same but very little, in order to avoid misjudgment, a large language model may be called for judgment, thereby achieving the effect of improving the accuracy of the judgment result.

[0085] The code to be tested and the reference code can be respectively executed by the execution mechanism, and whether the two are the same can be determined based on the execution results. If the judgment result needs to be verified, the large language model can be used for verification. Optionally, in the code testing method of this embodiment, before generating instruction information based on the judgment condition information, the code to be tested and the reference code, the method also includes: executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; when the first execution result and the second execution result are different, if a code accuracy verification instruction is received, the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code is executed; when the first execution result and the second execution result are the same, and the number of code lines of the first execution result and the second execution result are both greater than or equal to the preset number of lines, if a code accuracy verification instruction is received, the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code is executed.

[0086] Exemplarily, in the SQL code test scenario, the SQL code to be tested and the reference SQL code can be executed based on the database respectively. If the return data is obtained after execution, it is determined whether the first execution result and the second execution result are the same. If the first execution result and the second execution result are different, in order to avoid misjudgment, there is a need to verify again whether the internal processing logic of the two is the same. If a code accuracy verification instruction is received, a large language model can be called for judgment. If the first execution result and the second execution result are the same, and the number of lines of code of the two are greater than or equal to the preset number of lines, it means that the probability of the two being the same is high. In order to avoid misjudgment, there is a need to verify again whether the internal processing logic of the two is the same. If a code accuracy verification instruction is received, a large language model can be called for judgment.

[0087] It should be noted that, when the judgment result obtained through the execution result is consistent with the judgment result obtained by calling the large language model, it means that the judgment result obtained through the execution result is accurate. When the judgment result obtained through the execution result is inconsistent with the judgment result obtained by calling the large language model, the input instructions of the model can be updated by adding prompt information to further determine whether the code to be tested and the reference code are the same, or by calling multiple different large language models and taking the majority opinion as the final judgment result.

[0088] In addition to directly calling the large language model to determine whether the code to be tested and the reference code are the same, this embodiment can also first determine by other means and then verify by the large language model, thereby improving the accuracy of the determination.

[0089] In order to understand the reason why the processing logic of the code to be tested is different from that of the reference code, optionally, in the code testing method of this embodiment, when the judgment condition information includes at least two judgment conditions, the instruction information is also used to indicate that when the processing logic of the code to be tested is different from that of the reference code, the judgment condition that the code to be tested does not meet is prompted.

[0090] Exemplarily, in the SQL code test scenario, the judgment condition information may include four judgment conditions: whether the source tables are the same, whether the filtering conditions are the same, whether the fields extracted from the source table are the same, and whether the number of fields returned is the same. Request information may also be added to the instruction information to request the large language model to prompt the judgment conditions that the code to be tested does not meet. In this way, in addition to returning the similarity between the code to be tested and the reference code, the large language model can also give the judgment conditions that the code to be tested does not meet. For example, the large language model returns a similarity of 75%, and prompts that the filtering conditions of the code to be tested and the reference code are different. In this way, the user can modify the filtering conditions of the code to be tested according to the filtering conditions of the code to be tested, and test again whether the processing logic of the modified code to be tested and the reference code is the same, so that the code to be tested can accurately achieve the target function. For another example, if the 100 codes to be tested generated by the code generation service are tested by the large language model, the large language model prompts that the reason why 80 codes to be tested are different from the reference code is the field extracted from the source table, indicating that there is an abnormality in the code generation logic of the code generation service, and the code generation service needs to be adjusted to improve the accuracy of the code generated by the code generation service.

[0091] This embodiment adjusts the instruction information input into the large language model to request the large language model to prompt the judgment condition that the code to be tested does not meet when the processing logic of the code to be tested is different from that of the reference code. On the one hand, it is convenient to further modify the code to be tested so that the modified code to be tested can accurately implement the target function. On the other hand, the code generation service can also be optimized more targeted so that the code generation service can generate accurate code to implement the corresponding function.

[0092] Example 2

[0093] According to an embodiment of the present application, a code testing method is also provided. Figure 4 is a flowchart of an optional code testing method according to an embodiment of the present application, such as Figure 4 As shown, the method includes:

[0094] Step S402: The cloud server receives the code generated by the code generation service for implementing the target function, and obtains the code to be tested.

[0095] Among them, a large language model is deployed in the cloud service. The large language model can be a large language model of a general field, which is used to determine whether the processing logic of the code to be tested and the reference code associated with the target problem information are the same.

[0096] The code generation service may be a service that converts input data into code. In the scenario of generating SQL statements, the code generation service may be a service that converts natural language text into SQL code. Exemplarily, the code generation service may be a large language model, such as a text2sql large model. The target problem information and the target prompt information may be data in an evaluation set. The evaluation set is used to evaluate the quality of the code generated by the code generation service. The evaluation set includes multiple groups of data, each group of data includes problem information, prompt information, and reference code. The reference code refers to the code that implements the function indicated by the problem information.

[0097] In the scenario of generating SQL code, the target function may be a data query function, a data insertion function, a data update function, and a data deletion function executed on the data in the database. Accordingly, the code to implement the target function may be an SQL query statement, an SQL insert statement, an SQL update statement, an SQL delete statement, etc. The target question information requests the generation of SQL code to implement the target function by asking questions. For example, the target question information may be "Can I generate SQL query code to execute the query function on the data in the target database?" The target prompt information may include the table structure information, query dimension information, industry information, etc. of the source table in the target database. In the case where the code generation service is a text2sql large model, the target question information and the target prompt information are used to assemble the code generation instructions, and the code generation instructions are input into the text2sql large model to obtain the return result.

[0098] Since the text2sql large model will add some explanation content in the process of generating code, the returned result contains not only SQL code, but also SQL code can be intercepted from the returned result. For example, the SQL code can be intercepted by writing a good code interception script, and the code interception instruction can be input to instruct the large language model to intercept the SQL code, so as to obtain the code to be tested.

[0099] Step S404, the cloud server receives instruction information generated by the client based on the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code that implements the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same based on the judgment condition information.

[0100] Among them, the client can be a large language model software product installed on the terminal device. The user can generate instruction information on the client based on the judgment condition information, the code to be tested and the reference code, and call the large language model that is not deployed on the cloud server through the client.

[0101] Among them, the judgment condition information is used to determine whether the processing logic of the code to be tested and the reference code is the same. The judgment condition information may include multiple judgment conditions. For example, the judgment conditions may include: whether the source tables are the same, whether the filtering conditions are the same, whether the fields extracted from the source tables are the same, whether the number of fields returned is the same, etc.

[0102] Among them, the target problem information is the problem information used when generating the code to be tested. The reference code associated with the target problem information refers to the code that implements the target function indicated by the target problem information, that is, the true value corresponding to the target problem information obtained from the evaluation set. The reference code is the reference object used to test whether the code to be tested is accurate.

[0103] In order to enable the large language model to understand and accurately determine whether the processing logic of the test code and the reference code is the same, this embodiment, when constructing the instruction for inputting the large language model, not only considers the test code and the reference code itself, but also considers the judgment condition information and the target problem information used to generate the code to be tested. In addition, when constructing the instruction for inputting the large language model, the target prompt information used when generating the code to be tested can also be used. For example, the target prompt information used when generating the code to be tested includes table structure information, query dimension information, and industry information. All or part of the target prompt information can be added to the instruction to further improve the judgment accuracy of the large language model. In this way, when the amount of true value data corresponding to the target problem information in the evaluation set is small, this embodiment can also use the understanding ability and reasoning ability of the large language model to analyze the constructed instruction information, and compare the code to be tested and the reference code from the internal processing logic.

[0104] Step S406: The cloud server processes the instruction information through the large language model, and determines whether the code to be tested is accurate based on the return result of the large language model.

[0105] Among them, the large language model can be a large language model in a general field. The return result of the large language model indicates whether the processing logic of the code to be tested and the reference code in the evaluation set are consistent. The return result can be yes or no. When the return result is yes, it indicates that the processing logic of the code to be tested and the reference code is the same, and the code to be tested can accurately implement the target function, and the code to be tested can be determined to be accurate. When the return result is no, it indicates that the processing logic of the code to be tested and the reference code is different, and it is difficult for the code to be tested to accurately implement the target function, and the code to be tested can be determined to be inaccurate.

[0106] In the case where the judgment condition information includes multiple judgment conditions, the returned result may be similarity. For example, the judgment condition information includes 4 judgment conditions. When the code to be tested meets the 4 judgment conditions, the returned result is 1. When the code to be tested meets 3 judgment conditions, the returned result is 0.75. When the code to be tested meets 2 judgment conditions, the returned result is 0.5. When the code to be tested meets 1 judgment condition, the returned result is 0.25. When the code to be tested does not meet any judgment condition, the returned result is 0. After obtaining the similarity of the large language model, the accuracy of the code to be tested may be determined according to the similarity. For example, when the similarity is greater than 0.7, it indicates that the processing logic of the code to be tested and the reference code is the same, the code to be tested can accurately implement the target function, and the code to be tested can be determined to be accurate. When the similarity is equal to or less than 0.7, it indicates that the processing logic of the code to be tested and the reference code is different, the code to be tested cannot accurately implement the target function, and the code to be tested can be determined to be inaccurate.

[0107] Step S408: The cloud server sends the accuracy of the code to be tested to the client.

[0108] In this way, users can view the accuracy of the code under test in implementing the target function through the client, thereby analyzing the usability of the code under test or analyzing the performance of the code generation service.

[0109] In an embodiment of the present application, a large language model deployed on a cloud server is used to determine whether the processing logic of the code to be tested and the reference code are the same, and then the accuracy of the code to be tested in achieving the target function is determined based on the output result of the large language model, and the accuracy is returned to the client. When constructing the instruction input into the large language model, not only the code to be tested and the reference code itself are considered, but also the judgment condition information. By combining the judgment condition information, the code to be tested, and the reference code to generate instruction information, and inputting the generated instructions into the large language model, the large language model can accurately determine whether the processing logic of the code to be tested and the reference code is the same according to the input instructions, thereby achieving the technical effect of accurately determining the accuracy of the code to be tested to achieve the target function based on the output result of the large language model, thereby solving the technical problem of inaccurate judgment results in the related art by comparing the execution results or code texts of the test code and the reference code to determine whether the code to be tested generated by the code generation service is accurate.

[0110] Example 3

[0111] According to an embodiment of the present application, a code testing method is also provided. Figure 5 is a schematic diagram of an optional code testing method according to an embodiment of the present application, which is applicable to SQL code testing scenarios, such as Figure 5 As shown, the method includes:

[0112] 1. Use the questions and table structure information in the evaluation set to assemble code generation instructions.

[0113] The evaluation set includes multiple groups of data, each of which includes at least questions, truth values, and table structure information, where the truth value refers to the SQL code that implements the function requested by the question. The table structure information in the evaluation set is also the table structure information of the source table. The table structure information is used as prompt information for code generation in the code generation instruction. In addition to the table structure information of the source table, the prompt information in the code generation instruction can also include query dimensions, industry information of the industry to which the data in the source table belongs, etc.

[0114] 2. Call the evaluation object, and the input is the code generation instruction in the first step.

[0115] The participating object is a service for generating SQL codes, for example, it can be a text2sql participating object, which refers to a service for converting natural language text (text) into SQL codes. The code generation instruction in the first step is input into the text2sql participating object, and the text2sql participating object can generate SQL codes for implementing the functions requested by the question according to the code generation instruction.

[0116] 3. Obtain the returned results of the evaluation objects and extract the SQL code from them.

[0117] It should be noted that, since the text2sql evaluation object will add some explanation content in the process of code generation, the returned result contains not only the SQL code, but also can extract the SQL code from the returned result. For example, the SQL code can be intercepted by writing a good code interception script, and the code interception instruction can be input to instruct the large language model to intercept the SQL code, so as to obtain the SQL code to be tested.

[0118] 4. Assemble the judging instructions using the judging criteria, the SQL code returned by the participant, the table structure information in the evaluation set, the true value, the original question, etc.

[0119] Among them, the information in the evaluation set is used to assist in evaluating the quality of the SQL code generated by the text2sql evaluation object. The judgment standard is used to determine whether the processing logic of the SQL code is the same as the true value. The judgment condition information may include multiple judgment conditions. For example, the judgment conditions may include: whether the source tables are the same, whether the filtering conditions are the same, whether the fields extracted from the source tables are the same, whether the number of fields returned is the same, etc.

[0120] 5. Call the referee model and input the referee instructions generated in step 4.

[0121] The referee model can be a large language model in a general field. The large language model's understanding and reasoning capabilities are used to analyze the constructed instruction information, and the SQL code is compared with the true value based on the internal processing logic.

[0122] 6. Obtain the return result of the referee model, that is, the evaluation result of whether the processing logic of the generated code is consistent with the true value in the evaluation set.

[0123] The returned result can be yes or no. When the returned result is yes, it indicates that the processing logic of the SQL code is the same as that of the true value. When the returned result is no, it indicates that the processing logic of the SQL code is different from that of the true value. The returned result can also be similarity, indicating the probability that the processing logic of the SQL code is the same as that of the true value.

[0124] It should be noted that the solutions in the related art need to rely on a database when judging whether the code to be tested is consistent with the true value. Without a database, SQL cannot be executed, or certain engineering capabilities are required to perform SQL text comparison. The present embodiment uses the understanding and reasoning capabilities of a large language model to analyze whether the SQL code generated by the evaluation object is consistent with the internal processing logic of the true value. On the one hand, the evaluation speed is fast, with fewer dependencies, and there is no need to execute SQL. At the same time, it does not rely on the comparison of text representations. On the other hand, SQL comparison is performed based on the processing logic of data processing, thereby improving the accuracy of SQL comparison. When the amount of true value data in the evaluation set is small, the judgment accuracy is higher.

[0125] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0127] Example 4

[0128] According to an embodiment of the present application, a device for implementing the above code testing method is also provided. Figure 6 is a schematic diagram of a code testing device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0129] An acquisition unit 602 is used to acquire the code generated by the code generation service for realizing the target function, and obtain the code to be tested;

[0130] A generating unit 604 is used to generate instruction information according to the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code that implements the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information;

[0131] The determination unit 606 is used to input the instruction information into the large language model, and determine whether the code to be tested is accurate based on the return result of the large language model.

[0132] Optionally, in the code testing device of the present embodiment, the generation unit 604 includes: an information acquisition module, used to obtain target problem information and target prompt information used by the code generation service to generate the code to be tested, wherein the target problem information is used to request the generation of code to implement the target function, and the target prompt information describes the information required to generate the code to implement the target function; an instruction information generation module, used to generate instruction information based on the judgment condition information, the code to be tested, the reference code, the target problem information and the target prompt information.

[0133] Optionally, in the code testing device of this embodiment, the determination unit 606 includes: a first input module, used to input instruction information into N large language models to obtain N return results, where N is a positive integer greater than 2; a first determination module, used to determine that the code to be tested is accurate when a return result greater than or equal to a first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code; and a second determination module, used to determine that the code to be tested is inaccurate when a return result less than the first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code.

[0134] Optionally, in the code testing device of the present embodiment, the determination unit 606 includes: a second input module, used to input instruction information into the large language model to obtain a return result; a third determination module, used to determine that the code to be tested is accurate when the return result indicates that the processing logic of the code to be tested is the same as that of the reference code; a fourth determination module, used to update the instruction information by adding prompt information when the return result indicates that the processing logic of the code to be tested is different from that of the reference code, and input the updated instruction information into the large language model until the number of updates of the instruction information does not reach a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is the same as that of the reference code, and the code to be tested is determined to be accurate, or until the number of updates of the instruction information reaches a preset number of updates and the updated return result indicates that the processing logic of the code to be tested is different from that of the reference code, and the code to be tested is determined to be inaccurate.

[0135] Optionally, in the code testing device of the present embodiment, the device further includes: a first execution unit, used to respectively execute the code to be tested and the reference code to obtain a first execution result and a second execution result before generating instruction information based on the judgment condition information, the code to be tested and the reference code; and a second execution unit, used to execute the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code when both the first execution result and the second execution result are empty.

[0136] Optionally, in the code testing device of the present embodiment, the device further includes: a third execution unit, used to respectively execute the code to be tested and the reference code to obtain a first execution result and a second execution result before generating instruction information according to the judgment condition information, the code to be tested and the reference code; and a fourth execution unit, used to, when the first execution result and the second execution result are the same, if the number of code lines of the first execution result and the second execution result are both less than a preset number of lines, execute the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code.

[0137] Optionally, in the code testing device of the present embodiment, the device also includes: a fifth execution unit, used to execute the code to be tested and the reference code respectively before generating instruction information according to the judgment condition information, the code to be tested and the reference code to obtain a first execution result and a second execution result; a sixth execution unit, used to, when the first execution result and the second execution result are different, if a code accuracy verification instruction is received, execute the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code; a seventh execution unit, used to, when the first execution result and the second execution result are the same and the number of code lines of the first execution result and the second execution result are both greater than or equal to a preset number of lines, if a code accuracy verification instruction is received.

[0138] Optionally, in the code testing device of this embodiment, when the judgment condition information includes at least two judgment conditions, the instruction information is also used to indicate that the code to be tested does not meet the judgment condition when the processing logic of the code to be tested and the reference code are different.

[0139] It should be noted that the above-mentioned units or modules correspond to the steps in Example 1, and the examples and application scenarios implemented by the units or modules are the same as those of the corresponding steps, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0140] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0141] Example 5

[0142] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0143] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0144] In this embodiment, the computer terminal can execute the program code in the method.

[0145] Optionally, Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device A may include: one or more ( Figure 7 (only one is shown) processor 102, memory 104, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0146] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the code generated by the code generation service for realizing the target function and obtain the code to be tested; generate instruction information according to the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code to realize the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; input the instruction information into the large language model, and judge whether the code to be tested is accurate based on the return result of the large language model.

[0148] Using the embodiment of the present application, a large language model is used to determine whether the processing logic of the code to be tested and the reference code are the same, and then the accuracy of the code to be tested in achieving the target function is determined based on the output result of the large language model. When constructing the instruction input into the large language model, not only the code to be tested and the reference code themselves are considered, but also the judgment condition information. By combining the judgment condition information, the code to be tested, and the reference code to generate instruction information, and inputting the generated instructions into the large language model, the large language model can accurately determine whether the processing logic of the code to be tested and the reference code are the same based on the input instructions, thereby achieving the technical effect of accurately determining the accuracy of the code to be tested in achieving the target function based on the output result of the large language model, thereby solving the technical problem in the related art of comparing the execution results or code texts of the test code and the reference code to determine whether the code to be tested generated by the code generation service is accurate, and the judgment result is inaccurate.

[0149] Those skilled in the art will understand that Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 7 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Figure 7More or fewer components (such as network interfaces, display devices, etc.) shown in the figure, or having Figure 7 Different configurations are shown.

[0150] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0151] Example 6

[0152] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0153] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0154] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the code generated by the code generation service for implementing the target function to obtain the code to be tested; generating instruction information based on the judgment condition information, the code to be tested, and the reference code, wherein the reference code refers to the code to implement the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same based on the judgment condition information; inputting the instruction information into the large language model, and judging whether the code to be tested is accurate based on the return result of the large language model.

[0155] Example 7

[0156] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0157] Example 8

[0158] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0159] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0160] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0165] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A code testing method, characterized in that: include: Obtain the code generated by the code generation service for implementing the target function, and obtain the code to be tested; Generate instruction information according to the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code that implements the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; The instruction information is input into a large language model, and whether the code to be tested is accurate is determined based on a return result of the large language model.

2. The method according to claim 1, characterized in that: Generating instruction information according to the judgment condition information, the code to be tested and the reference code includes: Obtaining target problem information and target prompt information used by the code generation service to generate the code to be tested, wherein the target problem information is used to request generation of code to implement the target function, and the target prompt information describes information required to generate the code to implement the target function; Instruction information is generated according to the judgment condition information, the code to be tested, the reference code, the target problem information and the target prompt information.

3. The method according to claim 1, characterized in that Inputting the instruction information into a large language model, and judging whether the code to be tested is accurate based on a return result of the large language model includes: Input the instruction information into N large language models to obtain N return results, where N is a positive integer greater than 2; In the case where the returned results greater than or equal to the first preset ratio indicate that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; In a case where the returned result smaller than the first preset ratio indicates that the processing logic of the code to be tested is the same as that of the reference code, it is determined that the code to be tested is inaccurate.

4. The method according to claim 1, characterized in that Inputting the instruction information into a large language model, and judging whether the code to be tested is accurate based on a return result of the large language model includes: Input the instruction information into the large language model to obtain a return result; In the case where the returned result indicates that the processing logic of the code to be tested is the same as that of the reference code, determining that the code to be tested is accurate; In the case where the return result indicates that the processing logic of the code to be tested is different from that of the reference code, the instruction information is updated by adding prompt information, and the updated instruction information is input into the large language model until the number of updates of the instruction information does not reach the preset number of updates and the updated return result indicates that the processing logic of the code to be tested is the same as that of the reference code, and the code to be tested is determined to be accurate, or until the number of updates of the instruction information reaches the preset number of updates and the updated return result indicates that the processing logic of the code to be tested is different from that of the reference code, and the code to be tested is determined to be inaccurate.

5. The method according to any one of claims 1 to 4, characterized in that: Before generating instruction information according to the judgment condition information, the code to be tested and the reference code, the method further includes: Executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; When both the first execution result and the second execution result are empty, the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code is performed.

6. The method according to any one of claims 1 to 4, characterized in that Before generating instruction information according to the judgment condition information, the code to be tested and the reference code, the method further includes: Executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; When the first execution result and the second execution result are the same, and if the number of code lines of the first execution result and the second execution result are both less than a preset number of lines, the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code is executed.

7. The method according to any one of claims 1 to 4, characterized in that Before generating instruction information according to the judgment condition information, the code to be tested and the reference code, the method further includes: Executing the code to be tested and the reference code respectively to obtain a first execution result and a second execution result; In the case where the first execution result and the second execution result are different, if the code accuracy verification instruction is received, the step of generating instruction information according to the judgment condition information, the code to be tested and the reference code is executed; When the first execution result and the second execution result are the same, and the number of code lines of the first execution result and the second execution result are both greater than or equal to the preset number of lines, if the code accuracy verification instruction is received, the step of generating instruction information based on the judgment condition information, the code to be tested and the reference code is executed.

8. The method according to claim 1, characterized in that In the case where the judgment condition information includes at least two judgment conditions, the instruction information is further used to indicate that when the processing logic of the code to be tested is different from that of the reference code, a judgment condition that the code to be tested does not meet is prompted.

9. A code testing method, characterized in that: include: The cloud server receives the code generated by the code generation service for implementing the target function and obtains the code to be tested; The cloud server receives instruction information generated by the client according to the judgment condition information, the code to be tested and the reference code, wherein the reference code refers to the code that implements the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; The cloud server processes the instruction information through a large language model, and determines whether the code to be tested is accurate based on a return result of the large language model; The cloud server sends the accuracy of the code to be tested to the client.

10. A code testing device, characterized in that: include: An acquisition unit, used to acquire the code generated by the code generation service for realizing the target function, and obtain the code to be tested; A generating unit, configured to generate instruction information according to the judgment condition information, the code to be tested and a reference code, wherein the reference code refers to a code for realizing the target function, and the instruction information indicates whether the processing logic of the code to be tested and the reference code is the same according to the judgment condition information; The determination unit is used to input the instruction information into the large language model, and determine whether the code to be tested is accurate based on the return result of the large language model.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 9 when running.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

Images