Code generation method, code generation device and related products

Abstract

This disclosure provides a code generation method, a code generation device, and related products. The code generation method includes: generating code to be tested using a first model based on prompt information; generating a test strategy corresponding to the code to be tested using a second model based on the prompt information and / or the code to be tested; testing the code to be tested based on the test strategy to obtain test results; wherein the test results are used to perform adversarial updates on the first and second models; and obtaining target code based on the test results and the code to be tested. This method can generate code with strong generalization ability, conforming to code rules, high execution efficiency, and robustness.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the field of high-performance computing and parallel programming technology, and particularly to a code generation method, code generation device, electronic device, chip, readable storage medium, and program product. Background Technology

[0002] Currently, Large Language Models (LLMs) have been applied to automatically generate code based on natural language prompts. In the field of high-performance computing, researchers have begun exploring the use of LLMs to generate parallel code for GPUs (Graphics Processing Units) to lower the barrier to GPU programming. Currently, the code generated by LLMs is mainly evaluated based on reinforcement learning with a fixed reward function. However, this method has significant drawbacks due to the static invariance of the reward function. Summary of the Invention

[0003] This disclosure presents a code generation method, a code generation apparatus, and related products.

[0004] In a first aspect, embodiments of this disclosure propose a code generation method, comprising: generating code to be tested using a first model based on prompt information; generating a test strategy corresponding to the code to be tested using a second model based on the prompt information and / or the code to be tested; testing the code to be tested based on the test strategy to obtain test results; wherein the test results are used to perform adversarial updates on the first and second models; and obtaining target code based on the test results and the code to be tested.

[0005] Secondly, embodiments of this disclosure propose a code generation apparatus, comprising: a first generation module configured to generate code to be tested based on prompt information using a first model; a second generation module configured to generate a test strategy corresponding to the code to be tested using a second model based on prompt information and / or the code to be tested; a testing module configured to test the code to be tested based on the test strategy and obtain test results; wherein the test results are used to perform adversarial updates on the first model and the second model; and a third generation module configured to obtain target code based on the test results and the code to be tested.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement, when executing instructions stored in the memory, a code generation method as described in any implementation of the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a chip including a processor capable of executing a code generation method as described in any implementation of the first aspect.

[0008] Fifthly, embodiments of this disclosure provide a non-volatile computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, can implement the code generation method as described in any implementation of the first aspect.

[0009] In a sixth aspect, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the code generation method as described in any implementation of the first aspect.

[0010] According to the technical solution provided in this disclosure, firstly, a first model generates code to be tested based on prompt information; then, a second model generates a test strategy corresponding to the code to be tested based on the prompt information and / or the code to be tested, i.e., the test strategy is dynamically generated and strongly correlated with the code to be tested; finally, a comprehensive test is performed on the code to be tested based on the test strategy to obtain test results for adversarial updates of the first and second models, and the target code is obtained based on the test results and the code to be tested. In this solution, since the test strategy is dynamically generated rather than pre-set, test cases and constraints can be adaptively designed for the generated code to be tested to adversarially update the model based on the test results and obtain the target code. This overcomes the limitations of fixed reward mechanisms and solves the overfitting problem caused by fixed reward mechanisms, enhances the generalization ability and robustness of the code, ensures the quality and compliance of the code, and thus improves the performance and usability of code generation.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart of a code generation method provided in this disclosure embodiment Figure 1 ; Figure 3 A flowchart of a method for updating an actor model provided in embodiments of this disclosure; Figure 4 A flowchart of a method for updating an evaluator model provided in embodiments of this disclosure; Figure 5 A flowchart of a method for generating a test strategy provided in this embodiment of the disclosure; Figure 6 A flowchart of a code generation method provided in this disclosure embodiment Figure 2 ; Figure 7 A structural block diagram of a code generation apparatus provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for executing a code generation method, provided as an embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] This disclosure relates to the field of artificial intelligence technology, specifically to the subfield of high-performance computing and GPU parallel programming technology. It is applied to scenarios requiring the generation of high-quality GPU parallel code, such as M-architecture kernel functions or M-computing platform kernel functions. Here, M-architecture or M-computing platform refers to any general-purpose parallel computing platform and programming model that supports multiple programming languages ​​such as C, C++, and Python, and enables collaborative computing between the CPU and GPU. Examples include Meta-computing Unified System Architecture (MUSA) and Compute Unified Device Architecture (CUDA).

[0015] In high-performance computing and GPU parallel programming, automatically generating correct and efficient parallel code is key to lowering the development threshold and improving productivity. Currently, reinforcement learning schemes based on fixed reward functions are the mainstream approach to optimizing LLM code generation. The implementation of this scheme in M-architecture code generation tasks is typically as follows: 1) First, define a fixed reward function. The core logic of this reward function is to execute a fixed, pre-written set of M-architecture unit tests. If the code passes the tests, it receives a high reward; otherwise, it receives a low reward. 2) Set up a reinforcement learning loop: a) Receive task prompts through the actor model, i.e., the LLM model, such as "Write an M-architecture matrix multiplication kernel function," and generate code. b) Compile and execute the M-architecture code using the fixed reward function, and evaluate it using the fixed set of unit tests (pre-defined unit tests, such as tensors of fixed shapes). c) The reward function returns a scalar reward R, for example: if the test output is correct, then R = +1; otherwise, R = 0. 3) Model update: The actor model uses a reinforcement learning algorithm, such as the GRPO algorithm, to update its policy based on this scalar reward R, with the goal of maximizing the probability of obtaining R = +1.

[0016] However, this method has significant drawbacks: 1) The reward function is predefined and static, which makes the model prone to overfitting to specific test configurations and lacks robustness: The reward function does not change during reinforcement learning training. The optimization goal of the reinforcement learning algorithm is to maximize the probability of passing this fixed set of tests. Therefore, the model will overfit to these specific test cases. For example, when training the M-architecture kernel function, if the fixed test cases only use tensor shape tensor_shape=[256,256] and precision='float32', the M-architecture code generated by the model may only be correct under this specific configuration.

[0017] Therefore, the models trained by these reinforcement learning schemes are not robust. If the shape or precision of the input tensor changes, or if the input edge case changes, the model will report an error. That is, the model has not learned general M-architecture programming, but has only learned to pass the current specific test, which cannot reach the true industrial level.

[0018] 2) Ignoring the performance goals of M-architecture programming: The reward function of reinforcement learning schemes only checks logical correctness, completely ignoring code speed, i.e., execution efficiency. For M-architecture programming, its sole purpose is high performance. A logically correct but inefficient implementation, such as one that does not fully utilize shared memory, causes bank conflicts, or has improper grid block configuration, will still receive a reward of R=+1. This completely violates the original intention of high-performance computing, rendering reinforcement learning optimization meaningless.

[0019] 3) The optimization objective of reinforcement learning is flawed, causing the model to learn to exploit loopholes: This is another manifestation of overfitting. Because the reward function itself is static and flawed (only checking the output), the reinforcement learning algorithm can correctly guide the model to learn to cheat. For example, in the task of writing the Softmax kernel function of the M architecture, if the fixed reward function only checks the output, the model will quickly learn that calling the pre-compiled kernel function in a third-party library is the optimal strategy to obtain R=+1, making the generated code unable to achieve autonomous programming.

[0020] To address the aforementioned issues, this disclosure provides a code generation method, a code generation device, and related products, aiming to solve at least some of the technical problems in the prior art, such as weak generalization ability, insufficient robustness, poor performance, and weak compliance verification, through a dynamic and adaptive multi-dimensional testing strategy.

[0021] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the code generation method, code generation apparatus and related products of this disclosure can be applied.

[0022] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0023] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include code generation applications, model management applications, and instant messaging applications.

[0024] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0025] Server 105 can provide various services through its built-in applications. Taking a code generation application as an example, when running this application, Server 105 can achieve the following: First, it generates code to be tested based on prompts; then, it dynamically generates a test strategy based on the prompts and / or the code to be tested; finally, it performs comprehensive testing on the code to be tested based on the test strategy, and optimizes the target code based on the test results. Through dynamic test strategies, test cases and constraints can be adaptively designed for the generated code to be tested, and the code can be iteratively optimized based on the test results. This solves the overfitting problem caused by fixed reward mechanisms, enhances the generalization ability and robustness of the code, ensures code quality and compliance, and thus improves the performance and usability of code generation.

[0026] It should be noted that, in addition to being obtained from terminal devices 101, 102, and 103 via network 104, the prompt information can also be pre-stored locally on server 105 through various means. Therefore, when server 105 detects that this data is already stored locally (for example, generating test code directly based on locally stored prompt information), it can choose to retrieve this data directly from the local storage. In this case, the exemplary system architecture 100 may not include terminal devices 101, 102, and 103 and network 104.

[0027] Since testing the code to be tested based on the testing strategy requires significant computing resources and strong computing power, the code generation methods provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the code generation device is also generally located in the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also have sufficient computing power and resources, they can also complete the aforementioned calculations performed by the server 105 through the code generation application installed on them, and thus output the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the code generation application determines that the terminal device has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the code generation device and the code generation agent can also be located in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0028] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0029] Please refer to Figure 2 , Figure 2 A flowchart of a code generation method provided in this disclosure embodiment Figure 1 Process 200 includes the following steps: Step 201: Generate the code to be tested based on the prompt information using the first model.

[0030] In this embodiment, the prompt information refers to input data used to describe the task objectives, functional requirements, constraints, and contextual information of the code to be generated, providing the specific intent and boundary conditions required to generate the code. The prompt information is typically a piece of natural language text, and its content includes, but is not limited to: the functions the code needs to implement, the programming language and environment, performance requirements, and prohibitive constraints, such as: Please implement a Softmax function using Python and the M-architecture, do not use existing softmax functions from torch or cupy, and high efficiency is required, etc. This embodiment uses the M-architecture as an example for illustration.

[0031] The code to be tested refers to the source code initially generated by the first model based on the prompts, which has not yet undergone systematic verification. For example, this code might be a real M-architecture core that can pass a simple dynamic test of [128, 128]. However, its implementation is very rudimentary and lacks optimization for shared memory. Another example is M-architecture parallel computing code written for execution on a GPU. While the code to be tested is syntactically complete and can be parsed by the compiler, its logical correctness, generalization ability to input variations, constraint compliance, and execution performance are all undetermined and require multi-dimensional testing and evaluation using subsequent testing strategies.

[0032] The actor model, also known as the first model, is a pre-trained model that is fine-tuned or specifically trained for code generation tasks, such as GPU parallel code generation. The goal of the actor model is to receive M-architecture task prompts, generate M-architecture code, and maximize its probability of passing the test strategy. Specifically, after receiving prompts from the user or upstream system, the actor model encodes and understands the prompts, and then generates source code strings that conform to the syntax of the programming language—that is, the code to be tested.

[0033] It should be noted that the aforementioned actor model can be a model with corresponding analysis and code generation capabilities obtained by training on any general model architecture. This general model architecture includes, but is not limited to, one or more combinations of models such as machine learning models, deep learning models, neural network models, large language models (LLM) for processing text data, large vision models (LVM) for processing visual data, and multimodal large models (MLM) for processing multimodal data. The specific model structures of the various general model architectures mentioned above will not be elaborated here.

[0034] Step 202: Generate a test strategy corresponding to the code to be tested based on the prompt information and / or the code to be tested using the second model.

[0035] In this embodiment, the test strategy corresponding to the code to be tested refers to a set of machine-readable and executable instructions to perform multi-dimensional and targeted testing and evaluation of the code to be tested, including multiple sub-strategies such as dynamic analysis strategy, static analysis strategy and performance analysis strategy.

[0036] The testing strategy can be dynamically and adaptively generated through the evaluator model, i.e., the second model. Unlike the actor model that generates the code, this model is a pre-trained model that is fine-tuned or specifically trained to analyze and predict the current task requirements, i.e., the prompts, and / or the generated code to be tested, and generate a set of machine-readable, rigorous, multi-layered testing strategies. This testing strategy can effectively detect and verify potential defects in the code to be tested, such as logical errors, compliance vulnerabilities, or performance bottlenecks, and can adaptively increase the difficulty as the actor model evolves, for example, by designing more difficult tensor shapes and stricter performance time limits, so as to make the generated code more robust, compliant, and high-performance.

[0037] The evaluator model can generate test strategies corresponding to the code under test based solely on the prompt information. For example, if the prompt information is "Write an M-architecture matrix multiplication kernel function, and prohibit the use of the ××× library," the evaluator model can analyze this prompt information to predict that the code under test may have compliance defects in calling third-party library functions, and generate a static analysis strategy that includes the names of functions to be prohibited from being called. At the same time, based on the high-performance requirements of the task, it can predict that the code under test may have performance defects in the improper use of shared memory, and generate a performance analysis strategy that includes specific tensor shapes, etc., to generate a test strategy corresponding to the code under test.

[0038] The evaluator model can also receive and analyze the generated code to be tested. By understanding its algorithmic logic, code structure, API call patterns, etc., it can proactively predict potential weaknesses or cheating issues. For example, it might determine that a loop structure in the code will cause performance degradation under certain input shapes, or suspect that it may contain hidden illegal function calls. Based on this prediction, it generates test strategies corresponding to the code to be tested to attack these predicted weaknesses.

[0039] The evaluator model can also jointly generate a test strategy corresponding to the code under test based on the prompt information and the code under test. For example, if the prompt information requires "implementing an efficient Softmax kernel function", after parsing the prompt information, the evaluator model analyzes the loop structure and memory access pattern of the code under test, predicts that it may have a generalization defect of memory access out of bounds when handling tensor shapes that are not powers of 2, and generates a dynamic analysis strategy that includes odd-dimensional inputs accordingly. At the same time, based on the shared memory configuration of the code, it predicts that it may have a performance defect of thread bundle divergence when processing tall and thin matrices, and generates a performance analysis strategy that includes inputs with specific aspect ratios, etc., to generate a test strategy corresponding to the code under test.

[0040] It should be noted that the aforementioned evaluator model can be a model trained on any general model architecture to obtain the corresponding analysis and code generation capabilities. This general model architecture includes, but is not limited to, one or more combinations of models such as machine learning models, deep learning models, neural network models, large language models for processing text data, large visual models for processing visual data, and large multimodal models for processing multimodal data. The specific model structures of the various general model architectures mentioned above will not be elaborated here.

[0041] Step 203: Test the code to be tested based on the testing strategy and obtain the test results.

[0042] For example, the test results are used to perform adversarial updates on the first and second models.

[0043] In this embodiment, a verification service, i.e., an automated programmatic module, can compile, execute, and comprehensively test and evaluate the code under test based on a testing strategy. For example, it can use input data provided by the dynamic analysis strategy in the testing strategy to perform calculations and compare the output of the code under test with the standard answer (usually generated by known correct reference code) to obtain test results regarding the logical correctness and adaptability to specific inputs of the code. Alternatively, code analysis tools, such as abstract syntax tree analyzers and regular expression matchers, can be invoked to directly analyze the text or structure of the code without executing it, checking whether it violates the constraint rules defined by the static analysis strategy in the testing strategy, such as whether it contains prohibited import statements, thus obtaining test results regarding the code's compliance. Furthermore, input data provided by a performance analysis strategy can be used to accurately measure its runtime or resource consumption on actual hardware, such as a GPU, and compare this performance metric with preset thresholds in the strategy or the performance baseline of the reference code to obtain test results regarding the code's execution efficiency.

[0044] The final test result is a comprehensive conclusion report. It can be a simple binary scalar, such as pass or fail, or a structured data object that details the pass / fail status and reasons for failure for each test dimension, such as the input conditions under which the dynamic test failed, which static constraint was violated, the performance difference, and the possible quantification score. This test result not only characterizes the quality status of the code under test but also provides optimization directions for the adversarial updates of the actor and evaluator models. For example, reward values ​​can be determined for the actor and evaluator models based on the test results: if the code under test fails the test strategy, the actor model receives a low reward and is updated based on this reward value to optimize code generation capabilities; the evaluator model receives a high reward and is updated based on this reward value to increase the stringency of the test strategy; if the code under test passes the test strategy, the actor model receives a high reward, and the evaluator model receives a low reward. Through this adversarial update mechanism, the actor and evaluator models can mutually promote and co-evolve during the iteration process.

[0045] Step 204: Based on the test results and the code to be tested, obtain the target code.

[0046] In this embodiment, when the actor model is powerful or the task is simple, the code to be tested generated by the actor model in the first round can pass the test strategy generated by the evaluator model, and this code can be identified as the target code. However, in more common cases, the code to be tested generated by the actor model in the first round cannot pass the test strategy generated by the evaluator model. The actor model and the evaluator model then evolve continuously in an adversarial process. That is, the actor model continuously optimizes and updates the code to be tested based on the test results, and the evaluator model continuously generates new test strategies for the new code to be tested generated by the actor model and tests them to obtain new test results. Until, in a certain iteration, when the code to be tested generated by the evaluator model can stably pass the current test strategy, the code generated in that round is considered the target code.

[0047] By employing dynamic testing strategies, test cases and constraints can be adaptively designed for the generated code to be tested. Based on the test results, the code to be tested can be iteratively optimized, which solves the overfitting problem caused by fixed reward mechanisms, enhances the generalization ability and robustness of the code, and ensures the quality and compliance of the code, thereby improving the performance and usability of the generated code.

[0048] In some optional implementations of the embodiments of this disclosure, the testing strategy includes at least one of the following: a dynamic analysis strategy, a static analysis strategy, and a performance analysis strategy; wherein, the dynamic analysis strategy is used to test the correctness and / or generalization ability of the code under test; the static analysis strategy is used to test whether the code under test conforms to the rules; and the performance analysis strategy is used to test the execution efficiency of the code under test.

[0049] In this embodiment, the dynamic analysis strategy verifies whether the output of the code, given input, numerically matches the expected result (typically generated by a known correct reference implementation), ensuring that the basic functional logic is correct and testing the correctness of the code. The dynamic analysis strategy also incentivizes the evaluator to generate inputs including edge cases and unconventional precisions, such as tensor shapes other than powers of 2, odd-sized inputs, empty tensors, and different data precisions like float16, float32, and int8. By requiring the code under test to still produce correct outputs when processing these inputs, the robustness and generalization ability of the code under test are evaluated and improved, preventing overfitting to simple use cases.

[0050] Static analysis strategies can analyze the source code text or structure directly without running the code. For example, they can detect whether the code under test imports prohibited libraries or functions, contains necessary syntax elements, conforms to specific coding standards, or has potential unsafe patterns, in order to verify the compliance of the process and prevent cheating.

[0051] Performance analysis strategies are the core metrics of M-architecture tasks. They can verify whether the output of code under a given input configuration, such as a specific tensor shape, precision, etc. that slows down the M-architecture code output by the actor model, matches the expected result (usually generated by a known correct reference implementation) numerically, in order to evaluate the execution efficiency of the code under test.

[0052] By employing a multi-dimensional testing strategy, we can provide richer and more accurate test results for the actor model. This can also prevent the actor model from overfitting to fixed test cases or generating non-compliant, logically correct but inefficient code to be tested, thereby improving the robustness, compliance, and performance of the model.

[0053] In some optional implementations of the embodiments of this disclosure, the testing strategy includes: a dynamic analysis strategy; the step of testing the code to be tested based on the testing strategy and obtaining the test result includes: processing the first input data in the dynamic analysis strategy through the code to be tested to obtain first output data; wherein, the first input data is determined based on the boundary within the effective input range of the code to be tested; obtaining a first result based on the first output data and the second output data; wherein, the second output data is obtained by processing the first input data through reference code corresponding to the code to be tested, and the test result includes the first result.

[0054] In this embodiment, the evaluator model is incentivized to generate inputs for common edge cases of the M-architecture kernel function. That is, the first input data is determined based on the boundaries of the effective input range of the code under test. For example, tensor shapes that are not powers of 2, inputs of odd size, empty tensors, float16, float32, and int8, etc., with different data precisions, are used to specifically test the output correctness and generalization ability of the code under test, and to prevent the code under test generated by the actor model from overfitting to simple use cases.

[0055] The code to be tested is compiled and run using the first input data to obtain the corresponding output result, i.e., the first output data. Simultaneously, existing, correct reference code corresponding to the code to be tested is compiled and run using the same first input data to obtain the standard answer corresponding to that input, i.e., the second output result, thus forming an input-output pair. The first output data and the second output data are compared, for example, through element-by-element numerical comparison or structural checks. Based on the comparison result, a first result is generated. This first result can be a Boolean value, such as "test passed" or "test failed," or a continuous score, such as the similarity percentage between the first and second output results. Essentially, it reflects the correctness of the code to be tested when processing specific inputs and its generalization ability to unconventional inputs.

[0056] It should be noted that the reference code and the code under test are used to implement the same data processing, or the two have the same processing logic, to ensure that the first output result of the code under test and the second output result of the reference code are comparable.

[0057] By using diverse inputs generated by the evaluator model that include edge cases for testing, and comparing the output of the code under test with the standard answer, the correctness of the generated code's output and its generalization ability to unknown inputs can be effectively evaluated, thus solving the problem of model overfitting to a fixed test set.

[0058] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a static analysis strategy; in this case, the step of testing the code to be tested based on the testing strategy to obtain the test result includes: analyzing the code to be tested based on the static analysis strategy to obtain a second result; wherein the test result includes the second result, and the static analysis strategy includes: prohibited function names and / or required string names.

[0059] In this embodiment, the static analysis strategy includes multiple rule sets generated by the evaluator model, such as prohibited API calls, library imports, and function names that are prohibited from appearing or being called in the code and / or string names that must be included, in order to check the process compliance and honesty of the code and prevent cheating.

[0060] By invoking appropriate static code analysis tools, such as abstract syntax tree parsers, regular expression engines, or other code analysis tools, the plain text or syntax structure of the code to be tested can be analyzed to match and check the code content against each rule in the static analysis strategy. For example, statements like `{"type":"FORBIDDEN_IMPORT","module_name":"torch"}` prevent the use of pre-written tools instead of writing custom code to "cheate." Statements like `{"type":"REQUIRED_STRING_PATTERN","pattern":"__global__"}` ensure the generation of M-architecture kernel functions. Statements like `{"type":"FORBIDDEN_API", "call_name":"printf"}` disable printing within the kernel. Alternatively, checks can be performed to see if the string "import torch" appears in the code text or if the abstract syntax tree contains a "__global__" function definition node. Based on the results of all these checks, a second result is generated. This result indicates whether the code under test has passed the constraint checks of the static analysis strategy, and if violations exist, which specific rule has been violated.

[0061] By employing a static analysis strategy, the evaluator model can verify the code generation process, preventing the actor model from circumventing tests by calling illegal functions or omitting core structures, thus ensuring the compliance of the code generation process.

[0062] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a performance analysis strategy. In this case, the step of testing the code under test based on the testing strategy and obtaining the test result includes: processing the second input data in the performance analysis strategy through the code under test to obtain first performance information of the code under test; wherein the second input data includes tensor shape and data precision, and the first performance information is used to indicate the performance of the code under test in the process of processing the second input data; and obtaining a third result based on the first performance information and the second performance information; wherein the second performance information is obtained in the process of processing the second input data through reference code corresponding to the code under test, and the test result includes the third result.

[0063] In this embodiment, performance analysis can examine the execution efficiency of the code. For example, the evaluator model proactively generates input configurations that may lead to performance bottlenecks and sets GPU execution time limits. Performance analysis is a core metric for the M-architecture task. Specifically, the evaluator model proactively generates input configurations that it predicts might slow down the M-architecture code output by the actor model. That is, the second input data, such as specific tensor shapes and data precision that slow down the M-architecture code output by the actor model. The tensor shape represents the dimensional composition of the second input data, and the data precision represents the numerical storage format of the second input data. Its purpose is to expose potential performance bottlenecks in the code under test. For example, due to improper use of shared memory or thread bundle divergence, the evaluator model may find that the code output by the actor model is slow in processing "tall and thin" matrices. In this case, the evaluator model can specifically generate such tensor shapes to attack the performance of the actor model.

[0064] The code under test performs calculations using the second input data described above. During this process, runtime information such as GPU kernel execution time, memory throughput, or computational unit utilization can be collected using hardware performance counters or timers to obtain first performance information. Simultaneously, existing, correct reference code corresponding to the code under test is compiled and run using the second input data to test the normal time required for this input configuration, obtaining second performance information. The first and second performance information are compared to generate a third result. For example, comparing their execution times, if the M-architecture code written using the actor model requires more time to execute the input configuration than the normal time, the third result indicates that the code only optimizes the input for a specific tensor shape and lacks sufficient robustness.

[0065] It should be noted that the reference code and the code under test are used to implement the same data processing, or the two have the same processing logic, to ensure that the first performance information output by the code under test and the second performance information output by the reference code are comparable.

[0066] Performance analysis can transform code execution efficiency into a quantifiable and optimizable training objective, ensuring that the code generated by the actor model is not only logically correct but also performs well on actual hardware.

[0067] In some optional implementations of the embodiments of this disclosure, the test result is used to indicate whether the test passed or failed; based on the test result and the code to be tested, the target code is obtained, including: if the test result indicates that the test failed, performing the next round of code to be tested generation operation, test strategy generation operation, and test operation based on the updated first model and the updated second model; wherein, the updated first model and the updated second model are obtained by updating based on the test result; if the test result indicates that the test passed, the code to be tested is determined as the target code.

[0068] In this embodiment, the test result includes at least one of the first result, the second result, and the third result. Only when the first result indicates that the code under test has passed the dynamic analysis test in the test strategy, the second result indicates that the code under test has passed the static analysis test in the test strategy, and the third result indicates that the code under test has passed the performance analysis test in the test strategy, that is, the currently generated code under test has met the stringent requirements of the test strategy in all dimensions, such as correctness and / or generalization ability, whether it conforms to the rules, and execution efficiency, etc., then the test result is used to indicate that the test has passed, and the code under test can be directly identified as the target code.

[0069] When at least one of the first, second, and third results indicates that the code under test has failed the dynamic analysis test and / or static analysis test and / or line analysis test in the test strategy, it indicates that the currently generated code under test has one or more defects, and this test result is used to indicate that the test has failed. At this time, the actor model and evaluator model can be optimized iteratively based on the test results to obtain the updated actor model and the updated evaluator model, namely the updated first model and the updated second model, to execute the next round of code under test generation operation, test strategy generation operation, and test operation. For example, the feedback information contained in the test results, such as which test strategy failed and the specific reason for the failure, can be used to correct or update the model until, in a certain iteration, the code under test can stably pass the current test strategy, then the code generated in that round is regarded as the target code.

[0070] It should be noted that during an iterative update, both the first and second models can be updated simultaneously based on test results, or only one of them can be updated. For example, if the code under test fails the test, the actor model can be updated first to optimize code generation capabilities; or if the code under test passes the test multiple times consecutively, the evaluator model can be updated first to increase the rigor of the testing strategy. After updating the model, the newly generated code under test can be tested using the updated evaluator model based on the new testing strategy, or it can be left untested.

[0071] Furthermore, model updates are not limited to tests failing; successful tests can also lead to updates, although the direction and intensity of the updates may differ. For instance, if the code fails the test, the actor model receives a low reward and updates towards improving code correctness, while the evaluator model receives a high reward and updates towards improving test rigor. Conversely, if the code passes the test, the actor model receives a high reward and updates towards optimizing code efficiency, while the evaluator model receives a low reward and updates towards improving test targeting. Additionally, passing a test may not be the sole criterion for stopping iteration; even if a test passes, iteration may continue to further optimize the model.

[0072] Therefore, those skilled in the art can set and plan the above content according to the actual situation. This application does not limit this, as long as the technical principles of this application can be realized.

[0073] By processing the test code using the test results, the compliance, generalization ability, and performance of the final generated target code are ensured.

[0074] In some optional implementations of the embodiments of this disclosure, after obtaining the test results, the code generation method further includes: updating the first model based on the test results to obtain the updated first model; and updating the second model based on the test results to obtain the updated second model.

[0075] In this embodiment, the actor model and the evaluator model can continuously evolve in adversarial scenarios based on test results. For example, if the actor model fails a test, it receives a low reward and is updated based on that reward value to optimize its code generation capabilities, resulting in an updated actor model, i.e., the updated first model. If the evaluator model finds a vulnerability in the code under test, it receives a high reward and is updated based on that reward value to increase the stringency of the testing strategy, resulting in an updated evaluator model, i.e., the updated second model. Alternatively, if the actor model succeeds in a test, it receives a high reward and is updated based on that reward value to further optimize its code generation capabilities, resulting in an updated actor model, i.e., the updated first model. If the evaluator model fails to find a vulnerability in the code under test, it receives a low reward and is updated based on that reward value to increase the specificity and stringency of the testing strategy, resulting in an updated evaluator model, i.e., the updated second model.

[0076] Through the adversarial update mechanism, the actor model and the evaluator model can promote each other and evolve together, thereby improving the quality of code generation and the effectiveness of testing strategies.

[0077] Please refer to Figure 3 , Figure 3 A flowchart of a method 300 for updating an actor model provided in an embodiment of this disclosure, wherein method 300 may include the following steps: Step 301: Based on the test results, determine the first reward value of the first model.

[0078] In this embodiment, the first model is the actor model described above. For a detailed explanation of the first model, please refer to the above content, which will not be repeated here.

[0079] Multi-dimensional test results can be transformed into one or more scalars that can be used by reinforcement learning algorithms, namely, the first reward value. The first reward value is not a single signal, but a comprehensive quantification of the evaluation results across various dimensions of the test results, reflecting the quality of the code generated by the actor model for testing. Specifically, if the code for testing passes all test cases in the dynamic analysis strategy, such as edge case test cases, it receives a positive reward; if it fails on a test case, the corresponding reward can be deducted based on the difficulty or importance of the failed test case. If the code for testing passes the static constraint check, it receives a positive reward; any violation results in a failure, causing this part of the reward to be zero or negative. Rewards depend not only on whether the performance test is passed, but also on the degree of performance performance. For example, execution time far below the threshold receives a positive reward; just meeting the threshold receives a moderate reward; although the logic is correct, a severe timeout may result in a very low or even negative reward, thus guiding the model to optimize performance.

[0080] For example, given the prompt "Please implement a Softmax function using Python and the M-architecture, avoiding existing softmax functions from torch or cupy, and prioritizing high efficiency," the actor model generates corresponding test code. This code is a real M-architecture kernel function and passes a simple dynamic test on [128, 128]. However, its implementation is very rudimentary and lacks shared memory optimization. Upon receiving this test code, the evaluator model recognizes its poor robustness and generates a triple-validation strategy in JSON format: dynamic analysis, static analysis, and performance analysis, to defeat the code. In dynamic analysis, the test code is correct on [128, 128], but misconfigured memory accesses on odd-numbered tensor shapes such as [1, 133] lead to out-of-bounds memory access, causing the dynamic test to fail. In static analysis, the test code does not contain any illegal imports, thus passing the static test. In the performance analysis, the code under test on [8192, 2] had poor parallelism for this extreme tensor shape due to its naive implementation, taking 15ms to execute, which is greater than the standard time of 5ms. Therefore, the performance test failed. In summary, since both the dynamic test and the performance test failed, the overall validation failed, and the actor model obtained a low reward value.

[0081] Step 302: Update the first model based on the first reward value to obtain the updated first model.

[0082] In this embodiment, a reinforcement learning algorithm, such as a proximal policy optimization algorithm or other algorithms, can be used to update the actor model based on a first reward value. The goal is to adjust the model parameters to maximize the expected reward, that is, to maximize the probability that the actor model generates code that will obtain a high reward value given the prompt information, thus obtaining an updated actor model. Specifically, the policy gradient can be calculated based on the currently generated code and the obtained first reward value. This gradient indicates how the model parameters should be slightly adjusted so that the probability of generating code increases when obtaining a high reward and decreases when obtaining a low reward. The parameters are updated through gradient ascent (or its approximate optimization steps). After the parameter update, when faced with the same or similar task prompts, the statistical distribution of the code generated by the model will shift towards a direction that is easier to pass validation and obtain higher rewards. For example, if the code generated in the previous round was penalized for poor performance, the updated actor model will be more inclined to generate code structures with optimized memory access or parallel configuration.

[0083] By using reinforcement learning, multidimensional test results are transformed into reward values ​​that the actor model can optimize, and the model is driven to be updated. This achieves iterative optimization of code generation capabilities, enabling the actor model to generate code with strong generalization ability, compliance and high performance.

[0084] Please refer to Figure 4 , Figure 4 A flowchart of a method 400 for updating an evaluator model provided for embodiments of this disclosure, wherein method 400 may include the following steps: Step 401: Based on the test results, determine the second reward value for the second model.

[0085] In this embodiment, the second model is the evaluator model described above. For details regarding the second model, please refer to the above content, which will not be repeated here.

[0086] Unlike the actor model, the evaluator model's second reward value is based not only on the quality of its generated test strategy but also on the effectiveness of that strategy. In other words, the level of the second reward value depends on whether the evaluator model can successfully identify or defeat the code to be tested generated by the actor model. Specifically, if the test results indicate that the test strategy successfully detects any defect in the code across dimensions such as dynamic analysis, static analysis, or performance analysis, causing the code to fail the test, then the test strategy generated by the evaluator model is effective and should receive a high reward value. Conversely, if the code to be tested easily passes all tests in the test strategy, then the strategy is not rigorous enough or has failed to find any defects in the code, and should receive a low reward value.

[0087] Step 402: Update the second model based on the second reward value to obtain the updated second model.

[0088] In this embodiment, the parameters of the evaluator model can be updated using a reinforcement learning algorithm similar to that used to update the actor model, based on the calculated second reward value, to obtain an updated second model. This allows the testing strategy to be adjusted so that it can generate a testing strategy that is more likely to discover code defects when faced with similar tasks or code in the future.

[0089] By rewarding the generation of test strategies that effectively discover code defects, the evaluator model is driven to continuously learn and design more rigorous and accurate multi-dimensional tests, thereby forming a healthy competition with the actor model and improving the quality of generated code.

[0090] Optionally, the first and second models can be adversarially trained based on steps 301, 302, 401, and 402.

[0091] For example, a first model generates a test code sample based on a prompt information sample; a second model generates a test strategy sample corresponding to the test code sample based on the prompt information sample and / or the test code sample; the test strategy sample is used to test the test code sample to obtain a test result sample; the test result sample is used to perform adversarial updates on the first and second models; the reward value of the first model is determined based on the test result sample; the first model is updated based on the reward value of the first model to obtain the updated first model; the reward value of the second model is determined based on the test result sample; the second model is updated based on the reward value of the second model to obtain the updated second model. The model can be continuously optimized through adversarial training.

[0092] Optionally, prior to step 201, the first and second models may have been adversarially trained.

[0093] Optionally, code can be generated directly using a second model that has been trained adversarially, without requiring testing based on a testing strategy.

[0094] Please refer to Figure 5 , Figure 5 A flowchart of a method 500 for generating a test strategy provided in this disclosure embodiment is shown. Method 500 may include the following steps: Step 501: Generate predicted defect information based on the prompt information and / or the code to be tested using the second model; wherein the predicted defect information includes at least one of the following: the type of predicted defect, the location of the predicted defect, and the input data used to trigger the predicted defect.

[0095] In this embodiment, the evaluator model takes the received prompts and / or the code to be tested as input. By analyzing this information, it infers the most likely defect type, defect location, and input data used to trigger the predicted defect, among other defect information. Specifically, based on its training-gained knowledge of code patterns, common errors, performance pitfalls, and constraints, the evaluator model understands the intent and constraints in the prompts, extracts the task objectives and prohibitions from the prompts (e.g., prohibiting the use of torch), and analyzes the structure of the code to be tested, such as its algorithmic logic, API calls, memory access patterns, and parallel structures. Based on the above information, defect prediction is performed on the code to be tested, generating structured predicted defect information. For example, this code may only be optimized for inputs of powers of 2, and may have generalization defect prediction information such as accessing tensor [1, 133] out of bounds; or there may be compliance defect prediction information such as the code having an indirect path that calls "torch.nn.functional.softmax"; or there may be performance defect prediction information such as potential storage conflicts in the use of shared memory in this kernel function code, and the performance may drop sharply when processing tall and thin matrices [8192, 2].

[0096] Step 502: Generate a test strategy based on the predicted defect information using the second model.

[0097] In this embodiment, if the prediction code might err under specific tensor shapes or precision, the evaluator model creates corresponding unit test entries in the dynamic analysis strategy, using these prediction defect information as the first input data, and calculates or references the corresponding correct output as the comparison standard. If the prediction code might "cheate" in some way, the evaluator model adds or strengthens corresponding prohibition rules in the static analysis strategy to ensure that such behavior can be captured by static checks. If the prediction code might have inefficiency issues under specific input configurations, the evaluator model creates dedicated performance test entries in the performance analysis strategy, using this configuration as the second input data, and calculates or references the corresponding correct output time as the comparison standard. Finally, the evaluator model integrates and encapsulates all test entries and constraint rules designed to address prediction defects into a complete, machine-readable test strategy.

[0098] By enabling the evaluator model to predict potential defects in the code and then generate targeted test strategies, the accuracy and efficiency of testing are improved, thereby driving code quality optimization more effectively.

[0099] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a dynamic analysis strategy, a static analysis strategy, and a performance analysis strategy. Generating the testing strategy based on prompt information and / or the code to be tested includes: generating first input data in the dynamic analysis strategy based on the prompt information and / or the code to be tested; wherein the first input data is determined based on the boundaries within the valid input range of the code to be tested; generating a static analysis strategy based on the prompt information and / or the code to be tested; wherein the static analysis strategy includes at least one of the following: prohibited function names and / or required string names; generating a performance analysis strategy based on the prompt information and / or the code to be tested; wherein the performance analysis strategy includes second input data for testing the performance of the code to be tested, the second input data including tensor shape and / or data precision.

[0100] In this embodiment, in the dynamic analysis strategy, the actor model does not generate random inputs, but rather generates data related to the boundaries within the effective input range of the code under test. For example, for a matrix operation kernel function, the normal effective input might be a two-dimensional tensor of arbitrary regular shape. The evaluator model will specifically generate inputs of different precisions, such as tensor shapes of [1, 133] (unaligned odd dimensions), [0, 256] (zero element dimension), [4096, 4096] (maximum size), or data types of float16, int8, etc., to test the robustness of the code to edge cases and prevent it from overfitting to the regular inputs commonly used during training.

[0101] Static analysis strategies directly generate specific constraint rules, which may include: prohibited function names, such as "torch.nn.functional.softmax", "cupy._core.xxxx", or "printf", to prevent the model from calling library functions or printing within the kernel, which could be considered "cheating" or non-compliant behavior; and / or required string names, such as "__global__", which are necessary identifiers for M-architecture kernel functions, to ensure that the model generates true GPU kernel functions rather than ordinary CPU functions.

[0102] The second input data used in the performance analysis strategy to test the performance of the code under test differs from the first input data in the dynamic analysis strategy. This includes tensor shape and / or data precision, and it is specifically designed for performance evaluation. The evaluator model analyzes the structure of the code under test, predicting under what inputs its computation patterns, memory access characteristics, or parallel configurations might lead to performance degradation. For example, if it predicts that the code's shared memory usage strategy might cause severe memory conflicts or low thread bundle utilization when processing "tall and thin" matrices, this configuration will be used as the second input data.

[0103] By specifying boundary inputs, clarifying compliance rules, and designing performance tests, the testing strategy becomes more precise and efficient, improving the generalization ability of generated code to handle edge cases, ensuring the standardization and originality of code writing, and optimizing code execution efficiency.

[0104] For a deeper understanding, please refer to Figure 6 , Figure 6 A flowchart of a code generation method 600 provided in this disclosure embodiment Figure 2 Method 600 includes the following steps: Step 601: Input prompts. Prompts refer to input data that describes the task objectives, functional requirements, constraints, and contextual information of the code to be generated. They provide the model with the specific intent and boundary conditions required to generate the code. Prompts are typically a piece of natural language text, and their content includes, but is not limited to: the functions the code needs to implement, the programming language and environment, performance requirements, and prohibitive constraints.

[0105] Step 602: The actor model receives a prompt. The actor model is a pre-trained model that has been fine-tuned or specifically trained for the code generation task.

[0106] Step 603: After receiving a prompt from the user or upstream system, the actor model encodes and understands the prompt and then generates a source code string that conforms to the syntax of the programming language, i.e., the code to be tested.

[0107] Step 604: The evaluator model differs from the actor model that generates the code. This model is a pre-trained model that is fine-tuned or specifically trained to analyze the current task requirements and / or the generated code to be tested, and to generate a set of machine-readable, rigorous multi-test strategies. The model receives and analyzes the generated code to be tested, and by understanding its algorithm logic, code structure, API call patterns, etc., it actively predicts the possible weaknesses or cheating suspicions in the code.

[0108] Step 605: Based on the predictions of the code to be tested in Step 604, generate a targeted testing strategy to "attack" the weaknesses in the code to be tested. This testing strategy includes multiple sub-strategies such as dynamic analysis strategy, static analysis strategy, and performance analysis strategy, which can effectively detect and verify the potential defects in the code to be tested.

[0109] Step 606: The verification service can perform comprehensive testing and evaluation of the code to be tested based on the testing strategy to obtain test results. For example, the code output can be compared with the standard answer generated by the reference code to verify the correctness and generalization of the code; static analysis tools can be called to analyze the code to check whether it violates prohibitive rules and ensure compliance; the code execution time can be measured on hardware such as GPUs and compared with preset thresholds or benchmarks to evaluate the code execution efficiency.

[0110] Step 607: The reward value for the actor model depends on the degree to which the code passes the multidimensional tests: a high reward is given for complete pass; if there are logical errors, violations, or poor performance, the reward is reduced accordingly, thus guiding optimization. The reward value for the evaluator model is based on the effectiveness of its testing strategy in discovering defects: a high reward is given for successfully identifying code vulnerabilities; a low reward is given if the code passes easily, incentivizing it to generate more stringent testing strategies.

[0111] Step 608: In simpler cases, the code generated by the actor model in the first round passes the test strategy, and this code is identified as the target code. More commonly, the initial code fails the tests. In this case, both models enter an adversarial iterative update: the actor model updates based on the test results to optimize and generate new code, while the evaluator model updates based on the test results to generate more challenging new test strategies. This cycle continues until the code generated by the actor model in a certain round consistently passes the current test strategy; at this point, the code is considered the final target code.

[0112] It should be noted that similar objectives can be achieved through other means. For example, reinforcement learning can be based on a fixed reward function, where the reward function is predefined and fixed, such as directly using "passing all fixed unit tests" as the reward signal. Human experts can also be used to review the M-architecture code generated by the actor model and determine if it exhibits cheating behavior, performance bottlenecks, or non-compliance with constraints. A reward model can be trained using ratings or preference data provided by human experts, and then the evaluator model can optimize under the guidance of this reward model. A framework similar to generative adversarial networks can also be used, treating the actor model as a generator and the evaluator model as a discriminator, with the discriminator aiming to learn to distinguish between code generated by the actor model and standard code. Specific implementations of static analysis, such as AST, Linters, and Regex, can also achieve similar objectives, but these are different implementation paths within the framework of this application, not algorithmic alternatives. No specific limitations are imposed on the above implementation methods, as long as they achieve the corresponding functionality of this application.

[0113] Further reference Figure 7As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a code generation apparatus, which corresponds to the above method embodiments.

[0114] like Figure 7 As shown, the code generation apparatus 700 of this embodiment may include: a first generation module 701, a second generation module 702, a testing module 703 and a third generation module 704.

[0115] The first generation module 701 is configured to generate code to be tested based on prompt information using a first model; the second generation module 702 is configured to generate a test strategy corresponding to the code to be tested using a second model based on prompt information and / or the code to be tested; the test module 703 is configured to test the code to be tested based on the test strategy and obtain test results; wherein the test results are used to perform adversarial updates on the first model and the second model; and the third generation module 704 is configured to obtain target code based on the test results and the code to be tested.

[0116] In the code generation apparatus 700 of this disclosure embodiment, the specific processing of the first generation module 701, the second generation module 702, the testing module 703 and the third generation module 704 and the resulting technical effects can be referred to the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0117] In some optional implementations of the embodiments of this disclosure, the testing strategy includes at least one of the following: a dynamic analysis strategy, a static analysis strategy, and a performance analysis strategy; wherein, the dynamic analysis strategy is used to test the correctness and / or generalization ability of the code under test; the static analysis strategy is used to test whether the code under test conforms to the rules; and the performance analysis strategy is used to test the execution efficiency of the code under test.

[0118] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a dynamic analysis strategy; the testing module 703 is further configured to: process the first input data in the dynamic analysis strategy through the code to be tested to obtain first output data; wherein the first input data is determined based on the boundary within the effective input range of the code to be tested; and obtain a first result based on the first output data and the second output data; wherein the second output data is obtained by processing the first input data through reference code corresponding to the code to be tested, and the test result includes the first result.

[0119] In some optional implementations of the embodiments of this disclosure, the testing strategy includes: a static analysis strategy; the test module 703 is further configured to: analyze the code to be tested based on the static analysis strategy to obtain a second result, wherein the static analysis strategy includes: prohibited function names and / or required string names.

[0120] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a performance analysis strategy; the testing module 703 is further configured to: process the second input data in the performance analysis strategy through the code under test to obtain first performance information of the code under test; wherein the second input data includes tensor shape and data precision, and the first performance information is used to indicate the performance of the code under test in the process of processing the second input data; based on the first performance information and the second performance information, a third result is obtained; wherein the second performance information is obtained in the process of processing the second input data through reference code corresponding to the code under test, and the test result includes the third result.

[0121] In some optional implementations of the embodiments of this disclosure, the test result is used to indicate whether the test passed or failed; the third generation module 704 is further configured to: if the test result indicates that the test failed, perform the next round of code generation operation, test strategy generation operation, and test operation based on the updated first model and the updated second model; wherein, the updated first model and the updated second model are obtained by updating based on the test result; if the test result indicates that the test passed, determine the code to be tested as the target code.

[0122] In some optional implementations of the embodiments of this disclosure, after obtaining the test results, the code generation device 700 further includes: a first update module 705 (not shown in the figure) configured to update the first model based on the test results to obtain an updated first model; and a first update module 706 (not shown in the figure) configured to update the second model based on the test results to obtain an updated second model.

[0123] In some optional implementations of the embodiments of this disclosure, the first update module 705 is further configured to: determine a first reward value of the first model based on the test results; update the first model based on the first reward value to obtain the updated first model.

[0124] In some optional implementations of the embodiments of this disclosure, the second update module 706 is further configured to: determine the second reward value of the second model based on the test results; update the second model based on the second reward value to obtain the updated second model.

[0125] In some optional implementations of the embodiments of this disclosure, the second generation module 702 is further configured to: generate predicted defect information based on prompt information and / or code to be tested using a second model; wherein the predicted defect information includes at least one of the following: the type of predicted defect, the location of the predicted defect, and input data for triggering the predicted defect; and generate a test strategy based on the predicted defect information using a second model.

[0126] In some optional implementations of the embodiments of this disclosure, the testing strategy includes a dynamic analysis strategy, a static analysis strategy, and a performance analysis strategy; the second generation module 702 is further configured to: generate first input data in the dynamic analysis strategy based on the prompt information and / or the code to be tested; wherein the first input data is determined based on the boundaries within the valid input range of the code to be tested; generate a static analysis strategy based on the prompt information and / or the code to be tested; wherein the static analysis strategy includes at least one of the following: prohibited function names and / or required string names; generate a performance analysis strategy based on the prompt information and / or the code to be tested; wherein the performance analysis strategy includes second input data for testing the performance of the code to be tested, the second input data including tensor shape and / or data precision.

[0127] This embodiment exists as a device embodiment corresponding to the above-described code generation method embodiment. The code generation device 700 provided in this embodiment can adaptively design test cases and constraints for the generated code to be tested by dynamically generating test strategies rather than pre-setting them. It can also perform adversarial updates on the model based on the test results and obtain the target code. This overcomes the limitations of the fixed reward mechanism and solves the overfitting problem caused by the fixed reward mechanism. It enhances the generalization ability and robustness of the code, ensures the quality and compliance of the code, and thus improves the performance and usability of code generation.

[0128] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the code generation method described in any of the above embodiments when executing the instructions stored in the memory.

[0129] According to embodiments of this disclosure, this disclosure also provides a chip including a processor for executing the code generation method described in any of the above embodiments.

[0130] According to embodiments of this disclosure, this disclosure also provides a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, can implement the code generation method described in any of the above embodiments.

[0131] For example, the computer instructions corresponding to a code generation method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer instructions corresponding to a code generation method in the storage media are read or executed by a computer, the code generation method as described in any of the above embodiments can be implemented.

[0132] According to embodiments of this disclosure, this disclosure also provides a computer program product including a computer program that, when executed by a processor, can implement the code generation method described in any of the above embodiments.

[0133] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 according to embodiments of the present disclosure includes a processor 801 and a memory 802 storing an executable computer program. The processor 801, when executing the executable computer program stored in the memory 802, implements the code generation method provided in embodiments of the present disclosure.

[0134] In some optional implementations of the embodiments of this disclosure, the electronic device 800 may further include a communication interface 803 and a bus 804 for connecting the processor 801, the memory 802 and the communication interface 803.

[0135] In some optional implementations of the embodiments of this disclosure, bus 804 is used to connect communication interface 803, processor 801 and memory 802 to realize mutual communication between these devices.

[0136] In some optional implementations of the embodiments of this disclosure, the processor 801 may be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that, for different devices, the electronic device used to implement the functions of the processor 801 may also be other types, and the embodiments of this disclosure do not specifically limit its use.

[0137] The aforementioned memory 802 is used to store executable computer programs and data. The executable computer program includes computer operation instructions. Memory 802 may include high-speed RAM and may also include non-volatile memory, such as at least two disk drives. In practical applications, the aforementioned memory 802 can be volatile memory, such as Random-Access Memory (RAM); or non-volatile memory, such as Read-Only Memory (ROM), flash memory, Hard Disk Drive (HDD), or Solid-State Drive (SSD); or a combination of the above types of memory, and provides executable computer programs and data to the processor.

[0138] Furthermore, the functional modules in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0139] If the integrated units described above are implemented as software functional modules and not sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this disclosure embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] It should be understood that if this disclosure references any user data and personal information (including but not limited to device information, behavioral data, location information, etc.) and before applying the technical solutions described in the embodiments of this disclosure, the relevant products or services should comply with the laws and regulations concerning the protection of user data and personal information, strictly process users' personal information and data in accordance with the provisions of applicable laws and regulations throughout the entire data processing lifecycle, follow the principles of legality, legitimacy, necessity, good faith, openness, and transparency, and adopt reasonable privacy design schemes and technical measures to ensure the security of user data and personal information, protect users' legitimate rights and interests, and prevent the risks of leakage, theft, or tampering of user data and personal information.

[0141] Specifically, the company must publish and display its privacy policy in a prominent position on the user interface, clearly informing users of the types, purposes, uses, and methods of processing personal information, as well as other matters that should be disclosed as required by laws and regulations; obtain users' prior informed consent or explicit authorization regarding data processing through user-initiated interaction (such as confirmation pop-ups); process or store user data securely within the legally required timeframe; adopt a series of security technologies and management measures, including but not limited to data encryption and access control; share and transfer user data within the scope permitted by law and in a legally required manner; and process user rights, including the rights to query, access, correct, delete, withdraw authorization and consent, cancel registration, and obtain copies of personal information, within the legally required timeframe.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this disclosure, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this disclosure, the word "may" is used to mean "one or more embodiments of this disclosure." And the term "exemplary" is intended to refer to an example or illustration.

[0144] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that, unless expressly stated in this disclosure, terms as defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A code generation method, comprising: Based on the prompts, the first model generates the code to be tested. The second model generates a test strategy corresponding to the code to be tested based on the prompt information and / or the code to be tested. The code to be tested is tested based on the aforementioned testing strategy to obtain test results; wherein, the test results are used to perform adversarial updates on the first model and the second model; Based on the test results and the code to be tested, the target code is obtained.

2. The method according to claim 1, wherein, The testing strategy includes at least one of the following: dynamic analysis strategy, static analysis strategy, and performance analysis strategy; The dynamic analysis strategy is used to test the correctness and / or generalization ability of the code under test. The static analysis strategy is used to test whether the code under test conforms to the rules; The performance analysis strategy is used to test the execution efficiency of the code under test.

3. The method according to claim 2, wherein, The testing strategy includes: a dynamic analysis strategy; the testing of the code to be tested based on the testing strategy to obtain test results includes: The first input data in the dynamic analysis strategy is processed by the code under test to obtain the first output data; wherein the first input data is determined based on the boundary of the effective input range of the code under test. Based on the first output data and the second output data, a first result is obtained; wherein the second output data is obtained by processing the first input data through reference code corresponding to the code to be tested, and the test result includes the first result.

4. The method according to claim 2, wherein, The testing strategy includes: a static analysis strategy; the testing of the code to be tested based on the testing strategy to obtain test results includes: The code to be tested is analyzed based on the static analysis strategy to obtain a second result; wherein the test result includes the second result, and the static analysis strategy includes: prohibited function names and / or required string names.

5. The method according to any one of claims 2-4, wherein, The testing strategy includes: a performance analysis strategy; the testing of the code to be tested based on the testing strategy to obtain test results includes: The test code processes the second input data in the performance analysis strategy to obtain the first performance information of the test code; wherein the second input data includes tensor shape and data precision, and the first performance information is used to indicate the performance of the test code in the process of processing the second input data; Based on the first performance information and the second performance information, a third result is obtained; wherein the second performance information is obtained during the process of processing the second input data through reference code corresponding to the code to be tested, and the test result includes the third result.

6. The method according to claim 1, wherein, The test results are used to indicate whether the test passed or failed. The process of obtaining the target code based on the test results and the code to be tested includes: If the test result indicates that the test has failed, the next round of code generation, test strategy generation, and testing operations are performed based on the updated first model and the updated second model; wherein the updated first model and the updated second model are obtained by updating based on the test result; If the test result indicates that the test passed, the code to be tested is determined to be the target code.

7. The method according to claim 1 or 6, wherein, After obtaining the test results, the method further includes: The first model is updated based on the test results to obtain the updated first model; The second model is updated based on the test results to obtain the updated second model.

8. The method according to claim 7, wherein, The step of updating the first model based on the test results to obtain the updated first model includes: Based on the test results, determine the first reward value of the first model; The first model is updated based on the first reward value to obtain the updated first model.

9. The method according to claim 7, wherein, The step of updating the second model based on the test results to obtain the updated second model includes: Based on the test results, a second reward value for the second model is determined; The second model is updated based on the second reward value to obtain the updated second model.

10. The method according to any one of claims 1-4, wherein, The step of generating a test strategy based on the prompt information and / or the code to be tested using the second model includes: Based on the prompt information and / or the code to be tested, the second model generates predicted defect information; wherein the predicted defect information includes at least one of the following: the type of predicted defect, the location of the predicted defect, and the input data used to trigger the predicted defect; The test strategy is generated by the second model based on the predicted defect information.

11. The method according to claim 1, wherein the testing strategy includes a dynamic analysis strategy, a static analysis strategy, and a performance analysis strategy; The step of generating a test strategy based on the prompt information and / or the code to be tested includes: Based on the prompt information and / or the code to be tested, the first input data in the dynamic analysis strategy is generated; wherein, the first input data is determined based on the boundary of the effective input range of the code to be tested; Based on the aforementioned prompt information and / or the code to be tested, the static analysis strategy is generated; wherein, the static analysis strategy includes: the names of functions that are prohibited from being called and / or the names of strings that must be included; Based on the prompt information and / or the code to be tested, the performance analysis strategy is generated; wherein, the performance analysis strategy includes second input data for testing the performance of the code to be tested, the second input data including tensor shape and / or data precision.

12. A code generation apparatus, comprising: The first generation module is configured to generate the code to be tested based on the prompt information from the first model; The second generation module is configured to generate a test strategy corresponding to the code to be tested based on the prompt information and / or the code to be tested using the second model; The testing module is configured to test the code to be tested based on the testing strategy and obtain test results; wherein the test results are used to perform adversarial updates on the first model and the second model; The third generation module is configured to obtain the target code based on the test results and the code to be tested.

13. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 11 when executing instructions stored in the memory.

14. A chip comprising a processor for performing the method of any one of claims 1 to 11.

15. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images