A code generation detection method, apparatus, medium, and device

Abstract

The embodiment of the specification discloses a code generation detection method. The method performs semantic analysis on the code based on the training sample composed of the code intention description and the code, extracts the function description actually implemented by the code as the code semantic description, and combines the code to form a first training set. The original code intention description is rewritten by using a general large model to obtain a rewritten intention description with consistent semantics, and the code is combined to form a second training set. The two training sets are used to jointly train a detection large model, so that when detection is needed, the semantic description of the code and the consistency result of the intention description can be output by the detection large model. Without manual review or manual rule setting, the detection of whether the code logic meets the expected requirements can be realized, and the detection efficiency and accuracy are improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and device for generating and detecting codes. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, Large Language Models (LLMs) have been widely applied in various fields. In the field of code generation, LLMs can provide developers with powerful automated programming support, significantly improving software development efficiency.

[0003] However, LLM (Local Management Module) often generates code that is syntactically correct and semantically seemingly reasonable, yet fails to execute as expected or meet specified requirements. Code exhibiting logical problems is generally termed "logic illusion," meaning the generated code contains flaws in its business or algorithmic logic. For example, incorrect conditional statements or abnormal loop logic can cause the program to run but produce results deviating from the intended outcome.

[0004] In existing technologies, addressing the issue of logical illusions in code generation typically relies on manual code review. However, manual review is not only costly in terms of manpower but also dependent on personal experience, making it difficult to guarantee detection efficiency and accuracy. Similarly, code verification through test cases also requires manually writing test cases and is similarly affected by personal experience; for example, the completeness of the test cases can impact the detection results.

[0005] Therefore, how to efficiently detect the logical functions of the code generated by large models has become an urgent problem to be solved. Thus, this specification provides a method for detecting generated code. Summary of the Invention

[0006] This specification provides a method, apparatus, storage medium, and electronic device for generating code detection, in order to partially solve the problems existing in the prior art.

[0007] The embodiments in this specification adopt the following technical solutions: This specification provides a method for generating code detection, the method comprising: Obtain training samples consisting of code intent descriptions and code; Based on the code of the training samples, the descriptions of each function implemented by the code are determined through semantic parsing, which are used as the code semantic descriptions. Based on the code and the parsed code semantic descriptions, the first training set is obtained. The code intent description of the training samples is rewritten using a general large model to obtain a semantically consistent rewritten intent description. Based on the training samples and the rewritten intent description, a second training set is obtained. A large detection model is trained using the first and second training sets. In response to a code detection request, the intent description to be detected and the code to be detected are input into the detection model to obtain the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and to determine the detection result of the code to be detected.

[0008] This specification provides a code generation detection device, the device comprising: The acquisition module is used to acquire training samples consisting of code intent descriptions and code. The first sample module is used to determine the descriptions of each function implemented by the code of the training sample through semantic parsing, as the code semantic description, and to obtain the first training set based on the code and the parsed code semantic description. The second sample module is used to rewrite the code intent description of the training samples using a general large model to obtain a semantically consistent rewritten intent description, and to obtain a second training set based on the training samples and the rewritten intent description. The training module is used to train a large detection model using the first training set and the second training set; The detection module is used to respond to a code detection request by inputting the intent description to be detected and the code to be detected into the detection model, obtaining the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and determining the detection result of the code to be detected.

[0009] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described code generation detection method.

[0010] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described code generation detection method.

[0011] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: This specification discloses a code generation detection method. Based on training samples consisting of code intent descriptions and the code itself, the method performs semantic parsing on the code, extracting the actual functional description implemented by the code as its semantic description. This semantic description is then combined with the code to form a first training set. A general-purpose large-scale model is used to rewrite the original code intent description, resulting in a semantically consistent rewritten intent description, which is then combined with the code to form a second training set. The two training sets are used to jointly train a large-scale detection model. When detection is needed, the large-scale detection model outputs the semantic description of the code and its consistency with the intent description. This method eliminates the need for manual review or rule setting, enabling the detection of whether code logic meets expected requirements, thus improving detection efficiency and accuracy. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This specification provides a flowchart of a code generation detection process as an embodiment. Figure 2 A schematic diagram illustrating the process of determining consistency detection results for the large detection model provided in the embodiments of this specification; Figure 3 This is a schematic diagram of a code generation detection device provided in an embodiment of this specification; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0014] Large Language Models (LLMs), also known simply as large models, are models that perform logical reasoning on input natural language text and output user-understandable natural language descriptions. LLMs generally have high accuracy and efficiency in code generation. However, there are situations where the generated code has a reasonable structure but fails to achieve the intended purpose at runtime. This manifests as business logic defects or algorithmic logic flaws in the generated code. Examples include incorrect conditional statements, loop boundary exceptions, improper variable updates, missing or incorrectly ordered algorithm steps, etc. These cause the code's execution result to deviate from the intended purpose, but the code itself can pass compilation or syntax checks, making it highly covert and difficult to detect.

[0015] Existing methods for addressing the logical illusion of LLM-generated code clearly struggle with the speed of automated code generation, making manual review insufficient for large-scale applications. Furthermore, users' programming experience, business understanding, and attention to detail are crucial; different users may yield varying review results for the same code, leading to low confidence in the audit outcomes. While test case testing methods exist, they still require manual test case writing, and test cases may fail to fully cover all code functionalities. This results in persistently high labor and testing costs, and it remains difficult to determine whether all functionalities in the code have been tested.

[0016] Based on this, this specification provides a method for detecting generated code. The technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0017] Figure 1 A code generation detection flowchart provided in this specification includes the following steps: S100: Obtain training samples consisting of code intent descriptions and code.

[0018] In the embodiments described in this specification, the following are employed: Figure 1 The device used for code logic testing of the large model generation code, as shown, can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. For ease of description, the following explanation uses a server as an example only.

[0019] To detect logical errors in the LLM output code that could prevent it from achieving its intended functionality, the server needs to obtain the correct code and its corresponding code intent description. The code intent description is textual information expressed in natural language that explains the expected functionality or behavior of the code. It typically represents the task requirements submitted by the user or developer to the code generation tool and usually includes descriptions of input / output, processing logic, algorithm requirements, and constraints. Correct code is code that completely and accurately implements the functionality described in its code intent description. Correct code generally has no syntactic errors and can execute successfully in a compilation or interpretation environment. Most importantly, it semantically matches the code intent description.

[0020] For example, if the code is intended to describe writing a function that takes an integer n and returns the nth term of the Fibonacci sequence, the correct code could be a function implemented recursively or iteratively that can correctly calculate and return the value of the specified term.

[0021] Specifically, the server can first obtain a code generation dataset from an open-source database. This dataset contains multiple training samples, each consisting of a code intent description and the corresponding code. Since this dataset is designed to train a model to generate code, the code intent descriptions and code in the training samples correspond correctly, meaning the code fully implements the intended code.

[0022] In addition, in the embodiments of this specification, to improve data quality, the server can also perform preliminary screening and cleaning of the training samples in the code generation dataset. The server can check whether each training sample contains both a non-empty code intent description and parsable code content, and remove samples with format errors, incomplete code, or ambiguous descriptions. At the same time, the server can also perform standardization processing on the code intent description, such as removing extra spaces and standardizing terminology. The cleaned code generation dataset serves as the initial set of training samples, used for subsequently constructing the first training set and the second training set.

[0023] S102: Based on the code of the training samples, determine the descriptions of each function implemented by the code through semantic parsing, which are used as the code semantic descriptions, and obtain the first training set based on the code and the parsed code semantic descriptions.

[0024] In one or more embodiments of this specification, the server may construct a first training set based on the code of the training samples. The first training set is constructed based on the functions that the code can perform, and is used to enable the detection large model to learn the ability to extract the functions implemented by the code from the code, thereby enabling the extracted functions to be compared with the subjective intention description, and realizing the automated detection of logical illusions.

[0025] Specifically, to obtain semantic information that accurately reflects code behavior, the server can parse the code itself and extract its specific functionalities. This transforms unstructured code into a structured functional description, providing a reliable benchmark for subsequent training and testing of large-scale models.

[0026] First, the server can leverage the semantic parsing capabilities of a large model to extract code functionality. Specifically, the server can pre-configure a large parsing model, which is used to understand the semantics of the code and generate corresponding natural language descriptions.

[0027] The server can combine pre-defined prompts with code from training samples to form the input to the large-scale parsing model. The pre-defined prompts guide the model to focus on the core logic of the code, instructing it to perform a comprehensive semantic analysis of the input code and output descriptions of the functions implemented by the code. For example, for code implementing the bubble sort algorithm, the pre-defined prompt could be "Please analyze the function of the following code in detail and list each specific function implemented by the code." The server then concatenates the prompts with the code and inputs it into the large-scale parsing model.

[0028] Upon receiving input, this large-scale parsing model performs semantic understanding of the code line by line or block by block based on its pre-trained knowledge, identifying key structures such as loops, conditional statements, function calls, and variable operations, and abstracting them into high-level functional descriptions. The output of this large-scale parsing model is the description of the functions implemented by the code.

[0029] The server then receives the output returned by the parsing model and uses it as the code semantic description for the training sample. For example, for the bubble sort code mentioned above, the parsing model might output a set of functional descriptions such as "traverse the array to be sorted", "compare the size of adjacent elements", "swap the positions of two elements if the order is incorrect", and "repeat the traversal until the array is completely sorted". The server associates and stores this code with the parsed code semantic description to form the samples in the first training set.

[0030] The first training set consists of code and semantic descriptions of the code obtained through semantic parsing. The semantic descriptions of the code objectively reflect the various functions actually implemented by the code, providing an objective functional reference benchmark for the detection model that is independent of the intent descriptions in the original training samples.

[0031] Furthermore, by training the detection model on the first training set, the detection model learns how to extract semantic information from the code, enabling it to deeply understand the internal logic of the code and transform unstructured code into structured functional descriptions.

[0032] S104: Using a general large model, the code intent description of the training samples is rewritten to obtain a semantically consistent rewritten intent description, and a second training set is obtained based on the training samples and the rewritten intent description.

[0033] In the embodiments of this specification, although the first training set provides an objective functional description based on the actual semantics of the code, the expression form obtained by parsing the code is generally relatively fixed and differs from the diverse intent descriptions that users may propose in application scenarios. If only the code intent descriptions in the training samples are used for training, the detection model is prone to overfitting to specific language expression patterns.

[0034] Furthermore, the code intent descriptions in the training samples may suffer from a lack of diversity in expression, potentially presenting the same functionality using only a fixed sentence structure. It's also difficult to rule out the possibility of redundant information, vague descriptions, or descriptions unrelated to the actual function of the code. Moreover, different developers have significantly different descriptive habits for the same function, while the original datasets are usually built by specific teams, resulting in a convergence of language styles. Therefore, these issues mean that relying solely on code intent descriptions for training may lead to a large model failing to accurately determine consistency with the code when faced with inputs that differ in style from the training data, thus affecting the reliability of logical illusion detection.

[0035] Therefore, the server can rewrite the code intent description of the training samples using a general large model to obtain a semantically consistent rewritten intent description, and obtain a second training set based on the training samples and the rewritten intent description.

[0036] First, the server can analyze the code intent description in each training sample, identifying and extracting the original functional descriptions contained therein. Original functional descriptions refer to the independent functional points explicitly described in the code intent description that require code implementation. The server can use rule-based natural language processing methods, such as identifying subject-verb-object structures through syntactic analysis or locating the core verbs and objects of the functional descriptions through keyword matching, to split a continuous intent description text into several semantically complete original functional descriptions.

[0037] After obtaining the original functional descriptions, the server combines these descriptions with preset rewriting requirements to form input information. The rewriting requirements are prompts used to guide the generation of results by the general large model, containing specific constraints on the rewriting task and an explanation of the expected output format. For example, the rewriting requirements might be: "Given the following complete and fluent natural language text based on the provided functional descriptions. This text should contain all the given functional descriptions and maintain semantic consistency with each description, but may be reorganized using different expressions, word order, or sentence structures. The output text should be concise and clear, avoiding the introduction of functional points not present in the original descriptions."

[0038] The server concatenates the rewriting requirement with the extracted original functional descriptions to form a complete input prompt, which is then input into the general model. Based on its knowledge and understanding of natural language, the general model generates diverse expressions for the input original functional descriptions while preserving their semantics. For example, the general model can analyze the core semantics of each original functional description and then use various language expression techniques such as synonym substitution, sentence transformation, word order adjustment, and functional point merging or splitting to generate text that is semantically identical to the original functional description set but with a completely new expression.

[0039] Then, the server obtains the output text returned by the general large model. This output text is a set of sub-functional descriptions that are semantically consistent with the original functional descriptions, but the expression form has changed.

[0040] Finally, the server constructs a second training set based on the original training samples and the generated rewritten intent descriptions.

[0041] In one or more embodiments of this specification, the construction of the second training set may include positive samples and negative samples. For each training sample, the server combines the code of the training sample with its corresponding rewrite intent description to form a positive sample, because the rewrite intent description and the code have semantic consistency. Simultaneously, the server combines the code of the training sample with rewrite intent descriptions obtained from other training samples. Since the code functions of different samples are different, the rewrite intent descriptions of other samples do not have semantic consistency with the code of this sample, thus forming negative samples. The server aggregates all positive and negative samples to form the second training set.

[0042] S106: Train a large detection model using the first training set and the second training set.

[0043] S108: In response to the code detection request, the intent description to be detected and the code to be detected are input into the detection model to obtain the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and the detection result of the code to be detected is determined.

[0044] In the embodiments of this specification, the server can use the first training set and the second training set to perform supervised fine-tuning of the detection model to be trained, thereby obtaining the ability to extract semantic information from the code, and using the extracted semantic information to determine whether the code implements the function and the intention description are consistent.

[0045] Once a large detection model is trained, the server can perform code detection. When the server receives a code detection request, it determines the intent description and the code to be detected carried in the request. The intent description is the functional requirement proposed by the user in natural language, while the code to be detected is the code generated based on the intent description, typically generated by a code generation model or written by the developer.

[0046] Specifically, the server can take the intent description to be detected and the code to be detected as input, and feed them into the trained detection model. The detection model can first perform semantic parsing on the input code to be detected, and based on its own ability learned in the first training set, extract the functional points actually implemented by the code, and generate a semantic description of the code to be detected. This process is the same as the semantic parsing principle when the first training set was constructed.

[0047] Furthermore, this large-scale detection model can semantically compare the generated code semantic description with the input intent description to be detected. Based on its discriminative ability learned in the second training set, it determines the semantic similarity or matching degree between the two and outputs a consistency detection result. The consistency detection result can be a binary label, such as "semantically consistent" or "semantically inconsistent".

[0048] Finally, based on the code semantic description of the code to be detected output by the large detection model and the consistency detection result, the server determines the final detection result of the code to be detected. In the embodiments of this specification, if the consistency detection result is semantically inconsistent, the server determines that the code to be detected has logical illusions and returns the detection result to the client, while attaching the generated code semantic description as a reference to help the user locate the problem. If the consistency detection result is semantically consistent, the server determines that the code to be detected meets the requirements of the intent description and returns the conclusion that the detection has passed.

[0049] Figure 2This diagram illustrates the process of determining the consistency detection result for the large-scale detection model provided in this specification. The server inputs the intent description and the code to be detected into the large-scale detection model. The model generates a functional description based on the code, serving as the code semantic description. Then, the consistency between the code semantic description and the intent description to be detected is judged. If they are logically consistent, a normal consistency judgment result is determined; if there is a logical conflict, a mismatched consistency judgment result is determined. Finally, the consistency detection result is output.

[0050] based on Figure 1 The code detection method shown here performs semantic parsing on training samples consisting of code intent descriptions and the code itself. It extracts the actual functional description implemented by the code as its semantic description, which is then combined with the code to form a first training set. A general-purpose large-scale model is then used to rewrite the original code intent description, resulting in a semantically consistent rewritten intent description, which is then combined with the code to form a second training set. Both training sets are used to train a large-scale detection model. When detection is needed, the large-scale detection model outputs the semantic description of the code and its consistency with the intent description. This method eliminates the need for manual review or rule setting, effectively detecting whether code logic meets expected requirements, thus improving detection efficiency and accuracy.

[0051] Furthermore, in step S104 of the embodiments of this specification, when constructing the second training set, the server needs to generate both positive and negative samples simultaneously to train the detection model to distinguish between semantic consistency and inconsistency. Positive samples are used to teach the model the pattern of "code matching description," while negative samples teach the model the pattern of "code not matching description," thereby accurately identifying logical illusions during detection. If only positive samples are used, the model will be unable to establish a discrimination boundary, easily classifying all inputs as consistent, thus failing to achieve the detection function.

[0052] In addition to the combination of rewrite intent descriptions based on other samples as described in step S104, the server can also delete or modify at least one sub-functional description in the rewrite intent description of the training sample to determine negative samples.

[0053] Specifically, for each training sample, the server first combines the code contained in the sample with its corresponding rewrite intent description to generate a positive sample. The rewrite intent description and code in the positive sample have semantic consistency and are marked as semantically consistent.

[0054] Secondly, the server processes the rewrite intent descriptions for the same training sample. A rewrite intent description typically consists of multiple sub-functional descriptions, each corresponding to a specific functional point implemented in the code. The server can select at least one sub-functional description for deletion or modification. Deletion or modification may include deleting the sub-functional description, changing it to a description that does not match the actual function of the code, or replacing it with another functional description.

[0055] The modified rewrite intent description no longer matches the semantics of the code. The server combines the modified description with the code of the sample to obtain a negative sample, which is then marked as semantically inconsistent.

[0056] It's important to note that deletion and modification operations generate negative samples to obtain descriptions that are semantically inconsistent with the code, thus providing negative samples for the large-scale detection model. However, simple deletion and modification operations, such as deleting some sub-functional descriptions, do not necessarily lead to semantic inconsistency, because the modified description may still be a subset of the actual functionality of the code and still meet the requirement of semantic consistency. Therefore, the server can follow specific implementation requirements when performing deletion and modification operations to ensure that the final rewritten intention description substantially conflicts with or deviates from the semantics of the code, thereby satisfying the labeling conditions for negative samples.

[0057] Specifically, the server can use one or more of the following methods in combination to delete or modify the description of the secondary functions.

[0058] The server can modify the description of a secondary function to a description that contradicts the actual function of the code. For example, changing "sort the array in ascending order" to "sort the array in descending order" while the code actually sorts in ascending order, the modified description directly conflicts with the algorithm logic of the code, constituting semantic inconsistency.

[0059] Alternatively, the server might replace a description of a secondary function with a function description that doesn't exist in the code at all. For example, it might replace "calculate the sum of array elements" with "calculate the product of array elements," even though the code doesn't actually perform multiplication. In this case, the description introduces a function that the code doesn't provide, which doesn't match the code's actual capabilities.

[0060] Alternatively, the server can make the overall semantics of the description deviate from the scope of the code's functionality by deleting multiple secondary function descriptions and simultaneously modifying the remaining descriptions. For example, if the code implements both sorting and summation functions, the server can delete the description of the sorting function and modify the description of the summation function to "calculate the average of array elements," while the code only implements summation. In this case, the description lacks sorting functionality and incorrectly claims to have average calculation functionality, which is inconsistent with the actual functionality of the code.

[0061] Furthermore, in one or more embodiments of this specification, in step S104, the server can also construct negative samples by adding functional descriptions. While negative samples generated solely through cross-sample pairing are abundant, they often consist of completely unrelated combinations of code and descriptions, making it difficult to cover complex situations where "the description is partially related to the code but contains redundancy or errors." To enhance the model's ability to discriminate such fine-grained inconsistencies, the server can also make targeted modifications to the rewrite intent description by adding functional descriptions not implemented in the code, constructing negative samples that partially overlap with the original description but are semantically inconsistent overall. This enriches the diversity of training data and improves the model's generalization ability.

[0062] Specifically, the server can obtain the code of the training sample and its corresponding rewrite intent description, wherein the rewrite intent description consists of several sub-functional descriptions, and each sub-functional description corresponds to a specific functional point implemented by the code.

[0063] Subsequently, while keeping the original sub-functional descriptions unchanged, the server adds at least one functional description that is not implemented by the code to the rewrite intent description. The newly added functional descriptions can come from a pre-defined unrelated functional library, be extracted from the rewrite intent descriptions of other samples, or be randomly generated by the server based on the code context.

[0064] Finally, the server will include both the actual functionality described in the code and the functionality not implemented in the code in the modified rewrite intent description. Therefore, the overall semantics of the description are no longer consistent with the code.

[0065] Additionally, in one or more embodiments of this specification, in step S108, the detection big model may also output descriptions determined from the code semantic description that are inconsistent with and / or missing from the description of the intent to be detected.

[0066] Specifically, after receiving a code detection request, the server inputs the intent description to be detected and the code to be detected generated by the code generation model based on that intent description into the trained detection model. The detection model first performs semantic parsing on the code to be detected, and then compares its generated semantic description with the input intent description item by item. During the comparison, functions mentioned in the intent description but missing in the code semantic description are identified as missing descriptions, and functions present in the code semantic description but contradicting the intent description are identified as inconsistent descriptions. After comparing each function description, the identified missing or inconsistent descriptions are determined and output along with the consistency detection result.

[0067] Therefore, the server receives the consistency detection results output by the large detection model. When the consistency detection result is inconsistent, the server can generate an anomaly detection result based on the accompanying description of inconsistencies and / or missing information. This anomaly detection result not only indicates that there is an inconsistency between the code and the intent description, but also includes specific information about which functionalities are contradictory or missing. For example, the server can return specific information such as "Logical illusion detected: the intent requires ascending order sorting, but the code implements descending order sorting" or "The function that requires calculating the sum is missing in the code." The server then returns this anomaly detection result.

[0068] When the consistency check result is consistent, the server determines that the code to be checked matches the intent description to be checked, and there is no logical illusion. At this time, the server generates a normal check result and returns it.

[0069] The server not only automates the detection of code logic illusions, but also provides fine-grained anomaly location information, enabling the code development process to make targeted corrections to the code based on the returned inconsistencies or missing descriptions, further reducing the workload of manual investigation and improving the overall efficiency of code generation and verification.

[0070] In the embodiments of this specification, advanced knowledge programs, as a novel descriptive language that combines the flexibility of natural language with the precision of programming languages, emphasize the use of natural language to describe knowledge concepts, while reusing traditional programming languages, such as Python, for the main logic. While advanced knowledge programs possess the advantages of both, verifying the semantic consistency between the knowledge concepts described in natural language and the underlying code logic becomes a more complex problem. The detection method provided in this specification can efficiently and accurately solve these problems in advanced knowledge program scenarios. In other words, when faced with a generated advanced knowledge program, the method provided in the embodiments of this specification can more efficiently and accurately verify whether there are problems with the logic.

[0071] Within the framework of advanced knowledge programs, high-order programs consist of knowledge concept descriptions and executable code. The knowledge concept descriptions are presented in a pseudocode-like natural language, facilitating human understanding and knowledge transfer, while the executable code handles the actual computational tasks. However, the ambiguity and flexibility of natural language can lead to different developers interpreting the same knowledge concept in multiple ways, resulting in discrepancies in code implementation and a difference between the semantics of knowledge and code in advanced knowledge programs. This specification's embodiments effectively solve this problem by constructing a large-scale detection model to automatically identify the consistency between knowledge concept descriptions and actual code implementations in advanced programs. This ensures that advanced knowledge programs maintain both the knowledge-expressive power of natural language and the logical accuracy of traditional programming languages.

[0072] Specifically, the server can receive the advanced knowledge program as the object to be tested, which includes the knowledge concept part described in natural language and the code logic part written in languages ​​such as Python. The detection model performs semantic parsing on the code part, generates a semantic description of the code, and then compares it with the knowledge concept description, outputting the consistency detection result and a detailed description of inconsistencies or omissions. This not only verifies the correctness of the advanced knowledge program, but also provides developers with fine-grained correction guidelines, making the verifiability of advanced knowledge programs more obvious and easier to apply.

[0073] Furthermore, one of the purposes of advanced knowledge programs is to facilitate the large-scale expansion of knowledge systems, which makes traditional manual review or test case verification methods difficult to implement. The detection method described in this specification, however, offers higher efficiency and accuracy, meeting the detection needs after large-scale deployment of advanced knowledge programs. The server can receive advanced knowledge programs to be detected in batches, quickly complete consistency verification using a trained detection model, and output structured detection results, thereby supporting iterative optimization of the large model used to generate advanced knowledge programs.

[0074] The detection method provided in the embodiments of this specification realizes the consistency verification between the knowledge concept description and code logic in high-order knowledge programs, solves the problem of semantic misalignment between the flexibility of natural language and the precision of programming language, and at the same time, its inclusiveness and cognitive ability of natural language can match the large-scale application deployment of high-order knowledge programs.

[0075] The above is an example of a code generation detection method provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0076] Figure 3 This is a schematic diagram of a code generation detection device provided in an embodiment of this specification. The device includes: Module 201 is used to acquire training samples consisting of code intent descriptions and code. The first sample module 202 is used to determine the descriptions of each function implemented by the code corresponding to the code through semantic parsing, based on the code of the training sample, as the code semantic description, and to obtain the first training set based on the code and the parsed code semantic description; The second sample module 203 is used to rewrite the code intent description of the training samples using a general large model to obtain a semantically consistent rewritten intent description, and to obtain a second training set based on the training samples and the rewritten intent description. Training module 204 is used to train a large detection model using the first training set and the second training set; The detection module 205 is used to respond to a code detection request by inputting the intent description to be detected and the code to be detected into the detection model, obtaining the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and determining the detection result of the code to be detected.

[0077] Optionally, the first sample module 202 is used to combine preset prompt words with the code of the training sample, input the code into the parsing big model to perform semantic parsing, the prompt words are used to prompt the parsing big model to perform semantic analysis on the input code, and output the functional description implemented by the code; and determine the functional descriptions implemented by the code of the training sample output by the parsing big model.

[0078] Optionally, the second sample module 203 is used to extract the content of the code intent description of the training sample, determine that it contains each original function description; combine each function description and rewriting requirements, input them into a general large model, obtain the sub-function description output by the general large model that is semantically consistent with each original function description, and use each sub-function description as the rewriting intent description.

[0079] Optionally, the second sample module 203 is configured to, for each training sample, combine the code contained in the training sample with the rewrite intent description of the training sample to obtain a positive sample, wherein the positive sample is labeled as semantically consistent; combine the code contained in the training sample with the rewrite intent descriptions of other training samples to obtain a negative sample, wherein the rewrite intent descriptions of other training samples are semantically inconsistent with the code contained in the training sample, and the negative sample is labeled as semantically inconsistent; and construct a second training set based on the obtained positive and negative samples.

[0080] Optionally, the second sample module 203 is used to, for each training sample, combine the code contained in the training sample with the rewrite intent description of the training sample to obtain a positive sample, wherein the positive sample is labeled as semantically consistent; delete or modify at least one sub-functional description in the rewrite intent description of the training sample, and combine the deleted or modified rewrite intent description with the code contained in the training sample to obtain a negative sample, wherein the negative sample is labeled as semantically inconsistent; and construct a second training set based on the obtained positive and negative samples.

[0081] Optionally, the second sample module 203 is configured to, for each training sample, combine the code contained in the training sample with the rewrite intent description of the training sample to obtain a positive sample, wherein the positive sample is labeled as semantically consistent; for each training sample, add functionality to the rewrite intent description of the training sample by adding at least one functional description that is not implemented by the code to the rewrite intent description to obtain a modified rewrite intent description, wherein the modified rewrite intent description is semantically inconsistent with the code; combine the modified rewrite intent description with the code to obtain a negative sample; and construct a second training set based on the obtained positive and negative samples.

[0082] Optionally, the detection module 205 is configured to, in response to a code detection request, determine the intent description to be detected and the code to be detected generated by a code generation model based on the intent description to be detected; input the intent description to be detected and the code to be detected into the detection model to obtain a code semantic description generated by the detection model from the code to be detected; and output a semantic consistency detection result based on the code semantic description and the intent description to be detected generated by the detection model itself.

[0083] Optionally, the detection big model also outputs descriptions determined from the code semantic description that are inconsistent with and / or missing from the description of the intent to be detected; The detection module 205 is used to determine and return an abnormal detection result based on the description of inconsistency and / or missing information output by the detection model when the consistency detection result is inconsistent; and to determine that the code to be detected is normal when the consistency detection result is consistent, and to return a normal detection result.

[0084] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the code generation detection method provided above.

[0085] based on Figure 1 The generated code detection method shown in this specification also provides embodiments that... Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned code generation detection method.

[0086] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for detecting generated code, the method comprising: Obtain training samples consisting of code intent descriptions and code; Based on the code of the training samples, the descriptions of each function implemented by the code are determined through semantic parsing, which are used as the code semantic descriptions. Based on the code and the parsed code semantic descriptions, the first training set is obtained. The code intent description of the training samples is rewritten using a general large model to obtain a semantically consistent rewritten intent description. Based on the training samples and the rewritten intent description, a second training set is obtained. A large detection model is trained using the first and second training sets. In response to a code detection request, the intent description to be detected and the code to be detected are input into the detection model to obtain the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and to determine the detection result of the code to be detected.

2. The method as described in claim 1, wherein based on the code of the training sample, semantic parsing is used to determine the descriptions of the functions implemented by the code, specifically including: The preset prompt words are combined with the code of the training sample and input into the parsing model to perform semantic parsing on the code. The prompt words are used to prompt the parsing model to perform semantic analysis on the input code and output a description of the function implemented by the code. The code of the training samples corresponds to the functional descriptions of the output of the large parsing model.

3. The method as described in claim 1, wherein the code intent description of the training samples is rewritten using a general large model to obtain a semantically consistent rewritten intent description, specifically including: Content extraction is performed on the code intent description of the training samples to determine that it contains descriptions of each original function; The functional descriptions and rewriting requirements are combined and input into a general model to obtain sub-functional descriptions that are semantically consistent with the original functional descriptions. Each sub-functional description is then used as a rewriting intent description.

4. The method as described in claim 1, wherein a second training set is obtained based on the training samples and the rewrite intent description, specifically comprising: For each training sample, the code contained in the training sample is combined with the rewrite intent description of the training sample to obtain a positive sample, and the positive sample is labeled as semantically consistent. The code contained in the training sample is combined with the rewrite intent descriptions of other training samples to obtain a negative sample, wherein the rewrite intent descriptions of the other training samples are semantically inconsistent with the code contained in the training sample, and the negative sample is labeled as semantically inconsistent. Based on the obtained positive and negative samples, a second training set is constructed.

5. The method as described in claim 3, wherein a second training set is obtained based on the training samples and the rewrite intent description, specifically comprising: For each training sample, a positive sample is obtained by combining the code contained in the training sample with the rewriting intent description of the training sample, and the positive sample is labeled as semantically consistent. At least one sub-functional description in the rewrite intent description of the training sample is deleted or modified, and the deleted and modified rewrite intent description is combined with the code contained in the training sample to obtain a negative sample, wherein the negative sample is labeled as semantically inconsistent. Based on the obtained positive and negative samples, a second training set is constructed.

6. The method as described in claim 3, wherein a second training set is obtained based on the training samples and the rewrite intent description, specifically comprising: For each training sample, a positive sample is obtained by combining the code contained in the training sample with the rewriting intent description of the training sample, and the positive sample is labeled as semantically consistent. For each training sample, a function is added to the rewrite intent description of the training sample. At least one function description that is not implemented in the code is added to the rewrite intent description to obtain a modified rewrite intent description. The modified rewrite intent description is semantically inconsistent with the code. The modified rewrite intent description is combined with the code to obtain a negative sample; Based on the obtained positive and negative samples, a second training set is constructed.

7. The method as described in claim 1, in response to a code detection request, inputting the intent description to be detected and the code to be detected into the detection model to obtain a code semantic description of the code to be detected and a consistency detection result between the code semantic description and the intent description to be detected, specifically including: In response to a code detection request, determine the intent description to be detected, and generate the code to be detected based on the intent description to be detected through a code generation model; The intent description to be detected and the code to be detected are input into the detection model to obtain the code semantic description generated by the detection model from the code to be detected; and the semantic consistency detection result output by the detection model based on the code semantic description and the intent description to be detected.

8. The method of claim 7, wherein the detection big model further outputs descriptions determined from the code semantic description that are inconsistent with and / or missing from the description of intent to be detected; Determining the detection result of the code to be detected specifically includes: When the consistency detection result is inconsistent, the abnormal detection result is determined and returned based on the inconsistent and / or missing descriptions output by the detection model. When the consistency test result is consistent, the code to be tested is determined to be normal, and a normal test result is returned.

9. A code generation detection device, the device comprising: The acquisition module is used to acquire training samples consisting of code intent descriptions and code. The first sample module is used to determine the descriptions of each function implemented by the code of the training sample through semantic parsing, as the code semantic description, and to obtain the first training set based on the code and the parsed code semantic description. The second sample module is used to rewrite the code intent description of the training samples using a general large model to obtain a semantically consistent rewritten intent description, and to obtain a second training set based on the training samples and the rewritten intent description. The training module is used to train a large detection model using the first training set and the second training set; The detection module is used to respond to a code detection request by inputting the intent description to be detected and the code to be detected into the detection model, obtaining the code semantic description of the code to be detected and the consistency detection result between the code semantic description and the intent description to be detected, and determining the detection result of the code to be detected.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1-8.

Images