Method and device for training code verification model, storage medium and electronic device

Abstract

This specification discloses a method, system, storage medium, and electronic device for training a code verification model. First, first functional requirement specification information is input into multiple code generation models to obtain first code output by each model. Second, the first functional requirement specification information is split into multiple first specifications, and a first dataset is generated based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification, where the first target code includes all or part of the first code. Then, first label information corresponding to the first target code is obtained. Finally, the code verification model is trained based on the first dataset and the first label information.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to a method, apparatus, storage medium, and electronic device for training code verification models. Background Technology

[0002] Spec-Driven Development (SDD) uses detailed and explicit software requirements specifications as a starting point for development, and leverages large models to transform these specifications into executable code. This approach emerged in 2025 with the widespread adoption of AI tools, aiming to address the unpredictable code quality issues caused by ambiguous natural language prompts in traditional AI-assisted development. The core of SDD is "first define clear specifications, then develop / verify code based on those specifications." The illusion of large models, however, arises when the model generates seemingly reasonable content that actually contradicts facts / specifications (including fabrication, misattribution, and logical contradictions). Throughout the SDD process, this illusion precisely targets the core stages of "specification-implementation-verification." For example, the illusion of specification interpretation—the "misreading / filling in the gaps" of the specifications—is the most basic and fatal illusion in SDD. Its core is that the large model interprets the established specification documents in a way that deviates from reality.

[0003] For the illusion of interpretation of specifications, the common detection method is manual verification of each item, which is extremely labor-intensive and difficult to promote. Alternatively, converting the specifications from natural language into a machine-parsable structured format and directly verifying the compliance of the model output through code is one of the best solutions to alleviate the illusion of interpretation. However, this method cannot cover the illusion of interpretation at the business logic level (such as the logical order error of "first determine the membership level and then apply the coupon"). Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a method, apparatus, storage medium, and electronic device for training code verification models.

[0005] This specification provides a method for training a code verification model. It proposes a method to detect the illusion of a large code generation model in specification-driven development (SDD). This method focuses on illusions related to specification interpretation, training a dedicated code verification model to verify whether a piece of code meets a specific specification. Compared to directly judging whether a piece of code meets all specifications, this method is more focused and targeted. Compared to traditional methods, the code verification model can save valuable manpower and discover logical errors that rule-based verification methods struggle to cover. It specifically targets the most critical and fatal illusion in specification interpretation within SDD, abandoning generalized detection approaches and precisely addressing the core pain point of "specification-code" matching. This avoids the problem of insufficient detection accuracy caused by covering multiple types of illusions. It breaks through the traditional holistic verification mode of "judging whether code meets all specifications" and designs a model specifically for verifying the "matching degree between a single specific specification and its corresponding code." By breaking down the complete specification and verifying it specification by specification, it can accurately capture fine-grained illusions such as logical order errors and constraint omissions. The method includes: Input the first functional requirement specification information into multiple code generation models to obtain the first code output by each code generation model; The first functional requirement specification information is split into multiple first specifications, and a first dataset is generated based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification. The first target code includes all or part of the first code. Obtain the first tag information corresponding to the first target code, wherein the first tag information is used to indicate whether the first target code meets its corresponding first specification; The code verification model is trained based on the first dataset and the first label information.

[0006] Furthermore, the method also includes: Obtain a second code generated based on the first functional requirement specification information, wherein the second code has been manually verified to meet the first functional requirement specification information; A second dataset is generated, wherein the second dataset includes the plurality of first specifications and a second target code corresponding to each first specification, the second target code includes all or part of the second code, and the second label information corresponding to the second target code is set to satisfy its corresponding first specification; The step of training the code verification model based on the first dataset and the first label information includes: The code verification model is trained based on the first dataset, the first label information, the second dataset, and the second label information.

[0007] Further, training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information includes: In the first stage, the code verification model is trained based on the second dataset and the second label information. In the second stage, the code verification model is trained again based on the first dataset and the first label information.

[0008] Furthermore, the step of splitting the first functional requirement specification information into multiple first specifications includes: By performing semantic analysis on the first functional requirement specification information, multiple functional requirements are obtained. Based on the multiple functional requirements, the first functional requirement specification information is split into multiple first specifications, wherein each functional requirement is independent of each other and each functional requirement corresponds to a first specification.

[0009] Furthermore, the step of splitting the first functional requirement specification information into multiple first specifications includes: The first functional requirement specification information is split into multiple first specifications based on at least one preset punctuation mark.

[0010] Furthermore, the method also includes: For each first specification, the first code is parsed to remove code whose correlation with the first specification is less than or equal to a first threshold, and the remaining code in the first code is obtained. The remaining code is then used as the first target code corresponding to the first specification.

[0011] Further, for each first specification, by parsing the first code and removing code whose correlation with the first specification is less than or equal to a first threshold to obtain the remaining code in the first code, and using the remaining code as the first target code corresponding to the first specification, the process includes: For each first specification, the remaining code in the first code is obtained by parsing the first code and removing code whose correlation with the first specification is less than or equal to a first threshold. Recover the context code corresponding to the remaining code from the removed code, and use the recovered remaining code as the first target code corresponding to the first specification.

[0012] Further, obtaining the first tag information corresponding to the first target code includes: If the syntax accuracy of the first target code is greater than or equal to the second threshold, the first tag information corresponding to the manually annotated first target code is obtained; otherwise, the first tag information corresponding to the first target code is set to not meet its corresponding first specification.

[0013] Further, the step of obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy corresponding to the first target code is greater than or equal to the second threshold includes: If the syntax accuracy of the first target code is greater than or equal to the second threshold, and the correlation between the first target code and its corresponding first specification is greater than or equal to the third threshold, the first tag information corresponding to the manually annotated first target code is obtained.

[0014] Furthermore, the method also includes: The second functional requirement specification information is broken down into multiple second specifications; Each second specification and the code to be verified generated based on the second functional requirement specification information are input into the trained code verification model to obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification.

[0015] This specification also provides a method for verifying codes, the method comprising: The second functional requirement specification information is broken down into multiple second specifications; Each second specification and the code to be verified generated based on the second functional requirement specification information are input into the trained code verification model to obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification.

[0016] This specification also provides an apparatus for training a code verification model, comprising: The input module is used to input the first functional requirement specification information into multiple code generation models and obtain the first code output by each code generation model. The dataset module is used to split the first functional requirement specification information into multiple first specifications, and generate a first dataset based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification. The first target code includes all or part of the first code. A tag module is used to obtain first tag information corresponding to the first target code, wherein the first tag information includes a tag indicating whether the first target code meets its corresponding first specification; The training module is used to train the code verification model based on the first dataset and the first label information.

[0017] This specification also provides an apparatus for verifying codes, comprising: The split module is used to split the second functional requirement specification information into multiple second specifications; The output module is used to input each second specification and the code to be verified generated based on the second functional requirement specification information into the trained code verification model, and obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification.

[0018] This specification also provides a storage medium storing a computer program adapted to be loaded by a processor and to execute the steps of the method described above.

[0019] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.

[0020] This specification also provides a computer program product that stores at least one instruction, characterized in that the at least one instruction, when executed by a processor, implements the steps of the above-described method.

[0021] According to the embodiments of this specification, a method for detecting the illusion of a large code generation model in specification-driven development is proposed. This method focuses on illusions related to specification interpretation, training a code verification model specifically for verifying whether a piece of code meets a specific specification. Compared to directly judging whether a piece of code meets all specifications, this method is more focused and targeted. Compared to traditional methods, the code verification model can save valuable manpower and discover logical errors that rule-based verification methods struggle to cover. It specifically targets the most critical and fatal illusion in specification interpretation within specification-driven development (SDD), abandoning generalized detection approaches and precisely addressing the core pain point of "specification-code" matching. This avoids the problem of insufficient detection accuracy caused by covering multiple types of illusions. It breaks through the traditional overall verification mode of "judging whether the code meets all specifications," designing a model specifically for verifying the "matching degree between a single specific specification and its corresponding code." By splitting the complete specification and verifying it specification by specification, it can accurately capture fine-grained illusions such as logical order errors and constraint omissions. Attached Figure Description

[0022] Figure 1 A flowchart illustrating a method for training a code verification model, provided as an embodiment of this specification; Figure 2 A flowchart illustrating a method for verifying codes provided in an embodiment of this specification; Figure 3 A flowchart illustrating a method for training a code verification model, provided as an embodiment of this specification; Figure 4 A schematic diagram of a device for training a code verification model provided in an embodiment of this specification; Figure 5 A schematic diagram of a device for verifying codes provided in an embodiment of this specification; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0024] Please see Figure 1 This is a flowchart illustrating a method for training a code verification model provided in an embodiment of this specification. In this embodiment, the method for training the code verification model is applied to the apparatus for training the code verification model described in this embodiment (hereinafter referred to as the "code verification model training apparatus"). The following will focus on... Figure 1 The process shown will be described in detail. The method for training the code verification model may specifically include the following steps: S102, input the first functional requirement specification information into multiple code generation models, and obtain the first code output by each code generation model.

[0025] In some embodiments, the functional requirement specification information includes multiple functional requirement specifications. For example, the functional requirement specification information may be a list of functional requirement specifications or a set of functional requirement specifications. This example embodiment does not make any special limitations on this. The functional requirement specification is a standardized description of the business behaviors, functional logic, input and output, boundary constraints, and other information that the code must implement before code generation. Functional requirement specifications are usually written in natural language or semi-structured format, emphasizing domain-specific language and clarity to improve the reliability of code generation. For example, functional requirement specifications may be "Trigger locking after 5 consecutive incorrect password attempts, with a lock duration of 30 minutes", "Administrators can manually unlock", "Detection logic takes precedence over password verification; if the account is in a locked state, login is directly rejected and a clear prompt is returned", "Attempts to log in during the lock period are directly blocked", "Automatic unlocking after the timer ends", "Reset the number of incorrect attempts after unlocking", etc. This example embodiment does not make any special limitations on this.

[0026] In some embodiments, the code generation model is an artificial intelligence model that takes functional requirement specification information as input, automatically parses semantics, understands business logic, and automatically generates compliant, runnable, and structurally standard program code according to specified programming languages, engineering specifications, and syntax rules. In some embodiments, the first functional requirement specification information is input into multiple code generation models to obtain the first code output by each code generation model, thus obtaining multiple first codes. In some embodiments, the size of the code generation model is less than or equal to a preset threshold, that is, the code generation model is a small-scale model. Small-scale models are more prone to producing illusions and errors. Here, model size is a core quantitative indicator for measuring the number of parameters, structural complexity, computing power, and storage consumption of an artificial intelligence model, mainly used to characterize the model's capacity, learning ability, and resource consumption level.

[0027] S104, the first functional requirement specification information is split into multiple first specifications, and a first dataset is generated based on the first code and the multiple first specifications, wherein the first dataset includes the multiple first specifications and a first target code corresponding to each first specification, and the first target code includes all or part of the first code.

[0028] In some embodiments, the first functional requirement specification information is split into multiple functional requirement specifications, i.e., the first specification. For example, the first functional requirement specification information can be split into multiple first specifications by performing semantic analysis on it. Alternatively, the first functional requirement specification information can be split into multiple first specifications based on at least one preset punctuation mark. For example, the first functional requirement specification information is "5 consecutive incorrect password attempts trigger locking for 30 minutes; administrators can manually unlock. Detection logic takes precedence over password verification; if the account is locked, login is directly rejected and a clear prompt is returned. Abnormal scenarios include: direct blocking of login attempts during the lock period, automatic unlocking after the timer ends, and resetting the number of incorrect attempts after unlocking." The multiple first specifications that can be split into are: "5 consecutive incorrect password attempts trigger locking for 30 minutes," "administrators can manually unlock," "detection logic takes precedence over password verification; if the account is locked, login is directly rejected and a clear prompt is returned," "direct blocking of login attempts during the lock period," "automatic unlocking after the timer ends," and "resetting the number of incorrect attempts after unlocking."

[0029] In some embodiments, generating a first dataset based on each first code and multiple first specifications includes multiple first specifications and a first target code corresponding to each first specification, wherein the first target code corresponding to each first specification is all or part of one of the multiple first codes.

[0030] S106, obtain the first tag information corresponding to the first target code, wherein the first tag information is used to indicate whether the first target code meets its corresponding first specification.

[0031] In some embodiments, first label information (i.e., training labels of the first dataset) corresponding to the first target code is obtained. The first label information is used to indicate whether the first target code meets its corresponding first specification. The first label information is manually annotated. If the first target code meets its corresponding first rule, the content of the first label information can be "meets the specification". If the first target code does not meet its corresponding first rule, the content of the first label information can be "does not meet the specification". This example embodiment does not make any special limitation on the specific content of the first label information.

[0032] S108, The code verification model is trained based on the first dataset and the first label information.

[0033] In some embodiments, the code verification model is trained based on a first dataset and first label information to obtain a trained code verification model. This example embodiment does not specifically limit the model structure or parameters of the code verification model. For instance, supervised fine-tuning (SFT) is performed on the code verification model based on the first dataset and first label information. SFT is a common deep learning strategy, typically used on pre-trained large language models. Specifically, it involves fine-tuning the model using labeled data to adapt it to a specific task or domain. In some embodiments, the input to the code verification model is a specific functional requirement specification and the code to be verified. The output of the code verification model is a verification result indicating whether the code to be verified meets the functional requirement specification.

[0034] According to the embodiments of this specification, a method for detecting the illusion of a large code generation model in specification-driven development is proposed. This method focuses on illusions related to specification interpretation, training a code verification model specifically for verifying whether a piece of code meets a specific specification. Compared to directly judging whether a piece of code meets all specifications, this method is more focused and targeted. Compared to traditional methods, the code verification model can save valuable manpower and discover logical errors that rule-based verification methods struggle to cover. It specifically targets the most critical and fatal illusion in specification interpretation within specification-driven development (SDD), abandoning generalized detection approaches and precisely addressing the core pain point of "specification-code" matching. This avoids the problem of insufficient detection accuracy caused by covering multiple types of illusions. It breaks through the traditional overall verification mode of "judging whether the code meets all specifications," designing a model specifically for verifying the "matching degree between a single specific specification and its corresponding code." By splitting the complete specification and verifying it specification by specification, it can accurately capture fine-grained illusions such as logical order errors and constraint omissions.

[0035] In some embodiments, the method further includes: obtaining second code generated based on the first functional requirement specification information, wherein the second code has been manually verified to meet the first functional requirement specification information; generating a second dataset, wherein the second dataset includes the plurality of first specifications and a second target code corresponding to each first specification, the second target code including all or part of the second code, and the second label information corresponding to the second target code being set to meet its corresponding first specification; wherein training the code verification model based on the first dataset and the first label information includes: training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information. In some embodiments, second code generated by specification-driven development based on the first functional requirement specification information is collected, and the second code has been manually verified and confirmed to meet the first functional requirement specification information. In some embodiments, since the second code is manually verified, its code quality is higher than that of the first code. In some embodiments, a second dataset is generated based on the plurality of first specifications and the second code, the second dataset including the plurality of first specifications and the second target code corresponding to each first specification, the second target code corresponding to each first specification being all or part of the second code. In some embodiments, the second label information corresponding to the second target code (i.e., the training label of the second dataset) is used to indicate whether the second target code meets its corresponding first specification. The second label information is set to meet its corresponding first specification. For example, the content of the second label information can be "meets the specification". This example embodiment does not specifically limit the specific content of the second label information. In some embodiments, the first dataset and the second dataset are merged, and the code verification model is trained based on the first dataset, the first label information, the second dataset, and the second label information to obtain the trained code verification model. The scheme of this embodiment constructs a training set by combining the first dataset (illusion-generated data of a small-scale model, i.e., error-containing samples, i.e., a set of positive and negative samples) and the second dataset (high-quality compliance data, i.e., positive samples). Positive samples are constructed using manually verified compliance data, and error-containing samples are generated by taking advantage of the characteristic that small-scale models are prone to producing illusions. The error-containing samples are manually labeled to form a set of positive and negative samples, providing accurate and comprehensive scenario coverage for the training of the code verification model.

[0036] In some embodiments, training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information includes: firstly training the code verification model based on the second dataset and the second label information in a first stage, and then training the code verification model based on the first dataset and the first label information in a second stage. In some embodiments, the code verification model is trained in a two-stage hierarchical manner. In the first stage, the code verification model is trained only on the second dataset (i.e., high-quality samples that meet the specifications) and the second label information, allowing the code verification model to first master the "correct correspondence between specifications and code". Then, in the second stage, the code verification model is trained based on the first dataset (i.e., correct samples that meet the specifications + incorrect samples that do not meet the specifications) and the first label information, focusing on teaching the code verification model the judgment logic of "not meeting the specifications" to avoid the model being misled by incorrect samples.

[0037] In some embodiments, the step of splitting the first functional requirement specification information into multiple first specifications includes: performing semantic analysis on the first functional requirement specification information to obtain multiple functional requirements, and splitting the first functional requirement specification information into multiple first specifications based on the multiple functional requirements, wherein each functional requirement is independent of each other, and each functional requirement corresponds to one first specification. In some embodiments, the first functional requirement specification information is obtained by performing semantic analysis on the first functional requirement specification information, and each functional requirement is a specific functional point or requirement point related to business behavior, functional logic, input and output, boundary constraints, etc., and each functional requirement is independent of each other. In some embodiments, the first functional requirement specification information is split into multiple first specifications based on the multiple functional requirements, and each functional requirement corresponds to one first specification.

[0038] In some embodiments, splitting the first functional requirement specification information into multiple first specifications includes: splitting the first functional requirement specification information into multiple first specifications based on at least one preset punctuation mark. In some embodiments, the first functional requirement specification information is split into multiple first specifications based on at least one preset punctuation mark (e.g., comma, semicolon, period, etc., which is not specifically limited in this example embodiment).

[0039] In some embodiments, the method further includes: for each first specification, performing code parsing on the first code to remove code whose correlation with the first specification is less than or equal to a first threshold, obtaining the remaining code in the first code, and using the remaining code as the first target code corresponding to the first specification. In some embodiments, for each first specification, using a code parsing tool, code parser, or code parsing model to perform code parsing on the first code, determining code whose correlation with the first specification is less than or equal to the first threshold, removing this part of the code from the first code, and using the remaining code in the first code as the first target code corresponding to the first specification.

[0040] In some embodiments, the step of parsing the first code for each first specification, removing code with a correlation degree less than or equal to a first threshold from the first code to obtain the remaining code in the first code, and using the remaining code as the first target code corresponding to the first specification, includes: for each first specification, parsing the first code to remove code with a correlation degree less than or equal to the first threshold from the first code to obtain the remaining code in the first code; recovering the context code corresponding to the remaining code from the removed code, and using the recovered remaining code as the first target code corresponding to the first specification. In some embodiments, for each first specification, using a code parsing tool, code parser, or code parsing model to parse the first code, determining code with a correlation degree less than or equal to the first threshold from the first specification, removing this part of the code from the first code to obtain the remaining code in the first code, and then recovering the context code corresponding to the remaining code from the removed part of the code, wherein the context code refers to the surrounding code that can determine whether the remaining code "can run, what it means, and how it is executed", and the context code refers to the code other than the remaining code that is used to support the operation of the remaining code and explain the meaning of the remaining code. In some embodiments, the remaining code is merged with the restored context code, and the merged code (i.e. the restored remaining code) is used as the first target code corresponding to the first specification.

[0041] In some embodiments, obtaining the first tag information corresponding to the first target code includes: if the syntax accuracy of the first target code is greater than or equal to a second threshold, obtaining the manually annotated first tag information corresponding to the first target code; otherwise, setting the first tag information corresponding to the first target code to not meet its corresponding first specification. In some embodiments, code parsing tools, code parsers, or code parsing models are used to parse the first code to obtain the syntax accuracy of the first target code. Only when the syntax accuracy of the first target code is greater than or equal to the second threshold will the manually annotated first tag information corresponding to the first target code be obtained. Otherwise, the first tag information corresponding to the first target code is directly set to not meet its corresponding first specification, that is, completely invalid samples are removed. Completely invalid samples do not need to be manually annotated. For example, the content of the first tag information can be "not meeting the specification". This example embodiment does not specifically limit the specific content of the first tag information.

[0042] In some embodiments, obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy of the first target code is greater than or equal to a second threshold includes: obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy of the first target code is greater than or equal to a second threshold and the correlation between the first target code and its corresponding first specification is greater than or equal to a third threshold. In some embodiments, semantic parsing needs to be performed on the first target code and the first specification corresponding to the first target code to obtain the semantic correlation between the first target code and the first specification. For example, the first target code is first parsed at the syntax level to extract information such as function name, class name, variable name, interface name, calling method, branch logic, input parameters, output parameters, and business fields. Then, based on this information, the first target code is parsed at the semantic level to obtain the corresponding code function description text, i.e., code semantics, so as to translate code behavior into natural language description. For another example, a preset large language model is used to perform semantic parsing on the first specification corresponding to the first target code to obtain the corresponding specification semantics. Then, the similarity between the code semantics of the first target code and the specification semantics of the first specification is obtained (for example, the similarity can be the cosine similarity between the first semantic vector corresponding to the code semantics and the second semantic vector corresponding to the specification semantics). This similarity is used as the semantic correlation between the first target code and the first specification. In some embodiments, the first label information corresponding to the manually annotated first target code will only be obtained if the syntax accuracy of the first target code is greater than or equal to the second threshold and the correlation between the first target code and its corresponding first specification is greater than or equal to the third threshold. Otherwise, the first label information corresponding to the first target code will be set to not meet its corresponding first specification.

[0043] In some embodiments, the method further includes: splitting the second functional requirement specification information into multiple second specifications; inputting each second specification and the code to be verified generated based on the second functional requirement specification information into a trained code verification model to obtain a verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification. In some embodiments, the second functional requirement specification information is first split into multiple second specifications, and the specific splitting method is the same as or similar to the method of splitting the first functional requirement specification information into multiple first specifications, which will not be described in detail here. In some embodiments, each second specification and the code to be verified generated by specification-driven development based on the second functional requirement specification information are input into a trained code verification model, so that the code verification model verifies the code to be verified and the second specification, and outputs a verification result indicating whether the code to be verified meets the second specification.

[0044] Figure 2 This is a flowchart illustrating a method for verifying codes provided in an embodiment of this specification. In this embodiment, the method for verifying codes is applied to a code verification apparatus (hereinafter referred to as a "code verification apparatus") described in this embodiment. The following will focus on... Figure 2 The process shown will be described in detail. The method for verifying the code may specifically include the following steps: S202 breaks down the second functional requirement specification information into multiple second specifications. The relevant operations have been described in detail above and will not be repeated here.

[0045] S204, input each second specification and the code to be verified generated based on the second functional requirement specification information into the trained code verification model, and obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification. The relevant operations have been described in detail above and will not be repeated here.

[0046] Figure 3 This is a flowchart illustrating a method for training a code verification model, provided as an embodiment of this specification.

[0047] like Figure 3 As shown, a common specification-driven development code generation dataset D1 is collected. The functional requirement specification list of D1 is broken down into specific specifications, while the corresponding code remains unchanged, forming dataset D2. Several small-scale code models are selected, and the functional requirement specification list of D1 is input to generate the corresponding code, resulting in dataset D3. The functional requirement specification list of D3 (which is the same as the functional requirement specification list of D1) is broken down into specific specifications, while the corresponding code remains unchanged, forming dataset D4. D4 is manually labeled to obtain dataset D5. D2 and D5 are merged to obtain dataset D6. An open-source large model is selected, and SFT (Supervised Fine-tuning) is performed on D6.

[0048] Figure 4 This is a schematic diagram of a device for training a code verification model, provided as an embodiment of this specification. This device (hereinafter referred to as "code verification model training device 1") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the code verification model training device 1 includes an input module 11, a dataset module 12, a label module 13, and a training module 14.

[0049] Input module 11 is used to input the first functional requirement specification information into multiple code generation models and obtain the first code output by each code generation model.

[0050] The dataset module 12 is used to split the first functional requirement specification information into multiple first specifications, and generate a first dataset based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification. The first target code includes all or part of the first code.

[0051] Tag module 13 is used to obtain first tag information corresponding to the first target code, wherein the first tag information includes a method for indicating whether the first target code meets its corresponding first specification.

[0052] Training module 14 is used to train the code verification model based on the first dataset and the first label information.

[0053] In some embodiments, the code verification model training device 1 is further configured to: obtain a second code generated based on the first functional requirement specification information, wherein the second code has been manually verified to meet the first functional requirement specification information; generate a second dataset, wherein the second dataset includes the plurality of first specifications and a second target code corresponding to each first specification, the second target code includes all or part of the second code, and the second label information corresponding to the second target code is set to meet its corresponding first specification; wherein training the code verification model based on the first dataset and the first label information includes: training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information.

[0054] In some embodiments, training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information includes: firstly training the code verification model based on the second dataset and the second label information in a first stage, and then training the code verification model based on the first dataset and the first label information in a second stage.

[0055] In some embodiments, splitting the first functional requirement specification information into multiple first specifications includes: obtaining multiple functional requirements by performing semantic analysis on the first functional requirement specification information, and splitting the first functional requirement specification information into multiple first specifications based on the multiple functional requirements, wherein each functional requirement is independent of each other and each functional requirement corresponds to a first specification.

[0056] In some embodiments, splitting the first functional requirement specification information into multiple first specifications includes: splitting the first functional requirement specification information into multiple first specifications based on at least one preset punctuation mark.

[0057] In some embodiments, the code verification model training device 1 is further configured to: for each first specification, by performing code parsing on the first code, remove code whose correlation with the first specification is less than or equal to a first threshold, obtain the remaining code in the first code, and use the remaining code as the first target code corresponding to the first specification.

[0058] In some embodiments, the step of parsing the first code for each first specification, removing code with a correlation degree less than or equal to a first threshold from the first code to obtain the remaining code in the first code, and using the remaining code as the first target code corresponding to the first specification, includes: parsing the first code for each first specification, removing code with a correlation degree less than or equal to a first threshold from the first code to obtain the remaining code in the first code; recovering the context code corresponding to the remaining code from the removed code, and using the recovered remaining code as the first target code corresponding to the first specification.

[0059] In some embodiments, obtaining the first tag information corresponding to the first target code includes: if the syntax accuracy of the first target code is greater than or equal to a second threshold, obtaining the first tag information corresponding to the manually annotated first target code; otherwise, setting the first tag information corresponding to the first target code to not meet its corresponding first specification.

[0060] In some embodiments, obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy of the first target code is greater than or equal to a second threshold includes: obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy of the first target code is greater than or equal to a second threshold and the correlation between the first target code and its corresponding first specification is greater than or equal to a third threshold.

[0061] In some embodiments, the code verification model training device 1 is further configured to: split the second functional requirement specification information into multiple second specifications; input each second specification and the code to be verified generated based on the second functional requirement specification information into the trained code verification model to obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification.

[0062] The above-described apparatus embodiments correspond to the aforementioned method embodiments. For detailed descriptions, please refer to the description in the method embodiments section; further details will not be repeated here. The apparatus embodiments are derived from the corresponding method embodiments and have the same technical effects. For detailed descriptions, please refer to the corresponding method embodiments.

[0063] Figure 5 This is a schematic diagram of a device for verifying codes provided in an embodiment of this specification. This device (hereinafter referred to as "code verification device 2") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the code verification device 2 includes a splitting module 21 and an output module 22.

[0064] The splitting module 21 is used to split the second functional requirement specification information into multiple second specifications.

[0065] The output module 22 is used to input each second specification and the code to be verified generated based on the second functional requirement specification information into the trained code verification model to obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification.

[0066] The above-described apparatus embodiments correspond to the aforementioned method embodiments. For detailed descriptions, please refer to the description in the method embodiments section; further details will not be repeated here. The apparatus embodiments are derived from the corresponding method embodiments and have the same technical effects. For detailed descriptions, please refer to the corresponding method embodiments.

[0067] This specification also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in this specification.

[0068] This specification also provides a computer program product that stores at least one instruction, which is loaded by the processor and executes the method described in this specification embodiment.

[0069] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and execute the method described in the embodiments of this specification.

[0070] The embodiments in this specification also provide Figure 6 The diagram shows the structure of the electronic device. Figure 6 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above method.

[0071] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0072] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0078] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0079] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for training a code verification model, comprising: Input the first functional requirement specification information into multiple code generation models to obtain the first code output by each code generation model; The first functional requirement specification information is split into multiple first specifications, and a first dataset is generated based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification. The first target code includes all or part of the first code. Obtain the first tag information corresponding to the first target code, wherein the first tag information is used to indicate whether the first target code meets its corresponding first specification; The code verification model is trained based on the first dataset and the first label information.

2. The method according to claim 1, further comprising: Obtain a second code generated based on the first functional requirement specification information, wherein the second code has been manually verified to meet the first functional requirement specification information; A second dataset is generated, wherein the second dataset includes the plurality of first specifications and a second target code corresponding to each first specification, the second target code includes all or part of the second code, and the second label information corresponding to the second target code is set to satisfy its corresponding first specification; The step of training the code verification model based on the first dataset and the first label information includes: The code verification model is trained based on the first dataset, the first label information, the second dataset, and the second label information.

3. The method according to claim 2, wherein training the code verification model based on the first dataset, the first label information, the second dataset, and the second label information comprises: In the first stage, the code verification model is trained based on the second dataset and the second label information. In the second stage, the code verification model is trained again based on the first dataset and the first label information.

4. The method according to claim 1, wherein splitting the first functional requirement specification information into multiple first specifications includes: By performing semantic analysis on the first functional requirement specification information, multiple functional requirements are obtained. Based on the multiple functional requirements, the first functional requirement specification information is split into multiple first specifications, wherein each functional requirement is independent of each other and each functional requirement corresponds to a first specification.

5. The method according to claim 1, wherein splitting the first functional requirement specification information into multiple first specifications includes: The first functional requirement specification information is split into multiple first specifications based on at least one preset punctuation mark.

6. The method according to claim 1, further comprising: For each first specification, the first code is parsed to remove code whose correlation with the first specification is less than or equal to a first threshold, and the remaining code in the first code is obtained. The remaining code is then used as the first target code corresponding to the first specification.

7. The method according to claim 6, wherein for each first specification, by parsing the first code to remove code whose correlation with the first specification is less than or equal to a first threshold, obtaining the remaining code in the first code, and using the remaining code as the first target code corresponding to the first specification, comprises: For each first specification, the remaining code in the first code is obtained by parsing the first code and removing code whose correlation with the first specification is less than or equal to a first threshold. Recover the context code corresponding to the remaining code from the removed code, and use the recovered remaining code as the first target code corresponding to the first specification.

8. The method according to claim 1, wherein obtaining the first tag information corresponding to the first target code includes: If the syntax accuracy of the first target code is greater than or equal to the second threshold, the first tag information corresponding to the manually annotated first target code is obtained. Otherwise, the first tag information corresponding to the first target code is set to not satisfy its corresponding first specification.

9. The method according to claim 8, wherein obtaining the first tag information corresponding to the manually annotated first target code if the syntax accuracy corresponding to the first target code is greater than or equal to the second threshold includes: If the syntax accuracy of the first target code is greater than or equal to the second threshold, and the correlation between the first target code and its corresponding first specification is greater than or equal to the third threshold, the first tag information corresponding to the manually annotated first target code is obtained.

10. A method for verifying codes, comprising: The second functional requirement specification information is broken down into multiple second specifications; Each second specification and the code to be verified generated based on the second functional requirement specification information are input into the trained code verification model to obtain the verification result output by the code verification model. The verification result is used to indicate whether the code to be verified meets the second specification. The code verification model is trained based on the method described in any one of claims 1 to 9.

11. An apparatus for training a code verification model, comprising: The input module is used to input the first functional requirement specification information into multiple code generation models and obtain the first code output by each code generation model. The dataset module is used to split the first functional requirement specification information into multiple first specifications, and generate a first dataset based on the first code and the multiple first specifications. The first dataset includes the multiple first specifications and a first target code corresponding to each first specification. The first target code includes all or part of the first code. A tag module is used to obtain first tag information corresponding to the first target code, wherein the first tag information includes a tag indicating whether the first target code meets its corresponding first specification; The training module is used to train the code verification model based on the first dataset and the first label information.

12. An apparatus for verifying codes, comprising: The split module is used to split the second functional requirement specification information into multiple second specifications; An output module is used to input each second specification and the code to be verified generated based on the second functional requirement specification information into a trained code verification model to obtain the verification result output by the code verification model, wherein the verification result is used to indicate whether the code to be verified meets the second specification, and the code verification model is trained based on the method described in any one of claims 1 to 9.

13. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

14. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 10.

15. A computer program product having at least one instruction stored thereon, characterized in that, When the at least one instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 10.

Images