Demand refinement method based on large language model

Through the combination of large language model and vector index library, efficient refinement and consistency verification of required documents are achieved, the problems of high difficulty in manual modification and high error rate are solved, and the code quality is improved.

CN119047451APending Publication Date: 2024-11-29BEIHANG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411232857.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, the modification of the required document depends on manual labor, which is prone to errors and is inefficient, and cannot effectively verify the consistency of the requirements, resulting in syntax and logical errors in the generated code.

Method used

Using the requirements refinement method based on large language models, we can build the use case design knowledge base and property specification knowledge base, generate refined requirements documents, and use vector index database and auxiliary modification robots for syntax and logic verification to ensure the consistency of requirements and Correctness.

Benefits of technology

It improves the refinement efficiency and accuracy of the required documents, the generated code quality solves the problems of difficult and high error rate of manual modification, and ensures the syntax and logic of the required documents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047451A_ABST
    Figure CN119047451A_ABST
Patent Text Reader

Abstract

The invention relates to a demand refinement method based on a large language model, and belongs to the technical field of software engineering. Comprising the steps of generating a refined demand document based on an original demand document, a use case design knowledge base and a use case design cue word template; generating an original property specification document based on the original demand document, and generating a mode design document based on the refined demand document and the mode design cue word template, and generating a refined property specification document based on the mode design document and the property specification knowledge base, and performing grammar verification on the refined property specification document and consistency verification on the original property specification document and the refined property specification document to obtain the verified refined property specification document and a corresponding refined demand document. According to the method, refinement and property specification consistency verification are carried out on the original demand based on the large language model, the verified refined demand and refined property specification are iteratively generated, and the problems that at present, modification of the demand document depends on manual work, errors are prone to occurring, and efficiency is low are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of software engineering, and in particular to a requirement refinement method based on a large language model. Background Art

[0002] Requirements documents play a vital role in the software development process, and use case design is the core component of requirements documents, which is used to describe in detail the functional requirements, interaction processes, execution conditions, status results, etc. How to automatically generate code from requirements documents to improve the efficiency of software development has always been a hot research issue.

[0003] However, requirement documents are usually described in natural language. The code automatically generated based on the requirement documents often has the following problems: first, because the requirement description is too abstract, the requirement document may be ambiguous, resulting in the generated code being unable to meet the actual needs of users; second, the requirement document in natural language form cannot be directly verified, resulting in the generated code being prone to grammatical errors; finally, the requirement document in natural language form may have inconsistent descriptions, resulting in conflicting constraints in the document, leading to logical errors in the generated code.

[0004] Currently, the modification of requirement documents relies on manual work, which requires developers to have rich experience. As software systems become increasingly large, it is becoming increasingly difficult to modify requirement documents. Manual modification is prone to errors and inefficient. Therefore, how to efficiently refine requirement documents and verify the consistency of the refined requirements to ensure their correctness has become an urgent problem to be solved. Summary of the invention

[0005] In view of the above analysis, the present invention aims to provide a requirement refinement method based on a large language model to solve the problem that the modification of requirement documents currently relies on manual completion, is prone to errors and has low efficiency.

[0006] The present invention provides a method for demand refinement based on a large language model, the method comprising the following steps:

[0007] Build a use case design knowledge base and a use case design prompt word template, and generate a refined requirement document based on the original requirement document, the use case design knowledge base, and the use case design prompt word template;

[0008] Construct a property specification knowledge base and a pattern design hint word template, generate an original property specification document based on the original requirement document, generate a pattern design document based on the refined requirement document and the pattern design hint word template, and generate a refined property specification document based on the pattern design document and the property specification knowledge base; verify the grammatical correctness of the refined property specification document. If the grammar is incorrect, modify the refined property specification document based on the large language model until the grammar is correct or the maximum number of modifications is reached; verify the consistency of the original property specification document and the refined property specification document after grammar verification. If they are inconsistent, modify the refined property specification document after grammar verification and its corresponding refined requirement document based on the large language model until the two are consistent or the maximum number of modifications is reached, and obtain the refined property specification document after consistency verification and its corresponding refined requirement document.

[0009] Furthermore, the original requirement document is segmented into sentences to obtain a number of original requirement document fragments, and for each original requirement document fragment, a first vector matching it is obtained in the use case design knowledge base, and the matching first vector is converted into a sentence to obtain a first sentence matching it.

[0010] Furthermore, for each original requirement document fragment: convert the original requirement document fragment into a second vector; calculate the vector similarity between the second vector and all first vectors, and select the top k first vectors with the largest vector similarity; set the sentence window size, and establish the first sentence windows corresponding to the top k first vectors according to the sentence window size; calculate the semantic similarity between the second vector and all corresponding first sentence windows, select all first vectors corresponding to the first sentence windows with the largest semantic similarity as the first vector matching the original requirement document fragment, and convert the matched first vectors into sentences in context order to obtain the first sentence matching the original requirement document fragment.

[0011] Furthermore, a first keyword related to use case diagram design is established, and a use case design prompt word template is constructed based on the first keyword; the first keyword includes use case, participant, system boundary, relationship between participants, precondition, postcondition, use case scenario, event flow, brief description, special requirements; the use case design prompt word template includes the first keyword and its corresponding description.

[0012] Furthermore, all original requirement document fragments and their matching first sentences and use case design prompt word templates are input into the large language model, so that the large language model extracts the content corresponding to the first keyword from the original requirement document fragments according to the use case design prompt word template and fills it into the use case design prompt word template, thereby generating a refined requirement document.

[0013] Furthermore, the verification of the grammatical correctness of the refined property specification document includes: constructing a counterexample for each variable expression in the refined property specification document; if the constructed counterexample can obtain a correct output, then the variable expression corresponding to the counterexample is judged to be grammatically correct; if the grammars of all variable expressions are correct, then the grammar of the refined property specification document is judged to be correct; otherwise, the grammar is judged to be incorrect.

[0014] Furthermore, an auxiliary modification robot is constructed to check and modify the refined requirement documents and refined property specification documents; a maximum number of modifications is set; if the syntax is incorrect, the pattern design document, the refined property specification document, the counterexample of incorrect syntax and its error information are sent to the auxiliary modification robot, so that it modifies the refined property specification document; the grammatical correctness of the modified refined property specification document is verified until its syntax is completely correct or the maximum number of modifications is reached.

[0015] Furthermore, the logical relationship between the original property specification document and the corresponding variable expressions in the refined property specification document after grammatical verification is verified. If the variable expression in the refined property specification document after grammatical verification implies or is equivalent to the corresponding variable expression in the original property specification document, the two are determined to be consistent; otherwise, the two are determined to be inconsistent.

[0016] Furthermore, a maximum number of modifications is set; if there is any inconsistency, the pattern design document, the current refined property specification document and its corresponding refined requirement document, and the SPEC statement error information are sent to the auxiliary modification robot, so that it can modify the refined property specification document and its corresponding refined requirement document after syntax verification; verify the consistency of the modified refined property specification document with the original property specification document until the two are consistent or the maximum number of modifications is reached.

[0017] Furthermore, first knowledge documents related to use case diagram design are collected, segmented and stored in a vector index library to form a use case design knowledge base; the first knowledge documents are parsed and cleaned, the cleaned first knowledge documents are segmented by sentences, all the segmented sentences are converted into first vectors and stored in the vector index library, and all the stored first vectors constitute the use case design knowledge base.

[0018] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0019] 1. The present invention provides LLM with the specification of refined requirement documents by constructing a use case design prompt word template, so that LLM can fill in the objects, event flows, preconditions, postconditions and other information accurately identified from the requirement document into the use case design prompt word template according to the standard format, thereby generating a refined requirement document with engineering specifications and use case design format, solving the problem that the modification of requirement documents currently relies on manual completion, which is prone to errors and inefficient.

[0020] 2. The present invention converts a large amount of effective information into vectors for storage by establishing a vector index library, so that LLM can retrieve related vectors as context information through the vector index library without having to input this information into LLM as prompts, thereby improving the accuracy and efficiency of the answers output by LLM; and, by persisting the information in the database, it can be retrieved and provided to LLM at any time as needed, which is equivalent to providing LLM with long-term memory capability and improving the accuracy of LLM's answers.

[0021] 3. The present invention first searches for vectors with high similarity to the original requirement document vectors in the constructed vector index library, then searches for matching vectors with the highest semantic similarity to the original requirement document through the sentence windows of these vectors, and finally converts the matching vectors into sentences in their context order as part of the LLM context input, so that the LLM can obtain knowledge that is not mentioned in the training and is truly relevant to the input content without retraining before answering, thereby improving its accuracy in identifying relevant content in the original requirement document and providing a basis for improving the correctness of refined requirements and refined property specifications.

[0022] 4. The present invention sends the pattern design document, the current refined property specification document, the SPEC statement error information, and the refined requirement document to the auxiliary modification robot, so that the robot can check and modify the refined requirement document and the refined property specification document based on sufficient external information prompts; and through multiple rounds of verification-modification-in-the-loop operations, iteratively generates the refined property specification document and the refined requirement document after consistency verification, thereby further improving the correctness of the refined requirement document and providing a basis for improving the correctness of the automatically generated code.

[0023] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0025] Figure 1 A flow chart of a method for demand refinement based on a large language model according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a first vector in a knowledge base designed for a use case of an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of designing a prompt word template for a use case of an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a requirement document described in natural language according to an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a refined requirements document according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a prompt word template designed for an embodiment mode of the present invention;

[0031] Figure 7 A schematic diagram of a design document for an embodiment mode of the present invention;

[0032] Figure 8 A schematic diagram of a property specification document according to an embodiment of the present invention;

[0033] Fig. 9 A schematic diagram showing that a counterexample of a variable expression in an embodiment of the present invention can be correctly output;

[0034] Fig.10 A schematic diagram showing that a counterexample of a variable expression in an embodiment of the present invention cannot be correctly output;

[0035] Fig.11 A schematic diagram of an auxiliary modification robot making modification suggestions according to an embodiment of the present invention;

[0036] Fig.12 A schematic diagram of a modification result returned by an auxiliary modification robot according to an embodiment of the present invention;

[0037] Fig.13 A schematic diagram of a SPEC statement according to an embodiment of the present invention;

[0038] Fig.14 This is an example of a refined property specification document after consistency verification according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0040] A specific embodiment of the present invention discloses a method for refining requirements based on a large language model. Figure 1 As shown, the method comprises the following steps:

[0041] Step S1, constructing a use case design knowledge base and a use case design prompt word template, and generating a refined requirement document based on the original requirement document, the use case design knowledge base and the use case design prompt word template;

[0042] Step S2, constructing a property specification knowledge base and a pattern design prompt word template, generating an original property specification document based on the original requirement document, generating a pattern design document based on the refined requirement document and the pattern design prompt word template, and generating a refined property specification document based on the pattern design document and the property specification knowledge base; verifying the grammatical correctness of the refined property specification document, if the grammar is incorrect, modifying the refined property specification document based on the large language model until the grammar is correct or the maximum number of modifications is reached; verifying the consistency of the original property specification document and the refined property specification document after grammar verification, if they are inconsistent, modifying the refined property specification document after grammar verification and its corresponding refined requirement document based on the large language model until the two are consistent or the maximum number of modifications is reached, and obtaining the refined property specification document after consistency verification and its corresponding refined requirement document.

[0043] Specifically, in step S1, first knowledge documents related to use case diagram design are collected, and the first knowledge documents are segmented and stored in a vector index library to form a use case design knowledge library.

[0044] Specifically, the first knowledge documents related to the use case diagram design include electronic books, articles, patents and other documents related to the use case diagram design principles, specifications and examples. The first knowledge documents are parsed and cleaned, and the cleaned first knowledge documents are segmented by sentences. All the segmented sentences are converted into first vectors and stored in a vector index library. All the stored first vectors constitute a use case design knowledge library.

[0045] It should be noted that parsing the first knowledge document refers to parsing the first knowledge document of different formats (such as PDF, Word, HTML, etc.) into a unified data format (such as a string). Cleaning the first knowledge document includes removing irrelevant symbols, removing repeated parts, standardizing, and removing irrelevant information from the parsed first knowledge document; the removal of irrelevant symbols refers to removing symbols that have no impact on semantics, such as line breaks, HTML tags, page numbers, etc.; the removal of repeated parts refers to removing semantically repeated parts to avoid redundant information in subsequent retrievals; the standardization refers to unifying terms with the same semantics, such as unifying "LLM" and "Large Language Model"; the removal of irrelevant information refers to removing information without substantive content, such as headers, footers, signatures, etc. All the segmented sentences are converted into first vectors through the embedding model. For each first knowledge document: all the corresponding first vectors are stored in the vector index library in the order of the sentences in the first knowledge document; the vector index library supports real-time data updates. Figure 2 As shown, the first vector is a high-dimensional data vector, which can represent the semantic information of the document.

[0046] Exemplarily, the BAAI / bge-large-en-v1.5 model is used as the embedding model.

[0047] It can be understood that the present invention converts a large amount of effective information into vectors for storage by establishing a vector index library, that is, a knowledge base, so that LLM can retrieve related vectors as context information through the vector index library without having to input this information as prompts into LLM, thereby improving the accuracy and efficiency of the answers output by LLM; and, by persisting the information in the database, it can be retrieved and provided to LLM at any time as needed, which is equivalent to providing LLM with long-term memory capabilities and improving the correctness of LLM's answers.

[0048] Furthermore, a first keyword related to the use case diagram design is established, and a use case design prompt word template is constructed according to the first keyword.

[0049] Specifically, the first keywords include use cases, participants, system boundaries, relationships between participants, preconditions, postconditions, use case scenarios, event flows, brief descriptions, and special requirements. Figure 3 As shown, the use case design prompt word template includes the first keyword and its corresponding description.

[0050] It should be noted that the use case includes a detailed description of the name and function of the use case; the participant refers to an external user, organization or other system that interacts with the system; the system boundary refers to the boundary between systems, defining which use cases are internal to the system and which are external to the system; the relationship between the participants refers to the relationship between the use cases, including inclusion, extension and generalization. The inclusion relationship represents that the execution of one use case involves the execution of another use case, the extension relationship represents that a use case can be extended to another use case, and the generalization relationship represents that there is an inheritance relationship between use cases or participants; the precondition refers to the state that the system should be in before the use case is executed; the postcondition refers to the state that the system should reach after the use case is executed; the use case scenario includes success scenario and abnormal scenario, representing the execution instance of the use case; the event flow includes basic flow and alternative flow, the basic flow represents the basic process of successful execution of the use case, and the alternative flow represents other processes that may occur during the execution of the use case; the brief description refers to a brief description of the role and quantitative information of the use case; the special requirements refer to a detailed description of the non-functional requirements (such as reliability, performance, availability) and design constraints of the use case.

[0051] Furthermore, the original requirement document is segmented into sentences to obtain a number of original requirement document fragments, and for each original requirement document fragment, a first vector matching it is obtained in the use case design knowledge base, and the matching first vector is converted into a sentence to obtain a first sentence matching it.

[0052] Specifically, for each original requirement document fragment: convert the original requirement document fragment into a second vector; calculate the vector similarity between the second vector and all first vectors, and select the first k first vectors with the largest vector similarity; set the sentence window size, and establish the first sentence windows corresponding to the first k first vectors according to the sentence window size; calculate the semantic similarity between the second vector and all corresponding first sentence windows, select all first vectors corresponding to the first sentence windows with the largest semantic similarity as the first vector matching the original requirement document fragment, and convert the matched first vectors into sentences in context order to obtain the first sentence matching the original requirement document fragment.

[0053] Specifically, the original requirement document fragment is converted into a second vector through the embedding model, the cosine similarity between the second vector and all the first vectors is calculated, the cosine similarities are arranged from large to small, and the first vectors corresponding to the first k cosine similarities are selected.

[0054] Specifically, the size of the first sentence window is set, the number of context sentences is determined according to the size of the first sentence window, the first vector and the first vector corresponding to the corresponding number of context sentences are used as the first sentence window corresponding to the first vector, and the semantic similarity between the second vector and all corresponding first sentence windows is calculated through the semantic analysis model.

[0055] It should be noted that since all first vectors corresponding to each first knowledge document are stored in the vector index library in the order of sentences in the first knowledge document, the first vectors corresponding to the adjacent preceding and following sentences can be obtained through a first vector.

[0056] Exemplarily, the size of the first sentence window is set to 3, then the first sentence window corresponding to a certain first vector is the first vector corresponding to 3 sentences above + the first vector + the first vector corresponding to 3 sentences below, and the first sentence window contains the first vectors corresponding to 7 sentences. Assuming that the semantic similarity between the first sentence window and the second vector corresponding to a certain original demand document fragment is the largest, the 7 first vectors corresponding to the first sentence window are used as the first vectors matching the original demand document fragment, and the 7 first vectors are converted into sentences according to the context order of the first sentence window, and 7 first sentences matching the original demand document fragment are obtained.

[0057] It is understandable that in order to improve the accuracy of automatic code generation, the original requirement document in natural language form needs to be refined through a large language model. However, as manufacturing automation systems become increasingly complex, their requirement documents are also increasingly complex. It is difficult to achieve a good requirement refinement effect by only training and fine-tuning a large language model. Therefore, the present invention first searches for vectors with high similarity to the original requirement document vectors in the constructed vector index library, and then searches for matching vectors with the highest semantic similarity to the original requirement document through the sentence windows of these vectors, and finally converts the matching vectors into sentences as part of the LLM context input according to their context order, so that the LLM can obtain knowledge that is not mentioned in the training and is truly related to the input content without retraining before answering, thereby improving its accuracy in identifying relevant content in the original requirement document and providing a basis for improving the correctness of refined requirements and refined property specifications.

[0058] Furthermore, all original requirement document fragments and their matching first sentences and use case design prompt word templates are input into the large language model, so that the large language model extracts the content corresponding to the first keyword from the original requirement document fragments according to the use case design prompt word template and fills it into the use case design prompt word template, thereby generating a refined requirement document.

[0059] For example, Figure 4As shown in FIG. 1 , it is an original requirement document for “obtaining the absolute value of input data”. It can be seen that it is only roughly described in natural language. Through the above method, based on the original requirement document for “obtaining the absolute value of input data”, the use case design knowledge base, Figure 3 The use case design prompt word template shown in the figure generates the following by calling the GPT-4 API: Figure 5 As shown in the refined requirement document, it can be seen that the refined requirement document is a use case design document with engineering specifications.

[0060] It should be noted that the refined requirement document contains all use cases related to the requirement, and each use case is generated in the format of the use case design prompt template. If necessary, you can manually check whether the event flow, use case scenario, preconditions, postconditions and other information in the refined requirement document meet the requirements of the original requirement document.

[0061] It can be understood that the present invention provides LLM with specifications for refined requirement documents by constructing a use case design prompt word template, so that LLM can fill in the objects, event flows, preconditions, postconditions and other information accurately identified from the requirement document into the use case design prompt word template in a standard format, thereby generating a refined requirement document with engineering specifications and a use case design format, solving the problem that the modification of requirement documents currently relies on manual completion, which is prone to errors and inefficient.

[0062] Specifically, in step S2, second knowledge documents related to property specifications are collected, and the second knowledge documents are segmented and stored in a vector index library to form a property specification knowledge base.

[0063] Specifically, the second knowledge documents related to property specification include design principles of model checking tools, CTL and LTL syntax specifications, conversion rules between patterns and CTL or LTL, e-books, articles, patents and other documents related to CTL and LTL design examples. The second knowledge documents are parsed and cleaned, and the cleaned second knowledge documents are segmented by sentences. All the segmented sentences are converted into third vectors and stored in a vector index library. All the stored third vectors constitute a property specification knowledge base.

[0064] It should be noted that CTL (Computation Tree Logic) and LTL (Linear Temporal Logic) represent two types of temporal logic, and the property specification documents in CTL and LTL formats can be verified for consistency through model tools.

[0065] Furthermore, a second keyword related to pattern design is established, and a pattern design prompt word template is constructed according to the second keyword, the number of variables included in the pattern, and the use cases and event flows in the first keyword.

[0066] Specifically, the second keyword includes a pattern and a parameter to be filled in. The pattern design prompt word template includes the second keyword, a use case, a basic flow and an alternative flow and their corresponding descriptions. If the pattern corresponding to the use case contains only one variable, such as a Boolean variable, the number of parameters to be filled in the basic flow and the alternative flow is 1, such as Figure 6 As shown in the first use case property specification in ; if the pattern corresponding to the use case contains 2 variables, the number of parameters to be filled in the basic flow and the alternative flow is 2, such as Figure 6 This is shown in the second use case property specification.

[0067] It should be noted that the modes include implication mode, invariant mode, forbidden state mode, state change during a cycle, state change between cycles, reachability mode, repeatability mode, and leads to mode; the implication mode is used to verify conditional relationships; the invariant mode is used to verify that certain conditions must always be true during program execution; the forbidden state mode is used to verify that certain states are not allowed to be entered; the state change during a cycle mode is used to verify the change of state within a single cycle; the state change between cycles mode is used to verify whether the state is converted as expected between consecutive cycles; the reachability mode is used to verify whether a certain state can be reached from the current state; the repeatability mode is used to verify whether the same output can be repeatedly generated under a given input; and the leads to mode is used to verify causal relationships. The parameters to be filled in represent the corresponding variable states that need to be filled in.

[0068] Furthermore, the refined requirement document and the pattern design prompt word template are input into the large language model, so that the large language model extracts the content corresponding to the first keyword from the refined requirement document according to the pattern design prompt word template, and generates several corresponding patterns for each use case in the refined requirement document. Each pattern is filled with corresponding content based on the content corresponding to the extracted first keyword according to the pattern design prompt word template, thereby generating a pattern design document.

[0069] Exemplarily, through the above method, based on Figure 5 The refined requirements document shown, Figure 6 The pattern design prompt word template shown in the figure is generated by calling the GPT-4 API. Figure 7As shown in the pattern design document, it can be seen that the pattern design document defines the status of its variables according to different patterns, providing a basis for the subsequent generation of property specification files in CTL or LTL format.

[0070] It should be noted that the pattern design document contains all patterns corresponding to all use cases in the refined requirements. Each use case corresponds to one or more patterns, and each pattern is generated according to the format of the pattern design prompt word template.

[0071] Furthermore, the pattern design document is segmented according to the pattern to obtain a number of pattern design document fragments, and for each pattern design document fragment, a third vector matching it is obtained in the property specification knowledge base, and the matching third vector is converted into a sentence to obtain a third sentence matching it.

[0072] Specifically, for each pattern design document fragment: convert the pattern design document fragment into a fourth vector; calculate the vector similarity between the fourth vector and all third vectors, and select the top k third vectors with the largest vector similarity; set the sentence window size, and establish the third sentence windows corresponding to the top k third vectors according to the sentence window size; calculate the semantic similarity between the fourth vector and all corresponding third sentence windows, select all third vectors corresponding to the third sentence windows with the largest semantic similarity as the third vector matching the pattern design document fragment, and convert the matched third vectors into sentences in context order to obtain the third sentence matching the pattern design document fragment.

[0073] It should be noted that, according to the method in step S1, the first k third vectors with the largest vector similarity are selected, a third sentence window is established, and the semantic similarity between the fourth vector and all corresponding third sentence windows is calculated.

[0074] Furthermore, all pattern design document fragments and their matching third sentences are input into the large language model, so that the large language model extracts the variable expressions corresponding to the parameters to be filled in the pattern design document fragment from the third sentence according to the pattern design document fragment, thereby generating a refined property specification document.

[0075] Exemplarily, through the above method, based on Figure 7 The model design document and property specification knowledge base shown in the figure are generated by calling the GPT-4 API. Figure 8 The refined property specification document shown in FIG. 1 corresponds to Figure 7 Each pattern in the pattern design document is shown and is in CTL or LTL format.

[0076] It can be understood that the present invention provides LLM with rules for converting from schema design to CTL or LTL property specifications by constructing a property specification knowledge base, so that LLM can convert schema design documents into refined property specification documents in CTL or LTL format.

[0077] It should be noted that in order to perform consistency check on the refined property specification document, the original requirement document needs to be manually converted into an original property specification document in CTL or LTL format.

[0078] Furthermore, the verification of the grammatical correctness of the refined property specification document includes: constructing a counterexample for each variable expression in the refined property specification document; if the constructed counterexample can obtain a correct output, then the variable expression corresponding to the counterexample is judged to be grammatically correct; if the grammars of all variable expressions are correct, then the grammar of the refined property specification document is judged to be correct; otherwise, the grammar is judged to be incorrect.

[0079] For example, suppose a variable expression is: input<0->AF result=-input, and the variable expression is set to false as its counterexample. If the counterexample can get the correct output, such as Fig. 9 As shown, the variable expression syntax is determined to be correct; otherwise, Fig.10 As shown, it is determined that the variable expression syntax is incorrect.

[0080] Furthermore, an auxiliary modification robot is constructed to check and modify the refined requirement documents and refined property specification documents; a maximum number of modifications is set; if the syntax is incorrect, the pattern design document, the refined property specification document, the counterexample of incorrect syntax and its error information are sent to the auxiliary modification robot, so that it modifies the refined property specification document; the grammatical correctness of the modified refined property specification document is verified until its syntax is completely correct or the maximum number of modifications is reached.

[0081] Specifically, the auxiliary modification robot is constructed through the agent development and management framework based on LLM, and the robot is given the ability of multi-round dialogue, so that the auxiliary modification robot can check and modify the syntax errors of variable expressions according to the received information. If the refined property specification document still has syntax errors when the maximum number of modifications is reached, it will be modified manually.

[0082] Exemplarily, the first auxiliary robot is for Fig.10 Suggested corrections for syntax errors such as Fig.11 As shown; the first auxiliary robot is for Fig.10 The syntax error shown in the following example returns the modified result. Fig.12 shown.

[0083] It is understandable that although the auxiliary modification robot may not be able to correct all grammatical errors in the refined property specification within the set maximum number of modifications, it has also made considerable modifications to its grammatical errors, and then manually corrected the remaining grammatical errors on this basis, greatly reducing the workload of manual correction.

[0084] Furthermore, the logical relationship between the original property specification document and the corresponding variable expressions in the refined property specification document after grammatical verification is verified. If the variable expression in the refined property specification document after grammatical verification implies or is equivalent to the corresponding variable expression in the original property specification document, the two are determined to be consistent; otherwise, the two are determined to be inconsistent.

[0085] Specifically, a SPEC statement is constructed, wherein the front part of the SPEC statement is a variable expression in a refined property specification document after syntax verification, the middle part of the SPEC statement is a specific symbol, and the back part of the SPEC statement is a corresponding variable expression in the original property specification document; the specific symbol represents the logical relationship between the front part and the back part of the SPEC statement, including "->", "=", wherein "->" represents implication and "=" represents equivalent to; if the test result of the SPEC statement is true, it means that the logical relationship represented by the specific symbol holds, that is, the two are consistent.

[0086] For example, Fig.13 As shown, the front part of the SPEC statement is "(input<0->AF(result=-input))&(input=0->AF(result=0))&(input>0->AF(result=input))", the middle part of the SPEC statement is "->", and the back part of the SPEC statement is "result>=0".

[0087] It can be understood that if the original property specification document is consistent with the refined property specification document, the variable expressions in the two do not conflict, and the corresponding original requirement documents and refined requirement documents are also consistent, so that the integrity of the original requirements can be preserved in the refined requirements.

[0088] Furthermore, a maximum number of modifications is set; if there is any inconsistency, the pattern design document, the current refined property specification document and its corresponding refined requirement document, and the SPEC statement error information are sent to the auxiliary modification robot, so that it can modify the refined property specification document and its corresponding refined requirement document after syntax verification; verify the consistency of the modified refined property specification document with the original property specification document until the two are consistent or the maximum number of modifications is reached.

[0089] Specifically, if the maximum number of modifications is reached and the refined property specification document after consistency verification still has variable expressions inconsistent with the original property specification, it is manually modified.

[0090] For example, the refined property specification document after consistency verification is as follows: Fig.14 As shown, the corresponding original property specification document contains only one variable expression: "result>=0", which means "the return value is always greater than or equal to 0"; and the refined property specification document after consistency verification contains three variable expressions: "(input<0->AF(result=-input)", "input=0->AF(result=0)", "input>0->AF(result=input)", which means "for the three cases where the input value is positive, negative, and zero, the corresponding value of the return value". It can be seen that the refined property specification after consistency verification has expanded and improved the original property specification.

[0091] It can be understood that the present invention sends the pattern design document, the current refined property specification document and its corresponding refined requirement document, and the SPEC statement error information to the auxiliary modification robot, so that the robot can check and modify the refined requirement document and the refined property specification document on the basis of sufficient external information prompts; and through multiple rounds of verification-modification-in-the-loop operations, iteratively generates refined property specification documents and refined requirement documents after consistency verification, thereby further improving the correctness of the refined requirement document and providing a basis for improving the correctness of the automatically generated code.

[0092] Compared with the prior art, the method for demand refinement based on a large language model provided by the present invention has the following beneficial effects:

[0093] 1. The present invention provides LLM with the specification of refined requirement documents by constructing a use case design prompt word template, so that LLM can fill in the objects, event flows, preconditions, postconditions and other information accurately identified from the requirement document into the use case design prompt word template according to the standard format, thereby generating a refined requirement document with engineering specifications and use case design format, solving the problem that the modification of requirement documents currently relies on manual completion, which is prone to errors and inefficient.

[0094] 2. The present invention converts a large amount of effective information into vectors for storage by establishing a vector index library, so that LLM can retrieve related vectors as context information through the vector index library without having to input this information into LLM as prompts, thereby improving the accuracy and efficiency of the answers output by LLM; and, by persisting the information in the database, it can be retrieved and provided to LLM at any time as needed, which is equivalent to providing LLM with long-term memory capability and improving the accuracy of LLM's answers.

[0095] 3. The present invention first searches for vectors with high similarity to the original requirement document vectors in the constructed vector index library, then searches for matching vectors with the highest semantic similarity to the original requirement document through the sentence windows of these vectors, and finally converts the matching vectors into sentences in their context order as part of the LLM context input, so that the LLM can obtain knowledge that is not mentioned in the training and is truly relevant to the input content without retraining before answering, thereby improving its accuracy in identifying relevant content in the original requirement document and providing a basis for improving the correctness of refined requirements and refined property specifications.

[0096] 4. The present invention sends the pattern design document, the current refined property specification document, the SPEC statement error information, and the refined requirement document to the auxiliary modification robot, so that the robot can check and modify the refined requirement document and the refined property specification document based on sufficient external information prompts; and through multiple rounds of verification-modification-in-the-loop operations, iteratively generates the refined property specification document and the refined requirement document after consistency verification, thereby further improving the correctness of the refined requirement document and providing a basis for improving the correctness of the automatically generated code.

[0097] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0098] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for demand refinement based on a large language model, characterized in that: The method comprises the following steps: Build a use case design knowledge base and a use case design prompt word template, and generate a refined requirement document based on the original requirement document, the use case design knowledge base, and the use case design prompt word template; Construct a property specification knowledge base and a pattern design hint word template, generate an original property specification document based on the original requirement document, generate a pattern design document based on the refined requirement document and the pattern design hint word template, and generate a refined property specification document based on the pattern design document and the property specification knowledge base; verify the grammatical correctness of the refined property specification document. If the grammar is incorrect, modify the refined property specification document based on the large language model until the grammar is correct or the maximum number of modifications is reached; verify the consistency of the original property specification document and the refined property specification document after grammar verification. If they are inconsistent, modify the refined property specification document after grammar verification and its corresponding refined requirement document based on the large language model until the two are consistent or the maximum number of modifications is reached, and obtain the refined property specification document after consistency verification and its corresponding refined requirement document.

2. The method for demand refinement based on a large language model according to claim 1, characterized in that: The original requirement document is segmented into sentences to obtain a number of original requirement document fragments, and for each original requirement document fragment, a first vector matching it is obtained in the use case design knowledge base, and the matching first vector is converted into a sentence to obtain a first sentence matching it.

3. The method for demand refinement based on a large language model according to claim 2, characterized in that: For each original requirement document fragment: convert the original requirement document fragment into a second vector; calculate the vector similarity between the second vector and all first vectors, and select the first k first vectors with the largest vector similarity; set the sentence window size, and establish the first sentence window corresponding to the first k first vectors according to the sentence window size; Calculate the semantic similarity between the second vector and all corresponding first sentence windows, select all first vectors corresponding to the first sentence windows with the largest semantic similarity as the first vectors matching the original demand document fragment, and convert the matched first vectors into sentences in context order to obtain the first sentence matching the original demand document fragment.

4. The method for demand refinement based on a large language model according to claim 2, characterized in that: Establish the first keyword related to the use case diagram design, and construct a use case design prompt word template based on the first keyword; the first keyword includes use case, participant, system boundary, relationship between participants, precondition, postcondition, use case scenario, event flow, brief description, special requirements; the use case design prompt word template includes the first keyword and its corresponding description.

5. The method for demand refinement based on a large language model according to claim 4, characterized in that: All original requirement document fragments and their matching first sentences and use case design prompt word templates are input into the large language model, so that the large language model extracts the content corresponding to the first keyword from the original requirement document fragments according to the use case design prompt word template and fills it into the use case design prompt word template, thereby generating a refined requirement document.

6. The method for demand refinement based on a large language model according to claim 1, characterized in that: The verification of the grammatical correctness of the refined property specification document includes: constructing a counterexample for each variable expression in the refined property specification document. If the constructed counterexample can obtain a correct output, then the variable expression corresponding to the counterexample is judged to be grammatically correct. If the grammars of all variable expressions are correct, then the grammar of the refined property specification document is judged to be correct, otherwise, the grammar is judged to be incorrect.

7. The method for demand refinement based on a large language model according to claim 6, characterized in that: An auxiliary modification robot is constructed to check and modify the refined requirement document and the refined property specification document; a maximum number of modifications is set; if the syntax is incorrect, the pattern design document, the refined property specification document, the counterexample of incorrect syntax and its error information are sent to the auxiliary modification robot so that it can modify the refined property specification document; the grammatical correctness of the modified refined property specification document is verified until its syntax is completely correct or the maximum number of modifications is reached.

8. The method for demand refinement based on a large language model according to claim 1, characterized in that: Verify the logical relationship between the corresponding variable expressions in the original property specification document and the refined property specification document after grammatical verification. If the variable expression in the refined property specification document after grammatical verification implies or is equivalent to the corresponding variable expression in the original property specification document, the two are determined to be consistent; otherwise, the two are determined to be inconsistent.

9. The method for demand refinement based on a large language model according to claim 8, characterized in that: Set the maximum number of modifications; if there is inconsistency, send the pattern design document, the current refined property specification document and its corresponding refined requirement document, and the SPEC statement error information to the auxiliary modification robot, so that it can modify the refined property specification document and its corresponding refined requirement document after syntax verification; verify the consistency of the modified refined property specification document with the original property specification document until the two are consistent or the maximum number of modifications is reached.

10. The method for demand refinement based on a large language model according to claim 1, characterized in that: Collect first knowledge documents related to use case diagram design, segment the first knowledge documents and store them in a vector index library to form a use case design knowledge base; parse and clean the first knowledge documents, segment the cleaned first knowledge documents by sentences, convert all the segmented sentences into first vectors and store them in the vector index library, and all the stored first vectors constitute the use case design knowledge base.

Citation Information

Cited By

  • Verification code generation method and device, equipment and storage medium

    CN122242400A