Test case construction method, device and product based on large model

Abstract

The present disclosure provides a large model-based test case construction method, device, equipment, medium and product, relates to the technical field of artificial intelligence, and in particular to the technical field of intelligent cloud, large model and the like. The large model-based test case construction method comprises the following steps: adopting a large model, analyzing a function document of a target business system to obtain a target test intention; obtaining target difference knowledge of a target environment to be deployed of the target business system; adopting the large model, synthesizing the target test intention and the target difference knowledge to obtain an initial test case; and obtaining a target test case based on the initial test case. The present disclosure can improve the performance of the test case.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of intelligent cloud and large models, and specifically to a method, apparatus, device, medium and product for constructing test cases based on large models. Background Technology

[0002] Currently, under the background of information technology application innovation (IT innovation), power business systems need to migrate their information technology (IT) infrastructure, such as migrating traditional processor architectures and traditional operating systems to domestically produced processors and operating systems. This migration process involves changes at multiple levels, and in order to ensure the reliable operation of the migrated facilities, test cases are needed to verify the functionality of the migrated facilities.

[0003] How to construct test cases is a problem that needs to be solved. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, medium, and product for constructing test cases based on a large model.

[0005] According to one aspect of this disclosure, a test case construction method based on a large model is provided, comprising: using a large model to parse the functional documents of a target business system to obtain target test intent; obtaining target difference knowledge of the target environment to be deployed in the target business system; using the large model to synthesize the target test intent and the target difference knowledge to obtain initial test cases; and obtaining target test cases based on the initial test cases.

[0006] According to another aspect of this disclosure, a test case construction apparatus based on a large model is provided, comprising: a parsing module, used to parse the functional documents of a target business system using the large model to obtain target test intent; an acquisition module, used to acquire target difference knowledge of the target environment to be deployed by the target business system; a synthesis module, used to synthesize the target test intent and the target difference knowledge using the large model to obtain initial test cases; and a completion module, used to obtain target test cases based on the initial test cases.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.

[0009] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.

[0010] According to embodiments of this disclosure, the performance of test cases can be improved.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0014] Figure 2 The overall architecture diagram of the system is constructed based on the test cases provided in the embodiments of this disclosure;

[0015] Figure 3 This is a schematic diagram according to the second embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram illustrating the specific implementation process of the offline construction phase of the knowledge base module for the characteristics of the information technology innovation environment provided in the embodiments of this disclosure;

[0017] Figure 5 This is a schematic diagram illustrating the specific implementation process of the online query stage of the knowledge base module for the characteristics of the information technology innovation environment provided in this embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram illustrating the specific implementation process of the power business test intent parsing module provided in the embodiments of this disclosure;

[0019] Figure 7 This is a schematic diagram illustrating the specific implementation process of the intelligent synthesis module for test cases provided in the embodiments of this disclosure;

[0020] Figure 8 This is a schematic diagram illustrating the specific implementation process of the coverage adaptive completion module provided in the embodiments of this disclosure;

[0021] Figure 9 This is a schematic diagram according to the third embodiment of the present disclosure;

[0022] Figure 10 This is a schematic diagram of an electronic device used to implement the large-model-based test case construction method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In related technologies, the mainstream practice is for test engineers familiar with the domestic IT innovation environment to manually rewrite the original test case set item by item after reading the domestic operating system (OS) and chip technology documents, and to add compatibility test items based on experience.

[0025] However, manually drafting solutions relies heavily on the individual experience of engineers, resulting in low knowledge conversion efficiency and poor consistency.

[0026] While there are solutions that utilize Large Language Models (LLMs) to assist in generating test cases, these solutions primarily involve providing the LLM with functional requirement descriptions, which then generate the corresponding test steps and expected results. This approach is mainly aimed at functional testing scenarios for general software. Since the LLM generates test cases based on general training data, it cannot perceive the unique boundary conditions of domestic OSs or Central Processing Units (CPUs), resulting in poor compatibility.

[0027] In order to at least partially solve at least one of the above problems, this disclosure provides the following embodiments.

[0028] Figure 1 Based on a schematic diagram of the first embodiment of this disclosure, this embodiment provides a method for constructing test cases based on a large model. For example... Figure 1 As shown, the method includes:

[0029] 101. Use a large model to analyze the functional documents of the target business system in order to obtain the target testing intent.

[0030] 102. Obtain target difference knowledge of the target environment to be deployed for the target business system.

[0031] 103. Using the large model, synthesize the target testing intent and the target difference knowledge to obtain initial test cases.

[0032] 104. Based on the initial test cases, obtain the target test cases.

[0033] Large models refer to deep learning models with large scale. For example, a large model is a large language model (LLM). An LLM is a deep learning model trained with a large amount of text data. It can generate natural language text or understand the meaning of language text. The number of parameters usually reaches billions to trillions.

[0034] LLM is based on prompts, which are text instructions input into the LLM to guide the model to generate specific, accurate, or expected outputs.

[0035] The target business system refers to the business system that is to be migrated to the domestic IT innovation environment (domestic OS and / or domestic CPU), such as the power business system.

[0036] Functional documentation refers to existing documents that describe the functions of the target business system. Specifically, it may include functional documentation for each subsystem of the target business system. For example, if the target business system is a power business system, it typically includes subsystems such as an Operation Management System (OMS), a Production Management System (PMS), an Energy Management System (EMS), and a Supervisory Control and Data Acquisition (SCADA) system. Therefore, this functional documentation may include functional documentation for each of these subsystems, and can be uploaded by users (such as migration engineers).

[0037] Once the functional documentation is obtained, it can be input into the large model, which will then be instructed to perform intent recognition. The output will be the target test intent.

[0038] The target test intent refers to the intent obtained after the large model parses the functional documents. Specifically, the target test intent can be obtained by extracting the functional description information in the functional documents.

[0039] The target test intent can be structured data, which can include the following fields: functional module label, intent type, functional summary, verification target description, key constraints, and priority.

[0040] The target environment refers to the domestic OS and / or domestic CPU to which the target business system is to be migrated, which can be specified by the user.

[0041] Target difference knowledge refers to the difference knowledge between the target environment and the pre-migration environment (such as a Windows system).

[0042] Among them, a knowledge base can be pre-built, which stores the difference knowledge of all domestically developed operating systems, such as the first knowledge of the first domestically developed OS and the second knowledge of the second domestically developed OS. Then, assuming that the target environment is the first domestically developed OS, the first knowledge is used as the target difference knowledge.

[0043] After obtaining the target testing intent and target difference knowledge, it is input into the large model, and the large model is instructed to generate test cases. The output of the large model is the initial test cases.

[0044] Initial test cases are test cases generated from the large model, and may include the following fields: test case number, test case title, preconditions (including requirements for the domestic IT innovation environment), test steps (including domestic IT innovation-specific operation steps), expected results (including domestic IT innovation compatibility judgment criteria), verification dimension tags, and a list of item IDs corresponding to the covered difference knowledge.

[0045] The target test case refers to the final test case that is constructed, which can be obtained by completing the initial test cases.

[0046] In this embodiment, initial test cases are generated based on a large model, which improves efficiency and accuracy compared to manually configuring test cases. The large model synthesizes initial test cases based on the target test intent and target difference knowledge, taking into account the unique knowledge of the target environment. This allows for the perception of the unique boundary conditions of the target environment, resulting in more adaptable test cases. Therefore, test cases that simultaneously cover the correctness of business logic and compatibility with the target environment can be obtained, improving test case performance.

[0047] In some embodiments, the step of employing a large model to parse the functional documents of the target business system to obtain the target testing intent includes:

[0048] Extract the function description text from the function document;

[0049] Based on the function description text and the preset domain enhancement prompt word strategy, construct prompt words;

[0050] The prompt words are input into the large model to drive the large model to output the initial test intent;

[0051] Based on the initial test intent, the target test intent is obtained.

[0052] Among them, the functional documents can be uploaded by users. After obtaining the functional documents, they can be preprocessed to extract the functional description text. This can extract effective information and reduce the processing volume and difficulty of subsequent large models.

[0053] After obtaining the feature description text, it can be added to the prompt according to preset rules and input into the large model. The large model performs intent parsing based on the prompt, and the output is the initial test intent. After obtaining the initial test intent, it can be used as the target test intent; or, the initial test intent can be merged or processed to obtain the target test intent.

[0054] When constructing prompts, a domain-enhanced prompt strategy can be used. This strategy provides the large model with relevant information about the target business system, thereby driving the large model to generate test intents that are more consistent with the target business system.

[0055] In this embodiment, by extracting functional description text from functional documents and performing intent recognition based on functional description text, the effectiveness of the text processed by the large model can be improved, and efficiency and accuracy can be enhanced. Constructing prompt words based on domain enhancement prompt strategies can provide domain knowledge to the large model and improve the accuracy of test intent.

[0056] In some embodiments, the domain-enhanced prompting strategy includes at least one of the following:

[0057] The functional module labels in the target business system;

[0058] Intent type determination rules;

[0059] Rules for determining intent priority.

[0060] Among them, the functional module label is used to identify the functional modules in the target business system. By providing functional module labels, the large model can be driven to select from multiple provided functional module labels, thereby improving the accuracy of the final determined current functional module label.

[0061] The rules for determining intent type and intent priority can drive large models to accurately determine intent type and intent priority.

[0062] In addition, when constructing prompts, the role of the large model can be indicated, such as its use for intent recognition. The output format of the large model can also be indicated, such as the intent determination result including: functional module label, intent type and intent priority.

[0063] In this embodiment, by including functional module tags, intent type determination rules, and intent priority determination rules in the domain enhancement prompt strategy, the large model can be driven to generate test intents that are more in line with the target business system, thereby improving the accuracy of test intents.

[0064] In some embodiments, the initial test intent includes: verifying the target description;

[0065] The step of obtaining the target test intent based on the initial test intent includes:

[0066] For any two initial test intentions, calculate the semantic similarity between the two initial test intentions based on the corresponding verification target descriptions;

[0067] In response to the semantic similarity being greater than a preset threshold, any two initial test intentions are merged to obtain the target test intention.

[0068] The large model can generate an initial test intent for each function description text. Since the function document may contain multiple function description texts, the large model may obtain multiple identical or similar initial test intents after processing each function description text. At this time, identical or similar initial test intents can be merged to obtain the target test intent.

[0069] Specifically, the initial test intent includes a verification target description, which is the text used to describe the verification target. At this time, for each pair of initial test intents, if the semantic similarity of the verification target descriptions of the two initial test intents is greater than a preset threshold, they are merged and used as a single target test intent.

[0070] In this embodiment, when the semantic similarity of the verification target descriptions of any two initial test intentions is greater than a preset threshold, they are merged. This can merge the same or similar test intentions, avoid intention duplication, and improve the effectiveness of the intentions.

[0071] In some embodiments, the two initial test intents include: a first intent and a second intent, wherein the length of the verification target description of the first intent is greater than the length of the verification target description of the second intent;

[0072] The initial test intent also includes: constraints and intent priority;

[0073] The step of merging any two initial test intentions to obtain the target test intention in response to the semantic similarity being greater than a preset threshold includes:

[0074] In response to the semantic similarity being greater than a preset threshold, the first intent is taken as the target test intent, the constraints of the second intent are merged into the constraints of the first intent, and the intent priority of the target test intent is set to the highest priority among the intent priorities corresponding to the first intent and the second intent.

[0075] In this case, assuming that the initial test intentions for pairwise comparison are the first intention and the second intention, if the semantic similarity of the verification target descriptions of these two intentions is greater than a threshold, they are merged. In this case, the initial test intention with the longer verification target description can be used as the target test intention, and the constraints of the other intention can be merged into the target test intention, with the highest priority selected.

[0076] For example, if the description of the first intent's verification target is long, and the constraints of the first intent and the second intent are the first constraint and the second constraint, with priorities P1 and P0 respectively, and assuming that P0 has a higher priority, then the target test intent is the first intent, and its constraints include the first constraint and the second constraint, with priority P0.

[0077] In this embodiment, by using the above-described merging method, we can obtain a target test intent that is more clearly described in terms of the verification target, more comprehensive in terms of constraints, and more precise in terms of priority.

[0078] In some embodiments, obtaining the target difference knowledge of the target environment to be deployed for the target business system includes:

[0079] Based on the environmental tags of the target environment, existing differential knowledge in the pre-established knowledge base is filtered to obtain candidate differential knowledge corresponding to the target environment;

[0080] Based on the vector similarity between the target testing intent and the candidate knowledge, the target difference knowledge is obtained from the candidate knowledge.

[0081] The environment label can specifically include the target OS name and the target CPU name. Existing difference knowledge can include the environment label. Based on this, candidate difference knowledge that matches the target OS name and the target CPU name can be obtained. Then, the vector similarity is calculated based on the corresponding vectors, and a preset number (Top-K) of candidate difference knowledge with high similarity is selected as the target difference knowledge.

[0082] In this embodiment, filtering is first performed based on environmental labels, and then selection is performed based on vector similarity. This two-stage retrieval method can obtain target difference knowledge, thereby balancing accuracy and semantic relevance.

[0083] In some embodiments, the step of using the large model to synthesize the target testing intent and the target difference knowledge to obtain initial test cases includes:

[0084] Based on the target testing intent and the target difference knowledge, prompt words are constructed;

[0085] The prompt words are input into the large model to drive the large model to output synthetic test cases;

[0086] The quality of the synthesized test cases is verified.

[0087] In response to the synthetic test case passing quality verification, the synthetic test case is used as the initial test case.

[0088] After obtaining the target testing intent and target difference knowledge, prompt words can be constructed based on this and input into the large model. The prompt words can also include role information, that is, instructing the large model to synthesize test cases. The large model can synthesize test cases based on the prompt words, and the output test cases are called synthesized test cases. Then, the synthesized test cases are quality verified. If they pass the quality verification, they are used as the initial test cases. Otherwise, the large model can be driven to regenerate test cases, and the number of retries can be limited, that is, not exceeding the maximum number of retries. If test cases that pass the quality verification are still not generated after reaching the maximum number of retries, manual intervention can be triggered.

[0089] In this embodiment, by performing quality verification on the synthetic test cases and using the synthetic test cases that have passed the quality verification as the initial test cases, the quality of the initial test cases can be improved.

[0090] In some embodiments, the quality verification of the synthetic test cases includes:

[0091] Perform field integrity verification on the synthesized test cases;

[0092] In response to the synthetic test case passing the field integrity verification, a perceptual consistency verification is performed on the synthetic test case;

[0093] In response to the synthetic test case passing the perceived consistency verification, it is determined that the synthetic test case has passed the quality verification.

[0094] In the quality verification process, two layers of quality verification can be included: field integrity verification and perceived consistency verification. Field integrity verification is used to check whether the test case contains preset key fields. If the synthesized test case contains all the key fields and is not empty, the field integrity verification passes; otherwise, it fails. Perceived consistency verification is used to check whether it matches the post-migration environment. This can be done through keywords and preset rules. Taking the post-migration environment as an example of a domestically developed environment, it can be verified whether the specified prerequisite environment contains the keywords of the target environment (such as the target OS name and the target CPU name). If not, the perceived consistency verification fails.

[0095] In this embodiment, by using field integrity verification and perceived consistency verification, the comprehensiveness of quality verification can be improved, thereby improving the quality of the initial test cases.

[0096] In some embodiments, obtaining the target test case based on the initial test case includes:

[0097] Based on the initial test cases, construct a coverage matrix;

[0098] In response to the coverage matrix meeting the preset coverage requirements, the initial test case is used as the target test case.

[0099] The coverage matrix is ​​used to describe the dimensions covered by the test cases.

[0100] The dimensions can specifically include three dimensions: OS, CPU, and functional modules. If the initial test case includes the first OS, the first CPU, and the first functional module, it means that the unit of the first OS, the first CPU, and the first functional module is covered. If the second OS is not included, then the unit corresponding to the second OS is not covered.

[0101] There are usually multiple initial test cases, which can form a test case set. Different initial test cases correspond to different coverage units. The current coverage can be obtained by statistics. The coverage requirement can be a target coverage (such as 90%). When the current coverage reaches the target coverage, these initial test cases are used as target test cases.

[0102] Assuming the migrated environment uses n1 types of OS, n2 types of CPU, and n3 types of functional modules (n1~n3 are all positive integers), there are a total of n1*n2*n3 (* indicates multiplication) coverage units. The current number of coverage units can be calculated based on the dimension labels corresponding to the initial test cases. For example, if the initial test cases correspond to one OS, one CPU, and two functional modules, then the current number of coverage units = 1*1*2. The current coverage rate is obtained by dividing the current number of coverage units by the total number of coverage units.

[0103] In this embodiment, when the coverage matrix of the initial test cases meets the coverage requirements, it is used as the target test case, which can improve the coverage of the target test cases.

[0104] In some embodiments, the method further includes:

[0105] In response to the coverage matrix not meeting the preset coverage requirements, complete test cases are obtained;

[0106] The initial test case and the completed test case are merged to obtain the target test case.

[0107] Specifically, when the current coverage is less than the target coverage, supplementary test cases can be obtained. These supplementary test cases are then merged with the initial test cases to obtain the target test cases. For example, if there are 3 initial test cases and 2 supplementary test cases, they can be merged to obtain 5 target test cases.

[0108] When obtaining complete test cases, new target difference knowledge can be obtained based on the target environment to be completed. Test cases are then resynthesized according to the target test intent and the new target difference knowledge. After quality verification, complete test cases are obtained.

[0109] In this embodiment, when the coverage matrix does not meet the coverage requirements, supplementary test cases are obtained, and the initial test cases and supplementary test cases are merged to obtain target test cases, which can improve the coverage of target test cases.

[0110] In some embodiments, constructing the coverage matrix based on the initial test cases includes:

[0111] Extract multiple dimension labels from the initial test cases, including: target OS label, target CPU label, and functional module label;

[0112] The coverage matrix is ​​constructed based on the multiple dimensional labels.

[0113] The coverage matrix can be constructed based on the dimension labels in the initial test cases. For example, if the test cases include the first OS, the first CPU, the first functional module, and the second functional module, then the first unit (the three dimensions are the first OS, the first CPU, and the first functional module) and the second unit (the three dimensions are the first OS, the first CPU, and the second functional module) are constructed. If no other dimension labels are included, then other units can be set to empty. For example, the third unit (the three dimensions are the first OS, the second CPU, and the first functional module) is empty. The coverage matrix is ​​formed based on the constructed units (such as the first unit and the second unit) and the empty units (such as the third unit).

[0114] In this embodiment, a coverage matrix is ​​constructed based on multiple dimension labels in the initial test cases. These multiple dimension labels include target OS labels, target CPU labels, and functional module labels, which can achieve test case coverage on OS, CPU, and functional modules, reducing the problem of test case omissions.

[0115] The following explanation uses the example of a power business system as the target business system.

[0116] Figure 2 This is an overall architecture diagram of the system built based on the test cases provided in the embodiments of this disclosure.

[0117] like Figure 2 As shown, this embodiment takes the target business system as an example, which is the power business system. The system 200 mainly includes: a knowledge base module 201 for the characteristics of the information technology innovation environment, a power business test intent parsing module 202, a test case intelligent synthesis module 203, and a coverage adaptive completion module 204.

[0118] The knowledge base module 201 for the characteristics of the information technology innovation environment is used to store characteristic knowledge of the information technology innovation environment, that is, knowledge of the differences between it and the environment before migration; and to provide target difference knowledge of the target environment during use case synthesis.

[0119] The Power Business Test Intent Parsing Module 202 is used to parse the functional documents of the power business system using a large model to obtain the target test intent.

[0120] The test case intelligent synthesis module 203 is used to synthesize the target test intent and target difference knowledge using a large model to obtain initial test cases.

[0121] The coverage adaptive completion module 204 is used to detect whether the initial test cases meet the preset coverage requirements and to complete the missing test cases to obtain the target test cases.

[0122] Among them, such as Figure 2 As shown, the input to the knowledge base module for the characteristics of the information technology innovation environment can include information technology innovation environment documents and migration test execution results obtained from historical migration processes. Based on this input, differential knowledge of the information technology innovation environment can be constructed. The knowledge can be in the form of items or knowledge entries.

[0123] Specifically, the knowledge base module for the characteristics of the domestic IT innovation environment is the knowledge infrastructure of this system. It is responsible for organizing and managing the knowledge entries on the differences between domestic OS and / or domestic CPU and traditional environments (such as x86 and Windows systems) in a structured form, as well as the regression test failure patterns accumulated in historical domestic IT innovation migration projects, so as to provide accurate difference knowledge retrieval services for the synthesis module.

[0124] Input: Technical documents of domestic OS (including multiple domestic OS), architecture manuals of domestic CPU (including multiple domestic CPU), test reports of historical migration projects, and a list of known compatibility issues.

[0125] Core processing logic: The knowledge base is organized into multiple partitions according to the difference categories, including: system call behavior difference partition (covering file system interface, process scheduling, signal processing), character set and encoding difference partition (the behavior of GB18030 and UTF-8 in domestic database drivers), CPU architecture characteristics partition (ARM memory alignment rules, floating-point precision differences, single instruction multiple data (SIMD) instruction set differences), domestic middleware (such as domestic databases) compatibility partition, and historical failure record partition (stores failure cases and root causes found in historical migration projects). Each knowledge entry includes fields such as difference description, triggering scenario, affected power system functional modules, and historical failure record references. The module supports hybrid retrieval based on embedding vectors, enabling the test case synthesis module to accurately recall relevant difference knowledge according to test intent and target environment.

[0126] Output: A structured collection of discrepancy knowledge entries, including fields such as discrepancy category, triggering condition, and expected abnormal behavior.

[0127] Interaction with other modules: Provide the test case intelligent synthesis module with difference knowledge retrieval results related to the current test intent; provide the coverage adaptive completion module with the number statistics of difference knowledge entries under each target OS version and CPU model, which are used as a proxy indicator of "mainstream status of target OS version" in the coverage hole priority ranking (the enumeration list of target OS versions and CPU models is directly provided by the user when specifying the target coverage matrix).

[0128] Example: When the synthesis module requests information on the differences in information technology innovation related to "substation remote signaling status transmission function + CPU X (a domestic CPU)", the knowledge base returns:

[0129] {Difference Category: "ARM Memory Alignment", Triggering Scenario: "Structure members did not explicitly specify alignment attributes during IEC 61850 data frame parsing", Affected Function: "SCADA Remote Signaling Upload", Historical Failure Record Reference: "Migrating test failure record of a certain provincial network: Parsing a 4-byte aligned IEC message on CPU X triggered a bus signal abnormality"}.

[0130] The power business test intent parsing module is responsible for semantic parsing of the functional documents of the power business system. It extracts structured business test intents through a large language model, transforms natural language functional descriptions into standardized test intent representations that can be consumed by the intelligent test case synthesis module, and marks the intent type and priority.

[0131] Inputs: Functional documents for power business systems (covering OMS, PMS, EMS, and SCADA subsystems), system interface documents, and historical user stories.

[0132] Core processing logic: A large language model is used to perform segmented semantic parsing of the functional documents, identifying the test intent type (forward process verification, boundary condition verification, exception handling verification, or performance benchmark verification) corresponding to each functional description paragraph; for each identified functional point, the large language model extracts the test intent... Figure 3 Tuple: {Name of the function under test, verification target, key constraints}; For the function description patterns unique to the power industry (such as "remote signaling transmission cycle not exceeding X milliseconds" and "equipment ledger export supports GB18030 encoding"), the model improves the recognition accuracy of power-specific terms through a domain-enhanced prompt word strategy; after extraction, similar intents are semantically deduplicated and merged to avoid repetitive synthesis caused by multiple descriptions of the same function in the function document.

[0133] Output: A structured list of test intents, each intent containing: {functional module label, intent type, functional summary, verification target description, key constraints, priority (P0, P1 or P2)}.

[0134] Interaction with other modules: The output structured test intent is passed to the test case intelligent synthesis module as a core component of the synthesis context; the coverage adaptive completion module also queries the target test intent from this module when initiating a supplementary synthesis request.

[0135] Example: Input function document paragraph "The remote signaling status change information transmitted by the substation-side IED device should be received by the SCADA front-end machine and written into the real-time database within 500ms, and the database write success rate should not be less than 99.9%", Output: {Function module label: "SCADA-Remote Signaling Acquisition", Intent type: "Performance benchmark + Functional correctness", Function summary: "Verification of remote signaling status change transmission delay and database write success rate", Verification target description: "End-to-end delay not exceeding 500ms, write success rate not less than 99.9%", Key constraint: "IED-Front-end machine-Real-time database full link", Priority: "P0"}.

[0136] The test case intelligent synthesis module is responsible for combining the power business testing intent with knowledge of differences in the information technology innovation environment and historical failure records as context to drive the large language model to synthesize complete test cases that cover both the correctness of business logic and the compatibility risks unique to the information technology innovation environment, and to perform two-layer quality verification on the generated results.

[0137] Input: Structured test intent from the power business test intent parsing module, relevant difference knowledge items (Top-K) from the knowledge base of information technology innovation environment characteristics, target environment specifications (OS model / version + CPU model), and historical failure record summary (optional).

[0138] Core processing logic: For each input test intent, retrieve the most relevant difference knowledge entries from the knowledge base that relate to the functional module and target environment of that intent; construct segmented structured synthetic prompts, and inject the test intent, difference knowledge, historical failure cases, target environment information, and output format specifications into the large language model; the large language model generates test cases containing complete fields: {test case number, test case title, preconditions (including requirements for the domestic IT innovation environment), test steps (including domestic IT innovation-specific operation steps), expected results (including domestic IT innovation compatibility judgment criteria), verification dimension labels, and a list of covered difference knowledge entry IDs}; perform dual-layer verification of structural integrity and domestic IT innovation perception consistency on the generated results, and trigger regeneration if it fails, and mark it as manually reviewed if the retry limit is exceeded.

[0139] Output: The initial version of the complete test case set, with each test case carrying coverage dimension labels {OS version, CPU model, functional module}.

[0140] Interaction with other modules: Receive the output of the intent parsing module and the retrieval results from the knowledge base; pass the initial use case set to the coverage adaptive completion module; respond to supplementary synthesis requests initiated by the completion module.

[0141] Example: Input {Intent: "SCADA remote signaling acquisition latency verification", Target environment: "X1 and Y1", Difference knowledge: "ARM memory alignment causes IEC message parsing anomalies"}, Output test case: {Prerequisites: "Install X1 on Y1 server, deploy SCADA front-end service, configure IED simulator to generate remote signaling change messages, verify SCADA process runs in non-forced alignment mode", Test steps: ["Step 1: Use IED simulator to send 100 frames each of 4-byte aligned and unaligned IEC61850 messages", "Step 2: Observe SCADA front-end process log, check for bus signal or message parsing errors", "Step 3: Record the latency of writing remote signaling status to real-time database, and calculate P99 latency"], Expected result: "No bus signal in front-end process, P99 latency not exceeding 500ms, write success rate not less than 99.9%", Verification dimension label: ["Business logic", "Information technology innovation compatibility - ARM memory alignment"]}. Among them, X1 and Y1 are a domestically produced OS and a domestically produced CPU, respectively.

[0142] The coverage adaptive completion module is responsible for performing a three-dimensional coverage matrix analysis on the currently synthesized test case set, identifying coverage gaps in the three dimensions of OS version, CPU model, and business function module, and triggering targeted supplementary synthesis requests until the preset coverage completeness target is achieved, and finally outputting a complete test case set and coverage report.

[0143] Input: Current use case set (including coverage dimension labels for each use case), target coverage matrix definition (user-specified OS version list × CPU model list × functional module list), coverage completeness threshold (default requirement is 100% coverage of P0 functional modules in all target environment combinations).

[0144] Core processing logic: Scan the current test case set and extract a coverage dimension vector {OS version, CPU model, functional module} for each test case; construct a three-dimensional coverage matrix and calculate the current coverage rate, identifying combinations with a value of zero (i.e., no test case coverage) as coverage gaps; sort coverage gaps by business priority (P0 functional modules first) and environment dimension in multiple levels; initiate targeted supplementary synthesis requests {target intent, target OS version, target CPU model, priority coverage difference knowledge category} to the test case intelligent synthesis module; update the coverage matrix after supplementary synthesis is completed, and repeat until all P0 and P1 functional modules are covered under all target environment combinations.

[0145] Output: Complete use case set (coverage meets threshold requirements), 3D coverage report (including use case counts and coverage heatmaps for each matrix cell).

[0146] Interaction with other modules: Read the output test case set of the test case intelligent synthesis module; initiate a supplementary synthesis request to the synthesis module; query the knowledge base of characteristics of the information technology innovation environment to obtain the number statistics of existing difference knowledge entries under each target OS version and CPU model, which is used as the data source for the proxy indicator of "mainstream degree of target OS version" in the priority ranking of coverage holes (the enumeration list of target OS versions and CPU models is directly provided by the user when specifying the target coverage matrix).

[0147] Example: After initial synthesis, the coverage matrix found that the combination {Function: "EMS Scheduling Status Display", OS: "X2", CPU: "Y2"} lacked test case coverage. The module triggered a supplementary synthesis request. The intelligent test case synthesis module generated targeted test cases for display correctness and rendering performance based on the differences in the SIMD instruction set of Y2 and the graphics stack characteristics of X2, filling the coverage gap. Here, X2 and Y2 represent a different domestically produced OS and CPU compared to X1 and Y1 mentioned above.

[0148] In conjunction with the above overall architecture, this disclosure also provides the following embodiments.

[0149] Figure 3 This is a schematic diagram based on the second embodiment of the present disclosure. This embodiment provides a method for constructing test cases based on a large model. Figure 3 As shown, the method includes:

[0150] 301. User uploads relevant information.

[0151] Specifically, the relevant information may include: functional documents of the power business system, target environment information (such as target OS name and target CPU name), and target coverage threshold.

[0152] 302. Power Business Test Intent Parsing Module: Extracts target test intent based on LLM.

[0153] This module can use LLM to parse the aforementioned functional documents and obtain the target test intent.

[0154] 303. Test case intelligent synthesis module, which obtains target difference knowledge of the target environment.

[0155] This module can interact with the knowledge base module for the characteristics of the information technology innovation environment to obtain target difference knowledge of the target environment from the knowledge base module.

[0156] 304. The test case intelligent synthesis module uses LLM to synthesize initial test cases based on the target test intent and target difference knowledge.

[0157] 305. Coverage Adaptive Completion Module: Verifies whether the initial test cases have reached the target coverage threshold. If not, it completes the test cases and obtains the target test cases based on the initial test cases and the completed test cases.

[0158] Specifically, a coverage matrix can be constructed based on the coverage dimension labels contained in the initial test cases. If the coverage of the coverage matrix does not reach the target coverage, the test case intelligent synthesis module is requested to regenerate test cases as supplementary test cases until the coverage reaches the target coverage threshold. Then, the initial test cases and supplementary test cases are merged to obtain the target test cases.

[0159] After obtaining the target test cases, the target test cases and their corresponding coverage reports can be stored in the test case library and returned to the user.

[0160] Furthermore, the relevant modules mentioned above are explained in detail below.

[0161] Figure 4 This is a schematic diagram illustrating the specific implementation process of the offline construction phase of the knowledge base module for the information technology innovation environment provided in the embodiments of this disclosure.

[0162] like Figure 4 As shown, the processing steps of this offline build phase include:

[0163] 401. Extract plain text paragraphs from the original knowledge.

[0164] 402. Extracting differential knowledge items based on LLM.

[0165] 403. Generate entry embedding vectors.

[0166] 404. Write into the vector knowledge base.

[0167] The original knowledge may include existing documents of the target business system uploaded by the user, such as domestic OS documents and domestic CPU manuals, and may also include failure records obtained from historical tests.

[0168] After obtaining the above information, it can be preprocessed, that is, the plain text paragraphs can be extracted, and then the differential knowledge items can be extracted based on LLM.

[0169] Each entry includes the following fields:

[0170] An identifier, used to globally and uniquely identify this difference in knowledge;

[0171] Difference Category: An enumerated value used to identify the category of this difference knowledge, such as system call differences, memory alignment, character set encoding, database driver or middleware compatibility, etc.

[0172] Affected OS range: an enumeration value used to identify the OS to which this difference knowledge applies, such as X1 and X2, where X1 and X2 are both domestic OS names;

[0173] Affected CPU range: an enumeration value used to identify the CPUs to which this difference applies, such as Y1 and Y2, where Y1 and Y2 are both names of domestically produced CPUs;

[0174] Triggering prerequisites: Natural language description, including key technical feature tags, used to describe the specific prerequisites that trigger this difference;

[0175] Abnormal Behavior: Used to describe abnormal behavior in the domestic IT innovation environment, and to provide a comparison with existing environments (such as x86+Windows);

[0176] Impact Functional Module: A functional module used to indicate the impact of this difference in knowledge. It can contain structured information, including {impact module label, impact type, and severity (P0, P1, or P2)}.

[0177] Failure record: ID of the failure record used to characterize the associated historical test;

[0178] Vector: The embedding vector of this difference knowledge can be obtained by concatenating the difference category, triggering preconditions, abnormal behavior and influencing module labels into text and then embedding them.

[0179] During extraction: For technical documents (OS documents and CPU manuals), LLM is used to extract structured difference entries segment by segment using difference extraction prompts (system-defined entry field semantics + plain text paragraphs in user-input documents + required output format entries); For historical failure records, LLM is used to extract {root cause category, triggering scenario, affected functional modules} and map them to entry fields, while establishing reference links with the source failure records to ensure that knowledge entries can be traced back to specific historical cases.

[0180] That is, in the above fields, the identifier can be automatically generated, such as sequential numbering; the difference category, affected OS range, affected CPU range, affected functional modules and abnormal behavior can be obtained based on historical failure records and / or after parsing technical documents; the triggering preconditions can be obtained after analyzing technical documents; the failure record is an associated record obtained from historical failure records; and the vector is obtained by concatenating and embedding the above related fields.

[0181] Figure 5 This is a schematic diagram illustrating the specific implementation process of the online query stage of the knowledge base module for the characteristics of the information technology innovation environment provided in the embodiments of this disclosure.

[0182] like Figure 5 As shown, the processing steps for this online query phase include:

[0183] 501. Receive search request.

[0184] The retrieval request may include the target test intent and the environment tag of the target environment. Specifically, the environment tag may include the target OS name (e.g., X1) and the target CPU name (e.g., Y1).

[0185] 502. Perform hard filtering on search requests based on environment tags.

[0186] For example, if a knowledge entry has field X2, and the search request includes X1, then that entry will be filtered out and will not participate in subsequent semantic calculations. If another knowledge condition has fields including both X1 and Y1, then it will be retained and participate in subsequent semantic calculations.

[0187] 503. Ranked based on semantic similarity.

[0188] This can be achieved by calculating the semantic similarity, such as cosine similarity, between the target testing intent and the knowledge items after hard filtering, and sorting them from high to low based on this similarity.

[0189] The vector of the target testing intent can be obtained by concatenating the text of the functional module tags, verification target description, and constraints in the target testing intent, and then embedding them. Vectors of existing knowledge entries, as described above and directly recorded in the vector database, can be directly retrieved for similarity calculation.

[0190] 504. Obtain a preset number of target difference knowledge items with high semantic similarity.

[0191] In this context, assuming the preset number is represented by K (K is a positive integer), the items retained after hard filtering are called candidate difference knowledge. Then, the K candidate difference knowledge items with high semantic similarity (represented by Top-K) can be used as target difference knowledge.

[0192] Figure 6 This is a schematic diagram illustrating the specific implementation process of the power business test intent parsing module provided in the embodiments of this disclosure.

[0193] like Figure 6 As shown, the processing procedure of the power business test intent parsing module includes:

[0194] 601. Extract the function description text from the function document.

[0195] The functional documents can be uploaded by users. After obtaining the functional documents, the relevant descriptive paragraphs can be extracted as functional description text.

[0196] 602. Use LLM for intent extraction.

[0197] Among these methods, prompts can be constructed based on domain-enhanced prompt strategies, and intent recognition can be performed by driving LLM based on these prompts.

[0198] 603. Obtain the test intent.

[0199] Among them, the test intent includes intention Figure 3 The tuple is: {functional module label, verification target description, constraints}.

[0200] 604. Determine the intent type.

[0201] The intent type is selected from existing types, such as forward flow, boundary conditions, exception handling, or performance benchmark.

[0202] 605. Priority of annotation intent.

[0203] These can be identified according to priority determination rules, for example, including three priorities, represented by P0, P1 and P2 respectively, and the priority order is P0>P1>P2.

[0204] 606. Calculate the pairwise similarity of intentions.

[0205] In this process, the description of the verification target of the intent can be embedded to obtain a semantic vector, and the similarity can be calculated based on the semantic vector.

[0206] 607. Determine if the similarity exceeds the threshold and if the functional module labels are the same. If yes, proceed to 608; otherwise, proceed to 609.

[0207] 608. Perform the intended merger.

[0208] 609. Retained as an independent intent.

[0209] 610. Generate the final list of test intents based on the merged intents and the independent intents.

[0210] Specifically, considering the domain characteristics of functional documents in the power business system, the module adopts a domain-enhanced prompt strategy, injecting power industry semantic mapping rules into the system prompts. This enables the LLM to accurately parse power-specific terminology without requiring specific fine-tuning. The prompts consist of the following key sections:

[0211] Role Partitioning: Set LLM as "a senior test engineer proficient in power dispatch automation and distribution network production management system testing, familiar with the functional specifications of OMS, PMS, EMS, and SCADA systems" to activate the model's sensitivity to power industry terminology.

[0212] Module Mapping Partition: Provides a list of standard functional module labels, listing the standard module divisions of each subsystem (such as "SCADA-Remote Signal Acquisition", "SCADA-Remote Control Execution", "EMS-Dispatch Status Display", "PMS-Equipment Ledger Management", "PMS-Defect Management", "OMS-Work Order Dispatch", "OMS-Emergency Repair Command", etc.). The model is required to map the functional description to the most matching standard label. If there is no matching label, a new label is generated and follows the naming format "{Subsystem}-{Functional Module}".

[0213] Intent Type Partitioning: Defines the rules for determining four intent types:

[0214] Positive process validation: Describes the scenario where the system produces the expected result after a user completes an operation;

[0215] Boundary condition verification: includes boundary terms such as "maximum / minimum / no more than / no less than / maximum / minimum";

[0216] Exception handling verification: Includes exception keywords such as "failure / error / timeout / interruption / exception / rollback";

[0217] Performance benchmarking: includes specific performance metrics such as latency, throughput, and availability.

[0218] Priority Determination Partition: Define the rules for determining the priority of test intents (P0 / P1 / P2):

[0219] P0 (Safety Control Category): This category includes functions that directly affect the safe operation of the power grid, such as remote signaling and telemetry, dispatch control, and protection actions. If such functions fail in the information technology innovation environment, it may lead to power grid accidents. Typically, 100% coverage is required under all target environment combinations.

[0220] P1 (Daily Business Functions): Functions involving daily business operations such as equipment ledgers, work order dispatch, and defect reporting need to be covered in the mainstream target environment combination.

[0221] P2 (Auxiliary Configuration): Involves auxiliary functions such as report statistics and parameter settings, and should be covered appropriately according to available resources;

[0222] Output format partitioning: Define the output format, such as requiring all mandatory fields (functional module label, intent type, functional summary, verification target description, constraints, priority), and specify field value type constraints to ensure that downstream modules can directly parse and consume the output.

[0223] Figure 7 This is a schematic diagram illustrating the specific implementation process of the test case intelligent synthesis module provided in the embodiments of this disclosure.

[0224] like Figure 7 As shown, the processing steps of the test case intelligent synthesis module include:

[0225] 701. Construct prompt words based on the target test intent and target environment labels.

[0226] 702. Input the prompt words into the LLM to drive the LLM to output synthesized test cases.

[0227] Specifically, the LLM can be a local LLM to ensure data security.

[0228] 703. Perform the first-level verification on the synthetic test cases, which can specifically be field integrity verification.

[0229] 704. Determine if the first-level validation has passed. If yes, execute 705. Otherwise, if the maximum number of retries is not exceeded, retry, i.e., reconstruct the prompt words and drive the LLM to re-output the synthesized test cases.

[0230] 705. Perform a second layer of verification on the synthetic test cases, which can be a perceptual consistency verification.

[0231] 706. Determine whether the second-level validation has passed. If yes, execute 707. Otherwise, if the maximum number of retries is not exceeded, retry, i.e., reconstruct the prompt words and drive the LLM to re-output the synthesized test cases.

[0232] If either the first-level or second-level check fails, and a retry is performed, the current retry count is increased by 1. This allows the two check layers to share the preset maximum number of retries, reducing resource overhead.

[0233] 707. Use the synthetic test cases that have passed quality verification (first-level verification and second-level verification) as the initial test cases, and inject coverage dimension labels into them, such as OS name, CPU name and functional module labels.

[0234] Specifically, the aforementioned prompt words can be called synthetic prompt words. Synthetic prompt words adopt a segmented injection structure, which organizes multi-source context information according to functional partitions. By using clear partition markers to isolate information from different sources, information confusion in long contexts can be avoided, thus preventing it from affecting the generation quality of LLM.

[0235] The order of the prompt words and the content of each section are as follows:

[0236] Role Partitioning: LLM is defined as "a power system testing expert proficient in domestic migration testing, capable of generating targeted compatibility test cases based on knowledge of differences between domestic OS / chips".

[0237] Intent Partition: Injects complete structure data of the current structured test intent, including functional module labels, verification objectives, constraints, intent type, and intent priority.

[0238] Environment partitioning: Inject the target domestic IT innovation environment specification, in the format {OS name:X1, CPU name:Y1, CPU architecture:ARM}, which includes the display name and technical identifier used for natural language references in the prompt words.

[0239] Knowledge partitioning: Inject Top-K discrepancy knowledge entries retrieved from the knowledge base. Each entry is presented in a structured text format, including the discrepancy category, triggering conditions, abnormal behavior, and impact on the power scenario. The corresponding discrepancy knowledge identifier is marked during injection to generate use case references.

[0240] Failure record partitioning (optional): If there are historical failure records related to the current functional module in the knowledge base, inject a failure record summary (failure phenomenon description + root cause category), requiring LLM to focus on covering the reproduction path of historical failure scenarios when generating test steps.

[0241] Output format partitioning: Specifies the output format, including all required fields and their type constraints, and can include a small sample example (such as a standard test case for a power scenario with domestic IT innovation compatibility, and a counterexample of insufficient domestic IT innovation awareness), to guide LLM to generate test cases that meet the domestic IT innovation awareness quality standards.

[0242] Quality verification is divided into two layers:

[0243] First layer, field integrity validation: Verify that the following required fields all exist and, unless otherwise specified, must be non-empty:

[0244] Use case number; Use case title; Prerequisites; Test steps; Expected results; Validation dimension label, containing at least one element; List of covered difference knowledge item IDs, which can be empty.

[0245] If any of the above fields is missing or does not meet the constraints, the first-level validation is deemed to have failed. Constraints can also be set for each of the above fields, such as the constraint that the test case number is a string, and the constraint that the test case title includes a string and limits the maximum number of characters.

[0246] The second layer is the consistency verification of the information technology innovation perception: the quality of the information technology innovation perception of the verification test cases is checked and verified through keywords and rules. The verification rules are as follows:

[0247] Rule 1 (Prerequisite Environment Specification): Check if the prerequisite field contains the keywords "target OS name" (e.g., X1) and "target CPU name" (e.g., Y1) to confirm that the prerequisite explicitly specifies the target domestic IT environment. X1 and Y1 are respectively a specified domestic OS name and a domestic CPU name.

[0248] Rule 2 (Test Step Relationship with Information Technology Innovation): For each input difference knowledge item, check whether there is at least one step in the array of test steps that contains an operation description related to the key trigger words of that difference knowledge (matching is done through a preset keyword mapping table, such as when the difference category is "memory alignment", the step should contain related words such as "alignment", "bus signal", "structure" etc.).

[0249] Rule 3 (Expected Result Compatibility Standard): Check whether the expected result field contains judgment statements related to the compatibility of domestic IT innovation, rather than only containing functional correctness standards. The detection method is as follows: if the expected result does not contain any negative statements related to the differences between domestic IT innovation (such as "no garbled characters", "no abnormalities", "normal compatibility") or quantitative verification standards unique to domestic IT innovation, it is judged as containing only functional correctness standards, and the verification fails.

[0250] If any rule fails, the specific description of the failed rule is injected as an append instruction prompt (such as "Please explicitly specify the X1 and Y1 runtime environments in the prerequisites"), triggering a regeneration.

[0251] Figure 8 This is a schematic diagram illustrating the specific implementation process of the coverage adaptive completion module provided in the embodiments of this disclosure.

[0252] like Figure 8 As shown, the processing steps of the coverage adaptive completion module include:

[0253] 801. Construct a coverage matrix based on the dimension labels of the initial test cases.

[0254] 802. Calculate the overall coverage and P0 functional coverage.

[0255] 803. Determine if the P0 function coverage has reached the threshold. If yes, proceed to 804; otherwise, proceed to 807.

[0256] 804. Determine if the overall coverage rate has reached the threshold. If yes, proceed to step 805; otherwise, proceed to step 808.

[0257] 805. Use the initial test cases as target test cases and generate a coverage report.

[0258] 806. Output target test cases and coverage report.

[0259] 807. Identify coverage gaps in the P0 function and prioritize their handling.

[0260] 808. Identify coverage gaps in P1 and P2 functions and process them in order of priority.

[0261] 809. Batch construct supplementary synthesis requests.

[0262] 810. Call the test case intelligent synthesis module to generate complete test cases.

[0263] 811. Merge the initial test cases and the completed test cases, and re-execute 802 and its subsequent steps.

[0264] Specifically, the coverage matrix can be a three-dimensional coverage matrix, which is a coverage matrix constructed from the target OS, the target CPU, and the target functional modules.

[0265] For a given cell in the 3D coverage matrix, if the cell's dimension label exists in the initial test cases, the corresponding value for that cell is non-zero; otherwise, it is zero. The overall coverage is the number of non-zero cells divided by the total number of cells.

[0266] P0 function coverage is the number of non-zero units corresponding to P0 function / the total number of units corresponding to P0 function. In other words, it requires that all P0 function modules have at least one use case coverage in all target environment combinations.

[0267] The target thresholds for different priorities can be the same or different. For example, the target threshold for function P0 can be 100% by default, while the target thresholds for P1 and P2 can be specified by the user.

[0268] For units with zero elements, fill in the gaps as covering voids.

[0269] The test coverage can be completed in priority order (P0 > P1 > P2) to ensure that critical business functions receive supplemented test cases first. For tests with the same priority, the number of difference knowledge entries for that OS version in the knowledge base is used as the indicator, with OS versions with more entries (usually the more mainstream versions) being processed first. In this way, prioritizing test coverage completion on such OS versions can achieve the highest engineering practical value within a limited number of completion rounds. When both priority and number of entries are the same, the CPU model with the fewest covered test cases can be prioritized based on the coverage frequency of the target CPU model in the current test case set, ensuring balanced coverage across CPU models.

[0270] For multiple coverage gaps of the same functional module under different environment combinations, the test case intelligent synthesis module supports batch parallel processing, merging the supplementary synthesis requests corresponding to these gaps into a single batch call, and generating supplementary test cases for multiple target environments simultaneously in a single LLM inference call (by simultaneously injecting multiple target environment specifications in the prompt words, requiring the output of test case sets grouped by environment), reducing the cumulative latency introduced by serial calls.

[0271] After each domestic IT migration test is completed, test engineers annotate newly discovered failure records with root causes. These annotated failure records are automatically added to the historical failure record partition via the knowledge base writing interface, and a reference link is established with the corresponding difference knowledge entry (if the root cause already has a corresponding entry) or a new entry is created (if it is a newly discovered difference). After the addition is complete, the knowledge base updates the embedding vectors of the affected entries to ensure that the retrieval results reflect the latest knowledge accumulation during the next synthesis.

[0272] For example, suppose the target matrix consists of 2 OS versions × 2 CPU models × 6 functional modules = 24 matrix elements, including 4 P0 functional modules and 2 P1 functional modules. After initial synthesis, all 6 functional modules of {X1+Y1} are covered (12 elements are covered), and 4 P0 functional modules of {X2+Y2} are covered (8 elements are covered), but 2 P1 functional modules are missing under {X2+Y2}. In the first round of completion: P0 coverage = 1.0, no gaps; P1 functional modules have coverage gaps, so the 4 P1 gaps (2 modules × 2 OS versions × 1 CPU model) are processed first. After batch parallel synthesis, the total coverage = 24 / 24 = 100%, and the completion terminates.

[0273] By combining the overall architecture described above with the specific implementation of each module, the following technical effects can be achieved:

[0274] Driven by a combination of LLM and a knowledge base on the characteristics of the domestic IT innovation environment, test cases can be automatically synthesized. Compared to manually building test cases, this significantly improves generation efficiency and shortens the testing cycle. The knowledge base module can accumulate knowledge of the unique boundary conditions of domestic OSs and chips (CPUs) and historical failure records, ensuring that generated test cases cover unique verification dimensions of domestic IT innovation, such as ARM memory alignment, file system behavior differences, Chinese character set encoding, and compatibility with domestic database drivers, avoiding systematic omissions caused by manual writing. Through a power business test intent parsing mechanism, generated test cases simultaneously verify the correctness of business logic (such as the latency and success rate of substation remote signaling status transmission) and compatibility with the domestic IT innovation environment (such as data frame alignment processing under a specific domestic OS and CPU), preventing test cases from covering only one dimension while omitting risks in another. Through an adaptive coverage completion mechanism, the system automatically analyzes the three-dimensional coverage matrix of OS version × CPU model × business function module, identifies and automatically fills coverage gaps, ensuring that all key business modules have corresponding test cases under the target domestic IT innovation environment combination, fundamentally eliminating systematic coverage omissions caused by the explosion of dimension combinations. Each newly discovered failure record after a domestic IT innovation migration test is automatically fed back to the historical failure record partition of the domestic IT innovation environment feature knowledge base. This drives the system to generate more targeted supplementary test cases during the next test case synthesis, forming a positive feedback loop that continuously improves test quality and boundary condition coverage depth as the project progresses. The system runs on a locally deployed large language model inference engine. Sensitive data such as power business system functional documents and test case libraries are processed within the enterprise intranet environment and are not transmitted to external services, meeting the data security compliance requirements of power companies and adapting to the private deployment scenarios of the power industry.

[0275] Figure 9 The diagram is based on the third embodiment of this disclosure. This embodiment provides a test case construction device based on a large model. The device 900 includes: a parsing module 901, an acquisition module 902, a synthesis module 903, and a completion module 904.

[0276] The parsing module 901 is used to parse the functional documents of the target business system using a large model to obtain the target test intent; the acquisition module 902 is used to acquire the target difference knowledge of the target environment to be deployed by the target business system; the synthesis module 903 is used to synthesize the target test intent and the target difference knowledge using the large model to obtain initial test cases; and the completion module 904 is used to obtain target test cases based on the initial test cases.

[0277] In this embodiment, initial test cases are generated based on a large model, which improves efficiency and accuracy compared to manually configuring test cases. The large model synthesizes initial test cases based on the target test intent and target difference knowledge, taking into account the unique knowledge of the target environment. This allows for the perception of the unique boundary conditions of the target environment, resulting in more adaptable test cases. Therefore, test cases that simultaneously cover the correctness of business logic and compatibility with the target environment can be obtained, improving test case performance.

[0278] In some embodiments, the parsing module 901 is further configured to:

[0279] Extract the function description text from the function document;

[0280] Based on the function description text and the preset domain enhancement prompt word strategy, construct prompt words;

[0281] The prompt words are input into the large model to drive the large model to output the initial test intent;

[0282] Based on the initial test intent, the target test intent is obtained.

[0283] In this embodiment, by extracting functional description text from functional documents and performing intent recognition based on functional description text, the effectiveness of the text processed by the large model can be improved, and efficiency and accuracy can be enhanced. Constructing prompt words based on domain enhancement prompt strategies can provide domain knowledge to the large model and improve the accuracy of test intent.

[0284] In some embodiments, the domain-enhanced prompting strategy includes at least one of the following:

[0285] The functional module labels in the target business system;

[0286] Intent type determination rules;

[0287] Rules for determining intent priority.

[0288] In this embodiment, by including functional module tags, intent type determination rules, and intent priority determination rules in the domain enhancement prompt strategy, the large model can be driven to generate test intents that are more in line with the target business system, thereby improving the accuracy of test intents.

[0289] In some embodiments, the initial test intent includes: verifying the target description;

[0290] Parsing module 901 is further used for:

[0291] For any two initial test intentions, calculate the semantic similarity between the two initial test intentions based on the corresponding verification target descriptions;

[0292] In response to the semantic similarity being greater than a preset threshold, any two initial test intentions are merged to obtain the target test intention.

[0293] In this embodiment, when the semantic similarity of the verification target descriptions of any two initial test intentions is greater than a preset threshold, they are merged. This can merge the same or similar test intentions, avoid intention duplication, and improve the effectiveness of the intentions.

[0294] In some embodiments, the two initial test intents include: a first intent and a second intent, wherein the length of the verification target description of the first intent is greater than the length of the verification target description of the second intent;

[0295] The initial test intent also includes: constraints and intent priority;

[0296] Parsing module 901 is further used for:

[0297] In response to the semantic similarity being greater than a preset threshold, the first intent is taken as the target test intent, the constraints of the second intent are merged into the constraints of the first intent, and the intent priority of the target test intent is set to the highest priority among the intent priorities corresponding to the first intent and the second intent.

[0298] In this embodiment, by using the above-described merging method, we can obtain a target test intent that is more clearly described in terms of the verification target, more comprehensive in terms of constraints, and more precise in terms of priority.

[0299] In some embodiments, the acquisition module 902 is further configured to:

[0300] Based on the environmental tags of the target environment, existing differential knowledge in the pre-established knowledge base is filtered to obtain candidate differential knowledge corresponding to the target environment;

[0301] Based on the vector similarity between the target testing intent and the candidate knowledge, the target difference knowledge is obtained from the candidate knowledge.

[0302] In this embodiment, filtering is first performed based on environmental labels, and then selection is performed based on vector similarity. This two-stage retrieval method can obtain target difference knowledge, thereby balancing accuracy and semantic relevance.

[0303] In some embodiments, the synthesis module 903 is further configured to:

[0304] Based on the target testing intent and the target difference knowledge, prompt words are constructed;

[0305] The prompt words are input into the large model to drive the large model to output synthetic test cases;

[0306] The quality of the synthesized test cases is verified.

[0307] In response to the synthetic test case passing quality verification, the synthetic test case is used as the initial test case.

[0308] In this embodiment, by performing quality verification on the synthetic test cases and using the synthetic test cases that have passed the quality verification as the initial test cases, the quality of the initial test cases can be improved.

[0309] In some embodiments, the synthesis module 903 is further configured to:

[0310] Perform field integrity verification on the synthesized test cases;

[0311] In response to the synthetic test case passing the field integrity verification, a perceptual consistency verification is performed on the synthetic test case;

[0312] In response to the synthetic test case passing the perceived consistency verification, it is determined that the synthetic test case has passed the quality verification.

[0313] In this embodiment, by using field integrity verification and perceived consistency verification, the comprehensiveness of quality verification can be improved, thereby improving the quality of the initial test cases.

[0314] In some embodiments, the completion module 904 is further used for:

[0315] Based on the initial test cases, construct a coverage matrix;

[0316] In response to the coverage matrix meeting the preset coverage requirements, the initial test case is used as the target test case.

[0317] In this embodiment, when the coverage matrix of the initial test cases meets the coverage requirements, it is used as the target test case, which can improve the coverage of the target test cases.

[0318] In some embodiments, the completion module 904 is also used for:

[0319] In response to the coverage matrix not meeting the preset coverage requirements, complete test cases are obtained;

[0320] The initial test case and the completed test case are merged to obtain the target test case.

[0321] In this embodiment, when the coverage matrix does not meet the coverage requirements, supplementary test cases are obtained, and the initial test cases and supplementary test cases are merged to obtain target test cases, which can improve the coverage of target test cases.

[0322] In some embodiments, the completion module 904 is further used for:

[0323] Extract multiple dimension labels from the initial test cases, including: target OS label, target CPU label, and functional module label;

[0324] The coverage matrix is ​​constructed based on the multiple dimensional labels.

[0325] In this embodiment, a coverage matrix is ​​constructed based on multiple dimension labels in the initial test cases. These multiple dimension labels include target OS labels, target CPU labels, and functional module labels, which can achieve test case coverage on OS, CPU, and functional modules, reducing the problem of test case omissions.

[0326] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.

[0327] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.

[0328] It is understandable that, unless otherwise specified, the order of steps in the process indicates that the temporal relationship between these steps is not limited.

[0329] The technical solutions disclosed herein involve the collection, storage, use, processing, transmission, provision, and disclosure of various types of information, such as user personal information, in accordance with relevant laws and regulations and do not violate public order and good morals.

[0330] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0331] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0332] like Figure 10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0333] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0334] The computing unit 1001 can be various general-purpose and / or proprietary processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various proprietary artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the large-model-based test case construction method. For example, in some embodiments, the large-model-based test case construction method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the large-model-based test case construction method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured, by any other suitable means (e.g., by means of firmware), to execute a test case construction method based on a large model.

[0335] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), proprietary integrated circuits (ASICs), proprietary standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a proprietary or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0336] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable task processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0337] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0338] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0339] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0340] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0341] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0342] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A test case construction method based on a large model, comprising: A large model is used to analyze the functional documents of the target business system in order to obtain the target testing intent. Obtain the target difference knowledge of the target environment to be deployed for the target business system; Using the large model, the target testing intent and the target difference knowledge are synthesized to obtain initial test cases; Based on the initial test cases, the target test cases are obtained.

2. The method according to claim 1, wherein, The process employs a large model to parse the functional documents of the target business system to obtain the target testing intent, including: Extract the function description text from the function document; Based on the function description text and the preset domain enhancement prompt word strategy, construct prompt words; The prompt words are input into the large model to drive the large model to output the initial test intent; Based on the initial test intent, the target test intent is obtained.

3. The method according to claim 2, wherein, The domain-enhanced prompting strategy includes at least one of the following: The functional module labels in the target business system; Intent type determination rules; Rules for determining intent priority.

4. The method according to claim 2, wherein, The initial test intent includes: verifying the target description; The step of obtaining the target test intent based on the initial test intent includes: For any two initial test intentions, calculate the semantic similarity between the two initial test intentions based on the corresponding verification target descriptions; In response to the semantic similarity being greater than a preset threshold, any two initial test intentions are merged to obtain the target test intention.

5. The method according to claim 4, wherein, The two initial test intentions include: a first intention and a second intention, wherein the length of the verification target description of the first intention is greater than the length of the verification target description of the second intention; The initial test intent also includes: constraints and intent priority; The step of merging any two initial test intentions to obtain the target test intention in response to the semantic similarity being greater than a preset threshold includes: In response to the semantic similarity being greater than a preset threshold, the first intent is taken as the target test intent, the constraints of the second intent are merged into the constraints of the first intent, and the intent priority of the target test intent is set to the highest priority among the intent priorities corresponding to the first intent and the second intent.

6. The method according to claim 1, wherein, The acquisition of target difference knowledge of the target environment to be deployed for the target business system includes: Based on the environmental tags of the target environment, existing differential knowledge in the pre-established knowledge base is filtered to obtain candidate differential knowledge corresponding to the target environment; Based on the vector similarity between the target testing intent and the candidate knowledge, the target difference knowledge is obtained from the candidate knowledge.

7. The method according to claim 1, wherein, The process of using the large model to synthesize the target testing intent and the target difference knowledge to obtain initial test cases includes: Based on the target testing intent and the target difference knowledge, prompt words are constructed; The prompt words are input into the large model to drive the large model to output synthetic test cases; The quality of the synthesized test cases is verified. In response to the synthetic test case passing quality verification, the synthetic test case is used as the initial test case.

8. The method according to claim 7, wherein, The quality verification of the synthesized test cases includes: Perform field integrity verification on the synthesized test cases; In response to the synthetic test case passing the field integrity verification, a perceptual consistency verification is performed on the synthetic test case; In response to the synthetic test case passing the perceived consistency verification, it is determined that the synthetic test case has passed the quality verification.

9. The method according to any one of claims 1-8, wherein, The step of obtaining the target test case based on the initial test case includes: Based on the initial test cases, construct a coverage matrix; In response to the coverage matrix meeting the preset coverage requirements, the initial test case is used as the target test case.

10. The method of claim 9, further comprising: In response to the coverage matrix not meeting the preset coverage requirements, complete test cases are obtained; The initial test case and the completed test case are merged to obtain the target test case.

11. The method according to claim 9, wherein, The construction of the coverage matrix based on the initial test cases includes: Extract multiple dimension labels from the initial test cases, including: target OS label, target CPU label, and functional module label; The coverage matrix is ​​constructed based on the multiple dimensional labels.

12. A test case construction device based on a large model, comprising: The parsing module is used to parse the functional documents of the target business system using a large model in order to obtain the target testing intent. The acquisition module is used to acquire target difference knowledge of the target environment to be deployed for the target business system; The synthesis module is used to synthesize the target testing intent and the target difference knowledge using the large model to obtain initial test cases; The completion module is used to obtain the target test case based on the initial test case.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images