A system for automated test scenarios based on requirements documents using generative AI

Abstract

A requirements document-based test scenario automation system using generative AI is disclosed. The test scenario automation system according to an embodiment of the present invention comprises: a requirements input interface that receives a requirements document, converts it into text, and automatically collects the requirements document by linking with an external system; a natural language interpretation processor that extracts the actor, action, condition, and expected outcome of a sentence using an LLM-based language model; a scenario generation processor that automatically generates a Given-When-Then structured scenario based on the extracted information; a quality evaluation processor that evaluates the coverage, logical consistency, and clarity of the automatically generated scenario to calculate a quality score, and transmits scenarios below a preset standard to an AI feedback processor; an AI feedback processor that performs retraining of the LLM based on the quality evaluation results and corrects scenarios below the preset standard quality to be above the preset standard; and a result review output interface that visualizes the scenario generation results, quality score, and requirements mapping results in a dashboard format, outputs them to an administrator, and accepts the administrator's review and approval.

Description

Technology Field

[0001] The present invention relates to a test scenario automation system, and more specifically, to a test scenario automation system that automatically derives test scenarios from requirements documents using Generative Artificial Intelligence in the field of software quality management and test automation technology. Background Technology

[0002] In software development projects, test scenario design is a critical step determining quality, and it depends heavily on how accurately and consistently requirements documents are interpreted. In conventional software testing environments, it is common practice for testers to manually create test scenarios by directly reading and understanding documents such as requirements specifications, planning documents, and functional definitions. This approach leads to a problem where requirements analysis capabilities depend heavily on individual experience and proficiency, resulting in different test scenarios being derived depending on the author, even for the same requirements.

[0003] In particular, since requirements documents are mostly written in natural language, it is difficult to clearly interpret conditions, exceptions, and the scope of actors implicitly contained within the sentences. Conventional technologies lack clear standards for structuring these natural language requirements, leading to persistent problems where important conditions or results are omitted or oversimplified in test scenarios. Consequently, there are frequent instances where actual implemented software functions fail to meet the requirements, yet this is not detected during the testing phase.

[0004] Furthermore, the existing process of creating test scenarios is repetitive and labor-intensive. When requirements change or are added, existing scenarios must be completely reviewed and modified; however, because the impact analysis of these changes is not systematically conducted, problems arise such as unnecessary duplicate testing or, conversely, the omission of essential tests. Conventional technologies lack the means to automatically manage the linkage between requirement changes and test scenarios.

[0005] As project scale increases, the number of requirements and test scenarios grows exponentially; however, conventional technologies make it difficult to quantitatively evaluate how sufficiently test scenarios cover the requirements. Coverage evaluation is primarily performed based on code during the development phase, and structures for systematically calculating test coverage from a requirements perspective are lacking. Consequently, there is a problem in objectively judging the quality of test design.

[0006] Furthermore, conventional test scenario quality evaluation largely relies on the subjective judgment of reviewers. Since assessments regarding the absence of logical contradictions, the clarity of the scenario, and the feasibility of actual test execution are made without clear criteria, quality variations occur. Such subjective evaluation acts as a factor that undermines consistency in quality management in large organizations or environments where multiple projects are executed simultaneously.

[0007] With the recent advancements in natural language processing and artificial intelligence technologies, attempts to utilize AI for requirements analysis are emerging. However, AI-based approaches based on conventional technologies often remain limited to sentence summarization or keyword extraction, and have limitations in converting actors, conditions, actions, and results within requirements into a structure suitable for test scenarios. In particular, sufficient functionality is not provided for automatically converting into the Given-When-Then format, which is the standard structure for behavior-based test scenarios.

[0008] Furthermore, existing AI-based systems lack structures for quality verification and improvement of generated results. Even when test scenarios are automatically generated, systematic follow-up verification is not conducted to determine whether the scenarios are logically valid, sufficiently reflect requirements, or clear from a test execution perspective. This results in inefficiency, as the automatically generated results must be completely revised by humans.

[0009] Conventional technology lacks a system for quantifying and managing the quality of generated test scenarios. It fails to evaluate quality elements such as coverage, logical consistency, and clarity using quantitative indicators, and the structure for incorporating evaluation results into scenario improvement or learning processes is also limited. Consequently, it has the limitation of being unable to implement a learning structure where quality improves with repeated system use.

[0010] Furthermore, the difficulty in intuitively verifying the mapping relationship between test scenario results and requirements is also pointed out as a problem. In conventional technology, test results are managed in a scattered manner across individual documents or tools, making it difficult for administrators to comprehensively grasp the overall test quality status. Management functions through visualization-based dashboards or approval procedures are also not sufficiently provided.

[0011] As such, conventional test scenario creation and management technologies harbor complex problems, including manual operation, reliance on subjective judgment, inefficiency in responding to requirements changes, and the absence of quality evaluation and learning structures. The present invention aims to fundamentally resolve these issues of conventional technology through generative AI-based natural language interpretation, automatic generation of structured scenarios, quantitative quality evaluation, and a feedback learning structure. Prior art literature

[0012] Korean Patent Publication No. 10-2352162 (Registration Date: January 12, 2022) The problem to be solved

[0013] This invention aims to resolve issues such as interpretation subjectivity, omissions and duplication, delays in responding to changes, and quality deviations that arise from humans manually interpreting requirements documents and creating test scenarios. Furthermore, it aims to overcome the limitations of conventional technology, which fails to structurally derive actors, conditions, actions, and results from requirements written in natural language, and to provide an automated system capable of automatically updating test scenarios upon requirements changes and generating and managing consistent scenarios according to quantitative quality standards. means of solving the problem

[0014] A test scenario automation system according to one aspect of the present invention for achieving the above purpose comprises: a requirements input interface that receives requirements documents (PDF, DOCX, PPT, Markdown, etc.) as input, converts them into text, and automatically collects requirements documents by linking with a plurality of external systems including Jira and Confluence; a natural language interpretation processor that extracts actors, actions, conditions, and expected outcomes of sentences using an LLM-based language model; a scenario generation processor that automatically generates Given-When-Then structured scenarios based on the extracted information; a quality evaluation processor that evaluates the coverage, logical consistency, and clarity of the automatically generated scenarios to calculate a quality score, and transmits scenarios below a preset standard to an AI feedback processor; and an AI feedback processor that performs retraining of the LLM based on the quality evaluation results and corrects scenarios below the preset standard quality to be above the preset standard. The configuration may include a result review output interface that visualizes the scenario generation results, quality scores, and requirements mapping results in a dashboard format and outputs them to the manager, and accepts the manager's review and approval.

[0015] In one embodiment of the present invention, the requirements input interface may be configured to include: a document input module that receives requirements documents (PDF, DOCX, PPT, Markdown, etc.), organizes them according to a preset format, and transmits them to a text conversion module; an external integration module that links with a plurality of external systems including Jira and Confluence to collect requirements documents, organizes them according to a preset format, and transmits them to a text conversion module; and a text conversion module that converts data transmitted from the document input module and the external integration module into text data identical to the data format to be input into a natural language interpretation processor.

[0016] In one embodiment of the present invention, the natural language interpretation processor may comprise: a language model construction module that constructs an LLM-based language model, is updated in real-time according to retraining performed by an AI feedback processor, and is updated again according to the requirements of an administrator; a sentence interpretation extraction module that analyzes text data transmitted from a requirements input interface using the latest language model updated by the language model construction module and extracts an actor, action, condition, and expected outcome from the text data; and an extraction result sorting module that refines and converts the data extracted from the sentence interpretation extraction module into a data format identical to the data format to be input to a scenario generation processor.

[0017] In one embodiment of the present invention, the scenario generation processor may be configured to include: a first structure generation module that generates a structure corresponding to the Given part of a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor; a second structure generation module that generates a structure corresponding to the When part of a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor; and a third structure generation module that generates a structure corresponding to the Then part of a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor.

[0018] In one embodiment of the present invention, the quality evaluation processor may be configured to include: a coverage evaluation module that quantitatively measures, according to a preset standard, how much source code a test case has executed and how much the requirements have been satisfied in a software test; a logic evaluation module that quantitatively measures, according to a preset standard, whether a number of components, functions, and data of the software are logically consistent without contradiction; and a clarity evaluation module that quantitatively measures, according to a preset standard, how easy and obvious documentation, code, and test scripts are to understand.

[0019] The present invention may also provide a test scenario automation method using the test scenario automation system. A test scenario automation method according to one aspect of the present invention may be configured to include: a requirements document input step; a natural language analysis step; a scenario automatic generation step; a quality evaluation step; a feedback step; and a result output approval step. Effects of the invention

[0020] According to the present invention, requirements documents can be automatically analyzed through generative AI-based natural language understanding and automatically converted into behavior-based test scenarios with a Given-When-Then structure. This eliminates individual interpretation variances that occurred during the test scenario design process and effectively prevents issues of omitted or duplicate requirements. Furthermore, the quality of test scenarios can be objectively managed through quantitative quality evaluations regarding coverage, logical consistency, and clarity, and scenario quality continuously improves with repeated use through an AI feedback and relearning structure. Moreover, real-time response to changes in requirements is possible, and the efficiency of test design and management is significantly enhanced by visualizing scenarios and quality metrics on a dashboard. Brief explanation of the drawing

[0021] FIG. 1 is a block diagram showing a test scenario automation system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a test scenario automation method according to one embodiment of the present invention. Specific details for implementing the invention

[0022] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept consistent with the technical spirit of the present invention.

[0023] Throughout this specification, when it is stated that one component is located "on" another component, this includes not only cases where one component is in contact with another component, but also cases where another component exists between the two components. Throughout this specification, when it is stated that a part "includes" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0024] FIG. 1 shows a block diagram illustrating a test scenario automation system according to one embodiment of the present invention, and FIG. 2 shows a flowchart illustrating a test scenario automation method according to one embodiment of the present invention.

[0025] Referring to these drawings, the test scenario automation system (100) according to the present embodiment is equipped with a requirements input interface (110) that performs a specific role, a natural language interpretation processor (120), a scenario generation processor (130), a quality evaluation processor (140), an AI feedback processor (150), and a result review output interface (160), thereby providing an automation system that automatically analyzes requirements documents through LLM (Large Language Model)-based natural language understanding and converts the results into behavior-based test scenarios.

[0026] Hereinafter, with reference to FIGS. 1 and FIGS. 2, each component constituting the test scenario automation system (100) according to the present embodiment will be described in detail.

[0027] Detailed explanation regarding the combination of hardware resources and information processing means of each component

[0028] The requirements input interface (110), natural language interpretation processor (120), scenario generation processor (130), quality evaluation processor (140), AI feedback processor (150), and result review output interface (160) may be implemented as software function blocks executed on one or more computer devices, wherein the computer device performs information processing in combination with specific hardware resources such as a processor (CPU), memory (RAM), storage device (SSD / HDD), input / output interface (I / O), and network interface (NIC). The requirements input interface (110) receives a user's upload request or an external system integration request through a network interface, records the received file or external API response data in a storage device, and loads file metadata and collection history into memory to transmit to a subsequent processing pipeline. In this process, the processor executes control logic such as document reception, file format determination, access permission verification, and transmission error handling, and the input / output interface provides an input path such as a file upload UI, an administrator console, or an API gateway. The natural language interpretation processor (120) receives requirement text loaded in a storage device or memory as input and enables the processor to perform preprocessing operations such as tokenization, sentence separation, entity candidate extraction, and condition, action, and result decomposition, and can perform inference calls using GPU computing resources or a network interface depending on whether LLM inference is performed on a local model or a remote model. That is, the natural language interpretation processor (120) is implemented in a structure that combines preprocessing and postprocessing software executed on a CPU with language model computing resources provided by a GPU or an external inference server, thereby performing requirement analysis in a manner reproducible in a normal server environment.

[0029] The scenario generation processor (130) and the quality evaluation processor (140) receive structured data (Actor, Action, Condition, Expected Outcome) produced by the natural language interpretation processor (120) as a data structure in memory (JSON, DataFrame, Object Model, etc.), and the processors produce a Given-When-Then scenario by executing generation logic such as scenario template mapping, branch generation, sentence combination, scenario ID assignment, and requirement-scenario mapping table generation. The scenario generation processor (130) can call a language model post-processing function as needed to perform sentence naturalization or term unification; in this case, when using a local LLM, sentence generation is performed by combining GPU memory (VRAM) and computational resources, and when using an external LLM, inference requests and responses are exchanged through a network interface. The quality evaluation processor (140) retrieves the generated scenario and the original requirement text from the storage device or maintains them in memory, and the processor calculates a score by performing coverage calculation (whether the scenario exists relative to the requirement items), logic check (Given-When-Then consistency rule check), and clarity analysis (whether sentence components are included, detection of ambiguous expressions). In cases where vector embeddings are used for coverage mapping or similarity calculation, the CPU triggers the embedding call, and the vector generation and search can be configured to be handled by the GPU or a separate vector DB server. Even in such cases, a typical developer can implement it using only public libraries and standard server configurations. The calculated score and the reason for the defect are maintained as an evaluation report object in memory or recorded in a database table in the storage device and transmitted to a subsequent feedback stage.

[0030] The AI ​​feedback processor (150) and the result review output interface (160) specifically execute a correction loop and a human-centered approval loop in combination with hardware resources based on the quality evaluation results. The AI ​​feedback processor (150) analyzes the evaluation report to classify defect types, applies correction rules such as inserting missing conditions, generating boundary value branches, and refining ambiguous expressions, and calls a language model if necessary to rewrite the corrected scenario sentences. In this process, the local LLM configuration uses GPU resources, and the remote LLM configuration calls inference using a network interface; the corrected results are immediately transferred from memory to the re-evaluation pipeline or stored in a storage device in a version-controlled format. The result review output interface (160) provides a web dashboard or management console UI through an input / output interface and a network interface, and the processor displays the scenario list, quality score, and requirements mapping results on the screen by querying, sorting, and filtering them. The administrator's approval and rejection inputs are received again through the input / output interface, and the processor records the approval status in the database, and the reason for rejection is transmitted as input data to the AI ​​feedback processor (150) and reflected in the re-correction loop. This entire structure can be implemented by combining standard server hardware (CPU, RAM, SSD, NIC), optional acceleration resources (GPU), and a general-purpose software stack (web server, database, document parser, LLM inference server), and is concretized to a level where a person with ordinary knowledge of the technical field can configure, implement, and reproduce it without special knowledge or excessive repetitive experiments.

[0031] Detailed description of the requirements input interface (110)

[0032] The requirements input interface (110) is located at the forefront of the test scenario automation system and performs the role of securing requirements documents existing in various forms as consistent analysis targets. In a software development environment, requirements are generally not managed in a single format but are dispersed across various documents such as planning documents, functional specifications, and meeting materials; the requirements input interface (110) is designed based on this unstructured and multi-dimensional input environment.

[0033] The requirements input interface (110) supports both offline document input and requirements collection based on online collaboration tools, thereby ensuring universality applicable without being dependent on a specific development process or organizational environment. This provides a structure that can be easily applied not only to new projects but also to existing development environments.

[0034] In addition, the requirements input interface (110) is not limited to a simple document collection function, but serves as a starting point for preprocessing to transmit the collected requirements in a form that can be processed by the natural language interpretation processor (120). In this process, it is designed to minimize analysis deviations that may occur due to format differences between documents.

[0035] Since the requirements data obtained through the requirements input interface (110) is used as reference data for subsequent scenario generation, quality evaluation, and AI learning stages, the accuracy and consistency at the input stage have a direct impact on the overall performance of the system. Therefore, the requirements input interface (110) is configured to prevent data loss or distortion of meaning.

[0036] Consequently, the requirements input interface (110) serves as a gateway for integrating various requirements sources into a single analysis flow, and is a key component that structurally improves the problem of requirements management tools and test tools being operated separately in conventional technology.

[0037] Detailed description of the document input module (111)

[0038] The document input module (111) is responsible for receiving and processing requirement documents provided directly by the user into the system. The document input module (111) is configured to support various file formats such as PDF, DOCX, PPT, and Markdown, thereby minimizing restrictions on document creation methods used in actual business operations.

[0039] The document input module (111) performs the role of organizing the document into a preset format by considering the structure and content characteristics of the document, going beyond the function of simply receiving a file. For example, it recognizes tables, lists, title structures, body paragraphs, etc., and removes or normalizes unnecessary format elements that are disadvantageous for natural language interpretation.

[0040] In this process, the document input module (111) is designed so as not to distort the meaning of the requirements and logically separates the document content to enable analysis at the sentence level. This lays the foundation for the natural language interpretation processor (120) to more accurately extract the actor, condition, action, and result thereafter.

[0041] The document input module (111) is designed to take into account not only one-time input but also repetitive document update situations. When a revised version of the same document is input, it is also possible to apply an extension that can efficiently manage whether requirements have changed by comparing it with the previous input history.

[0042] Therefore, the document input module (111) has the effect of significantly reducing the time consumption and the possibility of errors that occurred in the pre-test scenario design stage by automating the inefficient preprocessing work that a person had to do in the conventional technology, where a person had to read and reorganize the document.

[0043] Detailed description of the external linkage module (112)

[0044] The external integration module (112) performs the function of automatically collecting requirements documents by linking with collaboration and requirements management systems such as Jira and Confluence. In modern software development environments, requirements are continuously changed and managed through collaboration tools, so the external integration module (112) is configured to directly reflect this practical environment.

[0045] The external integration module (112) collects requirement data periodically or on an event basis through the API or integration interface of each external system and organizes the collected data according to a preset format. Through this, the latest requirements can be automatically reflected without human intervention during the requirement input process.

[0046] This module is designed to be extensible to collect additional information such as requirements status, version information, author, and change history. This additional information can be utilized later for tracking requirements changes, determining scenario updates, and analyzing quality control.

[0047] The requirements collected through the external linkage module (112) are integrated into the same processing flow as the requirements entered through the document input module (111). This ensures consistency, allowing the natural language interpretation processor (120) to perform analysis based on the same criteria regardless of the source of the requirements.

[0048] Consequently, the external linkage module (112) is an important component that resolves the information disconnection problem that occurred when the requirements management tool and the test design tool were separated in the prior art, and realizes the core effects of the present invention, such as automatic response to requirements changes and updating of test scenarios.

[0049] Detailed description of the text conversion module (113)

[0050] The text conversion module (113) performs the role of converting requirement data transmitted from the document input module (111) and the external linkage module (112) into a text data form that the natural language interpretation processor (120) can directly analyze. Since requirement documents have very different expression methods depending on the file format, format structure, and writing tool, the text conversion module (113) is configured mainly with a function to normalize these heterogeneous inputs based on a single analysis standard.

[0051] The text conversion module (113) is configured to recognize the internal structure of each document format, such as PDF, DOCX, PPT, and Markdown, to extract text, and in the case of image-based documents, to obtain character data including the OCR processing results. At this time, it does not stop at simple character extraction, but also considers document structural elements such as titles, body text, tables, and lists to remove or rearrange elements that are disadvantageous for analysis.

[0052] Additionally, the text conversion module (113) performs a preprocessing function that improves the accuracy of natural language processing by separating the requirement sentence into paragraphs or sentences and cleaning up unnecessary symbols, redundant spaces, and formal phrases. This provides a foundation for making the distinction between the agent, condition, action, and result clearer in the subsequent natural language interpretation stage.

[0053] The text conversion module (113) ensures consistency of the text to be analyzed, thereby minimizing deviations in analysis results according to the input path of requirements. That is, by ensuring that documents directly uploaded by the user and requirements collected from external systems are converted into text data of the same format and processed, differences based on the data source are eliminated.

[0054] Consequently, the text conversion module (113) plays a key connecting role between the requirements input stage and the natural language interpretation stage, and structurally improves the problems of interpretation errors and analysis instability that occurred due to differences in document format in conventional technology.

[0055] A detailed description of the natural language interpretation processor (120).

[0056] The natural language interpretation processor (120) is a core processing unit that receives the normalized requirement text through the text conversion module (113) and structurally analyzes the semantic elements included in the requirement. The natural language interpretation processor (120) is configured to perform context-understanding-centered semantic interpretation rather than simple keyword analysis by utilizing an LLM-based language model.

[0057] The natural language interpretation processor (120) operates primarily by extracting key elements of a test scenario, such as actors, actions, conditions, and results, from a requirement sentence. This enables analysis that reflects even the preconditions or expected results implicitly included in the requirements, effectively reducing the problem of missing meaning that frequently occurred in conventional technology.

[0058] In addition, the natural language interpretation processor (120) is not limited to analyzing a single sentence, but performs analysis by considering the relationship between paragraphs or the contextual flow of consecutive requirements. Through this, complex requirements or exception conditions described across multiple sentences can also be integrated and interpreted as a single scenario unit.

[0059] The natural language interpretation processor (120) is designed to minimize analysis deviations based on the expression method of the requirements. Even if the same functional requirements are described in different expressions, language model-based inference is utilized so that semantically equivalent elements can be interpreted in the same structure.

[0060] The configuration of such a natural language interpretation processor (120) resolves the problem in conventional technology where requirements interpretation relies on human experience and intuition, and provides a basis for fundamentally improving the reliability and reproducibility of test scenario automation.

[0061] Detailed description of the language model construction module (121)

[0062] The language model construction module (121) performs the function of creating, maintaining, and managing an LLM-based language model used in a natural language interpretation processor (120). The language model construction module (121) is structured to be based on an initially trained language model, but is continuously updated by reflecting the results of test scenario creation and quality evaluation.

[0063] The language model construction module (121) is designed to reflect domain-specific requirements. For example, by reflecting expressions and requirements that are repeatedly used in specific domains, such as finance, e-commerce, and public systems, into the training data, it is possible to secure higher interpretation accuracy compared to a general language model.

[0064] Additionally, the language model building module (121) performs retraining based on the quality evaluation results transmitted from the AI ​​feedback processor (150). It analyzes the generation patterns of scenarios judged to be below quality standards and adjusts the parameters or inference paths of the language model so that the same errors are not repeated.

[0065] The language model building module (121) is configured to be extensible so that the model update cycle, learning scope, and types of applicable requirements can be controlled according to the administrator's settings. This allows for the implementation of a customized natural language interpretation environment suitable for project characteristics or organizational policies.

[0066] Consequently, the language model building module (121) enables a learning structure in which the accuracy of requirement interpretation and the quality of scenario generation improve as the system is used repeatedly, and is a key component that implements a continuous quality improvement mechanism that was not provided by conventional technology.

[0067] Detailed explanation of LLM-based language models

[0068] The LLM-based language model applied in this invention is trained to simultaneously acquire general-purpose language comprehension and software requirements interpretation capabilities in order to accurately understand natural language sentences included in requirements documents and convert them into test scenario structures. It is fundamentally based on a pre-training model utilizing large-scale public text data such as wiki documents, technical blogs, user manuals, and development guides, and subsequently performs fine-tuning using data specialized for the software development and quality management domains. Domain-specific training data may include requirements specifications (SRS), user stories, functional definitions, API specifications, acceptance criteria, existing test case documents, Given-When-Then (BDD) scenarios, bug reports, Jira issue descriptions, etc. For example, if a requirement sentence such as "If a user enters the password incorrectly 5 or more times, the account should be locked" is trained together with structural data such as Actor=User, Condition=5 or more password errors, Action=Login attempt, and Expected Outcome=Account locking, the model can automatically extract test scenario components from the requirement sentence. Furthermore, including data from test scenario repositories (TestRail, Zephyr, etc.) used in actual projects reflects the organization's unique terminology and expressions, thereby enhancing practical applicability.

[0069] The model structure utilizes a large language model based on the Transformer architecture and is composed of a core Self-Attention mechanism for context understanding and semantic inference. For implementation, it can be designed to leverage a decoder-centric structure similar to the GPT family to generate structured results (JSON-formatted Actor, Action, Condition, Expected Outcome) or Given-When-Then scenario statements when a requirement statement is input. For example, given the input statement "If a non-logged-in user accesses the order inquiry menu, they should be redirected to the login screen," the model outputs a structure where Actor = Non-logged-in user, Condition = Not logged in, Action = Access order inquiry menu, Expected Outcome = Redirect to login screen, or generates a form such as "If the user accesses the order inquiry menu while not logged in / they should be redirected to the login screen." To achieve this, supervised learning, which pairs requirement statements with structured ground truth data, and reinforcement learning-based fine-tuning (RLHF), utilizing the quality scores (logic, clarity, etc.) of the generated results, can be applied in parallel. This configuration can be implemented using open frameworks such as PyTorch-based Transformer implementations or the Hugging Face Transformers library, and is designed to be reproducible in a typical server environment (1-2 GPUs or a cloud environment).

[0070] Detailed description of the sentence interpretation extraction module (122)

[0071] The sentence parsing extraction module (122) uses the latest LLM-based language model updated by the language model building module (121) to perform the role of precisely decomposing text data transmitted from the requirements input interface (110) into semantic units. Since requirements documents are written in natural language, the sentence structure is inconsistent, and there are often cases where multiple conditions, actions, and results are mixed within a single sentence. The sentence parsing extraction module (122) is designed to analyze such unstructured sentences based on semantics rather than simple grammatical interpretation.

[0072] The sentence interpretation extraction module (122) identifies the actors, actions, conditions, and results that constitute the core elements of the test scenario in each requirement sentence. In this process, not only the explicitly revealed subjects and verbs, but also the user state, system state, and antecedent conditions implicitly assumed in the context are included as subjects of inference. Through this, hidden conditions or expected results that had to be interpreted by a human in conventional technology can be automatically derived.

[0073] Additionally, the sentence interpretation extraction module (122) is configured to interpret requirements described across complex sentences or consecutive paragraphs into a single logical flow. For example, if a specific condition is repeatedly referenced in multiple sentences, the condition is recognized as an independent element and structured so that it can be reused in the subsequent scenario generation stage. This contributes to preventing duplicate creation of test scenarios.

[0074] The sentence interpretation extraction module (122) comprehensively considers synonyms, similar expressions, and domain-specific terms to reduce interpretation errors caused by differences in expression methods. Even if requirements with the same meaning are written in different sentence structures, consistency in requirement interpretation is ensured by extracting them into the same or similar semantic structures.

[0075] Consequently, the sentence interpretation extraction module (122) is a component that substantially determines the accuracy of the automation of requirements document interpretation, and plays a key role in transforming the requirements analysis process, which relied on human experience and intuition in conventional technology, into a systematic and repeatable data extraction process.

[0076] Detailed explanation of the extraction result organization module (123)

[0077] The extraction result organization module (123) performs the role of converting the actor, action, condition, and result data derived from the sentence interpretation extraction module (122) into a structured data format that the scenario generation processor (130) can immediately utilize. Since the natural language interpretation result itself contains various expressions and inference results, instability may occur during the scenario generation process if it is not refined.

[0078] The extraction result sorting module (123) sorts the extracted semantic elements according to a pre-set data structure and verifies whether there are any missing elements. For example, if a condition or result is not clearly extracted, it marks the element as a target for supplementation or provides state information so that branching processing is possible during the scenario generation stage.

[0079] In addition, the extraction result organization module (123) integrates and manages elements that have the same or similar meanings. By organizing actors or system behaviors that appear repeatedly in various requirement sentences into common elements, consistent expressions and structures can be maintained when generating scenarios later. This contributes to reducing the problem of inconsistencies in expression between scenarios.

[0080] The extraction result organization module (123) maintains a mapping relationship between requirements and extraction results. Since each extraction result is managed in conjunction with information on the original requirement sentence or paragraph, it is possible to track which requirement a specific scenario was created based on during the scenario review process. This serves as important supporting data during the quality evaluation and manager review stages.

[0081] Consequently, the extraction result organization module (123) serves as an intermediate layer that converts natural language interpretation results into structured data capable of generating scenarios, and plays a role in ensuring the stability of the connection between requirements semantic analysis and automatic test scenario generation.

[0082] Detailed description of the scenario generation processor (130)

[0083] The scenario generation processor (130) receives structured requirement information through the natural language interpretation processor (120) and performs the function of automatically generating behavior-based test scenarios. The scenario generation processor (130) is designed based on the Given-When-Then structure widely used in the field of test automation, and is configured so that the generated result can be directly linked to actual test execution and automation tools.

[0084] The scenario generation processor (130) clearly defines the preconditions and initial state for test execution in the Given section. This is configured based on the conditions and actor information transmitted from the extraction result cleanup module (123) and is designed to clearly reflect preconditions such as whether the user is logged in, whether they possess specific authority, and whether the system is operating normally.

[0085] In the When section, test triggers are defined based on user actions or system events. The scenario generation processor (130) is configured to generate scenarios that combine not only single actions but also multiple actions by considering the order of action information. Through this, test scenarios similar to actual user flow can be automatically generated.

[0086] The "Then" section clearly describes the expected results to provide criteria for determining test success. This can be extended to include not only visible outcomes such as screen transitions, data output, and error message displays, but also internal state changes and data processing results. This enables precise test verification beyond simple functional checks.

[0087] As a result, the scenario generation processor (130) is a key component that significantly reduces test design time and costs and greatly improves the consistency and reproducibility of scenario quality by automatically generating test scenarios based on requirements, which were previously written manually by humans in conventional technology.

[0088] Detailed description of the scenario automatic generation means of the scenario generation processor (130)

[0089] A scenario generation processor (130) is a software-based processing configuration that receives Actor, Action, Condition, and Expected Outcome structure data transmitted from a natural language interpretation processor (120) as input and automatically generates a test scenario in the Given-When-Then format. The scenario generation processor (130) can be implemented by combining a rule-based template engine and a language model-based sentence generation function, and operates by mapping structured data to sentence patterns corresponding to each scenario component and then refining it into natural sentences. For example, when data such as Actor="Logged-in user", Condition="Cart amount is 50,000 won or more", Action="Click Order button", and Expected Outcome="Shipping fee of 0 won" is input, a Given-When-Then structure scenario is automatically generated through template mapping, stating "When the user is logged in and the cart amount is 50,000 won or more / when the Order button is clicked / shipping fee should be applied as 0 won". This template mapping logic can be easily implemented by a typical developer using a Python-based string template (Jinja2), rule-based sentence generation logic, or a JSON-to-sentence conversion script.

[0090] The scenario generation processor (130) can utilize a language model in the post-processing stage to improve the readability and expression consistency of the generated sentences. After a structured scenario draft is generated, the LLM inference function can be used to rewrite the sentences into natural test scenario expressions. For example, if the template result is generated in a mechanical form such as "Display order list when user clicks order inquiry menu while logged in," it is refined into a natural and clear sentence such as "When user clicks order inquiry menu while logged in, the order list should be displayed on the screen" through language model post-processing. This process can be implemented through a REST API-based language model service, an in-house inference server, or an inference pipeline based on Hugging Face Transformers, and can be implemented by including instructions such as "naturally rewrite into BDD scenario sentences" in the input prompt.

[0091] Additionally, the scenario generation processor (130) can be configured to automatically derive and generate multiple test scenarios from a single requirement. For example, if the requirement is defined as "non-logged-in users should be moved to the login page when accessing the order inquiry menu, and logged-in users should be able to view the order history screen," the natural language interpretation processor (120) transmits two result structures based on the login status condition, and the scenario generation processor (130) automatically generates two Given-When-Then scenarios based on this. The first scenario is generated as "If the user accesses the order inquiry menu while not logged in, they should be moved to the login page," and the second scenario is generated as "If the user accesses the order inquiry menu while logged in, the order history screen should be displayed." This branching scenario generation function can be implemented using iterative generation logic based on conditional branching rules (user status, authority level, boundary value conditions, etc.), and can be reproduced using only standard conditional branching processing and template application methods, allowing for implementation without excessive trial and error.

[0092] Detailed description of the first structure generation module (131)

[0093] The first structure generation module (131) is a component that exclusively generates the structure corresponding to the Given part within the scenario generation processor (130), and performs the role of clearly defining the initial state and preconditions that serve as the starting point of the test scenario. The first structure generation module (131) receives actor information and condition information transmitted from the extraction result sorting module (123) as input and constructs a natural language sentence describing what state the system and user must be in before the test is performed. At this time, rather than simply listing keywords such as "login state," it is designed to convert into a sentence structure that is easy to understand in the actual test context, such as "when the user has successfully logged in with a normal account." Through this, it provides a basis for the scenario reader to intuitively grasp the prior state.

[0094] The first structure generation module (131) takes into account cases where multiple preconditions are derived from a single requirement and reflects the priority and logical relationship between the conditions. For example, if a specific function is provided only to "users with administrator privileges" and simultaneously requires "a state of having agreed to the terms of service," the first structure generation module (131) appropriately combines the two conditions to construct a natural and logical Given clause such as "a state in which a user with administrator privileges has agreed to the terms of service." This reduces confusion in interpretation caused by omissions or unnatural connections between conditions.

[0095] In addition, the first structure generation module (131) also controls system-side conditions related to test environment settings. For example, cases where specific test data must exist in a database in advance, or where an external integration system must be in a state capable of normal response, are information that a person would otherwise note separately in a test document or write as an additional explanation. The first structure generation module (131) automatically reflects these system state conditions based on a separate template or rule, thereby enabling the preparatory requirements necessary for actual test execution to be explicitly included within the Given structure.

[0096] The first structure generation module (131) operates by taking into account that different expression practices exist depending on the domain. By reflecting precondition expressions frequently used in each domain, such as finance, public sector, commerce, and medical sectors, together with the language model and rules, domain-friendly Given sentences can be generated, such as "in the state where an account is opened," "in the state where a certificate is registered," and "in the state where patient information is registered." This leads to a practical advantage in that it allows test personnel to immediately utilize the scenario without modifying the domain terminology.

[0097] Consequently, the first structure generation module (131) plays a role in reducing the variation in precondition descriptions that were written differently by each person in conventional technology by systematically and clearly defining the premise of the test scenario, and significantly improving test reproducibility and comprehensibility. Through this module, a foundation is established that allows testing to be performed by assuming the same initial state regardless of when or by who reads the scenario for the same requirements.

[0098] Detailed description of the second structure generation module (132)

[0099] The second structure generation module (132) is a component that generates a structure corresponding to the "When" part of a Given-When-Then structure scenario, and defines the core operations of the test based on user actions or trigger events occurring within the system. Based on action information transmitted from the sentence parsing extraction module (122), the second structure generation module (132) converts actions performed by the actual user, such as operations, API calls, button clicks, menu selections, and data inputs, into natural language sentences. For example, if action information such as "clicking the payment history button" is transmitted, the second structure generation module (132) combines it into a form suitable for the "When" structure, such as "when the user clicks the payment history button."

[0100] The second structure generation module (132) is designed to express not only single actions but also sequential actions or conditional actions. In actual service environments, there are often cases where multiple steps of action are included in a single scenario, such as "log in, go to My Page, and press the notification settings button." In this case, the second structure generation module (132) generates a When sentence structure that is easy to understand from a test perspective by appropriately splitting or adjusting connecting words so that the sentence does not become excessively long while maintaining the order between actions. It is also possible to reconstruct multiple When structures into separate scenarios as needed.

[0101] Additionally, the second structure generation module (132) can represent not only user actions but also internal system events as triggers. For example, the time when a background batch job is executed, the time when a specific response arrives from an external system, or a task automatically executed by a scheduler occurs without direct user interaction but corresponds to an important "When" from a testing perspective. The second structure generation module (132) expresses these system events in forms such as "when a regular batch is executed" or "when an approval response is received from an external payment system," thereby naturally including test scenarios for automated processes.

[0102] The second structure generation module (132) is configured to convert even the same Action data into natural sentences suitable for the situation by considering domain-specific behavior expression patterns. For example, in the e-commerce domain, expressions such as "add products to the shopping cart" and "click the payment button" are mainly used, while in public administration services, expressions such as "submit a civil complaint application" and "complete an electronic signature" are used. The second structure generation module (132) learns these frequent patterns in advance or embeds them as rules to automatically generate When sentences that are close to the tone used in actual business operations.

[0103] Consequently, the second structure generation module (132) plays a role in standardizing behavior descriptions that test designers described in different ways based on experience in conventional technology by systematically defining "when to do what" which is the core of the test scenario, and greatly increasing the possibility of linkage with scenario automation tools. Through this module, the flow of operations of the scenario becomes clear, and the mapping work with automatic execution scripts is also simplified.

[0104] Detailed description of the third structure generation module (133)

[0105] The third structure generation module (133) is a component that generates a structure corresponding to the Then part in a Given-When-Then structure scenario, and performs the role of clearly describing the expected result after a user's action or a system event. Based on the expected outcome information transmitted from the sentence parsing extraction module (122) and the extraction result sorting module (123), the third structure generation module (133) generates a specific result sentence that can determine whether the test is successful. For example, when result information such as "display payment history screen" is given, the third structure generation module (133) converts it into a natural language sentence suitable for the Then structure, such as "the payment history screen should be displayed."

[0106] The third structure generation module (133) is designed to express not only simple screen displays or message outputs but also internal results such as data state changes, business rule application results, and whether logs are recorded. For example, sentences such as "points must be accumulated," "order status must be changed to 'payment completed'," and "error logs must be recorded" are important checkpoints that the test automation tool must verify. The third structure generation module (133) specifies these internal results at the level of natural language scenarios, thereby providing clear criteria for subsequent automation script design or verification logic design.

[0107] In addition, the third structure generation module (133) is configured to handle not only normal scenarios but also exception scenarios and error handling scenarios. For situations such as incorrect user input, network failures, or external system errors, results such as "an error message should be displayed," "a request should be rejected," or "a retry button should be enabled" are expected. By expressing these exception results as a separate Then structure, the third structure generation module (133) can automatically include error handling tests that were often omitted in conventional technology.

[0108] The third structure generation module (133) is designed to generate sentences that include quantitative or explicit criteria whenever possible to reduce ambiguity in the expression of results. For example, instead of abstract expressions such as "the list should be displayed normally," specific conditions such as "only payment history within the last 3 months should be displayed" or "only approved transactions should be included in the list" are reflected in the scenario. This reduces judgment deviations that may occur when a tester interprets the results and enables quality measurement to be done more objectively.

[0109] Consequently, the third structure generation module (133) clearly presents the "verification criteria" of the test scenario, thereby solving the problem in the prior art where it was difficult to consistently determine whether a test was passed due to ambiguous result definitions. Through this module, the situation and actions defined by Given and When clearly lead to Then, and the entire scenario acquires a complete structure, thereby ensuring a level of specificity that allows for immediate use in practical testing, even though it is an automatically generated scenario.

[0110] Detailed description of the quality evaluation processor (140)

[0111] The quality evaluation processor (140) is a core component that verifies the completeness of test scenarios automatically generated by the scenario generation processor (130) from various angles, and performs the role of quantitatively determining the usability and reliability of the scenarios. In conventional technology, the quality of test scenarios was evaluated based on the reviewer's experience or subjective judgment, but the quality evaluation processor (140) is designed to automatically calculate quality according to pre-set evaluation criteria and measurement indicators. Through this, consistency and objectivity in test scenario quality management are ensured.

[0112] The quality evaluation processor (140) operates by integrating multiple evaluation perspectives rather than a single standard. It independently evaluates whether the test scenario sufficiently reflects the requirements, whether it is logically structured without contradictions, and whether it is written in a form that is easy for the actual tester to understand, and then calculates a comprehensive score. This structure prevents the problem of a scenario being judged as good overall when only certain items are excellent.

[0113] In addition, the quality evaluation processor (140) is configured not only to output a simple score for the evaluation results but also to produce detailed results for each evaluation item. This allows for the clear identification of which factors degraded the scenario quality and is subsequently used as supporting data for the AI ​​feedback processor (150) to perform retraining or scenario correction. This structure enables a closed-loop quality management system in which automatic generation and improvement are repeated.

[0114] The quality evaluation processor (140) is configured to be expandable so that evaluation criteria can be flexibly set according to the quality policy of the project or organization. For example, the weight of coverage can be set high during the initial project phase, and the weight of logical completeness or clarity can be increased during the stabilization phase. This allows for adaptation to various development phases and test strategies.

[0115] As a result, the quality evaluation processor (140) enables numerical management of test scenario quality, which was not systematically managed in conventional technology, thereby improving the reliability of test design and significantly increasing the practical utility of automatically generated scenarios.

[0116] Detailed description of the means and method for calculating the quality score of the quality evaluation processor (140)

[0117] The quality evaluation processor (140) is a software-based evaluation configuration that calculates a quality score by quantitatively evaluating whether the test scenario generated by the scenario generation processor (130) sufficiently reflects the requirements, is logically valid, and is written at a level that is easy for the test performer to understand. The quality score can be implemented by calculating the total score based on weights for the evaluation results of each item, such as coverage, logical consistency, and clarity. For example, it can be set to calculate the total score by applying weights of 40% for coverage, 30% for logical consistency, and 30% for clarity, and the final score is calculated after each item score is calculated in the range of 0 to 100. This evaluation logic can be implemented as a Python-based evaluation script or a test quality evaluation engine, and can be configured to automatically calculate by receiving requirements-scenario mapping data in JSON format as input.

[0118] Coverage evaluation can be performed by analyzing the mapping relationship between the requirements document and the generated scenarios. It is implemented by breaking down requirements statements into functional or conditional units and checking whether each requirement element is reflected in one or more scenarios. For example, if a requirement "Lock account after 5 incorrect password attempts" exists, a low coverage score is calculated if only a normal login scenario exists and the account lock scenario is missing. To implement this, rule-based matching that compares requirement keywords (e.g., 'error', 'lock', 'restriction', 'exception') with scenario conditions, or natural language similarity analysis (e.g., semantic similarity measurement based on Sentence-BERT), can be used. As a practical application example, in an e-commerce system where a requirement "Free shipping on orders over 50,000 KRW" exists, the system can be configured to assign a high coverage score when both the free shipping scenario and the paid shipping boundary condition scenario exist.

[0119] Evaluations of logical consistency and clarity can be performed by combining rule-based verification and language model-based analysis. Logical consistency evaluation uses rule-based logic to check for contradictions between Given conditions and Then results, as well as the existence of conflicting results under the same conditions. For example, if the result "My Page Accessible" is generated under the condition "Non-logged-in state," it is judged as a logical error, and the score is lowered. Clarity evaluation is based on criteria such as sentence structure completeness, specificity of conditions, and clarity of result expression; for instance, a higher score is assigned to specific expressions like "Order history for the last 3 months should be displayed in chronological order" rather than ambiguous expressions like "Displays normally." In this process, sentence complexity analysis, verification of the inclusion of essential components, or scoring methods based on "clarity evaluation prompts" utilizing LLM can be applied. For instance, automatic score calculation is possible by inputting scenario sentences into a language model and applying an evaluation prompt that asks, "Evaluate on a scale of 0 to 100 whether the tester can understand it without confusion." This approach can be implemented by combining rule-based checks and language model evaluations, and is configured to be reproducible by ordinary developers using open NLP libraries and LLM APIs without excessive trial and error.

[0120] Detailed description of the coverage evaluation module (141)

[0121] The coverage evaluation module (141) is a component that quantitatively measures how sufficiently the generated test scenarios reflect the requirements. In conventional technology, coverage evaluation is mainly performed based on the code execution scope, making it difficult to directly determine the sufficiency of the test from the perspective of requirements. The coverage evaluation module (141) performs an evaluation based on the mapping relationship between the requirements document and the test scenarios.

[0122] The coverage evaluation module (141) analyzes whether each requirement is connected to one or more test scenarios and automatically identifies whether each requirement is reflected in the test. This allows for the immediate identification of whether a specific requirement is missing from the test scenario, thereby structurally preventing the problem of missing requirements that occurred during the test design phase.

[0123] In addition, the coverage evaluation module (141) evaluates not only whether there is a simple existence, but also how much conditional branching or exception situations within the requirements are included in the test scenario. For example, if only the normal flow is tested and the exception flow is omitted, the requirement is evaluated as having partial coverage and is configured to receive a low score. This allows for the management of the depth of the test scenario.

[0124] The coverage evaluation module (141) operates effectively even when requirements change. When the requirements document is updated, it automatically re-evaluates whether test scenarios exist for the changed or newly added requirements, thereby enabling early identification of test gaps caused by insufficient response to changes. This eliminates the inefficiency of manually tracking changes in conventional technology.

[0125] Consequently, the coverage evaluation module (141) plays a key role in ensuring the completeness of the requirements-based test design and provides the foundation for the test scenario automation system to function as a quality control tool rather than a simple creation tool.

[0126] Detailed description of the logic evaluation module (142)

[0127] The logic evaluation module (142) performs the function of verifying whether there are logical contradictions or inconsistencies within the generated test scenario. In conventional technology, discrepancies in conditions between scenario sentences or illogicality in the flow of actions were often discovered late during the review stage. The logic evaluation module (142) is designed to automatically detect these problems.

[0128] The logic evaluation module (142) performs an evaluation based on the consistency between the Given-When-Then structures. By analyzing whether the conditions defined in Given are logically connected to the actions in When and whether the results of the actions in When are consistent with the expected results described in Then, the logical flow of the entire scenario structure is verified. Through this, scenarios containing actions or results unrelated to the preceding conditions can be identified.

[0129] In addition, the logic evaluation module (142) also checks whether mutually exclusive conditions or contradictory state definitions exist within the same scenario or set of scenarios. For example, if a login state and a non-login state are simultaneously included in Given, or if a failure condition and a success result are simultaneously described in Then, this is judged as a logical error. This has the effect of blocking scenarios that cause confusion during test execution in advance.

[0130] The logic evaluation module (142) can be extended to include the logical relationship between multiple scenarios as an evaluation target. If scenarios expecting conflicting results for the same requirement are generated simultaneously, this is determined as a logical conflict and reflected in the quality score. This allows for the management of consistency across the entire set of test scenarios.

[0131] Consequently, the logic evaluation module (142) verifies the quality based on the validity of the logical structure rather than the linguistic expression of the test scenario, thereby automating the scenario logic review process that relied on human intuition in conventional technology and significantly improving the reliability and feasibility of the test scenario.

[0132] Detailed explanation of the clarity evaluation module (143)

[0133] The clarity evaluation module (143) is a component that quantitatively evaluates how easy it is for a test performer or manager to understand the generated test scenario. In the prior art, there was a problem where test scenarios were used without clear judgment criteria even when their expressions were ambiguous or abstract, but the clarity evaluation module (143) systematically analyzes clarity based on sentence structure and expression level.

[0134] The clarity evaluation module (143) analyzes and evaluates whether each sentence within a Given-When-Then structure clearly includes a subject, action, object, and result. It is designed to assign a low evaluation score to incomplete sentences, sentences with excessive omission of the subject, and expressions that can be interpreted in multiple ways. This prevents confusion that may occur during the process of interpreting the scenario by the tester.

[0135] In addition, the clarity evaluation module (143) also evaluates the consistency of terminology usage within the scenario. If the same action or result is described repeatedly using different expressions, or if domain terms and general terms are mixed, this is judged as a factor that reduces clarity. This evaluation contributes to improving the readability of the entire scenario and lowering the learning cost.

[0136] The clarity evaluation module (143) performs an evaluation by also considering the length, complexity, and degree of condition overlap of the sentence. If multiple conditions or results are excessively combined in a single sentence, it may be difficult to understand and execute during actual testing, so this is reflected as a factor that reduces clarity. This encourages the creation of scenarios that are concise yet clear in their core.

[0137] Consequently, the clarity evaluation module (143) supports ensuring that test scenarios are not merely generated but also secure a level of readability and expression quality that allows them to be immediately utilized in a practical environment. This has the effect of structurally improving the problem in conventional technology where the quality of scenario expression varied significantly depending on individual capabilities.

[0138] Detailed description of the AI ​​feedback processor (150)

[0139] The AI ​​feedback processor (150) is a component that performs the role of improving the quality of test scenarios and continuously improving the performance of the language model based on the evaluation results produced by the quality evaluation processor (140). While the automatic generation system of the prior art was limited to providing one-time results, the AI ​​feedback processor (150) is designed around a structure of gradual improvement through iterative learning and correction.

[0140] The AI ​​feedback processor (150) identifies targets for correction for scenarios that are judged to be below a preset standard among the results of coverage, logic, and clarity evaluations. At this time, rather than simple regeneration, it analyzes which element is the cause of the quality degradation and distinguishes problem types such as missing conditions, errors in behavior expression, and ambiguity of results.

[0141] Subsequently, the AI ​​feedback processor (150) adjusts the language model inference path according to the type of problem or reflects rules and examples to be referenced when regenerating the scenario. This prevents the problem of the same quality degradation scenario being repeatedly generated for the same requirements and gradually improves the stability of the generated results.

[0142] The AI ​​feedback processor (150) performs the role of providing retraining data to the language model building module (121) as well as correcting short-term scenarios. By accumulating repeatedly occurring error patterns as training data, the accuracy of natural language interpretation and the quality of scenario generation of the entire system can be improved in the long term.

[0143] Consequently, the AI ​​feedback processor (150) is a key component that transforms the test scenario automation system from a simple generation tool into a learning-type quality improvement system, and realizes a continuous quality improvement mechanism that was not provided by conventional technology.

[0144] A detailed description of the means and method for correcting the scenario of the AI ​​feedback processor (150).

[0145] The AI ​​feedback processor (150) is a software-based correction configuration that receives scenario quality scores and item-specific defect reasons transmitted from the quality evaluation processor (140) as input, and corrects substandard scenarios to regenerate them or refines sentences. The AI ​​feedback processor (150) can be implemented by combining a diagnostic logic that classifies defect types in a regular manner and a language model inference logic that generates correction proposals. For example, coverage defects are classified as "existence of missing conditions / exception cases," logic defects as "Given-Then contradictions" or "existence of conflicting results," and clarity defects as "contains ambiguous expressions" or "lack of verifiable criteria," after which correction rules corresponding to each defect type are applied. Since this structure can be implemented using only general branch processing, rule tables, and JSON-based defect reports, it can be reproduced by a typical developer without excessive repetitive experimentation.

[0146] The AI ​​feedback processor (150) can use both template-based automatic correction and LLM-based rewriting methods as correction methods. Template-based automatic correction is applied when missing elements can be clearly identified, for example, if a requirement condition is missing in the Given of a scenario, it can be configured to insert a Condition value into the Given clause. LLM-based rewriting is applied when the clarity score is low because the sentence is unnatural or ambiguous, and it is configured to improve the sentence using prompts such as "Modify the scenario below to be more specific and verifiable in BDD format." As a practical example, if the scenario "If a user fails to log in, an error should be displayed" is classified as having a clarity defect, the AI ​​feedback processor (150) refers together with the error message phrase and threshold count information from the original requirement text or acceptance criteria to correct it in a form such as "If a user enters an incorrect password / a 'Password is incorrect' message should be displayed" or "If a user enters the password incorrectly 5 times in a row / the account should be switched to a locked state and further logins should be blocked." This correction can be implemented simply by inserting a template and rewriting the language model, making it easy to apply in practice.

[0147] In addition, the AI ​​feedback processor (150) can improve coverage by not only correcting a single scenario sentence but also automatically generating missing branch scenarios for the same requirements. For example, if only a free shipping scenario exists for the requirement of "free shipping for over 50,000 won" and a "paid shipping for under 50,000 won" scenario is missing, the AI ​​feedback processor (150) generates a boundary opposite condition based on the "missing boundary value case" indication in the defect report and then adds a separate scenario. At this time, boundary value generation can be implemented as a software rule in the form of "if there is an ≥ condition, add a < condition," and the generated condition and Expected Outcome can be naturally completed into a scenario sentence by being passed to the language model as a prompt to "generate a test scenario of the opposite condition." Subsequently, the corrected scenario set is passed back to the quality evaluation processor (140) for re-evaluation, and can be configured so that the same loop is repeated until it reaches a threshold score or higher. This closed-loop structure can be implemented using only standard software pipelines and LLM inference calls, allowing a person with ordinary knowledge to reproduce it without excessive trial and error.

[0148] Detailed description of the result review output interface (160)

[0149] The result review output interface (160) is a component that provides the generated test scenarios and quality evaluation results to the manager and manages the review and approval process. Since the final quality responsibility generally rests with a person even for automatically generated results, the result review output interface (160) is designed as the final decision-making layer connecting people and systems.

[0150] The result review output interface (160) outputs the scenario generation results, the quality score of each scenario, and the mapping relationship with requirements in the form of a visualized dashboard. Through this, the manager can intuitively understand the test quality status of not only individual scenarios but also the entire project.

[0151] Additionally, the result review output interface (160) is configured to allow an administrator to request approval, rejection, or modification for specific scenarios. Scenarios that are rejected or requested for modification are passed to the AI ​​feedback processor (150) to undergo additional correction or regeneration processes. Through this, a verification flow is formed in which automation and human judgment are organically combined.

[0152] The result review output interface (160) ensures traceability and auditability of the test design process by recording and managing review history and approval results. This plays an important role in meeting the quality management documentation requirements demanded by large organizations or regulatory environments.

[0153] Consequently, the result review output interface (160) serves as the final gateway for converting automatically generated test scenarios into the organization's official test assets, thereby resolving the problem of insufficient reliability of automatically generated results in conventional technology and increasing the feasibility of introducing an automated test scenario system.

[0154] Below, a test scenario automation method (S100) using the above test scenario automation system (100) is described in detail.

[0155] The test scenario automation method (S100) of the present invention includes a requirement document input step (S110) for performing a specific process, a natural language analysis step (S120), a scenario automatic generation step (S130), a quality evaluation step (S140), a feedback step (S150), and a result output approval step (S160), thereby enabling automatic response to changes in requirements to maintain the latest scenario, ensuring consistent scenario quality by eliminating deviations in human interpretation, reducing the test design phase time by more than 70% compared to the existing method, improving quality management efficiency through visualization based on quality indicators, and securing a structure for gradual quality improvement through AI learning.

[0156] Detailed description of the requirements document input step (S110)

[0157] The requirements document input step (S110) is the first step performed in the test scenario automation method and serves the role of introducing requirements information, which exists in various forms and channels, into the system. In actual project environments, requirements are often scattered across various formats such as planning documents, functional specifications, meeting materials, screen design documents, spreadsheets, and presentation files; it is not uncommon for them to be dispersed even within emails, wiki pages, or comment sections of issue tracking systems. Considering this realistic dispersion, the requirements document input step (S110) is configured to integrate the process of direct file uploads by users and the automatic collection of requirements from external collaboration systems such as Jira and Confluence.

[0158] In the requirements document input step (S110), the user can first select and upload a file stored on a local PC or an internal document management system. For example, if the user selects a document created by a PM, such as "Online Payment System Function Specification.docx", "Screen Flow Design.pptx", or "API Requirements Definition Document.pdf", the document input module (111) receives it and registers it as an object manageable within the system. At this time, metadata such as the document's file format, author, version, and creation date is also collected so that it can be used for tracking requirements changes or managing history later. The user can input documents grouped by project, function, or sprint, and the system can organize test scenario bundles based on this.

[0159] Simultaneously, in the requirements document input stage (S110), requirements stored in tools such as Jira and Confluence are automatically collected through an external integration module (112). For example, tickets registered in the "User Story" issue type in Jira, "Functional Requirements Wiki Pages" and "API Spec Documents" written in Confluence can be regularly retrieved into the system via REST API. At this time, not only the simple body content but also the issue key, status (e.g., To Do, In Progress, Done), tags, responsible person, and linked subtask information are collected together, thereby securing a basis for analyzing which development stage a specific requirement is in and what test priority it should have.

[0160] In this stage, even if the document formats vary, a process is performed to organize them into "requirement candidate text" units suitable for subsequent natural language analysis. For example, in PPT files, slide titles and body text are extracted while shapes or decorative text are excluded from analysis; in DOCX files, title styles, numbering, and table structures are recognized to separate paragraphs into logical requirement units. In the case of Confluence documents, "requirement items" can be automatically recognized by utilizing header levels and bullet structures. As such, the requirements document input stage (S110) serves to safely collect sentences and paragraphs presumed to be requirements, regardless of file format or source.

[0161] For example, in an online shopping mall project, "cart function requirements" are written in natural language on a Confluence page, while at the same time, user stories such as "adding products to the cart," "summing up cart amounts," and "removing products from the cart" are registered in Jira as issues. The requirements document input step (S110) collects the body content of the Confluence page, the Jira issue description, and acceptance criteria, and integrates them into a single "cart requirements text set." In a subsequent step, the natural language interpretation processor (120) extracts actors, conditions, actions, and results based on this set, which serves as the starting point for the automatic generation of test scenarios. As such, the requirements document input step (S110) is characterized as a gateway that gathers information scattered across various sources into a consistent subject for analysis.

[0162] Detailed explanation of the natural language analysis step (S120)

[0163] The natural language analysis step (S120) is a process of analyzing text data obtained through the requirements document input step (S110) into semantic units using an LLM-based language model and converting it into structured information necessary for generating test scenarios. The key aspect of this step is not simply keyword extraction, but inferring and distinguishing actors, actions, conditions, and expected outcomes embedded within the requirements sentences based on context. To this end, a natural language interpretation processor (120) operates centrally, and a language model construction module (121), a sentence interpretation extraction module (122), and an extraction result organization module (123) cooperate organically to perform the analysis.

[0164] The first process performed in the natural language analysis step (S120) is to simultaneously analyze each sentence from the perspective of grammatical structure and semantic structure. For example, if a requirement sentence is written as "After the user successfully logs in, if they click the order history inquiry button on My Page, order history within the last 6 months should be displayed in descending order of date," the natural language interpretation processor (120) decomposes "user" as the agent, "after successfully logging in" as the antecedent condition, "clicking the order history inquiry button" as the action, and "order history within the last 6 months should be displayed in descending order of date" as the result. In this process, the boundary between condition-action-result is identified using conjunctions and auxiliary verbs such as "after," "if," and "should be" as clues.

[0165] The natural language analysis step (S120) considers not only single sentence units but also paragraph units and the context of the entire document. For example, if a condition such as "premium member" is already mentioned in a higher paragraph and a lower sentence only states "provides free shipping benefits," the natural language interpretation processor (120) can connect the condition in the higher paragraph with the result in the lower sentence to supplement the meaning into a structure such as "when a user who is a premium member orders a product, a free shipping benefit should be applied." Analysis that considers context in this way is an important function that automates parts that previously required human inference in conventional technology.

[0166] In the natural language analysis stage (S120), domain-specific terms and expressions are also processed. For example, in the financial domain, expressions such as "opening an account," "identity verification," and "registering for automatic transfer" appear repeatedly, while in the e-commerce domain, expressions such as "adding to cart," "adding to wishlist," "confirming order," and "approving payment" are mainly used. The natural language interpretation processor (120) utilizes domain-specific patterns learned in the language model building module (121) to classify these expressions into appropriate Action, Condition, and Outcome. For example, "in the state of having completed identity verification" is structured as a Condition, "registering for automatic transfer" as an Action, and "should be automatically withdrawn on a designated date every month" as an Expected Outcome.

[0167] To give another specific example, let's assume there is a requirement for a customer center system stating, "When an agent changes the status of a ticket received under the 'customer complaint' type to 'resolved,' an email notifying the customer that the issue has been resolved should be automatically sent." The natural language analysis step (S120) recognizes "agent" as the actor, "changing a ticket received under the 'customer complaint' type to 'resolved'" as the action, and "an email notifying the customer that the issue has been resolved should be automatically sent" as the result. Internally, it maps this information to structural expressions such as "ticket status = resolved" and "send email = true." Subsequently, in the scenario generation step (S130), this information is recombined into a Given-When-Then structure and automatically converted into a scenario such as, "Given: A customer complaint ticket exists and an agent is online / When: When an agent changes the ticket status to 'resolved' / Then: An email notifying the customer that the issue has been resolved should be sent." As such, the natural language analysis step (S120) is a core step that meticulously transforms the task of interpreting requirements—which is implicitly performed in a human mind—into a structure that a machine can understand.

[0168] Detailed description of the scenario automatic generation step (S130)

[0169] The scenario automatic generation step (S130) receives actor, behavior, condition, and result information extracted and organized in the natural language analysis step (S120) as input and automatically combines it in the Given-When-Then format, which is a behavior-based test scenario structure. In this step, rather than simply concatenating text, sentences are constructed by comprehensively considering sentence structure, domain expression, and clarity of the test perspective to generate natural language scenarios that can be actually utilized by the test designer. For example, if elements such as "User = General Member," "Condition = Login Success," "Behavior = Click Payment History Button," and "Result = Display List of Payment History for the Last 3 Months" are derived as a result of the analysis, the scenario automatic generation step (S130) arranges them naturally in accordance with the Given-When-Then structure in the form of "When the user has successfully logged in with a normal account / and clicks the Payment History Button / a list of payment history within the last 3 months should be displayed on the screen."

[0170] The scenario automatic generation step (S130) also handles situations where multiple scenarios can be derived from a single requirement. If a single requirement statement contains normal flow, exception flow, branching by authority, and different results depending on whether options are set, the system can automatically branch them to generate multiple scenarios. For example, if there is a requirement stating "Shipping is free for premium members, and a shipping fee of 3,000 won is charged for general members on orders under 30,000 won," the scenario automatic generation step (S130) generates a Given-When-Then scenario for premium members and a Given-When-Then scenario for general members, respectively, thereby automatically obtaining multiple test cases derived from a single requirement. This reduces the burden of having to manually divide and write branching cases one by one, which was a conventional method.

[0171] This stage is designed to enable the generation of not only user scenarios but also backend, batch, and integration scenarios. For example, when there is a requirement that "point accumulation should be processed based on the previous day's transaction history when the regular settlement batch is executed at midnight every day," the scenario automatic generation stage (S130) generates a server-side scenario such as "Given, while settlement target transaction data is stored / When, when the regular settlement batch is executed at midnight / Then, points should be accumulated for the previous day's transaction history," even in the absence of user interaction. A key feature is that the scope of testing is not limited to the user interface, as scenarios from a system operation perspective rather than a UI-centric one are automatically generated in this manner.

[0172] The automatic scenario generation step (S130) also considers the consistency of sentence expression and the selection of domain-friendly vocabulary. It utilizes a combination of templates and language models to ensure that sentences with similar patterns are always generated for the same type of action. For example, in the payment area, expressions such as "when the payment button is clicked" and "payment must be completed" are used repeatedly, while in the customer center area, expressions such as "when the ticket status is changed to 'resolved'" and "a notification email must be sent to the customer" are generated naturally. This allows test personnel to become familiar with the structure and expression methods while reading through various scenarios, and simplifies the review and automation script mapping processes.

[0173] As a specific example, it can be assumed that in an online shopping mall project, a requirement is given that "free shipping must be provided if the total amount of products in the shopping cart is 50,000 won or more." When "Actor = Logged-in user," "Condition = Total amount in shopping cart ≥ 50,000 won," "Action = Click the 'Order' button," and "Result = Shipping fee 0 won, Free shipping indicated in order summary" are extracted in the natural language analysis step (S120), the scenario automatic generation step (S130) generates a specific scenario based on this, stating, "If the user is logged in and the total amount of products in the shopping cart is 50,000 won or more / if the 'Order' button is clicked / the shipping fee should be displayed as 0 won in the order summary information and a 'Free Shipping' message should be displayed." This has a distinct difference from conventional technology in that it is automatically generated while possessing a level of specificity and testability equivalent to a scenario written by a human.

[0174] Detailed explanation of the quality evaluation stage (S140)

[0175] The quality evaluation step (S140) is a step in which a quantitative quality evaluation is performed on the test scenarios generated through the scenario automatic generation step (S130) in terms of coverage, logical consistency, and clarity. In this step, the coverage evaluation module (141), the logic evaluation module (142), and the clarity evaluation module (143) operate sequentially and in parallel, centered around the quality evaluation processor (140), to calculate scores and evaluation results for each scenario. This differs significantly from conventional methods in that the system automatically evaluates areas that were previously judged subjectively by test leaders or senior testers based on experience, such as "this scenario is okay" or "this scenario is lacking," according to certain criteria.

[0176] In the quality evaluation stage (S140), the coverage evaluation module (141) first analyzes the mapping relationship between the requirements document and the generated scenarios. For example, when various cases such as normal login, password error, account lock, and inactive account are described in the requirements document for "login function requirements," it checks whether the generated scenarios include all of these cases. If there is only a scenario dealing with normal login, the coverage of the requirement is evaluated as being only partially satisfied, and if there are enough scenarios that include error cases and exception cases, a high coverage score is assigned. Through this, instead of a vague judgment that "the requirements seem to be roughly reflected," it becomes possible to make numerical judgments such as "Requirement item X is 80% covered" or "Requirement item Y is 50% covered."

[0177] The logic evaluation module (142) verifies the logical consistency within each scenario and between scenarios. For example, if a scenario has "when the user is not logged in" written in Given and "when the order history button is clicked on My Page" written in When, it becomes a scenario that contradicts the actual service operation. The logic evaluation module (142) automatically detects such contradictions, assigns a low logic score to the corresponding scenario, and, if necessary, indicates that it is subject to correction by the AI ​​feedback processor (150). Additionally, if different scenarios describe conflicting expected results for the same requirement—for example, if one scenario expects free shipping while another expects shipping fees under the same conditions—this is recognized as a logical conflict and reflected in the quality evaluation result.

[0178] The clarity evaluation module (143) evaluates whether the scenario text is intuitively understandable to the tester. For example, abstract expressions such as "the list appears normally" are judged to have lower clarity than concrete expressions such as "order history within the last 3 months should be displayed in descending order based on the order date." The clarity evaluation module (143) calculates a score based on the specificity of conditions, targets, and expected results included in the scenario sentence, the length and structure of the sentence, and the consistency of the use of domain terms. Through this, scenarios that may be interpreted differently during testing due to ambiguous expressions are identified early on, and the AI ​​feedback processor (150) guides the sentences to be improved into clearer sentences.

[0179] For a specific example, it can be assumed that the quality evaluation stage (S140) evaluates the "Free Cart Shipping" scenario generated in the previous scenario automatic generation stage (S130). In the coverage evaluation, it is discovered that the scenario only covers the normal case of "Free shipping for 50,000 won or more," and that a scenario for the "Paid shipping for less than 50,000 won" case has not yet been generated; thus, a coverage level of 50% can be assigned to that requirement. In the logic evaluation, a high score is assigned because the Given condition "If the total amount in the cart is 50,000 won or more" and the Then result "Shipping fee should be displayed as 0 won" align with the business rule; however, if another scenario describes that a shipping fee of 3,000 won should be charged for the same condition, it is judged as a logical conflict between the two scenarios. In the clarity evaluation, a high clarity score can be assigned because specific expressions such as "within the last 3 months," "Shipping fee displayed as 0 won," and "Free shipping text displayed" are included. In this way, the quality evaluation step (S140) calculates a final quality score by synthesizing the results from each perspective and transmits the substandard scenarios to the AI ​​feedback processor (150) to enter the retraining and correction loop.

[0180] Detailed explanation of the feedback step (S150)

[0181] The feedback step (S150) is a step that performs the process of automatically correcting test scenarios that do not meet quality standards and retraining the language model based on the evaluation results produced in the quality evaluation step (S140). In this step, scenarios that fall below a certain standard score in terms of coverage, logical consistency, and clarity are classified as scenarios subject to feedback, and detailed analysis information on why the scenario failed to meet the standards is generated. For example, this applies to cases where a specific scenario receives a low coverage score because it does not reflect exception conditions within the requirements, or where a clarity score receives a low score due to abstract expressions in the result description, such as "should operate normally." The feedback step (S150) transmits these diagnostic results to the AI ​​feedback processor (150) so that improvements based on the cause are made, rather than simple regeneration.

[0182] In the feedback step (S150), a process of categorizing quality degradation factors for each scenario is performed. For example, in terms of coverage, they can be classified into types such as "missing exception cases," "non-reflection of boundary values," and "lack of branching scenarios by authority"; in terms of logic, into types such as "condition-action discrepancy," "result-condition contradiction," and "existence of conflicting result scenarios"; and in terms of clarity, into types such as "excessive abstract expression," "absence of specific values ​​or conditions," and "excessively complex sentence structure." By transmitting this categorized information to the AI ​​feedback processor (150), the feedback step (S150) provides a clear signal to the language model regarding the direction in which the scenario generation method should be modified.

[0183] The feedback step (S150) may also directly modify the scenario itself. For example, if a scenario generated for the requirement that "free shipping should be provided if the total amount in the shopping cart is above a certain amount" has a missing amount condition, such as "free shipping should be applied when the order button is clicked while the product is in the shopping cart," the feedback step (S150) triggers the AI ​​feedback processor (150) to automatically correct the scenario into the form "free shipping should be applied when the total amount in the shopping cart is 50,000 won or more / and the order button is clicked / while the total amount in the shopping cart is above 50,000 won." At this time, the modified scenario undergoes a quality evaluation again to check if it satisfies the standard score, and the feedback loop can be repeated multiple times if necessary.

[0184] Additionally, the feedback step (S150) is used to improve recurring error patterns at the language model level, going beyond single-scenario correction. For example, if a tendency is found in a specific domain to always vaguely express the result of "sending an email" as "a notification should be sent," the feedback step (S150) accumulates this pattern as data and provides retraining data to the language model building module (121) through the AI ​​feedback processor (150) to generate more specific expressions, such as "an email containing the recipient, subject, and main content should be sent." Through this, the quality is improved so that when a scenario in the same domain is generated later, a more specific and testable result sentence is naturally generated.

[0185] For a specific example, consider a case where the automatically generated scenario for the requirement of "displaying an error message upon login failure" is described only up to "if the user enters an incorrect password, login must fail," and does not reflect what message should be displayed on the actual screen or how many failures lead to account locking for security purposes. The quality evaluation stage (S140) determines that this is insufficient in terms of coverage and clarity, and the feedback stage (S150) transmits feedback information such as "missing error message and account locking policy" to the AI ​​feedback processor (150). Then, the AI ​​feedback processor (150) uses a language model to induce the scenario to be reconstructed as follows: "If the user enters an incorrect password 5 times in a row / login fails and an error message 'Password is incorrect' must be displayed / and the account must be switched to a locked state to block further logins." In this way, the feedback stage (S150) repeatedly corrects the automatically generated scenario and is a key stage for realizing a structure in which quality improves as the system is used.

[0186] Detailed description of the result output approval step (S160)

[0187] The result output approval stage (S160) is a stage in which the test scenarios that have undergone the quality improvement process up to the feedback stage (S150) and the quality evaluation results thereof are finally provided to the user, and are confirmed as official test assets through the manager's review and approval. In this stage, the result review output interface (160) operates centrally, and through a dashboard-type UI, the list of generated scenarios, the quality score of each scenario, the coverage status, the logic and clarity evaluation results, and the mapping information with requirements are provided so that they can be viewed on a single screen. Through this, the manager can intuitively determine which scenarios are at a level where they can be used immediately and which scenarios require additional review or manual modification.

[0188] In the result output approval step (S160), the administrator can select approval, hold, or rejection on a scenario-by-scenario or group-by-group basis. For example, if a set of scenarios related to "free shopping cart shipping" all receive high quality scores and are determined to be in compliance with internal policies and business rules, the administrator changes the status of the scenario set to approval. Conversely, if some of the scenarios related to "login failure" have discrepancies between the error message expression or the account lock policy and the actual operational policy, only those scenarios can be selected and rejected, and the reason for rejection can be noted in a memo. The rejected scenarios are then forwarded to the AI ​​feedback processor (150) to become targets for further correction or regeneration, and the comments left by the administrator serve to guide the direction of correction.

[0189] The result output approval step (S160) also performs the role of exporting approved scenarios to external test management tools or automation script management systems. For example, it is possible to convert approved scenarios to match the format of test management tools such as TestRail, Zephyr, or Xray, or to automatically convert and save them as feature files in the Cucumber format that supports BDD (Behavior Driven Development). A scenario such as "When the user has successfully logged in / When the payment history button is clicked / Then payment history within the last 3 months should be displayed" can be output as is in Gherkin format, which can be utilized by the development team or QA team to link directly to an automation test script. At this stage, the result review output interface (160) provides options to select the output format and target system, thereby flexibly responding to various development and test environments.

[0190] In addition, the result output approval stage (S160) aggregates the status of scenario quality at the project, sprint, and function levels and provides it in the form of management indicators. For example, by statistically displaying the average coverage score, logic score, clarity score, approval rate, and rejection rate of automatically generated scenarios for 20 requirements belonging to a specific sprint, it is possible to quantitatively determine the level at which test design quality has been achieved. Through this, managers can make decisions based on objective data, such as, "The login and sign-up function scenarios in this sprint are at a stable level, but the payment and refund function scenarios still need improvement."

[0191] For a specific example, it can be assumed that in an online shopping mall project, multiple scenarios were automatically generated for each of the five functions: "Sign-up," "Login," "Cart," "Payment," and "Order Inquiry." After passing through the quality evaluation stage (S140) and the feedback stage (S150), when the result output approval stage (S160) is reached, the result review output interface (160) displays a list of scenarios, quality scores, and approval recommendations for each function on the dashboard. The administrator collectively approves the scenarios related to "Sign-up" and "Login" because they all received high scores and are consistent with actual service policies. On the other hand, the administrator discovers that in some scenarios for the "Payment" function, the interest-free installment rules of a specific credit card company are reflected differently from the latest policy, records the reason for rejection for the relevant scenario as "Interest-free installment policy needs updating," and processes the rejection. This rejection information is transmitted back to the AI ​​feedback processor (150), serving as an opportunity for the language model to learn the latest data on payment policies, and subsequently, the regenerated scenario passes the quality evaluation stage (S140) again in a form reflecting the latest policies. In this way, the result output approval stage (S160) completes the automatic generation, evaluation, and correction processes in a form controllable by humans and performs the role of officially incorporating them as test assets of the organization.

[0192] As explained above, the test scenario automation system according to the present invention fundamentally resolves the problem of subjective judgment deviations that occurred in conventional technology when requirements documents were interpreted directly by humans. By analyzing semantic elements included in requirements documents using the same criteria through generative AI-based natural language interpretation, it eliminates the problem of different test scenarios being written depending on the tester or analyst, and enables the maintenance of consistent test quality throughout the project.

[0193] Furthermore, it effectively resolves the issues of omitted requirements and the generation of duplicate scenarios that frequently occurred in conventional technologies. By structurally analyzing requirements documents to systematically derive actors, conditions, actions, and results, it enables the management of each requirement to accurately correspond to test scenarios and significantly reduces human error that occurred during the test design process.

[0194] It also overcomes the limitations of conventional technology in terms of responding to changes in requirements. Previously, when requirements changed, test scenarios had to be modified manually, incurring significant time and cost. However, this invention automatically collects and analyzes requirements documents to immediately reflect changes, thereby enabling the continuous maintenance of up-to-date test scenarios. Consequently, efficient test management is possible even in large-scale projects or environments with frequent requirements changes.

[0195] The problem of relying on subjective judgment for the quality evaluation of test scenarios is also resolved. By quantitatively evaluating quality elements such as coverage, logical consistency, and clarity, the present invention enables the completeness of test scenarios to be managed based on numerical values. This systematizes the scenario review process, which was previously conducted without clear standards in conventional technology, and improves the objectivity and reliability of quality management.

[0196] It provides effects differentiated from conventional technology even when the quality of automatically generated scenarios is insufficient. By incorporating quality evaluation results into language model training via an AI feedback processor, it implements a structure that iteratively improves generated results. Consequently, it is possible to establish a learning-oriented automation environment where test scenario quality continuously improves as system usage accumulates, rather than merely stopping at simple automation.

[0197] Furthermore, it improves upon the problem of conventional technology, where it was difficult to intuitively grasp the relationship between test scenarios and requirements. By providing scenario generation results, quality scores, and requirements mapping information in a visualized dashboard format, this invention enables managers to grasp the overall test quality status at a glance and make efficient decisions.

[0198] Consequently, the present invention resolves the issues of time consumption, quality inconsistencies, and management inefficiency associated with manual test scenario creation methods by transitioning to a generative AI-based automation structure. By organically linking the entire process from requirements analysis to scenario generation, quality evaluation, feedback learning, and approval management, it provides the effect of effectively overcoming the fundamental problems of test design and quality management that conventional technologies have failed to address.

[0199] The above detailed description of the present invention describes only specific embodiments thereof. However, it should be understood that the present invention is not limited to the specific forms mentioned in the detailed description, but rather should be understood to include all variations, equivalents, and substitutions within the spirit and scope of the invention as defined by the appended claims.

[0200] In other words, the present invention is not limited to the specific embodiments and descriptions described above, and any person skilled in the art to which the present invention pertains can make various modifications without departing from the essence of the invention as claimed in the claims, and such modifications fall within the scope of protection of the present invention. Explanation of the symbols

[0201] 100: Test Scenario Automation System 110: Requirements Input Interface 111: Document Input Module 112: External Integration Module 113: Text conversion module 120: Natural Language Interpretation Processor 121: Language Model Construction Module 122: Sentence Interpretation Extraction Module 123: Extraction Result Cleanup Module 130: Scenario Generation Processor 131: 1st Structure Generation Module 132: Second Structure Generation Module 133: Third Structure Generation Module 140: Quality Evaluation Processor 141: Coverage Evaluation Module 142: Logic Evaluation Module 143: Clarity Evaluation Module 150: AI Feedback Processor 160: Result Review Output Interface S100: Test Scenario Automation Method S110: Requirements Document Input Step S120: Natural Language Analysis Step S130: Automatic Scenario Generation Step S140: Quality evaluation stage S150: Feedback stage S160: Result output approval step

Claims

Claim 1 A requirements input interface (110) that receives requirements documents, converts them into text, and automatically collects requirements documents by linking with an external system; a natural language interpretation processor (120) that extracts the actor, action, condition, and expected outcome of a sentence using an LLM-based language model; a scenario generation processor (130) that automatically generates a Given-When-Then structured scenario based on the extracted information; a quality evaluation processor (140) that evaluates the coverage, logical consistency, and clarity of the automatically generated scenarios to calculate a quality score, and transmits scenarios below a preset standard to an AI feedback processor (150); an AI feedback processor (150) that performs retraining of the LLM based on the quality evaluation results and corrects scenarios below the preset standard quality to be above the preset standard. A test scenario automation system characterized by including a result review output interface (160) that visualizes the results of scenario generation, quality scores, and requirements mapping results in the form of a dashboard and outputs them to an administrator, and accepts the administrator's review and approval; wherein the requirements input interface (110) includes a document input module (111) that receives requirements documents, organizes them according to a preset format, and transmits them to a text conversion module (113); an external linkage module (112) that links with multiple external systems including Jira and Confluence to collect requirements documents, organizes them according to a preset format, and transmits them to a text conversion module (113); and a text conversion module (113) that converts the data transmitted from the document input module (111) and the external linkage module (112) into text data identical to the data format to be input into a natural language interpretation processor (120). Claim 2 delete Claim 3 In claim 1, the natural language interpretation processor (120) comprises: a language model construction module (121) that constructs an LLM-based language model, is updated in real-time according to relearning performed by an AI feedback processor (150), and is updated again according to the requirements of the manager; a sentence interpretation extraction module (122) that analyzes text data transmitted from a requirement input interface (110) using the latest language model updated by the language model construction module (121) and extracts an actor, action, condition, and expected outcome from the text data; and an extraction result sorting module (123) that refines and converts the data extracted from the sentence interpretation extraction module (122) into a data format identical to the data format to be input to the scenario generation processor (130). Claim 4 In paragraph 3, the scenario generation processor (130) comprises: a first structure generation module (131) that generates a structure corresponding to the Given part in a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor (120); a second structure generation module (132) that generates a structure corresponding to the When part in a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor (120); and a third structure generation module (133) that generates a structure corresponding to the Then part in a Given-When-Then structure scenario based on data transmitted from the natural language interpretation processor (120); characterized in that the test scenario automation system comprises Claim 5 In claim 4, the quality evaluation processor (140) comprises: a coverage evaluation module (141) that quantitatively measures, according to a preset standard, how much source code a test case has executed and how much the requirements have been met in a software test; a logic evaluation module (142) that quantitatively measures, according to a preset standard, whether a number of components, functions, and data of the software are logically consistent without contradiction; and a clarity evaluation module (143) that quantitatively measures, according to a preset standard, how easy and clear the documentation, code, and test scripts are to understand.

Images