A method and system for generating requirements specifications based on multimodal understanding

Abstract

This invention provides a method and system for generating requirement specifications based on multimodal understanding, relating to the field of artificial intelligence. The method includes: performing content understanding operations on data from each modality using corresponding content extraction tools to generate a unified multimodal requirement representation document; establishing a requirement knowledge base based on an existing template library and corresponding rule base, matching and comparing the requirement knowledge base with the requirement representation document to check the completeness of the requirement representation document and generate decision-making questions; creating an overall framework for the specification document based on a standard specification template, and filling the overall framework with content based on the requirement representation document and the answers to the decision-making questions to generate the specification document; collecting feedback information on the specification document, analyzing the feedback information using a large language model, and converting the feedback information into actionable modification suggestions for the specification document. This invention improves the automation, completeness, and accuracy of requirement specification generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and system for generating requirement specifications based on multimodal understanding. Background Technology

[0002] In the software development lifecycle, requirements engineering is a crucial and extremely challenging first step. Requirements come from diverse sources and take many forms, often scattered across various media such as meeting recordings, product demo videos, UI design drafts, flowcharts, emails, and instant messages. Traditional requirements analysis relies heavily on manual labor; requirements analysts need to manually collect and organize this multimodal information, "translating" it into a structured requirements document. This process is not only extremely time-consuming and labor-intensive but also prone to errors, leading to information omissions, requirement conflicts, or misunderstandings. With the rise of Specification-Driven Development (SDD), AI code generators have placed unprecedented demands on the quality of input specifications (Specs). A vague, incomplete, or even contradictory specification document is the root cause of "garbage code" generated by AI.

[0003] In the process of realizing this invention, the applicant discovered that the most similar prior art to this invention is Chinese patent application CN120780275A, which discloses a "method and system for acquiring and analyzing industrial software requirements based on intelligent agents," including: constructing a requirements analysis system based on multi-agents. Using this system, the user's multimodal input is "converted" into "structured industrial software requirements." Based on these structured requirements, a "business process diagram" conforming to the BPMN standard is generated. Although the prior art represented by this patent application recognizes the problem of multimodal input, its solution has fundamental limitations and cannot meet the needs of modern AI-driven software development, exhibiting at least the following defects:

[0004] 1. The lack of "conversion" to "understanding and integration": The core of this solution is "conversion," that is, converting one format to another. It lacks a true "understanding" and "integration" process, especially in handling conflicts and inconsistencies between multimodal inputs. For example, when the description in the meeting recording conflicts with the layout in the UI design draft, the system cannot identify or resolve this issue.

[0005] 2. The output objective is too singular and incomplete: The final output of this solution is a "BPMN business process diagram." BPMN is merely a subset of business processes in software specifications. A complete specification for AI code generation must include numerous non-functional requirements (such as performance and security), API interface specifications, data models, acceptance criteria, etc. The BPMN output from current technologies is far from sufficient for AI development.

[0006] 3. Lack of a complete and consistent verification loop: This solution is an "open-loop" unidirectional process. It lacks a mechanism to check whether the generated requirements are complete (whether key constraints are missing), and also lacks a mechanism to check whether there are logical contradictions within the requirements.

[0007] 4. Lack of human-machine collaborative iterative refinement capabilities: Requirements engineering is essentially an iterative process that requires repeated communication and refinement. Existing technologies do not provide an interactive mechanism that allows users to easily provide feedback, clarification, and modification of the generated requirements.

[0008] Therefore, how to automatically, accurately, and completely generate a high-quality, machine-readable specification document from the original multimodal input has become a core pain point that urgently needs to be addressed in the current software engineering field. Summary of the Invention

[0009] This invention aims to solve at least one of the technical problems existing in the prior art or related technologies, and discloses a requirement specification generation method and system based on multimodal understanding. It is a new specification generation paradigm based on "deep understanding and fusion" and has the capabilities of "completeness check" and "iterative refinement", which greatly improves the accuracy of requirement specification documents.

[0010] Terminology Explanation:

[0011] Multimodal input refers to demand-related data from different information channels and in various formats, such as text (emails, chat logs), images (UI design drafts, hand-drawn sketches), audio (meeting recordings), and video (operation demonstrations).

[0012] Specification (Spec): A structured document (such as a Markdown file) that describes the functional requirements, non-functional requirements, constraints, and design decisions of a software system using natural or semi-formal language.

[0013] Unified Requirement Representation: An intermediate data structure (such as JSON) used to integrate unstructured, fragmented requirement information from multimodal inputs into a unified, structured, machine-readable format.

[0014] BPMN (Business Process Model and Notation): A graphical representation used to specify business processes. It is a subset of the requirements specification, focusing primarily on the process.

[0015] Cross-modal fusion: an AI technology used to integrate information from different modalities and critically identify and resolve inconsistencies or conflicts between them (e.g., audio descriptions do not match UI design drafts).

[0016] The first aspect of the present invention discloses a requirement specification generation method based on multimodal understanding, comprising: understanding and fusing multimodal data: receiving multimodal data containing requirement information of target software; performing content understanding operations on the data of each modality using corresponding content extraction tools to obtain requirement understanding information of the target software; performing conflict detection on the requirement understanding information, marking conflict points, and generating a conflict point list; and generating a unified multimodal requirement representation document based on the requirement understanding information and the conflict point list.

[0017] Requirements Completeness Check: A requirements knowledge base is established based on the existing software development entity template library and corresponding rule base. This knowledge base is then matched and compared with the requirements representation document to check its completeness. Based on the results of the completeness check, a list of missing requirements and a list of covered requirements are compiled. Questions seeking decisions are raised for the missing requirements, and specific issues in the conflict point list are transformed into questions seeking decisions, so that the requesting party or the development party can answer these questions.

[0018] Specification document generation: Create the overall framework of the specification document based on the standard specification template, and populate the overall framework with content based on the requirements representation document and the answers to the questions seeking decision-making, so as to generate the specification document;

[0019] Specification consistency verification: Perform internal consistency checks, external consistency checks, and feasibility checks on specification documents to generate a consistency verification report;

[0020] Interactive specification refinement: Provides a rich text web editor to collect feedback on specification documents, analyzes the feedback using a large language model, and transforms the feedback into actionable modification suggestions for the specification documents in order to optimize them.

[0021] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the requirement understanding information specifically includes: entities, entity relationships, functional descriptions, constraints, UI elements, process nodes, data model diagrams, and dialogue content between the requester and the developer regarding software interaction and software operation processes.

[0022] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the steps of understanding and fusing multimodal data specifically include: the multimodal data includes: text data, image data, audio data, and video data; the content understanding operation includes: for text data, using an NLP model to extract a first parsing document containing entities, entity relationships, software function descriptions, and software constraints; for image data, using a CV model to identify a second parsing document containing software UI elements, software process nodes, and software data model diagrams; and for audio and video data, using speech-to-text and video analysis techniques to extract dialogue information and software operation process information to generate... The third parsing document; timestamp alignment and semantic alignment are performed on the first, second, and third parsing documents, followed by conflict detection. Conflict detection specifically includes: checking whether the color, size, and status attributes of related entities are consistent, and marking inconsistent entities as conflict points; checking whether the entities or functions described in one modality are consistent with those in another modality, and marking inconsistent entities as conflict points; checking whether there are contradictions in the business processes or rules described in different modalities, marking contradictory points as conflict points, and generating a conflict point list; and generating a multimodal unified requirement representation document based on the information obtained from the content understanding operation and the conflict point list.

[0023] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the requirement integrity check step specifically includes:

[0024] Check for conflicts between the requirements representation document and the knowledge base: Based on the non-functional descriptions mentioned in the requirements representation document and the list of non-functional requirements in the knowledge base, perform keyword matching or quantitative indicator analysis to determine whether there are contradictions; match and compare the domain-related descriptions in the requirements representation document with the industry standard library in the knowledge base to check whether mandatory requirements are violated; for some requirements in the requirements representation document that do not have corresponding content in the knowledge base, extract the description differences of the same function or attribute in different modal inputs and determine them as cross-modal conflicts;

[0025] Check the completeness of the requirements representation document: For data objects present in the requirements representation document, check whether they are explicitly mentioned as being associated with "create, read, update, delete" operations. If an operation is found to be missing, mark it as potentially missing. For key user processes in the requirements representation document, compare them with standard process templates in the knowledge base to check whether any key steps are missing. For functional entity types in the requirements representation document, load the relevant constraint entries from the NFR check table, and then check whether each constraint entry exists in the requirements representation document.

[0026] Clarification Question Generation: Generate questions seeking decisions based on the list of conflict points, serving as clarification questions; generate questions seeking decisions based on the conflict between the requirement statement document and the knowledge base, serving as clarification questions; generate questions seeking decisions based on the missing items in the requirement statement document, serving as clarification questions.

[0027] Requirements integrity report generation: Detailed list of covered requirements, detected conflicts, missing items, severity and source rules, integrity score, and a list of clarification issues.

[0028] According to the multimodal understanding-based requirement specification generation method disclosed in this invention, preferably, the overall framework for creating a specification document based on a standard specification template specifically includes: functional specifications: generating user stories, functional descriptions, inputs and outputs, behavioral constraints, and acceptance criteria based on the requirement representation document; non-functional specifications: generating performance requirements, security requirements, and usability requirements based on the requirement representation document and answers to questions seeking decision-making; interface specifications: defining API endpoints, data models, and protocol specifications; and testing specifications: generating test scenarios and expected results based on acceptance criteria.

[0029] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the internal consistency check specifically includes: using a large language model to infer whether there are logical contradictions within the specification document, or performing logical contradiction detection according to a predefined set of contradiction rules.

[0030] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the external consistency check specifically includes: comparing the specification document with an existing specification library to check for conflicts with existing interfaces and data models. When an existing specification library is lacking, the external consistency check is to compare with industry standards.

[0031] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the feasibility check specifically includes: comparing the technical requirements in the specification document with the preset technology stack capabilities, and evaluating whether the requirements in the specification document are feasible under the current technology stack.

[0032] According to the requirement specification generation method based on multimodal understanding disclosed in this invention, preferably, the interactive specification refinement step specifically includes:

[0033] Feedback Understanding: A rich text web editor is provided, allowing users to highlight a section of specification text and add comments, or edit it directly. A lightweight text classification model or a large language model optimized for prompts is used to analyze the feedback information to determine whether the user's intent is to "clarify ambiguity," "correct errors," "add details," "add functionality," or "delete functionality." Entities and entity attributes are then extracted from the feedback information. Finally, based on the parsed intent, the large language model automatically converts ambiguous feedback into specific, actionable modification suggestions for the specification document.

[0034] Standardized updates: Directly overwriting the original document is not allowed. Instead, a preview of the changes is presented, and the user confirms or rejects the automatically generated modification suggestions. After user confirmation, Git is used in the background to manage spec.md. Each update executes git commit and generates semantic commit messages. Once a new version of spec.md is committed, the internal consistency check is automatically retried.

[0035] Historical tracking: Record every change to the standard to form a traceable evolutionary history and change description.

[0036] A second aspect of the present invention discloses a requirement specification generation system based on multimodal understanding, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory to implement the requirement specification generation method based on multimodal understanding as described above.

[0037] The beneficial effects of the present invention include at least the following:

[0038] (1) From “information silos” to “full integration”: This invention integrates the demand information scattered in audio, video, images and text into a unified and conflict-free whole through cross-modal fusion technology, thus solving the problem of information omission and conflict from the source.

[0039] (2) From “passive recording” to “active completion”: This invention transforms passive recording into active inquiry through a requirement integrity check and clarification question generation mechanism, which can uncover hidden and missing requirement points (especially non-functional requirements), greatly improving the integrity of the specification.

[0040] (3) From “single view” to “complete blueprint”: The “complete specification document” generated by this invention far exceeds the limitations of existing technologies that only generate BPMN, and provides all the context (functional, non-functional, API, data, test) required for AI code generation, making it a truly executable “development blueprint”.

[0041] (4) From “one-time delivery” to “continuous evolution”: This invention constructs a closed loop of requirement iteration through consistency verification and interactive refinement engine. The specification is no longer a rigid document, but a “living document” that can continuously evolve, verify and track with feedback, perfectly matching the agile development process. Attached Figure Description

[0042] Figure 1 A flowchart illustrating a requirement specification generation method based on multimodal understanding according to an embodiment of the present invention is shown.

[0043] Figure 2 A schematic diagram of the system architecture corresponding to the requirement specification generation method based on multimodal understanding according to another embodiment of the present invention is shown.

[0044] Figure 3 A schematic block diagram of a requirement specification generation system based on multimodal understanding according to an embodiment of the present invention is shown. Detailed Implementation

[0045] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] This invention aims to address the problems caused by the existing technology's "shallow transformation", "single output" and "lack of verification", and proposes a new specification generation paradigm based on "deep understanding and fusion" and with the capabilities of "completeness check" and "iterative refinement".

[0047] like Figure 1 As shown, according to an embodiment of the present invention, a method for generating requirements specifications based on multimodal understanding is disclosed, comprising:

[0048] Step S1, Understanding and Fusion of Multimodal Data: Receive multimodal data containing the requirements information of the target software; perform content understanding operations on the data of each modality using the corresponding content extraction tool to obtain the requirements understanding information of the target software; perform conflict detection on the requirements understanding information, mark the conflict points, and generate a conflict point list; generate a unified multimodal requirements representation document based on the requirements understanding information and the conflict point list.

[0049] Step S2, Requirements Completeness Check: A requirements knowledge base is established based on the existing software development entity template library and corresponding rule base. The requirements knowledge base is matched and compared with the requirements representation document to check the completeness of the requirements representation document. Based on the results of the completeness check, a list of missing requirements and a list of covered requirements are compiled. Questions seeking decisions are raised for the missing requirements, and specific issues involved in the conflict point list are transformed into questions seeking decisions, so that the requesting party or the development party can answer these questions.

[0050] Step S3, Specification Document Generation: Create the overall framework of the specification document based on the standard specification template, and fill in the content of the overall framework according to the requirements representation document and the answers to the questions seeking decision-making, so as to generate the specification document;

[0051] Step S4, Specification Consistency Verification: Perform internal consistency checks, external consistency checks, and feasibility checks on the specification document to generate a consistency verification report;

[0052] Step S5, Interactive Specification Refinement: Provide a rich text web editor to collect feedback on the specification document, analyze the feedback using a large language model, and convert the feedback into actionable modification suggestions for the specification document in order to optimize the specification document.

[0053] According to the above embodiments, the requirements understanding information further includes: entities, entity relationships, functional descriptions, constraints, UI elements, process nodes, data model diagrams, and dialogue content between the requester and the developer regarding software interaction and software operation processes.

[0054] According to the above embodiments, the steps for understanding and fusing multimodal data further include:

[0055] Multimodal data includes: text data, image data, audio data, and video data;

[0056] Content understanding operations include: for text data, using an NLP model to extract a first parsing document containing entities, entity relationships, software function descriptions, and software constraints; for image data, using a CV model to identify a second parsing document containing software UI elements, software process nodes, and software data model diagrams; and for audio and video data, using speech-to-text and video analysis techniques to extract dialogue information and software operation process information to generate a third parsing document.

[0057] The first, second, and third parsing documents are time-stamped and semantically aligned, followed by conflict detection. Conflict detection specifically includes: checking for consistency in the color, size, and status attributes of related entities, marking inconsistent entities as conflict points; checking for consistency between entities or functions described in one modality and entities or functions in another modality, marking inconsistent entities as conflict points; checking for contradictions in business processes or rules described in different modalities, marking contradictory points as conflict points, and generating a conflict point list; and generating a unified multimodal requirement representation document based on the information obtained from content understanding operations and the conflict point list.

[0058] According to the above embodiments, the requirement integrity check step further includes:

[0059] Check for conflicts between the requirements document and the knowledge base: Based on the non-functional descriptions mentioned in the requirements document and the list of non-functional requirements in the knowledge base, perform keyword matching or quantitative indicator analysis to determine if contradictions exist; match and compare the domain-related descriptions in the requirements document with the industry standard library in the knowledge base to check for violations of mandatory requirements; for some requirements in the requirements document that do not have corresponding content in the knowledge base, extract the description differences of the same function or attribute in different modal inputs and determine them as cross-modal conflicts; check the completeness of the requirements document: for data objects existing in the requirements document, check whether they have been explicitly mentioned and related to "creating, reading"... The system is linked to "update," "delete," and "update" operations. If an operation is missing, it is marked as a potential missing operation. For key user flows in the requirements representation document (NRV), they are compared with standard flow templates in the knowledge base to check for missing key steps. For functional entity types in the NRV, relevant constraint entries are loaded from the NFR check table, and then each constraint entry is checked to see if it exists in the NRV. Clarification question generation: Questions seeking decisions are generated based on the conflict point list, as clarification questions; questions seeking decisions are generated based on conflicts between the NRV and the knowledge base, as clarification questions; questions seeking decisions are generated based on missing items in the NRV, as clarification questions. Requirements integrity report generation: A detailed list of covered requirements points, detected conflict list, missing item list, severity and source rules, integrity score, and clarification question list are generated.

[0060] According to the above embodiments, the overall framework for creating specification documents based on standard specification templates further includes: Functional specifications: generating user stories, functional descriptions, inputs and outputs, behavioral constraints, and acceptance criteria based on the requirements representation document; Non-functional specifications: generating performance requirements, security requirements, and usability requirements based on the requirements representation document and answers to questions seeking decision-making; Interface specifications: defining API endpoints, data models, and protocol specifications; Test specifications: generating test scenarios and expected results based on acceptance criteria.

[0061] According to the above embodiments, the internal consistency check further includes: using a large language model to check for logical contradictions within the specification document, or performing logical contradiction detection based on a predefined set of contradiction rules. The external consistency check specifically includes: comparing the specification document with an existing specification library to check for conflicts with existing interfaces and data models; when an existing specification library is lacking, the external consistency check compares it with industry standards. The feasibility check specifically includes: comparing the technical requirements in the specification document with preset technology stack capabilities to assess whether the requirements in the specification document are feasible under the current technology stack.

[0062] According to the above embodiments, the further detailed steps of interactive specification refinement specifically include: Feedback understanding: providing a rich text web editor that allows highlighting a section of specification text and adding comments, or direct editing; using a lightweight text classification model or a large language model optimized for prompts to analyze the feedback information to determine whether the user's intent is "clarifying ambiguity," "correcting errors," "supplementing details," "adding functionality," or "deleting functionality"; then extracting entities and entity attributes from the feedback information, and finally, the large language model automatically converts the ambiguous feedback into specific, actionable modification suggestions for the specification document based on the parsed intent; Specification updates: not allowing direct overwriting of the original document, but presenting a preview of the changes, allowing the user to confirm or reject the automatically generated modification suggestions; after user confirmation, using Git to manage spec.md in the background; executing git commit for each update and generating semantic commit messages; once a new version of spec.md is committed, automatically re-triggering internal consistency checks; History tracking: recording every change to the specification to form a traceable evolution history and change descriptions.

[0063] like Figure 2 As shown, according to another embodiment of the present invention, a system architecture diagram corresponding to the requirement specification generation method based on multimodal understanding of the above embodiments is also disclosed. This embodiment will be based on... Figure 2 The system architecture diagram shown illustrates a specific implementation of the requirement specification generation method based on multimodal understanding proposed in this invention, including:

[0064] Step 1: Multimodal Input Understanding Engine (corresponding to the steps of understanding and fusing multimodal data)

[0065] Input fragmented, unstructured multimodal data, such as: Text: Requirements documents (Word), user stories, emails, chat logs. Images: UI design drafts (Figma, Sketch), hand-drawn flowcharts, architecture diagrams. Audio: Product meeting recordings, user interview recordings. Video: Product operation demonstrations, user operation screen recordings. Process the above multimodal data as follows:

[0066] 1. Single-mode analysis:

[0067] Text understanding: Using NLP models to extract entities, relationships, functional descriptions, and constraints.

[0068] The following processing steps are applied to inputs such as requirements documents (.docx, .pdf), user stories (.txt), emails, and exported chat logs (.json, .csv), ultimately outputting a set of text semantic units:

[0069] (1) Preprocessing and word segmentation: Clean the input text (such as removing special characters), segment words, tag parts of speech and standardize terms (e.g., unify "login" as "login").

[0070] (2) Named entity recognition: A sequence labeling model based on pre-trained language models (such as BERT, RoBERTa) is used to extract specific types of entities from the text.

[0071] (3) Relation extraction: Using dependency parsing or a relation classifier based on a pre-trained model, the semantic relationships between entities extracted in step (2) Named entity recognition are identified. For example, from the sentence "User clicks the login button", the triple <user, execution, clicks the login button> can be extracted.

[0072] Image understanding: Using CV models (such as OCR and layout analysis) to identify UI elements, process nodes, and data model diagrams.

[0073] Input UI design drafts (Figma, Sketch files, or exported PNGs), hand-drawn flowcharts, and system architecture diagrams (JPG, PNGs) and process them as follows to output a set of image semantic units:

[0074] (1) UI element / graphic detection: Use deep learning-based object detection models (such as YOLO or Faster R-CNN) to scan the image, identify and locate the graphic elements, such as buttons, input boxes, drop-down menus, flowchart nodes (start, end, process), database symbols, etc.

[0075] (2) Optical character recognition: Within the detected element area or the entire image, an OCR engine (such as PaddleOCR) is applied to extract all text content.

[0076] (3) Layout and structure analysis: By analyzing the relative position, size, overlap and alignment of the detected elements, the layout structure of the interface (such as navigation bar, sidebar, content area) or the logical flow of the flowchart can be inferred.

[0077] Audio / Video Understanding: Using ASR (Speech-to-Text) and video analytics technologies, key dialogues, operational processes, and user interactions are extracted.

[0078] The following processing steps are performed on the input of product meeting recordings (.wav, .mp3), user interview recordings, and product operation demonstration videos (.mp4), and the final output is a set of audio / video semantic units.

[0079] (1) Speech-to-text: Using an automatic speech recognition model (such as OpenAI Whisper), audio streams or audio tracks in videos are converted into timestamped text records. This step can process different speakers simultaneously (if the model supports it) and output segmented text.

[0080] (2) Video-specific keyframe analysis and operation sequence reconstruction: For video input, the following additional operations are performed:

[0081] Keyframe extraction: Extract video frames at fixed time intervals or based on scene change detection.

[0082] Frame analysis: Each keyframe is sent to the image understanding module (see above) to identify the UI state at that moment.

[0083] Interaction sequence generation: By combining UI state changes and cursor / touchpoint movement trajectories (if visible) in consecutive frames, the user's operation flow is reconstructed, for example: ['On the main page', 'Click the search box', 'Enter keywords', 'Click the search button'].

[0084] 2. Cross-modal fusion and conflict detection:

[0085] All information parsed from the single modality is time-stamped and semantically aligned before collision detection is performed.

[0086] Level 1: Timestamp Alignment (Coarse Screening, Creating Candidate Association Sets): The system divides all semantic units into different temporal context groups. For time-series streaming data (audio / video): The system creates a time window centered on the timestamp of a semantic unit (such as a sentence spoken in audio), and then marks other units in all modalities that fall within this window as candidate association sets. For static data (images / text), these data are placed in a "global context" pool. They will undergo subsequent semantic alignment with the candidate sets of all time windows and with each other. This significantly reduces the number of semantic units that need to be involved in the calculation, avoids unnecessary semantic comparisons spanning large time periods, improves system efficiency, and ensures the contextual relevance of associations. Technical Effect: Through this step, the system explicitly separates "the design diagram A shown when discussing function A" from "the requirement document fragment B mentioned when discussing function B," laying the foundation for subsequent precise matching.

[0087] Level 2: Semantic Alignment (Fine Matching, Establishing Precise Associations): The system uses a multimodal pre-trained model to map units in the candidate set to a shared semantic space and calculate vector similarity. The system prioritizes detailed semantic similarity calculations within the candidate set within the same time window. Simultaneously, the system also calculates similarity between static data in the "global context" pool and the candidate sets of each time window. When the similarity exceeds a certain threshold, the system ultimately establishes a robust association between the two units. This allows the system to determine which units within the "candidate association set" generated by timestamp alignment truly have a semantic referential relationship. Technical Effect: This step ensures the accuracy of the associations; the system not only knows that these pieces of information appeared simultaneously but also knows precisely that they describe the same thing.

[0088] Level 3: Conflict Detection (Consistency Verification): The system iterates through all the associations established in Level 2, performing consistency checks on the associated entities. It ultimately outputs a conflict list, detailing the conflict type, the associated units involved, the conflict content, and its severity level. This serves as a quality assurance layer, proactively identifying and marking inconsistencies in the merged data.

[0089] Collision detection mainly includes:

[0090] Attribute conflict detection: Check whether related entities are consistent in key attributes (such as color, size, status).

[0091] Existence conflict detection: Examine whether an entity or function described in detail in one modality is completely missing or explicitly denied in another related modality.

[0092] Logical conflict detection: Check for contradictions in business processes or rules described in different modalities.

[0093] Finally, after cross-modal fusion and conflict detection, a "unified requirement representation" (such as JSON format) is output, which contains all extracted requirement points, their relationships, and marked conflict points.

[0094] Step 2: Requirements Completeness Check Engine (corresponding to the requirements completeness check steps)

[0095] The unified requirement representation obtained in step one is input. This requirement representation is a structured JSON object containing extracted entities, processes, related links, and a list of detected conflicts. It is then checked using a requirement integrity detection engine, which has a pre-defined knowledge base and rule base. The following processing is performed:

[0096] 1. Knowledge base comparison:

[0097] The engine has a built-in configurable "requirements knowledge base" that contains common non-functional requirements (such as security, performance, and availability) and industry standard constraints.

[0098] Requirements Knowledge Base Structure: Entity Template Library: Predefined sets of required attributes for standard system components. For example, a "User Login" function template might contain fields such as {Authentication Factor: [], Session Timeout: "", Error Handling: ""}. Non-functional Requirements Checklist: A list of common constraints organized by category (performance, security, availability, reliability). For example: ["Response Time < 2s", "Supports HTTPS", "Compatible with Chrome Browser", "Data Backup Strategy"].

[0099] Industry standard constraint library: Compliance requirements for specific fields (such as healthcare and finance). For example, relevant clauses of the Payment Card Industry Data Security Standard (PCI DSS).

[0100] Rule base: Contains a series of "IF-THEN" logical rules used to trigger checks.

[0101] The core of knowledge base comparison is to compare the "input requirement representation" with the standard constraints in the "requirement knowledge base" to identify contradictory content. Specifically, the comparison is divided into three categories:

[0102] (1) Comparison of non-functional requirement conflicts:

[0103] Compare the non-functional descriptions mentioned in the input requirements (such as "response time < 5s" and "supports HTTP protocol") with the list of non-functional requirements in the knowledge base (such as "response time < 2s" and "must support HTTPS"). Directly match keywords or quantitative indicators to determine if there is a contradiction.

[0104] (2) Comparison of industry standard constraints:

[0105] Input the domain-related description in the requirements (e.g., "data storage period is 1 year" for a medical system) vs. the industry standard library in the knowledge base. Based on the system's domain (e.g., medical, financial), automatically match the corresponding industry standard clauses and check for violations of mandatory requirements.

[0106] (3) Cross-modal input conflict comparison:

[0107] The requirements after multimodal input transformation (e.g., text says "UI buttons should be red," audio says "buttons should be blue") vs. there are no explicit knowledge base entries, but the implicit rule of "the same attribute must be consistent" needs to be applied. Identify the description differences of the same function / attribute in different modal inputs and determine them as "cross-modal conflicts".

[0108] 2. Integrity check

[0109] The system compares the input requirement representation with the knowledge base to check the completeness of functional requirements (whether they cover all user scenarios) and the completeness of non-functional requirements (whether necessary performance and security constraints are missing). The engine performs the following checks, which essentially involve pattern matching between the input requirement representation and the knowledge base template.

[0110] (1) Functional scenario coverage check:

[0111] CRUD matrix analysis: For each identified core "data object" (such as "order" or "user"), the system checks whether it has been explicitly mentioned as being associated with "create, read, update, or delete" operations. If an operation is found to be missing (e.g., there is "create order" but no "cancel / update order"), it is marked as potentially missing.

[0112] User journey integrity: For identified key user flows (such as the "shopping flow"), the system compares them with standard flow templates in the knowledge base to check for missing key steps (e.g., the "payment" step is missing after "submitting the order").

[0113] (2) Non-functional requirement integrity check:

[0114] Based on the identified functional entity type, the system loads the relevant constraint entries from the NFR check table and then checks one by one whether these entries are covered in the input requirement representation.

[0115] 3. Clarify question generation

[0116] For the detected conflict points (from step one), generate an issue (e.g., "The UI displays a red button, but the recording mentions a blue one; which one should be considered correct?").

[0117] For any missing points detected (from step two), generate an issue (e.g., "No rate limit was found for the login interface, please add it." or "No password complexity requirement was defined, please add it.").

[0118] This module translates the inspection results into user-oriented natural language questions, driving the interaction.

[0119] (1) Clarification of the conflict:

[0120] The system transforms the technical descriptions in the conflict logs into neutral, decision-making questions.

[0121] Example: Conflict record: {Type: "Attribute conflict", Entity: "Login button", Details: "Color: Text (blue) vs Image (red)"}

[0122] Question generated: "There is a conflicting information regarding the color of the 'Login Button'. The documentation describes it as 'blue,' while the UI design draft shows it as 'red.' Which one should be taken as the correct one?"

[0123] (2) Clarification of missing items:

[0124] Based on the missing type and context, generate specific, actionable questions.

[0125] Example: Missing item: The feature "removing items from the shopping cart" is missing.

[0126] Question generated: "The system has recognized the 'shopping cart' function, but no description of 'removing items from the shopping cart' has been found. Do users need this function? If so, please provide details on how to use it."

[0127] Missing item: The non-functional requirement "cryptographic complexity strategy" is missing.

[0128] The generated question is: "For the 'User Login' function, no password complexity requirements (such as length or character type) are defined. Please provide specific requirements or confirm that there are no special requirements."

[0129] 4. Output a requirements completeness report

[0130] Output an integrity check report and a "list of clarification questions".

[0131] For example: Output a machine-readable report (such as JSON) that details: the requirements covered, the list of detected conflicts, the list of newly discovered potential missing items, and for each missing item, its type (functionality / NFR), severity, and source rule, as well as the overall completeness score (e.g., percentage based on coverage).

[0132] List of questions for clarification:

[0133] A prioritized list of natural language questions for direct user interaction. Each question includes: Question ID, Question content, Question type (conflict clarification / feature missing / NFR missing), associated requirement entity or conflict ID, and priority (usually determined by the severity of the conflict or the criticality of the missing item).

[0134] This invention transforms the traditionally subjective requirements review process, which relies on expert experience, into an automated, systematic, and traceable engineering quality control process, significantly improving the completeness and reliability of requirements engineering.

[0135] Step 3: Specification Generation Engine (corresponding to the specification document generation steps)

[0136] Enter the "Unified Requirements Representation" (from Step 1) and the user's answers to the "Clarification Questions List" (from Step 2). Create the overall framework for the specification document based on a standard specification template (such as OpenSpec or spec-kit format). Fill in the following:

[0137] Functional Specifications: Based on the requirements representation, generate user stories, function descriptions, inputs and outputs, behavioral constraints, and acceptance criteria.

[0138] Non-functional specifications: Based on the requirements and user clarification, generate requirements for performance (response time), security (encryption, permissions), availability, etc.

[0139] Interface Specification: Initially define API endpoints, data models (Schema), and protocol specifications.

[0140] Test specifications: Based on the acceptance criteria, generate key test scenarios and expected results.

[0141] Finally, output a complete, structured, AI-oriented specification document (such as Markdown format).

[0142] Step 4: Standardize the consistency verification engine (corresponding to the standard consistency verification steps)

[0143] Input the generated specification document (from step three), then perform internal consistency checks, external consistency checks, and feasibility checks respectively, finally outputting a consistency verification report that highlights potential contradictions and risks. Perform the following processing:

[0144] Internal consistency checks: Using formal methods or LLM reasoning, check for logical inconsistencies within the specification document (e.g., defining user ID as an integer in one place and as a string in another). Inconsistency checks can be performed based on a predefined set of inconsistency rules.

[0145] External consistency check: The generated specification is compared with the existing system's specification library (if it exists) to check for conflicts with existing interfaces and data models. When an existing specification library is not available, the external consistency check is not skipped, but its focus shifts from "compatibility with legacy systems" to "alignment with industry standards".

[0146] Feasibility check: Compare the technical requirements in the specification with the preset technology stack capabilities to make a preliminary assessment of whether the requirements in the specification are feasible under the current technology stack.

[0147] Step 5: Interactive Specification Refinement Engine (corresponding to the interactive specification refinement steps)

[0148] After inputting the specification document (from step three) and continuous user feedback (such as modification suggestions and supplementary information), and performing the following processing, output a refined, latest version of the specification document, along with a change log. The processing is as follows:

[0149] Feedback Understanding: Provide a rich text web editor (e.g., Markdown-based) that allows users to highlight a section of specification text and add comments, or edit it directly. The interface clearly displays the current version and comment thread. Then, use a lightweight text classification model or a suggestion-optimized LLM to analyze user feedback: determine whether the user intent is "clarifying ambiguity," "correcting errors," "adding details," "adding functionality," or "removing functionality." Extract key entities (such as "PDF," "Excel," "response time") and their attributes ("<200ms") from the feedback. Finally, based on the parsed intent, the LLM automatically converts ambiguous feedback into specific, actionable modification suggestions for the specification document.

[0150] Specification Updates: The system will not allow direct or arbitrary overwriting of the original document. It presents a preview of the changes to the user (similar to GitHub's Pull Request interface), allowing the user to confirm or reject the automatically generated modification suggestions. Users can also make manual adjustments. After user confirmation, the system uses Git in the background to manage spec.md. Each update executes a git commit and generates semantic commit messages (generated by LLM based on the changes). Once the new version of spec.md is committed, the system automatically re-triggers the consistency verification in step (four).

[0151] Historical tracking: Record every change to the standard to form a traceable evolutionary history and change description.

[0152] According to another embodiment of the present invention, the differences between the requirement specification generation method based on multimodal understanding provided in the above embodiments and the prior art are also described, taking an example of a development team needing to add a "user refund" function to an e-commerce app:

[0153] Multimodal inputs include: Video: A screen recording of a competitor's app demonstrating the process of "clicking the order -> requesting a refund -> selecting a reason -> submitting"; Audio: A recording of a product manager stating, "...refunds must be processed within 48 hours...only shipped orders are eligible for refunds..."; Image: A Figma design showing an "upload voucher" image upload button on the refund interface.

[0154] Text: Email from the legal team: "According to regulations, digital goods (such as e-books) are non-refundable once sold."

[0155] Existing technological approaches:

[0156] Transformation: The system receives the above input and "transforms" it into structured requirements.

[0157] Output: The system may ultimately generate only one BPMN flowchart, which is based on a video demonstration and shows the process of "Request a Refund -> Select Reason -> Submit".

[0158] Conflict: The system cannot handle the complex logic between audio (shipped orders) and email (digital goods).

[0159] Omission: The BPMN flowchart is completely missing key information such as "process within 48 hours" (performance NFR), "upload credentials" (UI-specific function), and "digital goods are non-refundable" (business constraint).

[0160] Result: Developing based on this BPMN will result in serious functional deficiencies and business violations.

[0161] Implementation of the present invention:

[0162] Understanding and Integration: The engine extracts the process from the video, constraints from the audio (T < 48h, status ==shipped), functionality from the image (upload_image), and constraints from the text (category !=digital). The system then links all these scattered requirements to the core intent of "refund".

[0163] Integrity check: The system found that "Refund Reason" appeared in the video but was not defined elsewhere. The system first generated clarification questions: "1. Please provide a detailed list of options for 'Refund Reason'; Is 'Upload Voucher' a required field; No user message was found for refund failure, please provide it."

[0164] Specification Generation: After the development team answers the clarification questions, the engine generates a complete specification document, which includes: Functional Specification: The user selects an item from the "Refund Reason" list;

[0165] Business constraints: IF (order.status == 'shipped' AND product.category != 'digital') THEN allow_refund ELSE reject.

[0166] Non-functional specification: processing_time < 48h.

[0167] API specification: POST / api / v1 / refunds { orderId, reason, images[]}.

[0168] Acceptance criteria: Test digital goods orders, expecting refunds to fail.

[0169] Validation and Refinement: The team reviewed the specification in the interactive interface, with the legal department adding, "'Virtual services' are also not eligible for refunds." The interactive specification refinement engine then updated the specification document and generated change logs.

[0170] like Figure 3 As shown, according to another embodiment of the present invention, a requirement specification generation system 300 based on multimodal understanding is also disclosed, comprising: a memory 301 for storing program instructions; and a processor 302 for calling the program instructions stored in the memory to implement the requirement specification generation method based on multimodal understanding as described in the above embodiment.

[0171] In summary, this invention provides a method for deep understanding and cross-modal fusion of multimodal requirements: it can not only parse text, images, and audio / video separately, but more importantly, it can fuse multi-source information and proactively identify and report conflicts between modalities. This invention provides a knowledge-based algorithm for requirement integrity checking and clarification question generation: it can automatically check for missing functional points and non-functional constraints in requirement representations and proactively generate "clarification questions" for user interaction to complete the requirements. This invention provides an AI-oriented method for automatically generating complete specification documents (Specs): unlike existing technologies that only generate BPMNs, this invention can generate complete, structured specification documents containing functional, non-functional, API, data models, and test cases. This invention provides an internal and external consistency verification mechanism for specifications: it can automatically detect whether there are logical contradictions within the generated specification and whether it conflicts with existing system specifications. This invention provides an interactive specification evolution and refinement tracking mechanism: it provides a human-computer collaborative interface, supports user feedback, specification version iteration, and change history tracking.

[0172] All or part of the steps in the various methods of the above embodiments can be implemented by a program controlling the relevant hardware. The program can be stored in a readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0173] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A requirement specification generation method based on multimodal understanding, characterized in that, include: Understanding and fusing multimodal data: receiving multimodal data containing the requirements information of the target software; performing content understanding operations on the data of each modality using corresponding content extraction tools to obtain the requirements understanding information of the target software; performing conflict detection on the requirements understanding information, marking conflict points, and generating a conflict point list; Based on the aforementioned requirements understanding information and the list of conflict points, a multimodal unified requirements representation document is generated. Requirements Completeness Check: A requirements knowledge base is established based on the existing software development entity template library and corresponding rule base. The requirements knowledge base is matched and compared with the requirements representation document to check the completeness of the requirements representation document. Based on the results of the completeness check, a list of missing requirements and a list of covered requirements are compiled. Questions seeking decisions are raised for the missing requirements, and specific issues involved in the conflict point list are transformed into questions seeking decisions, so that the requester or developer can answer the questions seeking decisions. Standard document generation: Create an overall framework for the standard document based on the standard specification template, and populate the overall framework with content based on the requirements representation document and the answers to the questions seeking decision-making, so as to generate the standard document; Specification consistency verification: Perform internal consistency checks, external consistency checks, and feasibility checks on the specification document to generate a consistency verification report; Interactive specification refinement: A rich text web editor is provided to collect feedback on the specification document, and a large language model is used to analyze the feedback, converting the feedback into actionable modification suggestions for the specification document in order to optimize the specification document; The steps of the requirement integrity check specifically include: Check for conflicts between the requirements representation document and the knowledge base: Based on the non-functional descriptions mentioned in the requirements representation document and the list of non-functional requirements in the knowledge base, perform keyword matching or quantitative indicator analysis to determine whether there are contradictions; match and compare the domain-related descriptions in the requirements representation document with the industry standard library in the knowledge base to check whether mandatory requirements are violated; for some requirements in the requirements representation document that do not have corresponding content in the knowledge base, extract the description differences of the same function or attribute in different modal inputs and determine them as cross-modal conflicts; Check the completeness of the requirements representation document: For data objects existing in the requirements representation document, check whether they are explicitly mentioned as being associated with "create, read, update, delete" operations. If an operation is found to be missing, mark it as potentially missing. For key user processes in the requirements representation document, compare them with standard process templates in the knowledge base to check whether any key steps are missing. For functional entity types in the requirements representation document, load the relevant constraint entries from the NFR check table, and then check whether each constraint entry exists in the requirements representation document. Clarification Question Generation: Generate questions seeking decisions based on the list of conflict points, serving as clarification questions; generate questions seeking decisions based on the conflict between the requirement statement document and the knowledge base, serving as clarification questions; generate questions seeking decisions based on the missing items in the requirement statement document, serving as clarification questions. Requirements integrity report generation: Detailed list of covered requirements, detected conflicts, missing items, severity and source rules, integrity score, and a list of clarification issues.

2. The requirement specification generation method based on multimodal understanding according to claim 1, characterized in that, The required understanding information specifically includes: The target software includes entities, entity relationships, functional descriptions, constraints, UI elements, process nodes, data model diagrams, and dialogue content between the requester and the developer regarding software interaction and software operation processes.

3. The requirement specification generation method based on multimodal understanding according to claim 2, characterized in that, The steps for understanding and fusing the multimodal data specifically include: The multimodal data includes: text data, image data, audio data, and video data; The content understanding operation includes: for text data, using an NLP model to extract a first parsing document containing entities, entity relationships, software function descriptions, and software constraints; for image data, using a CV model to identify a second parsing document containing software UI elements, software process nodes, and software data model diagrams; and for audio and video data, using speech-to-text and video analysis technologies to extract dialogue information and software operation process information to generate a third parsing document. The first, second, and third parsed documents are time-stamped and semantically aligned, followed by conflict detection. Conflict detection specifically includes: checking whether the color, size, and status attributes of associated entities are consistent, and marking inconsistent entities as conflict points; checking whether the entities or functions described in one modality are consistent with those in another modality, and marking inconsistent entities as conflict points; checking whether there are contradictions in the business processes or rules described in different modalities, marking contradictory points as conflict points, and generating a conflict point list; and generating a multimodal unified requirement representation document based on the information obtained from the content understanding operation and the conflict point list.

4. The requirement specification generation method based on multimodal understanding according to claim 1, characterized in that, The overall framework for creating specification documents based on standard specification templates specifically includes: Functional Specifications: Based on the requirements representation document, generate user stories, function descriptions, inputs and outputs, behavioral constraints, and acceptance criteria; Non-functional specifications: Based on the requirements representation document and the answers to questions seeking decisions, generate performance requirements, security requirements, and availability requirements; Interface Specification: Defines API endpoints, data models, and protocol specifications; Test specifications: Generate test scenarios and expected results based on acceptance criteria.

5. The requirement specification generation method based on multimodal understanding as described in claim 1, characterized in that, The internal consistency check specifically includes: Use large language models to infer and check for logical inconsistencies within specification documents, or perform logical inconsistency detection based on a predefined set of inconsistency rules.

6. The requirement specification generation method based on multimodal understanding as described in claim 1, characterized in that, The external consistency check specifically includes: The specification document is compared with the existing specification library to check for conflicts with existing interfaces and data models. When the existing specification library is not available, the external consistency check is to compare with industry standards.

7. The requirement specification generation method based on multimodal understanding as described in claim 1, characterized in that, The feasibility check specifically includes: The technical requirements in the specification document are compared with the preset technology stack capabilities to assess whether the requirements in the specification document are feasible under the current technology stack.

8. The requirement specification generation method based on multimodal understanding as described in claim 1, characterized in that, The steps for refining the interactive specification specifically include: Feedback Understanding: A rich text web editor is provided, allowing users to highlight a section of specification text and add comments or edit it directly. A lightweight text classification model or a prompt-optimized large language model is used to analyze the feedback information to determine whether the user's intent is to "clarify ambiguity," "correct errors," "add details," "add functionality," or "delete functionality." Entities and entity attributes are then extracted from the feedback information. Finally, based on the parsed intent, the large language model automatically converts ambiguous feedback into specific, actionable modification suggestions for the specification document. Standardized updates: Directly overwriting the original document is not allowed. Instead, a preview of the changes is presented, and the user confirms or rejects the automatically generated modification suggestions. After user confirmation, Git is used in the background to manage spec.md. Each update executes git commit and generates semantic commit messages. Once a new version of spec.md is committed, the internal consistency check is retried. Historical tracking: Record every change to the standard to form a traceable evolutionary history and change description.

9. A requirement specification generation system based on multimodal understanding, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke the program instructions stored in the memory to implement the requirement specification generation method based on multimodal understanding as described in any one of claims 1 to 8.

Images