Multi-source software engineering document analysis and evaluation report generation method and system based on generative AI

Abstract

The invention discloses a multi-source software engineering document analysis and evaluation report generation method and system based on generative AI, and the method comprises the steps: constructing a cross-document relation graph, and storing the cross-document relation graph in a knowledge base; establishing a dynamic evaluation rule engine, presetting a document format standard template in the industry, setting evaluation dimensions, and establishing a rule executor by setting a rule for judging whether a test is completed or not according to requirements; an AI report generation module is generated in a constraint form, and a traceability mark is embedded; a final evaluation report is generated by manually checking an AI report and a knowledge base evidence chain; the system comprises a multi-source document processing module, a joint information extraction and knowledge graph construction module, a dynamic evaluation rule engine module, a constrained generative AI report generation module and a man-machine collaborative auditing feedback module. According to the method, the problems of accuracy, compliance and traceability in automatic generation of the system evaluation report are solved, and efficient, standard and audible software evaluation report output is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence and software engineering, and in particular to a multi-source software engineering document analysis and evaluation report generation method and system based on generative AI. BACKGROUND

[0002] In the process of carrying out software capability, quality characteristics, technical system interoperability and other special evaluation, information system software needs to analyze a large number of basic document data such as software requirement documents, identification evaluation reports, joint reports, evaluation cases and evaluation defect lists through manual analysis to support the formation of special test reports. The evaluation process is time-consuming (usually several weeks) and error-prone. Moreover, when manually integrating multi-source documents, key information is easily missed, and the quality of the report is greatly influenced by the experience of engineers, and the standardization degree is low. At present, traditional automated tools (such as template filling) only support structured data, and cannot understand non-text semantics, associate cross-document logic, and identify potential risks. Through information extraction (IE) technology, only isolated entities can be extracted, and there is a lack of multi-document correlation analysis capability. Through the traditional rule engine method, it is difficult to cover complex evaluation logic due to the dependence on pre-defined hard-coded rules. Therefore, it is necessary to propose an AI-driven system evaluation report generation method based on multi-level information fusion and rule constraint generation. SUMMARY

[0003] The purpose of the present application is to provide a multi-source software engineering document analysis and evaluation report generation method and system based on generative AI; to solve the problems of accuracy, compliance and traceability in the automatic generation of system evaluation reports, and to realize efficient, standard and auditable software evaluation report output.

[0004] Technical scheme: The multi-source software engineering document analysis and evaluation report generation method based on generative AI comprises the following steps:

[0005] Step 1, constructing a cross-document relationship graph of associated requirements, tests and defect links, and storing it in a knowledge base;

[0006] Step 2, establishing a dynamic evaluation rule engine, pre-setting industry document format standard templates, setting evaluation dimensions, setting rules for whether the requirements are completed, and establishing a rule executor;

[0007] Step 3, constraint generative AI report generation module, generating a report draft by calling a large model, and embedding a traceability mark;

[0008] Step 4, the man-machine cooperative review feedback module checks the AI report and the knowledge base evidence chain through manual inspection, and generates a final evaluation report.

[0009] Further, the step 1 of constructing the cross-document relationship graph comprises the following steps:

[0010] Step 1-1, the multi-source document processing module imports software requirement documents, appraisal evaluation reports, joint reports, evaluation cases, and evaluation defect list heterogeneous data, and performs text extraction and format conversion on the documents;

[0011] Step 1-2, the joint information extraction and knowledge graph construction module uses a joint entity recognition and relationship extraction model to extract entity information.

[0012] Further, the step 2 evaluation dimensions include functionality, reliability, maintainability, testability, safety, supportability, and complex environment adaptability.

[0013] Further, the step 2 includes dynamically generating a requirement tracking matrix based on the knowledge base, calculating the case distribution density, test pass rate, and overall requirement coverage.

[0014] Further, the step 3 includes setting dynamic evaluation rules, risk judgment basis, output industry standard document templates and chapter structure according to step 2, generating a report draft by calling a large model, and embedding a traceability mark.

[0015] Further, the step 4 includes marking low confidence content, correcting conclusions, supplementing rules, and feeding back errors to generate a final evaluation report.

[0016] The multi-source software engineering document analysis and evaluation report generation system based on generative AI includes:

[0017] The multi-source document processing module is used to identify manually imported software requirement documents, appraisal evaluation reports, joint reports, evaluation cases, and evaluation defect list heterogeneous data;

[0018] The joint information extraction and knowledge graph construction module is used to extract entity information, construct a cross-document relationship graph of associated requirements, tests, and defect links, and store them in a knowledge base.

[0019] The dynamic evaluation rule engine module is used to preset industry document format standard templates, set evaluation dimensions, set rules for whether requirements are completed, and establish a rule executor.

[0020] The constraint generative AI report generation module is used to generate a report draft by calling a large model according to the chapter structure, and embed a traceability mark.

[0021] The human-computer collaborative review and feedback module is used to manually check AI reports and knowledge base evidence chains, mark low confidence content, and generate a final evaluation report.

[0022] Beneficial effects: Compared with the prior art, the present application has the following significant advantages: the method provided by the present application greatly improves the efficiency of system evaluation, and the generation time of system evaluation report is shortened from several weeks to several hours; the system evaluation method can cover 100% of the preset evaluation rules, and at the same time, 100% of the "conclusion-demand-test-defect" evaluation conclusion is traceable; compared with manual analysis, the AI generated report covers 98% of the demand evaluation points, and the false positive rate is less than 5%. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flow chart of the method of the present application;

[0024] Figure 2 is a cross-document relationship graph diagram of the present application;

[0025] Figure 3 is a dynamic rule engine execution logic diagram of the present application;

[0026] Figure 4 is a generative AI prompt word construction and constraint injection diagram of the present application. DETAILED DESCRIPTION

[0027] The technical solutions of the present application will be further described below in conjunction with the drawings.

[0028] As shown in Figure 1 , the multi-source software engineering document analysis and evaluation report generation method based on generative AI provided by the present application comprises the following steps:

[0029] Step 1, the multi-source document processing module imports software requirement documents (Word), identification and evaluation reports (Word), joint test reports (Word), identification and evaluation use cases (Word), identification and evaluation defect lists (Excel) and other heterogeneous data; the joint information extraction function extracts system requirements from software requirement documents, extracts test cases from identification and evaluation use case documents, extracts identification and evaluation defects from identification and evaluation defect lists, extracts identification and evaluation test conclusions corresponding to requirements from identification and evaluation report reports, extracts joint test cases and test results from joint test reports, and forms different entity information. The entity contains system / module name, requirement ID, test case ID, defect ID and other information; the knowledge graph construction module uses the relationship extraction function to establish the relationship between requirements, test cases and defects, and constructs a cross-document relationship graph of the "requirement-test-defect" link, as shown in Figure 2 , and stores the extracted entity, relationship and other information into the knowledge base.

[0030] Step 2: The dynamic evaluation rule engine reads the industry-standard document format templates, checklists, and evaluation dimensions (functionality, reliability, maintainability, testability, security, supportability, adaptability to complex environments, etc.) and judgment criteria (failed test, passed test) that the system evaluation must follow; such as Figure 3 As shown, by setting the rule that "if either the evaluation test or the test case fails the verification, the requirement is deemed to have failed the test, or if the requirement is not associated with a test case, it is also deemed to have failed the test", a rule executor is established. Based on the "requirement-test-defect" relationship graph formed in step 1, a requirement tracing matrix of "conclusion 1-requirement ID-test case ID-defect ID" is generated, and information such as test case distribution density, test pass rate, and overall requirement coverage is calculated.

[0031] Step 3, Constraint-based Generative AI Report Generation Module, such as Figure 4 As shown, the template for the system evaluation report output must conform to the industry standard document template. The output chapter structure includes evaluation content, evaluation methods, and evaluation conclusions. Based on the dynamic evaluation rules and judgment criteria set in step 2, the task description is input: "Based on the following structured information and knowledge base content, write a system evaluation report according to the requirements of the industry standard document template. The report content should include evaluation basis, evaluation content, and evaluation conclusions. Among them, the evaluation content should be classified and described according to dimensions such as function, performance, and general quality characteristics. The evaluation conclusions should explain the main problems found, risk assessment, conclusions, and recommendations. It is required to write based on the provided information, without adding unverified information, and mark uncertain content as pending confirmation. The conclusions and recommendations should provide traceable evidence information and embed traceability tags in the form of "<Conclusion 1>: Requirement ID-Test Case ID-Defect ID". The system report draft is generated by calling the large model, and traceability tags are embedded in the evaluation conclusion in the form of "<Conclusion 1>: Requirement ID-Test Case ID-Defect ID".

[0032] Step 4: The human-machine collaborative review and feedback module manually checks the AI ​​report and the knowledge base evidence chain, marks low-confidence content, corrects conclusions, supplements rules, provides feedback on errors, and generates the final evaluation report.

[0033] The present invention will be further described in detail below with reference to embodiments:

[0034] (1) In the software capability assessment of a certain information system, it is necessary to assess the completion status of the system software capabilities. The information system includes 24 subsystems, with 1006 software functional requirements, 83 performance requirements, 56 system requirements, and 284 general quality characteristic requirements. In the third-party assessment, a total of 95,684 test cases were executed, and 5,057 problems were found; in the joint test, 529 joint test items were executed, and 105 problems were found. In conducting software evaluation, the system's software requirements specification (Word), evaluation and testing report (Word), joint testing report (Word), evaluation and testing test cases (Word), and evaluation and testing defect list (Excel) are first imported into the multi-source document processing module. The joint information extraction module extracts system requirements, functional categories, system requirements, test cases, evaluation and testing defects, joint testing issues, and evaluation and testing conclusions from the software requirements document, evaluation and testing test case document, evaluation and testing defect list, evaluation and testing report, and joint testing report, forming entity information such as different requirements, test cases, defects, and evaluation and testing conclusions. The knowledge graph construction module uses the relationship extraction function to establish the relationship between requirements, test cases, defects, and evaluation and testing conclusions, constructing a cross-document relationship graph of the "requirement-test-defect" link, and storing the extracted entity and relationship information in the knowledge base.

[0035] (2) Set dynamic evaluation rules, set industry standard document templates that need to be followed in system evaluation, import industry standard document templates into the system, set the checklist as software requirements, and conduct capability evaluation based on the functional subsystem classification, according to the evaluation dimensions of functionality, reliability, maintainability, testability, security, supportability, and adaptability to complex environments. The judgment criteria are two results: failed test and passed test. The judgment rule is set as "if one of the identification test and test case fails the verification of the test case associated with the requirement, the requirement is judged to have failed the test, or if the requirement is not associated with a test case, it is also judged to have failed the test". Generate a requirement traceability matrix according to the above rules, and calculate information such as test case distribution density, test pass rate, and overall requirement coverage.

[0036] (3) In the constraint-based generative AI report generation module, set the template requirements and chapter structure for the output of the system evaluation report. Based on the dynamic evaluation rules and judgment criteria set in step 2, input the task description "Based on the following structured information and knowledge base content, write a system evaluation report according to the requirements of the industry standard document template. The report content should include the evaluation basis, evaluation content, and evaluation conclusions. Among them, the evaluation content should be classified and described according to the dimensions of function, performance, and general quality characteristics. The evaluation conclusions should explain the main problems found, risk assessment, conclusions and recommendations. It is required to write based on the provided information, not to add unverified information, and to mark uncertain content as pending confirmation. The conclusions and recommendations should provide traceable evidence information and embed traceability tags in the form of "<Conclusion 1>: Requirement ID-Test Case ID-Defect ID". Click "Generate" to form a draft of the system evaluation report.

[0037] (4) For the generated system evaluation report draft, check the implementation status, coverage and evaluation conclusions based on the embedded traceability mark. For the incorrectly marked content, modify it directly in the system to form the correct conclusion and generate the final evaluation report. Through manual verification, the AI-generated report covers 98% of the requirement evaluation points and has a false alarm rate of <5%.

[0038] The multi-source software engineering document analysis and evaluation report generation system based on generative AI described in this invention includes:

[0039] Multi-source document processing module: used to identify heterogeneous data such as manually imported software requirements documents, evaluation reports, joint reports, test cases, and test defect lists;

[0040] Joint Information Extraction and Knowledge Graph Construction Module: Used to combine entity recognition and relationship extraction models to extract entity information, construct a cross-document relationship graph that links requirements, tests, and defects, and store it in the knowledge base;

[0041] Dynamic evaluation rule engine module: used to pre-set industry-standard document format templates, set evaluation dimensions, define rules for whether requirements have been tested, and establish rule executors;

[0042] Constrained Generative AI Report Generation Module: Used to generate report drafts based on chapter structure by calling a large model and embed traceability tags;

[0043] Human-machine collaborative review and feedback module: used for manual inspection of AI reports and knowledge base evidence chains, marking low-confidence content, and generating a final evaluation report.

Claims

1. A method for generating multi-source software engineering document analysis and evaluation reports based on generative AI, characterized in that, Includes the following steps: Step 1: Construct a cross-document relationship graph of related requirements, tests, and defects, and store it in the knowledge base; Step 2: Establish a dynamic evaluation rule engine, pre-set industry-standard document format templates, set evaluation dimensions, and establish a rule executor by setting rules for whether the requirements have been tested. Step 3: Constraint-based AI report generation module generates a draft report by calling a large model and embeds traceability tags; Step 4: The human-machine collaborative review and feedback module generates a final evaluation report by manually checking the AI ​​report and the evidence chain in the knowledge base.

2. The method for generating multi-source software engineering document analysis and evaluation reports based on generative AI according to claim 1, characterized in that, Step 1, constructing the cross-document relationship graph, includes the following steps: Step 1-1: Import heterogeneous data such as software requirements documents, evaluation reports, joint reports, test cases, and test defect lists into the multi-source document processing module, and extract and convert the text format of the documents. Steps 1-2: The joint information extraction and knowledge graph construction module uses a joint entity recognition and relation extraction model to extract entity information.

3. The method for generating multi-source software engineering document analysis and evaluation reports based on generative AI according to claim 1, characterized in that, The evaluation dimensions in step 2 include functionality, reliability, maintainability, testability, security, supportability, and adaptability to complex environments.

4. The method for generating multi-source software engineering document analysis and evaluation reports based on generative AI according to claim 1, characterized in that, Step 2 includes dynamically generating a requirement tracing matrix based on a knowledge base, and calculating the test case distribution density, test pass rate, and overall requirement coverage.

5. The method for generating multi-source software engineering document analysis and evaluation reports based on generative AI according to claim 1, characterized in that, Step 3 includes generating a draft report by calling a large model based on the dynamic assessment rules, risk judgment criteria, output industry standard document templates and chapter structures set in step 2, and embedding traceability tags.

6. The method for generating multi-source software engineering document analysis and evaluation reports based on generative AI according to claim 1, characterized in that, Step 4 includes marking low-confidence content, revising conclusions, supplementing rules, providing feedback on errors, and generating a final evaluation report.

7. A multi-source software engineering document analysis and evaluation report generation system based on generative AI, implemented using the method described in any one of claims 1-6, characterized in that, include: Multi-source document processing module: used to identify heterogeneous data such as manually imported software requirements documents, evaluation reports, joint reports, test cases, and test defect lists; Joint Information Extraction and Knowledge Graph Construction Module: Used to combine entity recognition and relationship extraction models to extract entity information, construct a cross-document relationship graph that links requirements, tests, and defects, and store it in the knowledge base; Dynamic evaluation rule engine module: used to pre-set industry-standard document format templates, set evaluation dimensions, define rules for whether requirements have been tested, and establish rule executors; Constrained Generative AI Report Generation Module: Used to generate report drafts based on chapter structure by calling a large model and embed traceability tags; Human-machine collaborative review and feedback module: used for manual inspection of AI reports and knowledge base evidence chains, marking low-confidence content, and generating a final evaluation report.

Images