Method and system for generating grade protection evaluation report based on RAG

Abstract

The invention discloses a method and system for generating a grade protection evaluation report based on RAG, and the method comprises the steps: carrying out the text segmentation of an input evaluation related file, and forming a plurality of text blocks; converting the text block into a vector form through a vector embedding model, and storing the text block into a semantic vector database; generating a corresponding user question according to terms in a preset evaluation template, and retrieving a target text block related to the user question from the semantic vector database by using a retrieval algorithm; and inputting the target text block into a pre-trained large language model, and generating evaluation contents corresponding to terms in combination with the keywords and terms in the evaluation template. The method does not depend on specific data formats or contents of specific companies, can flexibly cope with different conditions of different companies through the semantic vector database, the retrieval algorithm and the powerful natural language understanding and generation capability of the large model, automatically matches related terms and generates corresponding evaluation reports.

Description

Technical Field

[0001] The present application relates to the technical field of evaluation report generation, and specifically to a method and system for generating a level protection evaluation report based on RAG. Background Art

[0002] With the development of information technology, level protection assessment has become an important means to ensure the security of information systems. For every company, enterprise, and government, level protection assessment is an indispensable link.

[0003] The evaluation service mainly includes two parts: the first is that the evaluator conducts on-site testing and conducts careful analysis and inspection of computers and servers; the second is to write an analysis report, summarize the analysis results and make recommendations. For most evaluators, although the on-site inspection part is relatively efficient, the writing and review of the report often becomes a problem they face.

[0004] The level protection assessment relies on the experience of the assessor, and there are problems such as strong subjectivity and low efficiency. In response to these urgent needs, at present, the assessment report is written and reviewed by the assessor, which not only takes a lot of time, but also makes it difficult to ensure the accuracy. In addition, the company hopes to automatically generate the assessment report and then conduct the final manual screening, which is difficult to promote. It is generated through human screening or rigid exhaustive programming of customized rules, but the effect is not good and cannot meet the actual application needs.

[0005] This paper proposes a method for generating a level protection evaluation report based on RAG. Retrieval-augmented generation (RAG) provides a new idea for the automatic generation of evaluation reports because it combines the powerful natural language understanding and generation capabilities of the large language model (LLM) and the ability to quickly search the vector database. Summary of the invention

[0006] In this embodiment, a method, system, electronic device and storage medium for generating a level protection evaluation report based on RAG are provided. Retrieval enhanced generation (RAG) provides a new idea for the automatic generation of evaluation reports due to its powerful natural language understanding and generation capabilities combined with the large language model (LLM) and the ability to quickly search the vector database.

[0007] In a first aspect, an embodiment of the present invention provides a method for generating a level protection evaluation report based on RAG, and the method for generating a level protection evaluation report based on RAG includes: Perform text segmentation on the input evaluation-related files to form multiple text blocks; Converting the text block into a vector form through a vector embedding model and storing it in a semantic vector database; Generate corresponding user questions according to the clauses in the preset evaluation template, and use a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; Input the target text block into the pre-trained large language model, combine the keywords and clauses in the evaluation template, and generate evaluation content for the corresponding clauses; Summarize the evaluation contents of all clauses to form a complete level protection evaluation report.

[0008] In an optional embodiment, generating corresponding user questions according to the clauses in the preset evaluation template, and retrieving target text blocks related to the user questions from the semantic vector database using a retrieval algorithm include: The initial set of relevant text blocks is obtained by performing rough sorting using the semantic similarity algorithm and the keyword matching algorithm; The initial set of relevant text blocks is processed by a rearrangement algorithm to screen out the target text block with the highest relevance.

[0009] In an optional embodiment, the semantic similarity algorithm is a maximum marginal correlation algorithm, the keyword matching algorithm is a BM25 algorithm, and the rearrangement algorithm is based on a comprehensive ranking of semantic similarity and keyword weights.

[0010] In an optional embodiment, the evaluation template includes level protection clauses and their corresponding keywords, and the keywords are used to combine with keyword matching of the retrieval algorithm to achieve automatic matching of clauses and content generation.

[0011] In an optional embodiment, the input assessment-related files include policy files and form files in PDF, Word or TXT formats, which are segmented into text blocks of fixed length using a text segmentation tool.

[0012] In an optional embodiment, the input of the large language model includes a prompt word template, and the prompt word template is used to limit the answer scenario and format of the model.

[0013] In an optional embodiment, when the large language model cannot generate evaluation content that meets a preset threshold, the corresponding clause is marked as a missing item and output to the evaluation report.

[0014] Compared with the prior art, the method for generating a graded protection evaluation report based on RAG of the present invention has the following beneficial effects: The present invention combines the database, retrieval algorithm, large model and evaluation template containing keywords in RAG technology, and can accurately process small and variable data. After manual verification, its accuracy rate is stable at an average of more than 97%. It can complete the processing of a batch of information for a single company or unit within 2-3 minutes, greatly improving the efficiency of report generation, which is more than ten times more efficient than traditional methods. The present invention does not rely on the specific data format or content of a specific company, but through semantic vector databases, MMR, BM25 and other retrieval algorithms, as well as the powerful natural language understanding and generation capabilities of the qwen1.5-14B large model, it can flexibly respond to different situations of different companies and units, automatically match relevant terms and generate corresponding evaluation reports. The automated process reduces the interference of human factors, reduces the evaluation bias caused by differences in personal experience, and ensures the consistency and reliability of the evaluation results.

[0015] In a second aspect, the present invention provides a system for generating a level protection evaluation report based on RAG, comprising: The preprocessing module is used to segment the input evaluation-related files into multiple text blocks; A storage module, used for converting the text block into a vector form through a vector embedding model, and storing it in a semantic vector database; A retrieval module generates corresponding user questions according to the terms in the preset evaluation template, and uses a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; A combining module, used for inputting the target text block into a pre-trained large language model, combining the keywords and clauses in the evaluation template, and generating evaluation content for the corresponding clauses; The generation module is used to summarize the evaluation contents of all clauses and form a complete level protection evaluation report.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for generating a level protection assessment report based on RAG as described in the first aspect.

[0018] Compared with the prior art, the beneficial effects of the system, electronic device and storage medium for generating a level protection evaluation report based on RAG of the present invention are the same as the method for generating a level protection evaluation report based on RAG described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of a method for generating a graded protection evaluation report based on RAG in an embodiment of the present invention; Figure 2 It is a schematic diagram of a method for generating a level protection evaluation report based on RAG in an embodiment of the present invention; Figure 3 Schematic diagram of the large model prompt words in this embodiment; Figure 4 This is a schematic diagram of the context chunk returned to the large model after retrieval in this embodiment; Figure 5 This is a schematic diagram of the final summary answer of the large model in this embodiment; Figure 6 It is a structural block diagram of a system for generating a graded protection evaluation report based on RAG in an embodiment of the present invention; Figure 7 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] In an embodiment of the present invention, a method for generating a level protection evaluation report based on RAG is provided. Figure 1 is a flow chart of the method for generating a graded protection evaluation report based on RAG of the present invention, such as Figure 1 and Figure 2 As shown, the process includes the following steps: S100, performing text segmentation on the input evaluation-related file to form multiple text blocks; It should be noted that the input assessment-related files include system files and form files in PDF, Word or TXT formats, which are segmented into text blocks of fixed length by a text segmentation tool.

[0024] Exemplarily, the lang-chain framework is used to segment initial data, such as PDF, word, txt and other files, into small text chunks.

[0025] S200, converting the text block into a vector form through a vector embedding model, and storing the vector in a semantic vector database; Each text chunk is converted into a mathematical form of a high-dimensional vector through a vector embedding model (Youdao translation vector model). In this embodiment, it is stored in the Chroma vector database to facilitate subsequent retrieval. The advantage is that the cosine similarity can be used to calculate the correlation between sentences.

[0026] S300, generating corresponding user questions according to the clauses in the preset evaluation template, and using a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; Specifically, a corresponding user question is generated according to the clauses in the preset evaluation template, and a target text block related to the user question is retrieved from the semantic vector database using a retrieval algorithm, including: The initial set of relevant text blocks is obtained by performing rough sorting using the semantic similarity algorithm and the keyword matching algorithm; The initial set of relevant text blocks is processed by a rearrangement algorithm to screen out the target text block with the highest relevance.

[0027] Furthermore, the semantic similarity algorithm is a maximum marginal correlation algorithm, the keyword matching algorithm is a BM25 algorithm, and the rearrangement algorithm is based on a comprehensive ranking of semantic similarity and keyword weights.

[0028] For example, the principle of the MMR retrieval algorithm is to retrieve sentences based on similarity. The advantage is that it can ensure high similarity while maintaining diversity and avoiding interference from repeated sentences. The BM25 retrieval algorithm uses Jieba word segmentation to divide sentences into keywords, thereby performing keyword matching retrieval and improving the accuracy of retrieval. The reranker re-ranking algorithm is based on the re-ranking after MMR and BM25 retrieval, which advances sentences with higher relevance, thereby improving the accuracy of the final summary.

[0029] Through semantic retrieval (similarity, MMR) algorithms, the input question or keyword is searched in the database to find sentences (text chunks) related to the question, and then input into the large model for use. The search involves rough sorting (semantic retrieval and keyword BM25 sorting) and fine sorting (Youdao translation re-ranking model).

[0030] In general, the application of retrieval algorithms can filter and select from a large number of text blocks to find the most relevant text blocks to help answer questions. Specifically, assuming that the current user's question is "What is the current company's overall policy?", and there are 1,000 segmented text blocks, how to find the most relevant 10 statements from these 1,000 text blocks to help the model finally summarize the answer is what the retrieval algorithm needs to do. Semantic retrieval in rough sorting uses the cosine theorem to calculate the similarity between sentences that have been compressed into vectors to find the 5 text blocks with the greatest similarity to the user's question. BM25 retrieval in rough sorting also finds 5 text blocks through keyword matching. Finally, fine sorting is to rearrange the final 10 text blocks, and the text blocks that are more in line with the actual situation are advanced to the front position to facilitate model judgment and weight allocation.

[0031] S400, inputting the target text block into a pre-trained large language model, combining the keywords and clauses in the evaluation template, and generating evaluation content corresponding to the clauses; Furthermore, the evaluation template includes the level protection clauses and their corresponding keywords, which are used to match the keywords of the search algorithm to achieve automatic matching of clauses and content generation. The input of the large language model includes a prompt word template, which is used to limit the answer scenario and format of the model.

[0032] It should be noted that the pre-trained large language model uses qwen1.5-14B. As the base large model, qwen1.5-14B only needs to provide its own summarization ability and does not require additional fine-tuning, which will lead to reduced summarization ability. However, since the model summarizes a large amount of context slowly, which affects user use, additional quantization acceleration is performed to improve the model's summarization speed.

[0033] The sentences returned to the big model are some sentences related to the user input or questions. The big model qwen1.5-14B will eventually combine the template, questions, and returned sentence chunks for the final summary and answer.

[0034] For example, Figure 3 , Figure 4 and Figure 5 As shown, Figure 3 Schematic diagram of the large model prompt words in this embodiment; Figure 4 is a schematic diagram of the context chunk returned to the large model after retrieval in this embodiment; Figure 5 This is the final summary answer of the large model in this embodiment; Figure 3 It is the prompt word template of the large model, giving the model identity and scene restrictions. Figure 4The following figure shows the final text block obtained after rough sorting and fine sorting based on the user's question, that is, the context. The context is likely to contain the final answer the user wants. For example, if the user's question is "What is the overall policy of the current company?", the large model qwen1.5-14B recognizes its identity and answer method through the prompt word template, and searches for relevant answers in the context (5 text blocks) according to the user's question and summarizes them. Figure 5 It should be noted that when the large language model cannot generate evaluation content that meets the preset threshold, the corresponding clause is marked as a missing item and output to the evaluation report.

[0035] The evaluation template is a collection of relevant terms and keywords, which facilitates the model to automatically and batch generate the content corresponding to each term of the evaluation report, thereby improving the quality and efficiency of generation.

[0036] The evaluation template is written by the company's internal level protection assessors based on their long-term evaluation experience. Level protection assessments need to be conducted in accordance with relevant laws and standards. Each evaluation clause has corresponding keywords, which are also constructed by assessors based on their long-term experience in order to be combined with rough sorting (mmr search and BM25 search). The keywords that appear in user questions will be matched with the keywords in the evaluation template. In this way, there is no need to manually check each clause one by one, but to directly and automatically match and generate answers.

[0037] S500. Summarize the evaluation contents of all clauses to form a complete level protection evaluation report.

[0038] Exemplarily, the compressed package provided by the evaluated unit (including the company's evaluation-related systems and forms) is input. At this time, the underlying logic code is directly called, first reading the contents of the above-mentioned system and form files, and then performing text segmentation to cut the long text into text blocks. At this time, a large number of text blocks are formed, which are converted into vectors and stored in the database. Then read the clauses in the template, convert the relevant content that needs to be evaluated in the clauses into question form, simulate it as a user question, perform semantic retrieval of the database based on the user question, and perform keyword matching retrieval at the same time. Finally, rearrange the context to obtain the context and return it to the large model for the final summary generation. If the model can generate a complete answer, it is determined that the clause is satisfied. If the model cannot generate a complete answer or the answer score is too low, it is determined that the clause is missing. By analogy, the template is continuously read to complete the above process, and an evaluation report is finally generated.

[0039] The present invention combines the database, retrieval algorithm, large model and evaluation template containing keywords in RAG technology, and can accurately process small and variable data. After manual verification, its accuracy rate is stable at an average of more than 97%. It can complete the processing of a batch of information for a single company or unit within 2-3 minutes, greatly improving the efficiency of report generation, which is more than ten times more efficient than traditional methods. The present invention does not rely on the specific data format or content of a specific company, but through semantic vector databases, MMR, BM25 and other retrieval algorithms, as well as the powerful natural language understanding and generation capabilities of the qwen1.5-14B large model, it can flexibly respond to different situations of different companies and units, automatically match relevant terms and generate corresponding evaluation reports. The automated process reduces the interference of human factors, reduces the evaluation bias caused by differences in personal experience, and ensures the consistency and reliability of the evaluation results.

[0040] The embodiment of the present invention also provides a system for generating a level protection evaluation report based on RAG, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0041] like Figure 6 As shown, Figure 6 It is a structural block diagram of a system for generating a graded protection evaluation report based on RAG in the present invention, and the system includes: The preprocessing module 101 is used to perform text segmentation on the input evaluation-related files to form multiple text blocks; A storage module 102, used to convert the text block into a vector form through a vector embedding model, and store it in a semantic vector database; A retrieval module 103 generates corresponding user questions according to the terms in the preset evaluation template, and uses a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; A combining module 104 is used to input the target text block into a pre-trained large language model, combine the keywords and clauses in the evaluation template, and generate evaluation content corresponding to the clauses; The generation module 105 is used to summarize the evaluation contents of all clauses and form a complete level protection evaluation report.

[0042] Figure 7 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: Perform text segmentation on the input evaluation-related files to form multiple text blocks; Converting the text block into a vector form through a vector embedding model and storing it in a semantic vector database; Generate corresponding user questions according to the clauses in the preset evaluation template, and use a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; Input the target text block into the pre-trained large language model, combine the keywords and clauses in the evaluation template, and generate evaluation content for the corresponding clauses; Summarize the evaluation contents of all clauses to form a complete level protection evaluation report.

[0043] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0044] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: Perform text segmentation on the input evaluation-related files to form multiple text blocks; Converting the text block into a vector form through a vector embedding model and storing it in a semantic vector database; Generate corresponding user questions according to the clauses in the preset evaluation template, and use a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; Input the target text block into the pre-trained large language model, combine the keywords and clauses in the evaluation template, and generate evaluation content for the corresponding clauses; Summarize the evaluation contents of all clauses to form a complete level protection evaluation report.

[0045] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a level protection evaluation report based on RAG, characterized in that: The method for generating a level protection evaluation report based on RAG includes: Perform text segmentation on the input evaluation-related files to form multiple text blocks; Converting the text block into a vector form through a vector embedding model and storing it in a semantic vector database; Generate corresponding user questions according to the clauses in the preset evaluation template, and use a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; Input the target text block into the pre-trained large language model, combine the keywords and clauses in the evaluation template, and generate evaluation content for the corresponding clauses; Summarize the evaluation contents of all clauses to form a complete level protection evaluation report.

2. The method for generating a graded protection evaluation report based on RAG according to claim 1, characterized in that: Generate corresponding user questions according to the clauses in the preset evaluation template, and use the retrieval algorithm to retrieve the target text block related to the user question from the semantic vector database, including: The initial set of relevant text blocks is obtained by performing rough sorting using the semantic similarity algorithm and the keyword matching algorithm; The initial set of relevant text blocks is processed by a rearrangement algorithm to screen out the target text block with the highest relevance.

3. The method for generating a graded protection evaluation report based on RAG according to claim 2 is characterized in that: The semantic similarity algorithm is the maximum marginal correlation algorithm, the keyword matching algorithm is the BM25 algorithm, and the rearrangement algorithm is based on the comprehensive sorting of semantic similarity and keyword weight.

4. The method for generating a graded protection evaluation report based on RAG according to claim 1, characterized in that: The evaluation template includes level protection clauses and their corresponding keywords, and the keywords are used to combine with keyword matching of the retrieval algorithm to achieve automatic matching of clauses and content generation.

5. The method for generating a graded protection evaluation report based on RAG according to claim 1, characterized in that: The input assessment-related files include system files and form files in PDF, Word or TXT format, which are segmented into text blocks of fixed length by a text segmentation tool.

6. The method for generating a graded protection evaluation report based on RAG according to claim 1, characterized in that: The input of the large language model includes a prompt word template, and the prompt word template is used to limit the answer scenario and format of the model.

7. The method for generating a graded protection evaluation report based on RAG according to claim 1, characterized in that: When the large language model cannot generate evaluation content that meets the preset threshold, the corresponding clause is marked as a missing item and output to the evaluation report.

8. A system for generating a level protection evaluation report based on RAG, characterized in that: include: The preprocessing module is used to segment the input evaluation-related files into multiple text blocks; A storage module, used for converting the text block into a vector form through a vector embedding model, and storing it in a semantic vector database; A retrieval module generates corresponding user questions according to the terms in the preset evaluation template, and uses a retrieval algorithm to retrieve a target text block related to the user question from the semantic vector database; A combining module, used for inputting the target text block into a pre-trained large language model, combining the keywords and clauses in the evaluation template, and generating evaluation content for the corresponding clauses; The generation module is used to summarize the evaluation contents of all clauses and form a complete level protection evaluation report.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method for generating a level protection assessment report based on RAG as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a level protection assessment report based on RAG as described in any one of claims 1 to 7 are implemented.

Images