Official document abstract generation quality evaluation method and system based on large language model

Abstract

The invention provides an official document abstract generation quality evaluation method and system based on a large language model. The method comprises the steps of obtaining an original official document and a to-be-evaluated abstract; analyzing the original text to obtain key information elements and constructing a verification problem set; calling a large language model, and independently evaluating the abstract from a plurality of dimensions such as semantic consistency, information coverage, language normalization and the like; wherein the information coverage degree is calculated by comparing the similarity of answers to the same question based on an original text and an abstract through a question and answer verification mechanism; and finally, fusing the scores of all dimensions to obtain a comprehensive quality result, and generating a structured report containing defect positioning and improvement suggestions. The method does not need to depend on reference abstracts, realizes comprehensive, objective and explainable automatic quality evaluation of the document abstracts, solves the problems of single evaluation dimension, lack of explanation and dependence on standard answers in the prior art, and is suitable for intelligent document processing scenes.

Images