Knowledge coverage rate evaluation method of large language model and related device

Abstract

The embodiment of the invention provides a knowledge coverage rate evaluation method of a large language model and a related device. The embodiment of the invention can be applied to various scenes such as artificial intelligence. The method comprises the steps of obtaining a to-be-detected entity, performing knowledge retrieval on the to-be-detected entity to obtain a knowledge text of the to-be-detected entity, predicting the to-be-detected entity through a target large language model to obtain a disguise text of the to-be-detected entity, and performing correctness judgment on the disguise text in combination with the knowledge text retrieved by a knowledge base. According to the correctness judgment result of the disgulation text, the classification set is generated, the knowledge coverage rate of the target large language model to the entity is evaluated through the classification set, the influence of the illusion problem of the target large language model on the evaluation of the knowledge coverage rate is reduced, and the evaluation accuracy of the knowledge coverage rate is improved. And the evaluation accuracy and reliability of the knowledge coverage rate of the target large language model are improved.

Images