Method and apparatus for calibrating a language model

The temperature scaling method addresses the overconfidence issue in aligned language models by calibrating their predictive distributions, improving their reliability and consistency, and enabling safer applications.

WO2025059913A9 Publication Date: 2025-06-12ROBERT BOSCH GMBH +1
0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/120048
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-09-20
Publication Date
2025-06-12

AI Technical Summary

Technical Problem

Aligned language models tend to be overconfident in their predictions, making it difficult to discern between truthful and hallucinated answers, which hinders their application in safety-critical domains.

Method used

A simple, effective, and sample-efficient temperature scaling method is proposed to calibrate aligned language models by refining their predictive distributions based on a temperature scaling coefficient, aligning them with the predictive distributions of pre-trained models.

Benefits of technology

The proposed method effectively calibrates aligned language models, reducing overconfidence and improving the consistency between predictive confidence and accuracy, thereby enhancing their reliability in practical applications.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The present disclosure provides a computer implemented method for calibrating a language model (LM), comprising: receiving, by a first LM, a set of language questions and generating a first set of predictive answer confidence distributions corresponding respectively to the set of language questions; receiving, by a second LM, the set of language questions and generating a second set of predictive answer confidence distributions corresponding respectively to the set of language questions, wherein the generating the second set of predictive answer confidence distributions comprising calibrating the second set of predictive answer confidence distributions based on a temperature scaling coefficient; determining a distance between the first set of predictive answer confidence distributions and the second set of predictive answer confidence distributions; and updating the temperature scaling coefficient based on the distance.
Need to check novelty before this filing date? Find Prior Art