Large model fine tuning method and device based on high belief illusion detection

Abstract

The invention discloses a large model fine tuning method and device based on high belief illusion detection, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an error question and answer knowledge set for a large model; determining the belief degree of the large model for each wrong question and answer pair; determining each error question and answer pair with the corresponding belief degree exceeding a preset belief threshold in the error question and answer knowledge set as a delusional error question and answer pair with high belief degree illusion; and performing modeling fine tuning on the large model according to each delusional error question and answer pair. Therefore, outputs which are most difficult to perceive and have the strongest misleading to the user can be accurately positioned, and the occurrence rate of high belief errors of a large model in similar input scenes is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a large model fine-tuning method and device based on high-confidence hallucination detection. Background Art

[0002] Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, driving significant progress in multiple fields such as machine translation, dialogue agents, code generation, etc. These models are trained on large amounts of text data, using deep neural network architectures to capture complex language patterns and world knowledge. However, despite their impressive performance, LLMs still face some key limitations, especially when generating factually incorrect but seemingly reasonable outputs, a phenomenon commonly referred to as hallucination.

[0003] Although hallucinations in LLMs have been widely studied as inaccurate, unfaithful, or absurd outputs, the inventors of this application found during the practice of this application that: LLMs exhibit unusually high confidence in certain hallucinatory outputs, and this phenomenon can be observed under different belief probing methods. Inspired by the concept of delusion in psychiatry, this phenomenon is defined as LLM delusion. In psychiatry, delusion is defined as an apparently false belief held with extraordinary conviction, indicating an abnormality in the content of an individual's thinking. Similarly, in this article, LLM delusion is formally defined as hallucination with high confidence. Different from hallucinations that usually involve higher uncertainty, delusions exhibit lower uncertainty, making them particularly difficult to detect and mitigate.

[0004] In response to the above problems, the industry has not yet proposed a better solution. Summary of the Invention

[0005] This application provides a large model fine-tuning method, an electronic device, and a storage medium based on high-confidence hallucination detection, so as to at least alleviate the problem of LLM delusion or high-confidence hallucination in the current related technologies.

[0006] In a first aspect, an embodiment of the present application provides a large model fine-tuning method based on high-confidence hallucination detection, including: obtaining a set of incorrect Q&A knowledge for the large model; the set of incorrect Q&A knowledge is a subset of the data sample set of the large model, and the set of incorrect Q&A knowledge contains at least one incorrect Q&A pair, and each incorrect Q&A pair contains an input question sample and a corresponding incorrect output answer; determining the confidence of the large model for each of the incorrect Q&A pairs; determining each incorrect Q&A pair in the set of incorrect Q&A knowledge whose confidence exceeds a preset confidence threshold as a delusional incorrect Q&A pair with high confidence; and performing modeling fine-tuning on the large model according to each of the delusional incorrect Q&A pairs.

[0007] In a second aspect, an electronic device is provided, which includes: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the large model fine-tuning method based on high-confidence hallucination detection according to any embodiment of the present application.

[0008] In a third aspect, an embodiment of the present application provides a storage medium, on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the large model fine-tuning method based on high-confidence hallucination detection according to any embodiment of the present application are implemented.

[0009] The beneficial effects of the embodiments of the present application are as follows: By constructing a set of incorrect Q&A knowledge and calculating the confidence of each Q&A pair, a confidence threshold is introduced to quantify and distinguish different types of output errors, so as to distinguish those incorrect outputs with abnormally high confidence. Using a quantitative confidence detection method, it is possible to accurately locate those outputs that are most difficult to detect and most misleading to users, and perform modeling fine-tuning on the large model to targetedly adjust the model parameters, so as to reduce the occurrence frequency of high-confidence errors in similar input scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 Shows a comparison schematic diagram between delusion and hallucination under different belief estimation methods; Figure 2The figure shows a flowchart of an example of a large model fine-tuning method based on high-confidence hallucination detection according to an embodiment of the present application; Figure 3 The figure shows a schematic diagram of the simulation effect of an example of the logit-based belief distribution of Llama3.1-8B-Instruct on the TriviaQA test set; Figure 4 The figure shows a schematic diagram of the simulation effect of an example of the comparison of rejection rates under prompts with different honesty levels; Figure 5 The figure shows a schematic diagram of the simulation effect of an example of the comparison of rejection rates under different model sizes; Figure 6 The figure shows a schematic diagram of the simulation effect of an example of the comparison of rejection rates under different SFT rejection data ratios; Figure 7 The figure shows a schematic diagram of the simulation effect of an example of the distribution of paranoia and hallucination reflection results; Figure 8 The figure shows a schematic diagram of the simulation effect of an example of the comparison of the percentage of paranoia and accuracy under different noise ratios and noise intensities; Figure 9 The figure shows a schematic diagram of the simulation effect of an example of the distribution of the number of similar samples for each paranoia sample; Figure 10 The figure shows a schematic diagram of the simulation effect of an example of the trend of various indicators during the training period; Figure 11 The figure is a schematic structural diagram of an embodiment of the electronic device of the present application. Detailed implementation manners

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present application belong to the scope of protection of the present application.

[0013] As Figure 1 shown, it shows a comparison of the phenomena of hallucination and delusion in the LLM. The question is "Who wrote the novel A Song of Ice and Fire?", the correct answer is "George R. R. Martin", and the wrong answers include "Robert Jordan" and "Neil Gaiman". Figure 1The upper part shows the situation of hallucination, that is, the large model may give wrong answers when answering, but its confidence level is relatively low. For example, for the wrong answer "Robert Jordan", the confidence value of the model is only 0.2, which indicates that the model is not very confident about this answer and there is a large degree of uncertainty. In Figure 1 The lower part shows the situation of delusion. In this case, the large model shows an abnormally high confidence in the wrong answer. For example, for the wrong answer "J.R.R. Tolkien" of the model, its confidence value is 0.89, showing a relatively high confidence. Therefore, this kind of hallucination with high confidence is defined as LLM delusion, which is different from general hallucination. General hallucination is usually accompanied by a high degree of uncertainty, while delusion shows a strong adherence to wrong beliefs. Therefore, delusion shows high confidence and is difficult to be detected and corrected, which is a more hidden and persistent challenge in LLM.

[0014] It should be noted that the current research on related technologies mainly focuses on identifying and optimizing the hallucinations of large models, aiming to improve the generation accuracy of the models or reduce the occurrence of single errors, without considering the distinction between high-confidence hallucinations (i.e., delusions) and other ordinary hallucinations of large models. In this paper, the definition and systematic detection mechanism of "high-confidence hallucination" are introduced for the first time, and a high-confidence hallucination detection and mitigation framework is proposed.

[0015] It should be understood that the purpose of the above description of the current related technologies is only to facilitate the public to better understand the inventive spirit and motivation of this application, and is not regarded as a limitation of this application. In addition, the technical solutions described in the above current related technologies are not prior art and may also be unpublished technical solutions, such as those under research or in the laboratory stage.

[0016] Figure 2 The flowchart of an example of a large model fine-tuning method based on high-confidence hallucination detection according to an embodiment of the present application is shown. In this embodiment, through the "high-confidence hallucination detection and mitigation framework", the detection and mitigation of high-confidence hallucinations are effectively achieved.

[0017] As Figure 2 shown, in step S210, an error Q&A knowledge set for the large model is obtained.

[0018] Here, the incorrect Q&A knowledge set is a subset of the large model's data sample set, and the incorrect Q&A knowledge set contains at least one incorrect Q&A pair. Each incorrect Q&A pair includes an input question sample and a corresponding incorrect output answer. Exemplarily, Q&A pairs can be screened out from the large model's original data sample set, and user feedback logs, online interaction records, or manually labeled data can be combined to ensure that the screened Q&A pairs actually have factual errors or logical inconsistencies. The input question samples can be various task questions, such as calculation or reasoning, etc., and the incorrect output answers can be answers that do not conform to facts or have logical loopholes. By screening incorrect Q&A pairs, it is ensured that the incorrect Q&A knowledge set can better reflect the defect of the incorrect knowledge distribution of the large model's hallucination.

[0019] In step S220, determine the confidence degree of the large model for each incorrect Q&A pair.

[0020] It should be noted that the measurement method of the confidence degree can be diversified. For example, the confidence can be measured by recording the probability distribution when the large model outputs an answer, so as to be used as a confidence index. In some examples, the confidence degree can also be quantified by having the large model regenerate the answer to the question in each incorrect Q&A pair and recording the intermediate probability or logit value during the output process.

[0021] In step S230, each incorrect Q&A pair in the incorrect Q&A knowledge set whose confidence degree exceeds the preset confidence threshold is determined to be a delusional incorrect Q&A pair with a high confidence degree hallucination.

[0022] Here, the confidence threshold can be preset, and a preset confidence threshold (such as 0.8 or 80%) can be determined through historical data and model performance as the dividing line between normal errors and high-confidence errors. In addition, the confidence threshold can also be dynamically adjusted according to the task scenario and actual requirements to achieve a more flexible screening strategy.

[0023] In some implementation manners, the confidence degrees of each incorrect Q&A pair are compared. If the confidence degree exceeds the set threshold, it is marked as a delusional incorrect Q&A pair. In addition, additional logical judgments can be added, such as multi-dimensional screening in combination with the error type, semantic deviation degree, etc., to improve the screening accuracy. Thus, by comparing the confidence degrees, the delusional incorrect Q&A pairs are screened out and classified separately to form a data subset specifically used for fine-tuning to accurately screen out those incorrect outputs accompanied by excessive confidence.

[0024] In step S240, the large model is fine-tuned according to each delusional incorrect Q&A pair.

[0025] In some embodiments, by using delusional error Q&A pairs to construct a training dataset, the data quality and representativeness can be ensured, and by designing a data augmentation strategy, similar error cases can be augmented to improve the diversity of fine-tuning samples. Thus, through fine-tuning specifically for high-confidence error cases, the answers generated by the model in similar scenarios no longer show excessive confidence, effectively reducing the incidence of high-confidence hallucinations in the large model.

[0026] Through the embodiments of the present application, constructing an error Q&A knowledge set, accurately calculating the confidence degree, screening high-confidence (delusional) error Q&A pairs, and targeted fine-tuning form a complete closed-loop fine-tuning mechanism from data screening to model calibration. Thus, it can not only effectively identify and correct the "confidence errors" in the generation process of the large model, but also fundamentally improve the model's expression of uncertainty about incorrect information, ultimately enhancing the safety, reliability, and user trust of the model in various actual application scenarios.

[0027] Regarding the implementation details of step S220, in some embodiments, for each error Q&A pair, the uncertainty score of the large model for the error Q&A pair is evaluated based on at least one uncertainty estimation method, and the confidence degree of the large model for the error Q&A pair is determined according to each uncertainty score.

[0028] It should be understood that the types of methods for uncertainty estimation of the large model can be diverse, such as uncertainty estimation based on probability distribution, uncertainty estimation based on Bayesian neural network, and so on.

[0029] In some examples of the embodiments of the present application, the uncertainty estimation method includes at least one of the following: logit-based method, consistency-based method, or verbalized confidence method. Detecting high-confidence incorrect outputs generated by the model through multiple uncertainty estimation methods (such as logit-based, consistency-based, verbalized confidence) can identify high-confidence hallucinations generated during the model generation process, ensuring that the system can promptly discover and handle these problems.

[0030] Specifically, multiple uncertainty estimation techniques are adopted, including the logit-based method (evaluating the probability distribution of output tokens), the consistency-based method (evaluating the consistency of the output through multiple samplings), and the verbalized confidence method (the model explicitly expresses its confidence). Further, statistical or normalization processing is performed on the uncertainty scores calculated by each method to obtain the final model belief degree. Thus, by fusing multiple uncertainty estimation methods, high-confidence hallucinations can be identified with high precision, significantly improving the system's detection ability for incorrect outputs and enhancing the robustness and reliability of the system.

[0031] In some examples of the embodiments of the present application, the belief threshold is determined according to the average confidence of each correct Q&A pair in the correct Q&A knowledge set, and the correct Q&A knowledge set is a subset of the data sample set of the large model. Each correct Q&A pair includes an input question sample and the corresponding correct output answer.

[0032] Here, the belief threshold is used to distinguish high-confidence incorrect outputs (i.e., delusions) from ordinary hallucinations. By using the average confidence of the correct Q&A knowledge set as a benchmark, the threshold can fit the confidence distribution of the model itself, avoiding the subjective bias of artificially setting the threshold, and can also adaptively determine a reasonable belief threshold according to different tasks or types of data sample sets, having strong generalization ability.

[0033] Regarding the implementation details of step S240, in an example of the embodiments of the present application, each delusional incorrect Q&A pair is removed from the data sample set to obtain a refined data sample set. Further, based on the refined data sample set, fine-tuning of the large model is performed.

[0034] It should be noted that since delusional incorrect Q&A pairs usually cause the model to maintain too high a confidence when generating incorrect answers in the future, making it easier for the model to be more inclined to adhere to its delusional beliefs during the reflection process. In the embodiments of the present application, by removing high-confidence incorrect Q&A pairs, the purity of the training data is improved, so that the model no longer contacts this incorrect information, reducing the confidence of the model in incorrect answers and reducing the occurrence probability of LLM delusions.

[0035] As a further preferred implementation manner, for each delusional incorrect Q&A pair, at least one similar incorrect Q&A pair with a semantic similarity higher than a preset similarity threshold to the delusional incorrect Q&A pair is screened from the data sample set. Further, each delusional incorrect Q&A pair and each similar incorrect Q&A pair are removed from the data sample set to obtain a refined data sample set.

[0036] Here, the semantic similarity calculation method can be diversified, such as cosine similarity or Jaccard similarity, etc., which will not be restricted here for the time being. Through the embodiments of the present application, in addition to removing delusional wrong question-and-answer pairs (high-confidence hallucination samples), similar wrong question-and-answer pairs with semantic similarity higher than a preset threshold to these wrong question-and-answer pairs will be further screened out from the data sample set and removed together to construct a more refined data sample set. Thereby, potential wrong information related to high-confidence hallucinations is eliminated, preventing the model from indirectly learning similar wrong patterns during the training process, and thus more effectively reducing the occurrence probability of LLM delusions.

[0037] Regarding the implementation details of step S240, in another example of the embodiments of the present application, based on the input question samples in the delusional wrong question-and-answer pairs, matching external knowledge is retrieved, and the large model is triggered to generate RAG answers for the input question samples according to the external knowledge. Then, the wrong output answers in the delusional wrong question-and-answer pairs are replaced with the RAG answers to obtain RAG-updated question-and-answer pairs. Furthermore, the large model is fine-tuned according to the RAG-updated question-and-answer pairs.

[0038] Here, for delusional wrong question-and-answer pairs (i.e., question-and-answer pairs where the model has high confidence in the wrong answers), the RAG (Retrieval-Augmented Generation) mechanism is used to retrieve matching authoritative knowledge from the external knowledge base, and the large model is triggered to regenerate answers (RAG answers) based on the retrieved knowledge. Then, the original wrong answers are replaced with the RAG answers to form RAG-updated question-and-answer pairs, and the large model is fine-tuned based on this data. Thereby, by combining the external knowledge base or retrieval system with the large language model, the wrong answers to specific questions are corrected using authoritative external knowledge, realizing knowledge correction for high-confidence hallucinations (LLM delusions), and making the model more inclined to align with the retrieved true knowledge when generating answers in the future, improving the output credibility.

[0039] Through the embodiments of the present application, for the detected high-confidence hallucinations, verification is performed by retrieving the external knowledge base. For example, an efficient retrieval algorithm is used to quickly retrieve relevant information from the large-scale knowledge base. If there is information in the external knowledge base that contradicts the generated content, the system will correct or filter the wrong output. This process uses external knowledge to enhance the generation results of the model, ensuring the accuracy and reliability of the output. Thereby, through the verification of the external knowledge base, the accuracy of the generated content and the richness of the knowledge background are significantly improved, ensuring the reliability and accuracy of the system output.

[0040] As an attachable or replaceable implementation, candidate output answers for the input question samples in the delusional wrong Q&A pairs are obtained based on multiple surrogate models respectively, and the surrogate-generated answers for the input question samples are screened from the respective candidate output answers through multi-agent voting. Then, according to the surrogate-generated answers, the wrong output answers in the delusional wrong Q&A pairs are replaced to obtain surrogate-updated Q&A pairs. Furthermore, based on the surrogate-updated Q&A pairs, fine-tuning of the large model is performed.

[0041] Here, multiple surrogate models are used to generate candidate answers, and the best surrogate-generated answers are screened through a voting mechanism to replace the wrong answers in the delusional wrong Q&A pairs, forming surrogate-updated Q&A pairs, which are then used to fine-tune the large model, leveraging the knowledge complementarity of multiple surrogate models to improve the reliability of the answers. Exemplarily, if the answers generated by multiple surrogate models are exactly the same or highly similar, the answer with the highest occurrence frequency is selected as the final answer.

[0042] Since different surrogate models have different training data and inference methods, by integrating the judgments of multiple models and using a voting mechanism to filter out outliers (i.e., extremely wrong answers), the most reliable answers are selected, reducing high-confidence hallucinations and enhancing the factual consistency of the model. Exemplarily, debates are held among multiple models or agents, and the models can verify the generated content with each other, reducing the errors of a single model. After each agent generates an answer, the system determines the final output through voting or consistency checking. Thus, by replacing the original wrong answers with surrogate-generated answers, the wrong cognition of the model is corrected, and the output credibility is improved.

[0043] Through the embodiments of the present application, multiple models or agents are used to debate the generated content, the final output is determined through voting or consistency checking, and through the collaboration and debate of multiple models, the errors of a single model are further reduced, ensuring the accuracy of the output. Thus, by using the multi-model collaboration and debate mechanism, the accuracy of the generated content is significantly improved, the robustness and reliability of the system are enhanced, and at the same time, the diversity of the output is increased.

[0044] In some embodiments, the high-confidence hallucination detection mechanism of the model can also be optimized according to user feedback and task execution results, improving the execution effect of future tasks, ensuring that the system can continuously improve and adapt to the changing user needs. Thus, through a user-friendly result output and feedback mechanism, the user experience is improved, and at the same time, the model is continuously optimized according to user feedback and task execution results, ensuring that the system can adapt to the changing user needs, improving the overall performance and reliability.

[0045] In some examples of the embodiments of the present application, a detection and mitigation framework for high-confidence hallucinations in large models is also proposed. The model can identify high-confidence incorrect outputs during the generation process. By introducing the definition of "high-confidence hallucinations" and a systematic detection mechanism, combined with external verification and multi-agent debate, the model can correct or filter these high-confidence hallucinations, and more effectively detect and mitigate high-confidence hallucinations. In addition, in the embodiments of the present application, uncertainty estimation is combined with external verification to further improve the reliability and efficiency of the model in complex tasks.

[0046] Through the embodiments of the present application, based on optimizing the detection and mitigation mechanism of high-confidence hallucinations, the reliability and accuracy of large language models in generation tasks are significantly improved, and the occurrence rate of high-confidence incorrect outputs is reduced. The chain reaction includes enhancing users' trust in AI systems, reducing manual intervention and correction costs, and improving the overall efficiency of the system. In addition, this solution can also promote the wide application of AI technology in various industries, especially in fields that require high reliability and accuracy (such as healthcare, finance, law, etc.), and drive the popularization and in-depth application of AI technology. At the same time, this solution provides support for the optimization and application expansion of future intelligent systems, and accelerates cross-industry cooperation and innovation.

[0047] Specifically, the embodiments of the present application have discovered a more concealed phenomenon, namely LLM delusion, which is defined as high-confidence hallucinations - that is, generating incorrect outputs with abnormally high confidence, making it more difficult to detect and mitigate. Different from ordinary hallucinations, delusions exhibit extremely low uncertainty, thus posing a major challenge to the reliability of the model. Through empirical analysis of different model families and scales on multiple question-and-answer tasks, it is found that the delusion phenomenon is widespread and significantly different from hallucinations. The "honesty" of LLM in the case of delusion is lower, and it is difficult to correct through fine-tuning or self-reflection. The formation of delusions is linked to training dynamics and dataset noise, and mitigation strategies such as retrieval-augmented generation and multi-agent debating are explored. By systematically studying the nature, prevalence, and mitigation methods of LLM delusions, the research in this paper reveals the potential causes of this phenomenon and points out the future development direction for improving the reliability of the model.

[0048] 1. Introduction To systematically analyze LLM delusions, uncertainty estimation is adopted as a proxy for model beliefs, using three complementary methods: (1) logit-based methods, by evaluating the token probability distribution; (2) verbalized confidence, where the model explicitly expresses its degree of belief; and (3) consistency-based methods, by evaluating the stability of beliefs through multiple samplings. To quantify delusions, a belief threshold is introduced, which is empirically determined by the average confidence of the correct answers on a given dataset. If the confidence of a wrong answer exceeds this threshold, it is classified as delusion.

[0049] In this study, an extensive empirical investigation was conducted on multiple LLM families and knowledge-intensive Q&A benchmarks, and the results show that delusions are prevalent across different models and scales and persist under changes in uncertainty estimation methods (§4.1). Additionally, a series of contrast experiments between delusions and hallucinations were designed to reveal that LLM delusions are similar to delusions in psychiatry in some characteristics (§4.3). Moreover, the formation and dynamics of delusions were analyzed from the perspectives of data and training (§5). Finally, several potential mitigation strategies were explored, and external validation was introduced (§6). The main findings during the inventors' practice of this application include: - The "honesty" of LLMs in delusional situations is lower than that in hallucinations. When prompted to reject unknown knowledge, the model is more likely to reject standard hallucinations while retaining delusional outputs.

[0050] - Delusions are more difficult to correct through fine-tuning than hallucinations. Even after training LLMs to reject wrong answers, delusions still persist at a higher rate than hallucinations.

[0051] - Self-reflection mechanisms are ineffective in mitigating delusions. When prompted to reconsider previous answers, LLMs are more likely to stick to delusional outputs rather than modify them.

[0052] - The formation of delusions is affected by training dynamics and dataset noise. Through experiments on synthetic datasets, it was found that the proportion of misinformation and consistency in training data exacerbate the tendency of delusions.

[0053] - External validation methods, such as retrieval-augmented generation and multi-agent debate systems, offer potential ways to reduce delusions but still face significant challenges in completely eliminating delusions.

[0054] By systematically studying the nature, prevalence, and mitigation of delusions in LLMs, our research provides profound insights into the potential causes of this phenomenon and points to future directions for improving model reliability. Our findings underscore the need for robust validation mechanisms and adaptive confidence calibration to ensure more reliable deployment of LLMs in real-world applications.

[0055] 2 Related Work 2.1 Delusions Delusions have long been a subject of study in psychiatry, defined as beliefs with extraordinary subjective certainty that are resistant to contrary evidence and whose content is often unachievable. A key feature of delusions is that individuals remain firmly convinced of their beliefs even in the face of overwhelming contradictory evidence. Recently, delusions have received attention in the field of Reinforcement Learning (RL), where candidate goal generators and goal estimators are used to simulate the belief formation and evaluation processes, and the inconsistencies between these components are analogous to the mechanisms of delusions in the human brain. However, in Natural Language Generation (NLG) models, the phenomenon of delusions remains under-explored. Different from delusions in RL, which typically occur in out-of-distribution scenarios, delusions in NLG models mainly involve inaccurate factual information in the real world.

[0056] 2.2 Uncertainty Estimation Uncertainty estimation plays a crucial role in evaluating LLM confidence and addressing the hallucination problem. Traditional methods, such as Bayesian methods and ensemble techniques, have been widely explored but are often computationally expensive and difficult to scale. Recent research has adopted methods such as logit-based uncertainty (analyzing the token probability distribution), consistency-based uncertainty (evaluating the stability of responses through multiple samples), and verbalized confidence (model self-assessment). Although these techniques have improved the accuracy of hallucination detection, they are not foolproof - models can still generate incorrect outputs in cases of low uncertainty. This limitation has prompted an in-depth study of delusions, a phenomenon in which models exhibit high confidence in false claims and resist traditional uncertainty-based filtering methods.

[0057] 2.3 LLM Hallucinations Hallucinations in large language models (LLMs) refer to the generation of content that is unfaithful or factually incorrect, posing challenges to the reliability of applications such as question answering and knowledge retrieval. To mitigate the hallucination problem, extensive efforts have been made, including training for enhanced factuality, retrieval-augmented generation (RAG), and confidence calibration. This study identifies a more problematic type of hallucination - delusion. Different from standard hallucinations, delusions persist even after being exposed to counter-evidence and are more difficult to eliminate, requiring new methods for detection and mitigation.

[0058] 3 Delusions in LLMs 3.1 Definition of Delusions Figure 3 The simulation effect schematic diagram showing an example of the logit-based belief distribution of Llama 3.1 - 8B-Instruct on the TriviaQA test set is presented. Figure 3 Below is the normalized confidence.

[0059] LLMs exhibit a phenomenon where they generate incorrect factual information while maintaining a high level of confidence in this inaccurate information. This unwavering confidence persists even when the model is prompted to re-evaluate or confirm its answer. Borrowing an analogy from psychiatry, delusion refers to a strong false belief. The concept of delusion in LLMs is introduced as a systematic extension of hallucination. Specifically, delusion is defined as a high-confidence hallucination where the model shows an unusually strong belief in its incorrect output. As Figure 3 shown, all incorrect predictions are classified into ordinary hallucinations and delusions according to the estimated belief scores and a specific belief threshold. The methods for estimating LLM belief and the belief threshold will be discussed in the next two subsections.

[0060] 3.1.1 Uncertainty as LLM Belief Uncertainty estimation has been widely used in machine learning as a measure to evaluate the confidence of a model in its predictions, thus providing a metric for the reliability of the model output. Therefore, uncertainty estimation is adopted as a proxy for the model's belief in its answer. Three main uncertainty estimation methods are used: - Logit-based estimation: This method derives the model's belief from the probability distribution of the output tokens, where a higher probability assigned to a certain token indicates a stronger belief in the correctness of that token. Specifically, the raw logits are used, and the generated probability is directly regarded as the confidence.

[0061] - Consistency-based estimation: This method evaluates beliefs by assessing the stability of the model's answers across multiple sampling iterations. If the responses are consistent across multiple trials, it indicates strong beliefs; while if the responses vary, it indicates uncertainty. In this case, answer vote consistency is used as a measure of consistency.

[0062] - Verbalized confidence: In this method, the model is explicitly prompted to express its confidence in the response, providing a self-reported measure of belief. Three methods are used to quantify verbalized confidence: P(true), verb. 1S top-1, and verb. 2S top-1.

[0063] These methods together provide a comprehensive understanding of the model's belief system, recognizing that human beliefs also vary across different contexts. For detailed information on the uncertainty calculation methods.

[0064] Normalization of uncertainty scores: Due to the large distribution differences in the uncertainty scores calculated by each method, an attempt was also made to normalize these uncertainty scores as the model's belief. This normalization method facilitates the comparison and averaging of uncertainty scores from different methods, and is intuitively normalized according to the uncertainty score rankings of all test data. Higher uncertainty scores correspond to higher model rankings and higher normalized beliefs.

[0065] 3.1.2 Belief Threshold for Delusions To systematically identify delusions, a belief threshold is defined to distinguish high-confidence incorrect outputs from ordinary hallucinations. This threshold is empirically set by analyzing the belief distribution of correctly answered questions on a given dataset. Specifically, the belief threshold is determined as the average belief level of all correctly answered questions. When the model's belief in an incorrect answer exceeds this threshold, that belief is considered abnormally high and is classified as a delusion. Intuitively, if the belief level assigned to an incorrect answer exceeds the average confidence of the correct answers, it indicates that the model has an abnormal belief in that error.

[0066] 3.2 Delusions and Hallucinations Both delusions and hallucinations in LLMs are fundamental errors in the model's output, indicating that the model fails to align its answers with factual accuracy. The key difference between hallucinations and delusions lies in the model's confidence: hallucinations are usually accompanied by low or medium confidence, while delusions are characterized by the model's firm high confidence in its incorrect output. Additionally, recent psychiatric research shows that delusions recur more frequently after remission than hallucinations, which usually remit first. This pattern is consistent with the findings in this paper: delusions in the model exhibit lower honesty and are more difficult to reject, and the model is more inclined to adhere to its delusional beliefs during the reflection process (see §4.3).

[0067] 4 Empirical Study of Delusions This section presents an empirical study of delusions in LLMs. The distribution of delusions across different datasets and models is investigated, using different belief estimation methods (§4.1) and belief aggregation techniques (§4.2). Additionally, the distinction between delusions and hallucinations is discussed by exploring the challenges LLMs face when rejecting or reflecting on delusions (§4.3).

[0068] Experimental Setup Experiments were conducted on two knowledge-based question-answering datasets: TriviaQA and Natural Questions. Three well-known open-source model families were used: Qwen-2.5-Instruct (1.5 / 3 / 7 / 14 / 30 / 70B), Llama-3.1-8b-Instruct, Llama-3.3-70b-Instruct, and Mistral-7B-Instruct-v0.1.

[0069] Evaluation Metrics Overall performance was mainly evaluated using accuracy and error rate (ER). Model responses were classified into three types: correct, wrong, and reject, with the reject cases being relatively rare. Wrong responses were further classified as delusions or hallucinations based on specific belief scores. The proportion of delusions in the entire dataset or all wrong responses was used as the delusion metric. For different belief calculation methods, the same model responses obtained through greedy search were used.

[0070] 4.1 Distribution of Delusions Table 1: Delusion ratios based on different belief estimation strategies and their combinations.

[0071] As shown in Table 1, the results of the empirical study reveal that the rate of delusions is consistently high across different model architectures and evaluation strategies, indicating that this phenomenon poses a general challenge beyond model configuration. For example, on the TriviaQA dataset, under the non-integrated belief condition, the overall delusion rate of the Qwen2.5 (7B) model ranges from 8.3% to 31.0%, where delusions account for 20.8% to 77.8% of all incorrect responses. Notably, after scaling the architecture size to Qwen2.5 (72B), the overall delusion rate drops to 5.8% to 17.8%, showing that the enhanced model capabilities improve the faithfulness of the responses. However, delusions still appear in 23.4% to 79.1% of the incorrect outputs, indicating that architecture scaling mainly improves the general hallucination phenomenon and has limited effect on alleviating the delusion tendency in incorrect responses.

[0072] Among all evaluation metrics, the speech-based methods show significantly higher delusion rates, highlighting the inherent challenges in relying on speech responses for belief estimation. These methods exhibit a significant difference between the expected and actual results, suggesting that alternative methods may be needed to address these issues more effectively.

[0073] 4.2 Can Belief Integration Eliminate Delusions? To evaluate whether the integration of different belief estimation methods can alleviate delusions, three of the most effective belief estimation techniques were integrated: P(True), consistency, and logits. The research results show that the integration method does help reduce delusions. For example, on the TriviaQA dataset, the delusion rate in the errors of the Qwen2.5 (72B) model drops from 71.9% to 33.5%. However, despite these improvements, which indicate that the integration technique provides some benefits, belief estimation itself is not the main cause of delusions. Other factors, perhaps the model architecture itself or the nature of the task, may contribute significantly to the observed high delusion rates.

[0074] 4.3 The Difference between Delusions and Hallucinations As discussed in § 3.2 and mentioned in the psychiatric literature, delusions are more difficult to handle and more prone to recurrence than hallucinations. Therefore, the difference between delusions and hallucinations is experimentally confirmed here.

[0075] 4.3.1 LLMs Show Lower Honesty in Handling Delusions Figure 4 A simulation effect diagram showing an example of the comparison of rejection rates under different honesty level prompts is presented. In this experiment, the model used is Llama3.1-8b-Instruct.

[0076] Figure 5 The schematic diagram of the simulation effect showing an example of the comparison of rejection rates under different model sizes. In this experiment, the model uses the Qwen2.5 family.

[0077] LLMs show a lower level of honesty when dealing with delusions. By using prompts with different honesty levels, the willingness of the model to refuse to answer unknown questions is evaluated, as Figure 4 shown. The results indicate that although the overall error rate varies under different prompts, the rejection rate of delusions is consistently lower than that of hallucinations and is consistent across all settings. This suggests that LLMs are more inclined to reject ordinary hallucinations but are less willing to reject when faced with delusion problems. Additionally, as Figure 5 shown, although larger models show higher accuracy and lower error rates, the rejection rate of delusions still consistently exceeds that of hallucinations. These findings emphasize that LLMs are more inclined to have stronger internal beliefs about delusions, making them less willing to reject delusional content compared to hallucinations.

[0078] 4.3.2 LLMs are more difficult to reject delusions Figure 6 The schematic diagram of the simulation effect showing an example of the comparison of rejection rates under different proportions of SFT refusal data. In this experiment, the model uses Llama3.1 - 8b - instruct.

[0079] The incorrect output was further modified by replacing it with "I don't know", and the model was fine - tuned to learn to reject unknown questions. The training set was constructed by adjusting the proportion of refusal data, which includes delusions and hallucinations, to create a balanced training set. As Figure 6 shown, the results indicate that even after training, the model is still more inclined to reject ordinary hallucinations, and the proportion of delusions remains high. This shows that although the model's ability to reject certain types of incorrect outputs has improved, they still have difficulty rejecting delusional content. Compared to hallucinations, they maintain stronger internal beliefs about delusions.

[0080] 4.3.3 Delusions are more difficult to reflect on than hallucinations Figure 7 The distribution of the reflection results of delusions and hallucinations is shown, where LLMs are more inclined to adhere to delusions rather than hallucinations.

[0081] Let the model reflect on their previous answers and evaluate whether they are willing to change the original answers. As Figure 7As shown, the results indicate that the model is significantly more firm in adhering to delusional answers than in non-delusional hallucinatory answers. This behavior suggests that LLMs exhibit stronger internal beliefs in delusional content as they are more resistant to changing their answers when faced with delusional content. In contrast, when dealing with hallucinations, the model shows a higher willingness to reconsider its answers, reflecting less entrenched beliefs in these types of errors.

[0082] 5. Causes and influencing factors In this section, the focus is mainly on studying the causes of delusion formation and the factors that influence delusion changes. The research focuses on two main aspects: data and training. The experiments were mainly conducted on the ALCUNA dataset, and all questions were related to fictional entities. This allows for the easy introduction of noise and other modifications without worrying about the impact of the model's existing knowledge on the results.

[0083] 5.1 Impact of data noise on delusion Figure 8 A schematic diagram of the simulation effect showing the comparison of delusion percentages and accuracies under different noise ratios and noise intensities is presented.

[0084] - Noise ratio: By training the model with different ratios of noisy data and clean data, the impact of data noise on delusion formation is analyzed. As Figure 8 shown, the x-axis represents the ratio of noisy data to clean data mixed during training, while the y-axis shows the delusion rate and the prediction accuracy of the model. The results indicate that as the ratio of noisy data increases, the delusion rate rises while the accuracy of the model decreases.

[0085] - Noise intensity: The study examines how noise intensity affects delusion formation. High noise intensity is characterized by a concentrated distribution of wrong answers, while low noise intensity leads to a more dispersed distribution of errors. It is found that when the noisy data has a higher intensity (i.e., more concentrated errors), as the ratio of noisy data increases, the delusion rate rises significantly, and the accuracy decreases more rapidly. In contrast, when the noise intensity is low, the delusion rate rises more slowly, and the accuracy also decreases more gently. Interestingly, in the case of low noise intensity, although the error rate decreases with the addition of more noisy data, the delusion rate also shows a decline. This suggests that when the wrong data is more evenly distributed, the model is less likely to form strong beliefs in any particular wrong answer, resulting in a higher hallucination rate but a lower likelihood of delusion occurrence.

[0086] 5.2 How does training affect delusion? Figure 9 A schematic diagram of the simulation effect showing the distribution of the number of similar samples for each delusion sample is presented.

[0087] The analysis in this paper further shows that even when the proportion of noisy data is zero, some experiments still exhibit a relatively high rate of delusions. This indicates that delusions may arise in the default state of the model without relying on the introduction of noise. When using the paraphrase-MiniLM-L6-v2 model for embedding and calculating cosine similarity, it is found that certain questions in the training set have relatively high similarity scores with each other. These questions correspond to known delusion examples, leading to interference in the model learning process. On average, each delusion example is associated with 27 training data points with similar questions and the same answers. As shown in Figure 9, this data interference may disrupt the model training and increase the difficulty of the training and learning process.

[0088] The hypothesis was further verified through the following two experimental methods: Table 2: Model performance in data ablation study.

[0089] 1) Reducing training interference: When the training set was refined by removing the question-and-answer pairs with high cosine similarity (greater than 0.9) to known delusions, the delusion rate decreased significantly, as shown in Table 2. Retraining the model with this refined dataset shows that dataset refinement plays a crucial role in reducing delusions.

[0090] Figure 10 A simulation effect diagram showing an example of the trend of various indicators during the training cycle.

[0091] 2) Improving training sufficiency: In Figure 10 , it was observed that as the training progresses, the accuracy of the model improves, the error rate decreases, and the delusion rate (both standardized and non-standardized) decreases significantly. These findings indicate that sufficient training and careful management of dataset interference play an important role in reducing the occurrence of delusions.

[0092] 6 Mitigating Delusions through External Validation In this section, how to mitigate delusions is explored. Since delusions are more difficult to identify through uncertainty-based detection methods compared to hallucinations, an attempt is made to solve this problem by introducing external knowledge to verify the correctness of the answers. Specifically, two methods are explored in the following subsections: model debate and retrieval-augmented generation (RAG).

[0093] 6.1 Mitigating Delusions through Multi-Agent Debate Table 3: Mitigating Delusions through Multi-Agent Debate at Different Models and Thresholds.

[0094] Here, a multi-agent voting method is adopted, where the output of each model is verified by other models. By aggregating the collective knowledge of multiple models, this voting method helps to filter out wrong answers. The experimental results are shown in Table 3, indicating that multi-agent voting effectively alleviates delusions and reduces the error rate and the proportion of delusions of all models. Notably, the Mistral-7B model achieved the largest reduction, from 14.6% to 1.3%. Although delusions are generally more difficult to alleviate than hallucinations, a significant portion of delusions was effectively addressed, highlighting the effectiveness of multi-agent voting in reducing delusions and hallucinations. These findings suggest that multi-agent voting is an effective method to alleviate delusions and improve the reliability of models.

[0095] 6.2 Alleviating Delusions via RAG Table 4: Alleviating Delusions via RAG.

[0096] RAG was applied on the TriviaQA dataset to alleviate delusions. For each question, the model enhanced its answer by retrieving 20 relevant passages from the knowledge base. This allows the model to incorporate external knowledge during the generation process, helping to alleviate delusions by grounding the answer in relevant information. The experimental results are shown in Table 4, indicating that this method significantly reduces the incidence of delusions of all models. This shows that incorporating the retrieval function can improve the factual accuracy of the answers generated by the model, thus reducing the occurrence of delusions. Additionally, the reduction in delusions is comparable to the reduction in hallucinations, highlighting the effectiveness of RAG in addressing delusions and hallucinations.

[0097] 7 Conclusion This paper introduces the concept of LLM delusions, a phenomenon that is more covert and persistent than traditional LLM hallucinations. The characteristics of delusions are demonstrated - having a high degree of confidence in factually incorrect answers, which makes delusions a unique challenge due to their lower uncertainty and resistance to detection. Empirical analysis of multiple LLM families reveals the widespread existence of delusions, highlighting the need for targeted alleviation strategies. Although self-reflection and fine-tuning methods have limited effectiveness in reducing delusions, external validation methods such as RAG and multi-agent debate systems offer potential solutions. These findings emphasize the importance of strong model validation and confidence calibration in enhancing the reliability and credibility of LLMs in practical applications.

[0098] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a combination of a series of actions. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In some embodiments, the embodiments of this application also provide a computer program product. The computer program product includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is enabled to execute any one of the above-mentioned large model fine-tuning methods based on high-confidence hallucination detection.

[0100] In some embodiments, the embodiments of this application also provide an electronic device, which includes: at least one processor, and a memory communicatively connected to the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the large model fine-tuning method based on high-confidence hallucination detection.

[0101] Figure 11 FIG. is a schematic hardware structure diagram of an electronic device for executing the large model fine-tuning method based on high-confidence hallucination detection provided by another embodiment of this application. As Figure 11 shown, the device includes: One or more processors 1110 and a memory 1120. Figure 11 Here, one processor 1110 is taken as an example.

[0102] The device for executing the large model fine-tuning method based on high-confidence hallucination detection may further include: an input device 1130 and an output device 1140.

[0103] The processor 1110, the memory 1120, the input device 1130, and the output device 1140 may be connected through a bus or other means. Figure 11 Here, taking the connection through a bus as an example.

[0104] The memory 1120 serves as a non-volatile computer-readable storage medium and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the large model fine-tuning method based on high-confidence hallucination detection in the embodiments of the present application. The processor 1110 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1120, that is, implements the large model fine-tuning method based on high-confidence hallucination detection in the above method embodiments.

[0105] The memory 1120 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 1120 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 1120 may optionally include a memory remotely set relative to the processor 1110, and these remote memories can be connected to the electronic device through a network. Examples of the above networks include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and their combinations.

[0106] The input device 1130 can receive input digital or character information and generate signals related to the user settings and function control of the electronic device. The output device 1140 may include a display device such as a display screen.

[0107] The one or more modules are stored in the memory 1120 and, when executed by the one or more processors 1110, execute the large model fine-tuning method based on high-confidence hallucination detection in any of the above method embodiments.

[0108] The above product can execute the method provided in the embodiments of the present application and has the corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference can be made to the method provided in the embodiments of the present application.

[0109] The electronic device in the embodiments of the present application exists in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by having mobile communication functions and mainly aim to provide voice and data communication. Such terminals include: smart phones, multimedia phones, functional phones, and low-end phones, etc.

[0110] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have the characteristic of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, etc.

[0111] (3) Portable entertainment devices: Such devices can display and play multimedia content. This type of device includes: audio and video players, handheld game consoles, e-books, as well as smart toys and portable in-vehicle navigation devices.

[0112] (4) Other airborne electronic devices with data interaction functions, such as in-vehicle device installed on a vehicle.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A large model fine-tuning method based on high-belief hallucination detection, comprising: Obtaining knowledge sets of incorrect question answers for large models; The incorrect question and answer knowledge set is a subset of the data sample set of the large model, and the incorrect question and answer knowledge set includes at least one incorrect question and answer pair, each of which includes an input question sample and a corresponding incorrect output answer; Determining the degree of belief of the large model for each of the incorrect question-answer pairs; Determine each incorrect question-answer pair in the incorrect question-answer knowledge set whose corresponding belief degree exceeds a preset belief threshold as a delusional incorrect question-answer pair with a high belief degree illusion; According to each of the delusional and incorrect question-answer pairs, the large model is fine-tuned.

2. The method according to claim 1, wherein: Determining the belief degree of the large model for each of the erroneous question-answer pairs includes: For each of the incorrect question-answer pairs, the uncertainty score of the large model for the incorrect question-answer pair is evaluated based on at least one uncertainty estimation method, and the belief degree of the large model for the incorrect question-answer pair is determined according to each of the uncertainty scores.

3. The method according to claim 2, wherein: The uncertainty estimation method includes at least one of the following: a logit-based method, a consistency-based method, or a verbalized confidence method.

4. The method according to claim 1, wherein: The belief threshold is determined based on the average confidence of each correct question and answer pair in the correct question and answer knowledge set, where the correct question and answer knowledge set is a subset of the data sample set of the large model, and each correct question and answer pair includes an input question sample and a corresponding correct output answer.

5. The method according to claim 1, wherein: The method of fine-tuning the large model according to each of the delusional erroneous question-answer pairs includes: Removing each of the delusional erroneous question-answer pairs from the data sample set to obtain a refined data sample set; Based on the refined data sample set, the large model is fine-tuned.

6. The method according to claim 5, further comprising: For each of the delusional and erroneous question-answer pairs, selecting from the data sample set at least one similar erroneous question-answer pair whose semantic similarity with the delusional and erroneous question-answer pair is higher than a preset similarity threshold; The removing of each of the delusional erroneous question-answer pairs from the data sample set to obtain a refined data sample set includes: Each of the delusional incorrect question-answer pairs and each of the similar incorrect question-answer pairs are removed from the data sample set to obtain a refined data sample set.

7. The method according to claim 1, wherein: The method of fine-tuning the large model according to each of the delusional erroneous question-answer pairs includes: Based on the input question sample in the delusional wrong question-answer pair, matching external knowledge is retrieved, and according to the external knowledge, the large model is triggered to generate a RAG answer for the input question sample; Replacing the wrong output answer in the delusional wrong question and answer pair according to the RAG answer to obtain a RAG updated question and answer pair; The question-answer pairs are updated according to the RAG, and the large model is fine-tuned.

8. The method according to claim 1, wherein: The method of fine-tuning the large model according to each of the delusional erroneous question-answer pairs includes: Based on multiple proxy models, candidate output answers for the input question samples in the delusional wrong question-answer pair are obtained respectively, and the proxy generated answers for the input question samples are selected from each of the candidate output answers through multi-proxy voting; Replacing the wrong output answer in the delusional wrong question and answer pair according to the agent-generated answer to obtain an agent-updated question and answer pair; The large model is fine-tuned based on the agent updating the question-answer pairs.

9. A storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

10. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 8.

Images