Object hallucination suppression method and system for multi-modal large language model

Abstract

The application is suitable for the technical field of artificial intelligence, and provides an object hallucination suppression method and system for a multi-modal large language model. The method comprises the following steps: obtaining input image information and text prompt information; determining image-related attention score information based on the input image information and the text prompt information; generating main path candidate output distribution information based on a global main path; if the current generation position belongs to a position to be enhanced, generating correction path candidate output distribution information based on a local correction path; comparing reliability information to further determine final output distribution information; and generating complete answer text information. The application can reduce object hallucination while maintaining or improving the recall ability and description integrity of the model for real image objects without retraining the model or adding an external training module, thereby improving the consistency, reliability and safety between the output content and the input image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and system for suppressing object illusions in multimodal large language models. More specifically, it relates to a tool, method, and system for suppressing object illusions by performing global-local dual-path correction on visual attention from text generation position to image visual lexical units during the inference stage of a multimodal large language model, and by gating object illusions based on prediction uncertainty and confidence. Background Technology

[0002] With the development of Multimodal Large Language Models (MLLMs), these models are now capable of simultaneously processing multimodal inputs such as images and text, and generating natural language responses in tasks such as image captioning, visual question answering, multimodal dialogue, scene understanding, assisted diagnosis, and autonomous driving scene explanation. These models typically include a visual encoder, a large language model, and connection modules for mapping visual features to the language model's representation space. After a user inputs an image and a text question, the MLLM can progressively generate the corresponding text response based on the image's visual features and the text context.

[0003] However, multimodal large language models are still prone to the "object illusion" problem when generating text, meaning the content generated by the model is inconsistent with the input image, or it generates objects, attributes, or relationships that do not exist in the input image. Object illusion is a relatively common and impactful form of illusion, specifically manifesting as the model describing objects that do not exist in the input image during image description or visual question answering. For example, objects such as "pedestrian," "table," or "chair" do not exist in the image, but the model still generates related descriptions. Object illusion reduces the credibility of the model's output, and can pose a particular risk in applications with high security and realism requirements, such as medical diagnosis, autonomous driving, security monitoring, and industrial quality inspection.

[0004] In the existing technology, there are several types of processing methods for the hallucination problem in multimodal large language models. The first type is the optimization method in the training stage, such as constructing an instruction fine-tuning dataset containing hallucination-sensitive samples, or using reinforcement learning, human feedback, preference optimization, contrastive learning and other methods to improve the model's ability to utilize visual information. The first category consists of methods that typically require additional training, labeled data, or computational resources, resulting in high implementation costs and limited transferability and deployment convenience across different models. The second category comprises contrastive decoding methods during the inference phase, such as comparing output distributions between the original and perturbed images, under different visual conditions, or between different decoding branches to suppress illusions caused by language priors. These methods usually do not require retraining the model but often require additional inference branches or multiple forward computations, resulting in significant inference overhead and potentially affecting the coherence of the output text. The third category comprises visual attention enhancement or visual feature re-introduction methods, such as enhancing cross-modal attention during inference or re-injecting visual features during decoding to reduce misrepresentations caused by language priors. These methods can improve the model's ability to utilize visual information to some extent, but they typically focus on enhancing visual dependence without fully considering the potential side effects of excessively enhancing visual attention.

[0005] The applicant's research found that in tasks such as open-ended image description, enhancing visual attention does not always reduce hallucinations. Conversely, while increasing visual attention from low to medium-high levels may improve the model's recall of real objects, it may decrease the proportion of responses that do not include hallucinatory objects. In other words, while the model can describe more real objects, it is also more likely to introduce objects that are not present in the image. Further analysis revealed that excessive visual attention can cause the model to add new, non-existent objects in the latter part of the generated response; this phenomenon can be termed additive object hallucination or incremental object hallucination. Therefore, simply enhancing visual attention can lead to a contradiction between perceptual integrity and factual consistency.

[0006] Currently, the main drawbacks of existing technologies include: First, optimization methods during the training phase require additional data, additional training, or manual annotation, resulting in high costs and long cycles, making them difficult to quickly deploy to existing models; second, comparison and decoding during the inference phase typically require multiple forward computations or the introduction of perturbation inputs, leading to significant inference overhead and potentially affecting text coherence; third, existing visual attention enhancement methods mostly employ a uniform enhancement strategy, lacking adaptive judgments for different generation locations and uncertainties, easily introducing unnecessary interventions at locations where the model can already reliably predict; fourth, existing methods do not adequately consider the additional object illusions caused by excessively strong visual attention; and fifth, the intensity of a single visual attention is insufficient to simultaneously address object recall and illusion suppression, resulting in poor reliability, which requires further improvement. Summary of the Invention

[0007] Based on this, embodiments of this application provide a method and system for suppressing object illusions in multimodal large language models, in order to solve the problem of poor reliability in the prior art.

[0008] In a first aspect, embodiments of this application provide a method for suppressing object illusions in a multimodal large language model, the method comprising: Obtain input image information and text prompt information; Based on the input image information and text prompt information, during the process of generating answer text word by word in the multimodal large language model, the image-related attention score information is determined according to the current generation position; Based on the preset global main path, the image-related attention score information is conservatively adjusted to generate main path candidate output distribution information; Based on the main path candidate output distribution information, determine whether the currently generated position belongs to the position to be enhanced; If the current generated position belongs to the position to be enhanced, the image-related attention score information is enhanced and adjusted based on the preset local correction path to generate correction path candidate output distribution information, wherein the visual attention intensity of the enhanced adjustment is higher than the visual attention intensity of the conservative adjustment. Compare the reliability information of the candidate output distribution information of the correction path with the reliability information of the candidate output distribution information of the main path; If the reliability information of the corrected path candidate output distribution information is higher than that of the main path candidate output distribution information, then the corrected path candidate output distribution information is used as the final output distribution information of the current generation position; otherwise, the main path candidate output distribution information is used as the final output distribution information of the current generation position. Based on the final output distribution information, the current word information is generated, and the process proceeds to the next generation position until the preset stopping condition is met, generating complete answer text information.

[0009] Compared with existing technologies, the beneficial effects are as follows: The object illusion suppression method for multimodal large language models provided in this application allows the terminal device to first acquire input image information and text prompt information. Then, based on the input image information and text prompt information, during the word-by-word generation of response text in the multimodal large language model, the image-related attention score information is effectively determined according to the current generation position. Then, based on a preset global main path, the image-related attention score information is conservatively adjusted to quickly generate main path candidate output distribution information. Then, based on the main path candidate output distribution information, it is determined whether the current generation position belongs to a position to be enhanced. If the current generation position belongs to a position to be enhanced, the image-related attention score information is enhanced and adjusted based on a preset local correction path to reliably generate correction path candidate output distribution information, which is then compared... The reliability information of the candidate output distribution information of the correction path is compared with that of the candidate output distribution information of the main path. If the reliability information of the candidate output distribution information of the correction path is higher than that of the candidate output distribution information of the main path, then the candidate output distribution information of the correction path is used as the final output distribution information of the current generation position. Otherwise, the candidate output distribution information of the main path is used as the final output distribution information of the current generation position. Finally, based on the final output distribution information, the current word information is generated and the next generation position is entered until the preset stopping condition is met, effectively generating complete answer text information. This can reduce object illusion while maintaining or improving the model's recall ability and description completeness of real image objects, improving the consistency, reliability and security between the output content and the input image, and solving the problem of poor reliability to a certain extent.

[0010] It should be noted that this application does not simply enhance or weaken the visual attention of multimodal large language models. Instead, it focuses on the new discovery that "excessive visual attention leads to additional object illusions" and constructs a dynamic inference mechanism that uses the conservative global main path as the default path, the local correction path as the exception enhancement path, and uses prediction entropy and confidence for gating selection. Furthermore, this application re-models the problem of "excessive visual attention leading to additional object illusions." Existing multimodal large language model illusion suppression methods typically attribute object illusions primarily to insufficient utilization of visual information, overly strong language priors, or insufficient visual grounding. Therefore, common approaches include enhancing visual information, introducing contrastive decoding, re-injecting visual features, or aligning during the training phase. This application differs in that the applicant has discovered that higher visual attention intensity is not always better: weak visual attention may lead to the omission of real objects, while excessive visual attention may cause the model to over-interpret local textures, background regions, blurred regions, or weakly correlated visual signals, thus adding objects not present in the input image in the latter part of the response. This application treats this phenomenon as a specific technical problem, avoiding the introduction of new, non-existent objects while improving the recall of real objects using existing visual enhancement methods.

[0011] Without loss of generality, this application establishes an asymmetric dual-path structure with a "conservative main path priority." That is, the global main path and local correction paths are not simply parallel or averaged together, but rather an asymmetric structure with clear priorities. The global main path is the default generation path, which conservatively adjusts image-related attention, ensuring the model maintains moderate and stable visual dependence in most generation locations. The local correction path is merely an exception correction path, providing a candidate result with stronger visual attention only when correction is necessary on the main path. Therefore, this application is not a simple combination of "global attention + local attention," but rather suppresses excessive visual attention through the global main path, restores visual evidence at necessary locations through the local correction path, and ultimately the output defaults to the main path, only being covered by the local path when certain conditions are met.

[0012] Furthermore, this application limits local visual enhancement to exception-based coverage based on uncertainty, rather than default enhancement. Specifically, at the current generation position, the model first generates candidate outputs through a conservative main path and calculates the prediction uncertainty of the candidate output distribution. Only when the current generation position meets a preset correction necessity condition, such as the main path prediction entropy exceeding a threshold, is a more visually attentive local correction path activated. This design limits local visual enhancement to locations where erroneous generation is more likely, rather than indiscriminately enhancing visual information at all generation positions, thereby reducing the risk of strong visual attention introducing the illusion of new objects at unnecessary locations.

[0013] Without loss of generality, this application proposes a dual-condition gating mechanism combining prediction entropy and confidence, meaning that this application does not blindly trust local correction paths. Even if the current generation position is a high-uncertainty position, the correction path result is only adopted when the candidate output distribution of the local correction path shows higher reliability than the main path. The reliability of this gating mechanism can be measured by indicators such as maximum probability, the difference between the top two probabilities, the cumulative probability of the top k candidates, the negative entropy of the output distribution, the magnitude of entropy decrease, the consistency between candidate object terms and image regions, and multi-level or multi-head consistency. Through dual-condition gating, this application can avoid introducing new object illusions when strong visual attention is unclear in visual evidence, blurry in image regions, complex in background, or when the model mistakenly focuses on the wrong region.

[0014] Furthermore, this application provides a training-independent, plug-and-play inference framework adaptable to various model structures. Specifically, this application can be applied to the image-related attention score and output distribution selection process during the model decoding stage, typically without requiring retraining of the original model or modification of its main parameters, thus offering good deployment convenience and model compatibility. This method can be applied to cross-attention multimodal models, self-attention multimodal models concatenating image and text terms, and multimodal large language models that incorporate visual information through visual adapters, projection layers, visual prefixes, or visual memory units.

[0015] Secondly, embodiments of this application provide an object illusion suppression system for multimodal large language models, the system comprising: Input image information acquisition module: used to acquire input image information and text prompt information; Image-related attention score information determination module: Based on the input image information and text prompt information, during the process of generating answer text word by word in a multimodal large language model, the module determines the image-related attention score information according to the current generation position. Main path candidate output distribution information generation module: Based on the preset global main path, it conservatively adjusts the image-related attention score information to generate main path candidate output distribution information; Current generated location determination module: used to determine whether the current generated location belongs to the location to be enhanced based on the main path candidate output distribution information; Correction path candidate output distribution information generation module: If the current generation position belongs to the position to be enhanced, it enhances and adjusts the image-related attention score information based on the preset local correction path to generate correction path candidate output distribution information, wherein the visual attention intensity of the enhancement adjustment is higher than that of the conservative adjustment. Reliability Information Comparison Module: Used to compare the reliability information of the candidate output distribution information of the correction path with the reliability information of the candidate output distribution information of the main path; Final output distribution information determination module: If the reliability information of the candidate output distribution information of the correction path is higher than that of the candidate output distribution information of the main path, then the candidate output distribution information of the correction path will be used as the final output distribution information of the current generation position; otherwise, the candidate output distribution information of the main path will be used as the final output distribution information of the current generation position. The complete answer text information generation module is used to generate the current word information based on the final output distribution information, and then proceed to the next generation position until the preset stopping condition is met to generate the complete answer text information.

[0016] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0018] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a schematic flowchart of an embodiment of the object illusion suppression method provided in this application; Figure 2 This is a flowchart illustrating step S200 in an embodiment of the object illusion suppression method provided in this application; Figure 3 This is a flowchart illustrating step S300 in an embodiment of the object illusion suppression method provided in this application; Figure 4 This is a flowchart illustrating step S400 in an embodiment of the object illusion suppression method provided in this application; Figure 5 This is a flowchart illustrating step S500 in an embodiment of the object illusion suppression method provided in this application; Figure 6 This is a flowchart illustrating step S700 in an embodiment of the object illusion suppression method provided in this application; Figure 7 This is a schematic diagram of the overall execution process provided in an embodiment of this application; Figure 8 This is a block diagram of an object illusion suppression system provided in an embodiment of this application; Figure 9 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0024] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating the object illusion suppression method for multimodal large language models provided in this embodiment. In this embodiment, the execution subject of the object illusion suppression method is a terminal device. It is understood that the types of terminal devices include, but are not limited to, tablet computers, laptops, Ultra-Mobile Personal Computers (UMPCs), netbooks, Personal Digital Assistants (PDAs), etc., and this embodiment does not impose any restrictions on the specific type of terminal device.

[0026] Please see Figure 1The object illusion suppression method provided in this application includes, but is not limited to, the following steps: In S100, input image information and text prompt information are obtained.

[0027] Specifically, the terminal device can obtain input image information and text prompt information.

[0028] For example, the terminal device can acquire an image I and a text prompt X input by the user, wherein the text prompt can be an open-ended image description instruction, a visual question-and-answer question, a multi-turn dialogue context, an object recognition question, or other multimodal task instruction.

[0029] In S200, based on the input image information and text prompt information, during the process of generating the answer text word by word in the multimodal large language model, the image-related attention score information is determined according to the current generation position.

[0030] Specifically, after the terminal device acquires the input image information and text prompt information, it can quickly determine the image-related attention score information based on the current generation position during the process of generating the answer text word by word in the multimodal large language model, based on the input image information and text prompt information. The image-related attention score information describes the image-related attention score from the current generation position to the image visual word; the image-related attention score information is the attention score before softmax normalization.

[0031] For example, the terminal device can use a preset visual encoding module to encode the input image to obtain an image visual word set V, and use a text encoding module to perform lexicalization and embedding processing on the text prompt to obtain a text word representation. Then, the multimodal large language model decoding module enters the decoding process at the current generation position t based on the image visual words, text prompt words, and the historical words that have been generated, and obtains the image-related attention score Z_img from the text generation position to the image visual words.

[0032] In some possible implementations, for determining image-related attention score information, please refer to [link to relevant documentation]. Figure 2 Step S200 includes, but is not limited to, the following steps: In S210, the input image information is encoded to generate a set of image visual lexical terms.

[0033] Specifically, the terminal device can encode the input image information to effectively generate image visual lexical set information, which includes multiple image visual lexical information.

[0034] In S220, the text prompt information is lexicalized and embedded to generate text lexical information.

[0035] Specifically, after the terminal device generates the image visual lexical set information, the terminal device can perform lexicalization and embedding processing on the text prompt information to quickly generate text lexical information.

[0036] In S230, when the multimodal large language model uses a cross-attention mechanism to fuse image visual lexical information, the current generation position is used as the query, and the image visual lexical is used as the key and value. The cross-attention scores from the current generation position to each image visual lexical information are extracted to determine the image-related attention score information.

[0037] Specifically, when a multimodal large language model uses a cross-attention mechanism to fuse image visual lexical information, the terminal device can use the current generation location as a query and the image visual lexical as a key and value to extract the cross-attention scores from the current generation location to each image visual lexical information, effectively determining the image-related attention score information, thereby achieving the acquisition of image-related attention scores.

[0038] For example, a multimodal large language model can use a cross-attention mechanism to introduce visual information. In this case, the current text generation position is used as the query end, and the image visual words are used as keys and values. The model calculates the cross-attention scores from the current generation position to each image visual word.

[0039] Alternatively, in S240, when the multimodal large language model concatenates image visual lexical information and text lexical information and then fuses them through a self-attention mechanism, the attention scores from the current generation position to each image visual lexical information are extracted from the self-attention matrix to determine the image-related attention score information.

[0040] Specifically, when a multimodal large language model concatenates image visual lexical information with textual lexical information and then fuses them through a self-attention mechanism, the terminal device can extract the attention scores from the current generation position to each image visual lexical information from the self-attention matrix, effectively determining the image-related attention score information.

[0041] For example, the multimodal large language model concatenates image visual words and text words and inputs them into the large language model, and performs multimodal fusion through a self-attention mechanism. At this time, the terminal device can distinguish image visual words and text words according to the word position index, and extract the attention score from the current text generation position to the image visual word position from the self-attention matrix as the image-related attention score.

[0042] In some possible implementations, in order to obtain image-related attention scores, the multimodal large language model maps visual features to the language model representation space through a visual adapter, projection layer, or connection module. In this case, the terminal device can use the attention scores of the current generation position to the visual adapter output features, visual memory units, or visual prefix words as image-related attention scores. Preferably, this application adjusts the image-related attention scores before softmax normalization, adjusting the relative dependence of the current generation position on image information and language context information from the generation mechanism level, rather than performing simple text filtering after the generation results.

[0043] In S300, based on the preset global main path, the image-related attention score information is conservatively adjusted to generate the main path candidate output distribution information.

[0044] Specifically, after the terminal device determines the image-related attention score information, it can conservatively adjust the image-related attention score information based on the preset global main path to quickly generate the main path candidate output distribution information.

[0045] In some possible implementations, to generate primary path candidate output distribution information, please refer to [link / reference]. Figure 3 Step S300 includes, but is not limited to, the following steps: In S310, based on the preset first visual attention correction parameters, the image-related attention score information is scaled to generate the first processing data information.

[0046] Specifically, the terminal device can scale the image-related attention score information based on preset first visual attention correction parameters to effectively generate first processed data information.

[0047] In S320, the first processed data information is forward-computed to generate the main path candidate output distribution information.

[0048] Specifically, after the terminal device generates the first processed data information, the terminal device can perform forward calculations on the first processed data information to quickly generate the main path candidate output distribution information.

[0049] For example, the terminal device can conservatively adjust the image-related attention score through the global main path; in one possible implementation, the main path attention score is... Based on the adjusted attention score, the forward computation of the current generation position is performed to obtain the main path candidate output distribution P_g=Softmax(Logit_g).

[0050] In S400, based on the distribution information of the main path candidate output, it is determined whether the currently generated position belongs to the position to be enhanced.

[0051] Specifically, after the terminal device generates the main path candidate output distribution information, the terminal device can determine whether the currently generated position belongs to the position to be enhanced based on the main path candidate output distribution information. The position to be enhanced is used to describe the high uncertainty position.

[0052] In some possible implementations, to effectively determine whether the currently generated position belongs to the position to be enhanced, please refer to [link / reference needed]. Figure 4 Step S400 includes, but is not limited to, the following steps: In S410, the prediction entropy information of the candidate output distribution information of the main path is calculated.

[0053] Specifically, the terminal device can first calculate the predicted entropy information of the main path candidate output distribution information.

[0054] In S420, the predicted entropy information is compared with the preset enhancement threshold.

[0055] Specifically, after the terminal device calculates the predicted entropy information, the terminal device can compare the predicted entropy information with the preset enhancement threshold.

[0056] In S430, if the predicted entropy information is greater than the preset enhancement threshold, then the current generated position is determined to be a position to be enhanced; otherwise, the current generated position is determined not to be a position to be enhanced.

[0057] Specifically, if the predicted entropy information is greater than the preset enhancement threshold, the terminal device can determine that the current generated position belongs to the position to be enhanced; otherwise, the terminal device can determine that the current generated position does not belong to the position to be enhanced.

[0058] For example, the terminal device can calculate the uncertainty of the current generated position based on the candidate output distribution P_g of the main path. Preferably, the terminal device can use the prediction entropy H(P_g)=-ΣP_g(v)logP_g(v) as the uncertainty index and compare the uncertainty with a preset threshold delta to determine whether the current generated position belongs to a high uncertainty position.

[0059] In S500, if the current generated position belongs to the position to be enhanced, the image-related attention score information is enhanced and adjusted based on the preset local correction path to generate correction path candidate output distribution information.

[0060] Specifically, if the current generated position is a position to be enhanced, the terminal device can enhance and adjust the image-related attention score information based on a preset local correction path, effectively generating candidate output distribution information of the correction path. The visual attention intensity of the enhanced adjustment is higher than that of the conservative adjustment.

[0061] For example, when the current generation location is a high-uncertainty location, the terminal device can enable a local correction path to strongly adjust the image-related attention score; in some possible implementations, the correction path attention score is... , where gamma2>gamma1, and the corrected path candidate output distribution P_l=Softmax(Logit_l) is obtained based on the adjusted attention score.

[0062] In some possible implementations, to generate candidate output distribution information for the correction path, please refer to [link / reference needed]. Figure 5 Step S500 includes, but is not limited to, the following steps: In S510, if it is determined that the current generated position belongs to the position to be enhanced, the image-related attention score information is scaled based on the preset second visual attention correction parameters to generate second processing data information.

[0063] Specifically, if it is determined that the current generated location belongs to the location to be enhanced, the terminal device can scale the image-related attention score information based on the preset second visual attention correction parameters to effectively generate the second processing data information.

[0064] In S520, the second processed data information is forward-computed to generate the candidate output distribution information of the correction path.

[0065] Specifically, after the terminal device generates the second processed data information, the terminal device can perform forward calculations on the second processed data information to quickly generate candidate output distribution information for the correction path, wherein the second visual attention correction parameter is greater than the first visual attention correction parameter.

[0066] In S600, the reliability information of the candidate output distribution information of the correction path is compared with the reliability information of the candidate output distribution information of the main path.

[0067] Specifically, after the terminal device generates the correction path candidate output distribution information, the terminal device can compare the reliability information of the correction path candidate output distribution information with the reliability information of the main path candidate output distribution information.

[0068] For example, the terminal device can compare the reliability of the main path candidate output distribution P_g and the correction path candidate output distribution P_l; the reliability can be determined by indicators such as the maximum probability C=max(P), the difference between the top two probabilities, the cumulative probability of the top k candidate words, the negative entropy of the output distribution, the magnitude of entropy decrease, or the consistency between candidate words and image regions.

[0069] In S700, if the reliability information of the corrected path candidate output distribution information is higher than that of the main path candidate output distribution information, then the corrected path candidate output distribution information is used as the final output distribution information of the current generated position; otherwise, the main path candidate output distribution information is used as the final output distribution information of the current generated position.

[0070] Specifically, if the reliability of the corrected path candidate output distribution information is higher than that of the main path candidate output distribution information, the terminal device can use the corrected path candidate output distribution information as the final output distribution information of the current generated position; otherwise, the terminal device can use the main path candidate output distribution information as the final output distribution information of the current generated position.

[0071] For example, when H(P_g)>delta and the reliability of the correction path is higher than that of the main path, the terminal device uses P_l as the final output distribution; otherwise, the terminal device uses P_g as the final output distribution.

[0072] In some possible implementations, to determine the final output distribution information of the previous generation positions, please refer to [link / reference]. Figure 6 Step S700 includes, but is not limited to, the following steps: In S710, if the reliability information of the correction path candidate output distribution information is higher than the reliability information of the main path candidate output distribution information, then it is determined whether the difference between the reliability information of the correction path candidate output distribution information and the reliability information of the main path candidate output distribution information is greater than the preset confidence difference threshold information.

[0073] Specifically, if the reliability information of the correction path candidate output distribution information is higher than that of the main path candidate output distribution information, the terminal device can determine whether the difference between the reliability information of the correction path candidate output distribution information and the reliability information of the main path candidate output distribution information is greater than a preset confidence difference threshold. For example, if an optional confidence difference threshold lambda is set, the terminal device can further limit the local correction path to covering the main path only after C_l > C_g + lambda. In S720, if the difference between the reliability information of the candidate output distribution information of the corrected path and the reliability information of the candidate output distribution information of the main path is greater than the preset confidence difference threshold, then the candidate output distribution information of the corrected path will be used as the final output distribution information of the current generated position.

[0074] Specifically, if the difference between the reliability information of the candidate output distribution information of the correction path and the reliability information of the candidate output distribution information of the main path is greater than the preset confidence difference threshold, the terminal device can use the candidate output distribution information of the correction path as the final output distribution information of the current generation location.

[0075] In S800, based on the final output distribution information, the current word information is generated, and the process moves to the next generation position until the preset stopping condition is met, generating complete answer text information.

[0076] Specifically, after the terminal device determines the final output distribution information of the current generation position, it can quickly generate the current word information based on the final output distribution information and proceed to the next generation position until the preset stopping condition is met, effectively generating complete answer text information. This proposes a multimodal large language model inference method that does not require model retraining, can dynamically adjust visual attention during the inference stage, and can maintain perceptual integrity while suppressing object illusion. It achieves dynamic visual attention control at the word level. Compared with the method of uniformly applying visual enhancement to the entire answer, this application only enhances visual attention at the position where the main path prediction is uncertain and the local correction path is more credible. Therefore, it can reduce unnecessary intervention, reduce the risk of additional object illusion, and control additional inference overhead.

[0077] For example, please refer to Figure 7 , Figure 7 This describes the process in this application: acquiring the image-related attention score Z_img at each generation position, performing global main path calculation, determining whether to enable the local correction path based on the prediction entropy, determining P_final through reliability gating, and iteratively generating the current word until a complete answer is output. The terminal device can generate the current word y_t based on the final output distribution P_final. The generation method can be greedy decoding, sampling decoding, top-k sampling, kernel sampling, beam search, or other text generation methods. Then, the current word is added to the already generated text sequence, and the process proceeds to the next generation position, repeating the above steps until an end marker is generated, the maximum generation length is reached, the output content meets the user's question answer requirements, or other preset stopping rules are triggered, ultimately outputting a complete text answer.

[0078] It should be noted that, for the convenience of understanding the present application, the terms and variables are described again as follows: the terminal device may denote the input image as I, and denote the text prompt input by the user as X; after encoding the input image I by the visual encoding module, an image visual token set V is obtained, which may include m image visual tokens from v1 to vm; at the t-th generation position, the multimodal large language model generates a current token y_t based on the image visual tokens, text prompt tokens and already generated historical tokens; the image-related attention score from the current generation position to the image visual tokens is denoted as Z_img. Said score may be an attention score before softmax normalization, or may be another score that can characterize the degree of association between the current text generation position and the image visual tokens before attention normalization; the candidate output distribution of the main path is denoted as P_g, which represents the probability prediction distribution of the model for the current generated token under the condition of global main path visual attention; the candidate output distribution of the correction path is denoted as P_l, which represents the probability prediction distribution of the model for the current generated token under the condition of local correction path visual attention; the final output distribution is denoted as P_final, which represents the output distribution actually used for generating the token at the current generation position; gamma1 represents the first visual attention correction parameter of the global main path, gamma2 represents the second visual attention correction parameter of the local correction path, and gamma1 is usually set to be less than gamma2; delta represents an uncertainty threshold; lambda represents an optional confidence difference threshold. All the above variable names are exemplary expressions, and other symbols or parameter names may also be adopted in actual implementation.

[0079] In some possible implementations, "conservative adjustment" in the global main path is not necessarily limited to absolutely reducing visual attention, but refers to adopting a visual adjustment method with lower intensity, higher stability, and less aggressiveness compared with the local correction path; in addition to multiplicative scaling, the main path can also be implemented by means of central scaling, additive bias, temperature adjustment, re-calibration before normalization, re-weighting after normalization, or selective enhancement of visual tokens, etc.

[0080] Without loss of generality, the local correction path is also not limited to a single multiplicative scaling method, and can be implemented by means of central scaling, additive bias, temperature adjustment, visual token filtering, visual feature gating, or visual context re-injection, etc. The local correction path can be enabled only at high-uncertainty positions, and can also be triggered in combination with the type of generated content, for example, it is enabled only when the candidate token may be an object noun, an attribute word, a spatial relation word, a quantifier, or a word related to medical diagnosis.

[0081] Specifically, the final output distribution can be determined by choosing between the main path candidate output distribution and the correction path candidate output distribution. Alternatively, weights can be assigned to the two paths based on prediction uncertainty and reliability, and the candidate output distributions of the two paths can be weighted and fused. Furthermore, multi-level local correction paths can be set up to use visual correction of different intensities under different degrees of uncertainty.

[0082] Therefore, this application can perform visual attention correction on all decoding layers of a multimodal large language model, or it can perform correction only on some decoding layers, some visually related attention heads, or some image visual lexical units. For example, in a preferred embodiment, the original attention mechanism of the first few layers is retained unchanged, and the correction method of this application is performed only from the middle layers or the last few layers to achieve a balance between illusion suppression effect and inference efficiency.

[0083] In some possible implementations, the first visual attention correction parameter gamma1 is used to control the visual attention intensity in the global main path, preferably set to a relatively conservative visual attention intensity so that the model does not over-amplify the visual information of the image during the default generation process; the second visual attention correction parameter gamma2 is used to control the visual attention intensity in the local correction path, preferably set to be greater than gamma1, so as to enhance the visual evidence of the image at high uncertainty positions; the uncertainty threshold delta is used to determine whether the current generation position belongs to a high uncertainty position. delta can be determined by searching the validation set, or it can be dynamically adjusted according to the input complexity, task type or generation stage; in a specific experimental setting, delta=1.5 is fixed for multiple tasks, and ablation experiments show that the method can remain effective in a wide threshold range.

[0084] In some possible implementations, an optional confidence difference threshold, lambda, is used to avoid the problem of frequent switching of output distributions even when the confidence differences between two paths are small. When lambda is set, the main path is only allowed to be covered if the confidence of the corrected path exceeds the confidence of the main path by a certain margin.

[0085] In some possible implementations, parameters can be determined through validation set search; specifically, different parameter combinations can be tested on a set of image descriptions or visual question answering validation samples, and the optimal parameters can be selected based on indicators such as object illusion rate, real object recall rate, answer accuracy, text fluency, and average inference time.

[0086] It should be noted that Table 1 lists some typical experimental results of this application on the CHAIR benchmark, which are used to illustrate the effect of this application in suppressing object illusion and maintaining object recall. As can be seen from Table 1, this application has, but is not limited to, the following six positive effects: (1) This application does not require retraining of the model and is easy to deploy. That is, this application adjusts the image-related attention score and performs candidate output gating selection during the inference stage. Usually, no additional training data, manual annotation, model retraining or external correction modules are required. It can be integrated into the existing multimodal large language model system as a plug-in inference strategy.

[0087] (2) This application specifically suppresses the illusion of additional objects. Unlike methods that simply enhance visual attention, this application adopts a conservative main path of visual attention by default, which can prevent the model from being in a state of strong visual attention throughout the generation process, thereby reducing the probability of adding non-existent objects in the later stages of the answer due to excessive visual attention.

[0088] (3) This application maintains or enhances the ability to recall real objects. That is, this application does not simply reduce visual attention, but enables local correction paths at high uncertainty positions on the main path to enhance the model’s ability to utilize visual evidence in images, so as to correct the omission and incorrect generation of real objects caused by insufficient language priors, co-occurrence bias of training corpus, or visual dependence.

[0089] (4) This application achieves dynamic control at the word level, reducing unnecessary intervention. That is, this application determines whether to use the local correction result at each generation position based on the prediction entropy and reliability. Compared with the method of uniformly applying visual enhancement to the whole paragraph, it can better balance factual consistency, object recall ability, text fluency and reasoning efficiency.

[0090] (5) This application has strong cross-model and cross-task applicability. That is, this application can be used for tasks such as image description, visual question answering, multimodal dialogue, medical image interpretation, autonomous driving scene understanding, robot visual interaction, industrial quality inspection image interpretation, remote sensing image interpretation, e-commerce product image description generation, etc., and can be adapted to different visual encoders, connection modules and large language model decoding structures.

[0091] (6) The experimental verification results of this application are clear. That is, this application has verified the LLaVA-v1.5-7B, LLaVA-v1.5-13B and MiniGPT-4 models on benchmarks such as CHAIR, AMBER and MM-Vet. Taking the CHAIR benchmark as an example, GLVAC reduces CHAIRs and CHAIRi on multiple models and suppresses object hallucinations while maintaining or even improving Recall, indicating that it can achieve a better balance between hallucination suppression and perceptual integrity.

[0092] Table 1. Experimental Results of the CHAIR Baseline Section Model method CHAIRS ↓ CHAIRi ↓ Recall ↑ LLaVA-v1.5-7B regular 50.4 14.2 80.6 LLaVA-v1.5-7B GLVAC 46.2 13.9 81.2 LLaVA-v1.5-13B regular 48.2 13.2 79.4 LLaVA-v1.5-13B GLVAC 44.4 12.2 80.9 MiniGPT-4 regular 37.6 14.4 59.4 MiniGPT-4 GLVAC 36.6 11.7 63.5 In some possible implementations, this application can be combined with the following specific application scenarios: (1) Object illusion suppression system in open image description service: In the cloud image description service, the system deploys an existing multimodal large language model, such as a model composed of a visual encoder, a projection connection layer and a large language model decoder. After the user uploads an image and enters a prompt such as "Please describe the image in detail", the system first uses the visual encoder to obtain the image visual lexicon set V, and uses the text encoding module to obtain the text prompt and the historical response lexicon representation.

[0093] Specifically, at the t-th generation position, the system extracts the normalized pre-attention score Z_img from the current text generation position to the image visual word from the model's cross-attention layer or self-attention layer. The global main path conservatively adjusts Z_img with a first scaling factor gamma1, obtaining the main path candidate output distribution P_g. The system calculates the prediction entropy H(P_g) of P_g. When H(P_g) does not exceed the threshold delta, P_g is directly used to generate the current word; when H(P_g) exceeds the threshold delta, the local correction path is enabled.

[0094] Without loss of generality, the local correction path applies a stronger visual adjustment to the same Z_img using a second scaling factor gamma2, resulting in a candidate output distribution P_l. The system compares P_l with P_g based on the maximum probability or other reliability metrics; only when the reliability of P_l is higher than that of P_g is P_l adopted as the final output distribution; otherwise, P_g is still used. This process iterates through each word until a terminator is generated or the maximum length is reached, thereby outputting descriptive text that better reflects the facts of the input image.

[0095] (2) Visual question answering reasoning deployment for edge computing devices: On in-vehicle computing devices, security terminals, or industrial quality inspection edge devices, due to limitations in computing power and latency requirements, the system can execute the correction method of this application only in the middle and later decoding layers or in attention heads with high visual relevance, while retaining the original attention mechanism of the first few layers unchanged. This setting can reduce additional computation while ensuring that the semantic layers that have a significant impact on the final lexical decision are dynamically corrected.

[0096] Specifically, when the input question is something like "Does an object exist in the image?", "How many objects are in the image?", or "Please describe the potential risks in the scene", the system can perform local correction paths only on candidate object nouns, attribute words, spatial relation words, quantity words, or high-risk diagnostic / judgment words; for common function words, conjunction words, punctuation marks, etc., the main path result is used first. This implementation can further reduce unnecessary local corrections and improve real-time inference efficiency.

[0097] (3) Parameter determination and effect verification process: Before model deployment, engineers can search for parameters such as gamma1, gamma2, delta, and optional lambda on a small number of validation samples. The search target can comprehensively consider the object illusion rate, real object recall rate, visual question answering accuracy, text fluency, and average inference time. It is usually required that gamma1 is less than gamma2 to keep the global main path relatively conservative, while the local correction path provides stronger visual evidence at necessary locations.

[0098] In one possible implementation, the terminal device can use prediction entropy as a necessity indicator for correction and the maximum probability of the candidate distribution as a reliability indicator. In another possible implementation, the terminal device can use probability interval, logit interval, negative entropy of candidate distribution, consistency between candidate object lexical and visual region, or multi-level / multi-head consistency as alternative indicators. After confirming the parameters through the validation set, the inference strategy is solidified into the online inference service or offline batch generation process.

[0099] The implementation principle of the object illusion suppression method for multimodal large language models in this application embodiment is as follows: The terminal device can first acquire input image information and text prompt information. Then, based on the input image information and text prompt information, during the word-by-word generation of response text in the multimodal large language model, the image-related attention score information is effectively determined according to the current generation position. Then, based on the preset global main path, the image-related attention score information is conservatively adjusted to quickly generate the main path candidate output distribution information. Then, based on the main path candidate output distribution information, it is determined whether the current generation position belongs to the position to be enhanced. If the current generation position belongs to the position to be enhanced, the image-related attention score information is enhanced and adjusted based on the preset local correction path to reliably generate the correction path candidate output distribution information. Then, the reliability information of the correction path candidate output distribution information is compared with the reliability information of the main path candidate output distribution information. If the reliability information of the correction path candidate output distribution information is higher than the reliability information of the main path candidate output distribution information, the correction path candidate output distribution information is... The final output distribution information of the current generation position is used as the final output distribution information of the current generation position; otherwise, the candidate output distribution information of the main path is used as the final output distribution information of the current generation position. Finally, based on the final output distribution information, the current word information is generated and the next generation position is entered until the preset stopping condition is met, effectively generating complete answer text information. Thus, without retraining the model or adding external training modules, the dependence of the multimodal large language model on visual information at different generation positions can be dynamically adjusted during the inference stage. The global main path suppresses the additional object illusion caused by excessive visual attention through conservative visual attention adjustment, and the local correction path can enhance visual attention at high uncertainty positions of the model to correct the erroneous generation caused by strong language priors or insufficient visual dependence. At the same time, the dual-condition gating mechanism based on prediction entropy and confidence can reduce unnecessary or unreliable corrections. Therefore, this application can maintain or improve the model's recall ability and descriptive integrity of real image objects while reducing object illusion, and improve the consistency, reliability and security between the output content and the input image.

[0100] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] Embodiments of this application also provide an object illusion suppression system for multimodal large language models. This system can be deployed in cloud servers, local servers, edge computing devices, smart terminals, in-vehicle computing devices, medical image analysis equipment, industrial quality inspection equipment, or other electronic devices with artificial intelligence model reasoning capabilities. For ease of explanation, only the parts relevant to this application are shown. Figure 8As shown, the system 80 includes: Input image information acquisition module 81: used to acquire input image information and text prompt information; Image-related attention score information determination module 82: used to determine image-related attention score information based on the current generation position during the process of generating answer text word by word in a multimodal large language model, based on input image information and text prompt information; Main path candidate output distribution information generation module 83: Based on the preset global main path, it conservatively adjusts the image-related attention score information to generate main path candidate output distribution information; Current generated location determination module 84: used to determine whether the current generated location belongs to the location to be enhanced based on the main path candidate output distribution information; Correction path candidate output distribution information generation module 85: If the current generation position belongs to the position to be enhanced, it enhances and adjusts the image-related attention score information based on the preset local correction path to generate correction path candidate output distribution information, wherein the visual attention intensity of the enhancement adjustment is higher than that of the conservative adjustment. Reliability Information Comparison Module 88: Used to compare the reliability information of the correction path candidate output distribution information with the reliability information of the main path candidate output distribution information; Final output distribution information determination module 89: If the reliability information of the candidate output distribution information of the correction path is higher than that of the candidate output distribution information of the main path, then the candidate output distribution information of the correction path will be used as the final output distribution information of the current generation position; otherwise, the candidate output distribution information of the main path will be used as the final output distribution information of the current generation position. The complete answer text information generation module 88 is used to generate the current word information based on the final output distribution information, and then proceed to the next generation position until the preset stopping condition is met to generate the complete answer text information.

[0102] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0103] This application also provides a terminal device, such as... Figure 9 As shown, the terminal device 90 of this embodiment includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91. When the processor 91 executes the computer program 93, it implements the steps in the above-described embodiment of the object illusion suppression method, for example... Figure 1Steps S100 to S800 are shown; or, when processor 91 executes computer program 93, it implements the functions of each module in the above-described device, for example... Figure 8 The functions of modules 81 to 88 are shown.

[0104] The terminal device 90 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The terminal device 90 includes, but is not limited to, a processor 91 and a memory 92. Those skilled in the art will understand that... Figure 9 This is merely an example of terminal device 90 and does not constitute a limitation on terminal device 90. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 90 may also include input / output devices, network access devices, buses, etc.

[0105] The processor 91 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0106] The memory 92 can be an internal storage unit of the terminal device 90, such as the hard disk or memory of the terminal device 90. The memory 92 can also be an external storage device of the terminal device 90, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 90. Furthermore, the memory 92 can include both internal storage units and external storage devices of the terminal device 90. The memory 92 can also store computer program 93 and other programs and data required by the terminal device 90. The memory 92 can also be used to temporarily store data that has been output or will be output.

[0107] One embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0108] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the methods, principles and structures of this application should be covered within the scope of protection of this application.

Claims

1. A method for suppressing object illusions in multimodal large language models, characterized in that, The method includes: Obtain input image information and text prompt information; Based on the input image information and text prompt information, during the process of generating answer text word by word in the multimodal large language model, the image-related attention score information is determined according to the current generation position; Based on the preset global main path, the image-related attention score information is conservatively adjusted to generate main path candidate output distribution information; Based on the main path candidate output distribution information, determine whether the currently generated position belongs to the position to be enhanced; If the current generated position belongs to the position to be enhanced, the image-related attention score information is enhanced and adjusted based on the preset local correction path to generate correction path candidate output distribution information, wherein the visual attention intensity of the enhanced adjustment is higher than the visual attention intensity of the conservative adjustment. Compare the reliability information of the candidate output distribution information of the correction path with the reliability information of the candidate output distribution information of the main path; If the reliability information of the corrected path candidate output distribution information is higher than that of the main path candidate output distribution information, then the corrected path candidate output distribution information is used as the final output distribution information of the current generation position; otherwise, the main path candidate output distribution information is used as the final output distribution information of the current generation position. Based on the final output distribution information, the current word information is generated, and the process proceeds to the next generation position until the preset stopping condition is met, generating complete answer text information.

2. The method according to claim 1, characterized in that, The image-related attention score information is used to describe the image-related attention score from the current generation position to the image visual unit; the process of generating the answer text per word in the multimodal large language model based on the input image information and text prompt information, determining the image-related attention score information according to the current generation position, includes: The input image information is encoded to generate image visual lexical set information, wherein the image visual lexical set information includes multiple image visual lexical information. The text prompt information is lexicalized and embedded to generate text lexical information; When the multimodal large language model uses a cross-attention mechanism to fuse image visual lexical information, the current generation position is used as the query, and the image visual lexical is used as the key and value. The cross-attention scores from the current generation position to each image visual lexical information are extracted to determine the image-related attention score information. or, When the multimodal large language model concatenates image visual lexical information and text lexical information and fuses them through a self-attention mechanism, it extracts the attention scores from the current generation position to each image visual lexical information from the self-attention matrix to determine the image-related attention score information.

3. The method according to claim 2, characterized in that, The image-related attention score information is the attention score before softmax normalization.

4. The method according to claim 1, characterized in that, The process of conservatively adjusting the image-related attention score information based on a preset global main path to generate main path candidate output distribution information includes: Based on the preset first visual attention correction parameters, the image-related attention score information is scaled to generate first processed data information; The first processed data information is used to perform forward calculations to generate the main path candidate output distribution information.

5. The method according to claim 1, characterized in that, The step of determining whether the currently generated position belongs to the position to be enhanced based on the main path candidate output distribution information includes: Calculate the prediction entropy information of the main path candidate output distribution information; Compare the predicted entropy information with a preset enhancement threshold; If the predicted entropy information is greater than the preset enhancement threshold, then the current generated position is determined to be a position to be enhanced; otherwise, the current generated position is determined not to be a position to be enhanced.

6. The method according to claim 1, characterized in that, If the currently generated position belongs to the position to be enhanced, then the image-related attention score information is enhanced and adjusted based on a preset local correction path to generate candidate output distribution information for the correction path, including: If it is determined that the current generated position belongs to the position to be enhanced, then based on the preset second visual attention correction parameters, the image-related attention score information is scaled to generate second processing data information; The second processed data information is used to perform forward calculation to generate candidate output distribution information for the correction path, wherein the second visual attention correction parameter is greater than the first visual attention correction parameter.

7. The method according to claim 1, characterized in that, If the reliability information of the candidate output distribution information of the corrected path is higher than the reliability information of the candidate output distribution information of the main path, then the candidate output distribution information of the corrected path is used as the final output distribution information of the current generation position, including: If the reliability information of the correction path candidate output distribution information is higher than the reliability information of the main path candidate output distribution information, then it is determined whether the difference between the reliability information of the correction path candidate output distribution information and the reliability information of the main path candidate output distribution information is greater than a preset confidence difference threshold. If the difference between the reliability information of the candidate output distribution information of the correction path and the reliability information of the candidate output distribution information of the main path is greater than the preset confidence difference threshold, then the candidate output distribution information of the correction path will be used as the final output distribution information of the current generation position.

8. A system for suppressing object illusions in multimodal large language models, characterized in that, The system includes: Input image information acquisition module: used to acquire input image information and text prompt information; Image-related attention score information determination module: used to determine image-related attention score information based on the input image information and text prompt information during the process of generating answer text word by word in a multimodal large language model, according to the current generation position; Main path candidate output distribution information generation module: used to conservatively adjust the image-related attention score information based on a preset global main path, and generate main path candidate output distribution information; Current generated location determination module: used to determine whether the current generated location belongs to the location to be enhanced based on the main path candidate output distribution information; Correction path candidate output distribution information generation module: if the current generation position belongs to the position to be enhanced, then based on the preset local correction path, the image-related attention score information is enhanced and adjusted to generate correction path candidate output distribution information, wherein the visual attention intensity of the enhancement adjustment is higher than the visual attention intensity of the conservative adjustment. Reliability information comparison module: used to compare the reliability information of the correction path candidate output distribution information with the reliability information of the main path candidate output distribution information; The final output distribution information determination module is used to determine the final output distribution information of the current generation position if the reliability information of the candidate output distribution information of the correction path is higher than that of the candidate output distribution information of the main path; otherwise, it uses the candidate output distribution information of the main path as the final output distribution information of the current generation position. The complete answer text information generation module is used to generate the current word element information based on the final output distribution information, and then proceed to the next generation position until the preset stopping condition is met, thereby generating complete answer text information.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images