Illusion phenomenon mitigation method and system of visual language large model, terminal and medium

Abstract

The invention discloses an illusion phenomenon mitigation method and system for a visual language large model, a terminal and a medium, and the method comprises the steps: obtaining input image-text data, processing the image-text data to obtain a text token and an image token, and aligning the text token with the image token to obtain an input sequence of the image-text data; inputting the input sequence into the visual language large model, calculating attention distribution through a cross-modal attention mechanism, and obtaining initial vocabulary probability distribution of the current time step based on the attention distribution; and based on the initial vocabulary probability distribution and the attention distribution, performing multi-scale enhancement and fusion reasoning to obtain target vocabulary probability distribution of the current time step. According to the method, through key technologies such as image-text alignment, attention guidance, multi-scale enhancement and fusion reasoning, the understanding and generation capability of the visual language model in a complex image-text scene is effectively improved, and the illusion problem possibly occurring in the cross-modal reasoning process of the large visual language model is relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language models, and particularly relates to a hallucination alleviation method and system for a visual language large model, a terminal and a medium. BACKGROUND

[0002] In the field of multi-modal artificial intelligence, visual language large models (VLMs) have attracted extensive attention due to their excellent performance in image description generation, visual question answering, multi-modal dialogue and other tasks. Such models are trained on large-scale image-text alignment data and can achieve understanding and textual expression of complex visual content. However, due to the existence of noise or semantic bias in the training data, the mainstream visual language large models often exhibit hallucination phenomenon during inference, that is, the generated text does not match the true semantics of the input image, and even is completely irrelevant. This phenomenon reflects the bias or error of the model in the process of image understanding or multi-modal information fusion, which may lead to inconsistency between the output content and the actual information of the image. Such problems may cause serious misleading in high-risk application scenarios such as medical diagnosis and autonomous driving, and effective mitigation mechanisms are urgently needed. To address this challenge, researchers have proposed various mitigation methods from multiple perspectives such as data optimization, training strategy adjustment and post-processing in the inference stage. Some techniques introduce high-quality labeled data or improve the loss function to enhance the image-text alignment capability of the model; other methods attempt to combine confidence assessment or external knowledge base in the inference stage to assist in judgment, in order to filter or correct unreliable output. In addition, some research proposes a self-correction mechanism based on multi-round question and answer consistency analysis to identify and correct hallucinated content in model generation; another method introduces multi-modal perturbation and contrastive decoding to calibrate the output results; and still another scheme adopts attention consistency modeling and combines with an early exit mechanism to alleviate the hallucination phenomenon. However, these methods still have obvious limitations: on the one hand, most of the methods rely on model retraining or parameter fine-tuning, which is computationally expensive and difficult to generalize; on the other hand, existing inference stage strategies lack fine-grained modeling capability for local uncertain regions of the image, and cannot accurately identify and correct key visual misjudgment areas; in addition, some methods have a long inference chain and complex process, and excessively rely on model self-consistency, which may cause error accumulation and affect the stability and reliability of the final output.

[0003] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0004] The technical problem solved by the present application is to provide a hallucination phenomenon reduction method, system, terminal and medium for a visual language large model, aiming to solve the problem that hallucination phenomenon occurs in the reasoning process of a visual language large model due to training data noise or semantic deviation, resulting in inconsistency between generated content and the real semantics of the input image.

[0005] To solve the above technical problems, the technical solutions adopted by the present application are as follows: In a first aspect, the present application provides a hallucination phenomenon reduction method for a visual language large model, wherein the method comprises: Obtaining input graphic-text data, processing the graphic-text data to obtain text tokens and image tokens, and aligning the text tokens and the image tokens to obtain an input sequence of the graphic-text data; Inputting the input sequence into a visual language large model, calculating an attention distribution through a cross-modal attention mechanism, and obtaining a preliminary vocabulary probability distribution of a current time step based on the attention distribution; Based on the preliminary vocabulary probability distribution and the attention distribution, multi-scale enhancement and fusion reasoning are performed to obtain a target vocabulary probability distribution of the current time step.

[0006] In an implementation manner, the inputting the input sequence into a visual language large model and calculating an attention distribution through a cross-modal attention mechanism comprises: Inputting the input sequence into a visual language large model, calculating a standard attention score of each layer of Transformer based on the text tokens and image tokens of the input sequence through a cross-modal attention mechanism; Based on the standard attention score of each layer of Transformer, the standard attention score of the middle layer of Transformer is extracted to obtain a middle layer attention score, and the standard attention score of the deep layer of Transformer is extracted to obtain a deep layer attention score; The deep layer attention score and the middle layer attention score are weighted and fused, and visual semantic amplification is performed to obtain a deep layer recalibration attention score; The deep layer recalibration attention score and the standard attention scores of the remaining Transformer layers are normalized to obtain the attention distribution of each layer of Transformer.

[0007] In an implementation manner, the multi-scale enhancement and fusion reasoning based on the preliminary vocabulary probability distribution and the attention distribution to obtain the target vocabulary probability distribution of the current time step comprises: Information entropy of the preliminary vocabulary probability distribution is calculated to determine a multi-scale amplification region; perform multi-scale enlargement on the multi-scale enlarged region based on the attention distribution to obtain a multi-scale image set; perform multi-scale fusion reasoning on the multi-scale image set to obtain a target vocabulary probability distribution of the current time step.

[0008] In an implementation manner, the information entropy calculation on the preliminary vocabulary probability distribution and the determination of the multi-scale enlarged region include: based on the preliminary vocabulary probability distribution, the probability distributions of the top pre-set number of text tokens are selected in descending order of probability to obtain pre-set text token probability distributions; perform information entropy calculation on the pre-set text token probability distributions to obtain pre-set text token information entropy values; If the pre-set text token information entropy value is greater than the entropy value threshold, the image token corresponding to the pre-set text token is the multi-scale enlarged region.

[0009] In an implementation manner, the multi-scale enlargement on the multi-scale enlarged region based on the attention distribution to obtain a multi-scale image set includes: based on the attention distribution, the attention distributions of the middle-layer Transformer layer and the shallow-layer Transformer layer are selected to obtain a middle-shallow-layer attention distribution; perform normalization, activation, and weighting processing on the middle-shallow-layer attention distribution to obtain an attention map; perform multi-scale enlargement on the multi-scale enlarged region based on the attention map to obtain a multi-scale image set.

[0010] In an implementation manner, the multi-scale fusion reasoning on the multi-scale image set to obtain a target vocabulary probability distribution of the current time step includes: perform multi-scale prediction on each multi-scale image in the multi-scale image set to obtain a probability distribution; perform processing on the probability distribution through information entropy calculation, consistency prediction, and normalization operation to obtain a weight coefficient; perform weighted summation on the probability distribution by using the weight coefficient to obtain a target vocabulary probability distribution of the current time step.

[0011] In an implementation manner, the processing on the probability distribution through information entropy calculation, consistency prediction, and normalization operation to obtain a weight coefficient includes: calculate information entropy based on the probability distribution to obtain an information entropy value; perform consistency prediction on the probability distribution by using a cosine similarity matrix to obtain a consistency score; Based on the consistency score and the information entropy value, a confidence weight is calculated, and the confidence weight is normalized to obtain a weight coefficient.

[0012] In a second aspect, the embodiment of the present application further provides a hallucination reduction system of a visual language large model, wherein the system comprises: An input sequence acquisition module is configured to acquire input graphic-text data, process the graphic-text data to obtain text tokens and image tokens, and align the text tokens and the image tokens to obtain an input sequence of the graphic-text data. A preliminary vocabulary probability distribution acquisition module is configured to input the input sequence into a visual language large model, calculate an attention distribution through a cross-modal attention mechanism, and obtain a preliminary vocabulary probability distribution of a current time step based on the attention distribution. A target vocabulary probability distribution module is configured to perform multi-scale enhancement and fusion reasoning based on the preliminary vocabulary probability distribution and the attention distribution to obtain a target vocabulary probability distribution of the current time step.

[0013] In a third aspect, the embodiment of the present application further provides a terminal, wherein the terminal comprises a memory, a processor, and a hallucination reduction program of a visual language large model stored in the memory and executable on the processor, and when the processor executes the hallucination reduction program of the visual language large model, the steps of the hallucination reduction method of the visual language large model in any of the above solutions are implemented.

[0014] In a fourth aspect, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a hallucination reduction program of a visual language large model, and when the hallucination reduction program of the visual language large model is executed by a processor, the steps of the hallucination reduction method of the visual language large model in any of the above solutions are implemented.

[0015] Beneficial effects: The present application provides a hallucination reduction method for a visual language large model. Compared with the prior art, the present application first acquires input graphic text data, processes the graphic text data to obtain text tokens and image tokens, and aligns the text tokens and the image tokens to obtain an input sequence of the graphic text data. This step helps to establish a fine-grained correspondence between graphics and texts, and improves the understanding ability of the model for multi-modal information. Then, the input sequence is input into the visual language large model, and the attention distribution is calculated through the cross-modal attention mechanism, so as to dynamically allocate attention weights between different modalities, so that the model can more accurately capture semantic information related to the current task, and then generate a more reasonable preliminary vocabulary probability distribution. Then, based on the preliminary vocabulary probability distribution and the attention distribution, multi-scale enhancement and fusion reasoning are performed to obtain the target vocabulary probability distribution at the current time step. This step enhances the information expression ability of the key visual area, helps to improve the accuracy of subsequent reasoning, and comprehensively considers the complementarity of image features at different scales to obtain the target vocabulary probability distribution at the current time step, so that the finally output vocabulary is more consistent with the overall graphic text context, and has higher semantic consistency and logical coherence. Since the above process is completely based on the forward output of the model during the reasoning stage (including the attention distribution and the preliminary vocabulary probability distribution) for enhancement operation, all calculations can be completed after a single forward propagation, without the need for gradient back propagation or iterative updating of model parameters. Therefore, the entire method does not need to fine-tune the model parameters, and does not rely on external labeling or data enhancement means, and can be directly applied to the reasoning stage. This effectively avoids the high computational cost and limited generalization ability caused by model retraining, and has good training freedom. At the same time, this method only relies on the intermediate output interface (such as attention weight and logits) of the visual language model, does not involve modification of the internal structure of the model or access to the loss function, and does not need to obtain gradient information, so it can realize dynamic calibration of the generation process, and therefore has high modularity and pluggability. It can seamlessly adapt to various existing visual language models, showing wide applicability and deployment flexibility. In addition, the present method adopts a single-step, non-iterative reasoning enhancement mechanism, which directly completes multi-scale information fusion and probability recalibration on the original generation path, avoiding complex reasoning chains such as multi-round self-consistent verification or repeated correction, and fundamentally reducing the error accumulation risk caused by relying on model self-feedback, significantly improving the stability and reliability of the output results, thereby systematically solving the key problems of "high computational cost and difficult generalization" and "over-reliance on model self-consistency, easy to cause error accumulation, and affect output stability" in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the specific implementation manner of the hallucination reduction method for a visual language large model provided by the embodiments of the present application is shown.

[0017] Figure 2 The flowchart of obtaining the attention distribution in the illusion phenomenon alleviation method of the visual language large model provided by the embodiment of the present application.

[0018] Figure 3 The flowchart of obtaining the multi-scale image set in the illusion phenomenon alleviation method of the visual language large model provided by the embodiment of the present application.

[0019] Figure 4 The flowchart of obtaining the target word probability distribution of the current time step in the illusion phenomenon alleviation method of the visual language large model provided by the embodiment of the present application.

[0020] Figure 5 The principle block diagram of the illusion phenomenon alleviation system of the visual language large model provided by the embodiment of the present application.

[0021] Figure 6 The principle block diagram of the internal structure of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0023] In the field of multi-modal artificial intelligence, vision-language models (VLMs) have attracted extensive attention due to their outstanding performance in image description generation, visual question answering, and multi-modal dialogue tasks. These models are trained on large-scale image-text aligned data and can understand and textualize complex visual content. However, due to the presence of noise or semantic bias in the training data, current mainstream VLMs often exhibit hallucination phenomena during inference, where the generated text content is inconsistent with the true semantics of the input image, or even completely unrelated. Such problems may cause serious misleading in high-risk application scenarios such as medical diagnosis and autonomous driving, and therefore effective mitigation mechanisms are urgently needed. To address this challenge, researchers have proposed various mitigation methods from multiple perspectives, including data optimization, training strategy adjustment, and post-processing at the inference stage. Some techniques improve the model's image-text alignment ability by introducing high-quality labeled data or modifying the loss function. Other methods attempt to filter or correct unreliable outputs by incorporating confidence assessment or external knowledge bases at the inference stage. In addition, some studies propose self-correction mechanisms based on multi-round question-answer consistency analysis to identify and correct hallucinated content in the model's generation. Other methods introduce multi-modal perturbation and contrastive decoding to calibrate the output results. Some approaches use attention consistency modeling and combine early exit mechanisms to alleviate hallucination phenomena. However, these methods still have obvious limitations: on the one hand, most methods rely on model retraining or parameter fine-tuning, which is computationally expensive and difficult to generalize. On the other hand, existing inference stage strategies lack fine-grained modeling capabilities for local uncertain regions of the image, making it difficult to accurately identify and correct key visual misjudgment areas. Furthermore, some methods have long inference chains and complex processes, relying heavily on model self-consistency, which can lead to error accumulation and affect the stability and reliability of the final output.

[0024] To solve the above problems, the embodiment provides a hallucination phenomenon reduction method of a visual language large model. In specific implementation, the embodiment first acquires input graphic-text data, processes the graphic-text data to obtain text tokens and image tokens, and aligns the text tokens and the image tokens to obtain an input sequence of the graphic-text data. This step helps to establish a fine-grained correspondence between graphics and texts and improve the understanding ability of the model for multi-modal information. Then, the input sequence is input into the visual language large model, and an attention distribution is calculated through a cross-modal attention mechanism, so as to dynamically allocate attention weights between different modalities, so that the model can more accurately capture semantic information related to the current task, and then generate a more reasonable preliminary vocabulary probability distribution. Then, based on the preliminary vocabulary probability distribution and the attention distribution, multi-scale enhancement and fusion reasoning are performed to obtain a target vocabulary probability distribution at the current time step. This step enhances the information expression ability of the key visual area, helps to improve the accuracy of subsequent reasoning, and comprehensively considers the complementarity of image features at different scales to obtain the target vocabulary probability distribution at the current time step, so that the finally output vocabulary is more consistent with the overall graphic-text context and has higher semantic consistency and logical coherence. Since the above process is completely based on the forward output of the model in the reasoning stage (including the attention distribution and the preliminary vocabulary probability distribution) for enhancement operation, all calculations can be completed after a single forward propagation, without the need for gradient back propagation or iterative updating of model parameters. Therefore, the entire method does not need to fine-tune the model parameters and does not rely on external labeling or data enhancement means, and can be directly applied to the reasoning stage, effectively avoiding the high computational cost and limited generalization ability caused by model retraining, and has good training freedom. At the same time, the method only relies on the intermediate output interface (such as attention weights and logits) of the visual language model, does not involve modification of the internal structure of the model or access to the loss function, and does not need to obtain gradient information, so as to realize dynamic calibration of the generation process, and therefore has high modularization and pluggability, can be seamlessly adapted to various existing visual language models, and has wide applicability and deployment flexibility. In addition, the method adopts a single-step, non-iterative reasoning enhancement mechanism to complete multi-scale information fusion and probability recalibration directly on the original generation path, avoids complex reasoning chains such as multi-round self-consistent verification or repeated correction, and fundamentally reduces the error accumulation risk caused by dependence on model self-feedback, significantly improving the stability and reliability of the output result, thereby systematically solving the key problems of "high computational cost and difficult generalization" and "over-reliance on model self-consistency, easy to cause error accumulation, and affect output stability" in the prior art.

[0025] The hallucination phenomenon reduction method of the visual language large model provided by the embodiment can be applied to an intelligent terminal, such as a smart phone. Figure 1 The method specifically includes the following steps: In step S100, inputted image-text data is acquired, the image-text data is processed to obtain text tokens and image tokens, and the text tokens and the image tokens are aligned to obtain an input sequence of the image-text data.

[0026] In this embodiment, first, inputted image-text data is acquired. This step collects multi-modal information to provide a raw data basis for subsequent processing, ensuring that the system can understand both text and image content. Then, the image-text data is processed to obtain text tokens and image tokens. This step extracts and encodes features of the raw data through natural language processing techniques and computer vision techniques, converting unstructured data into structured representations that can be processed by the model, thereby improving information expression accuracy and computational efficiency. Subsequently, the text tokens and the image tokens are aligned. This step establishes semantic associations between text and images through cross-modal alignment mechanisms, enabling information from different modalities to be fused and interacted in a unified semantic space, enhancing the model's ability to understand complex semantics. Finally, an input sequence of the image-text data is generated. This step organizes the processed and aligned text tokens and image tokens into a unified structured form, forming a multi-modal input representation that can be accepted by the model. This input sequence not only preserves the semantic information in the original image-text data, but also realizes semantic association and fusion between different modalities through cross-modal alignment mechanisms. This process provides high-quality, semantically consistent multi-modal joint representations for subsequent downstream tasks, enabling the model to more accurately understand complex visual and language interaction content, thereby improving task execution accuracy and intelligence level. Overall, the above steps achieve effective integration and collaborative processing of multi-modal information, improving the model's understanding and reasoning ability for complex scenarios, and enhancing the system's intelligence level and application adaptability.

[0027] Specifically, the step S100 includes the following steps: In step S101, the text in the image-text data is segmented, mapped, and embedded to obtain text tokens. In step S102, the image in the image-text data is divided, encoded, and projected to obtain image tokens.

[0028] In an implementation manner, firstly, inputted graphic-text data composed of both text and image is acquired, the main role of this stage is to collect information from different modalities uniformly, providing original materials for subsequent multi-modal fusion, the advantage of this process is that it can capture language and visual information simultaneously, and build the foundation for cross-modal understanding. Then, the text part of the graphic-text data is processed by word segmentation, mapping and embedding to obtain text tokens. Specifically, the word segmentation operation divides continuous text into basic units with semantic meaning (such as words or subwords); the mapping process corresponds these lexical units to a pre-defined vocabulary; the embedding process further maps each word to a vector representation in a high-dimensional space. The role of this step is to convert unstructured natural language into structured numerical form, which is convenient for the model to perform semantic modeling and reasoning. Its advantage is not only to retain the semantic information of the text, but also to capture the complex relationships between words through context learning, thereby enhancing the language understanding ability of the model. Subsequently, the image part is processed by blocking, encoding and projecting to generate image tokens. Specifically, the image (original picture) is divided into several image blocks (patches), each image block is regarded as a visual "word"; then, an image encoder (such as a convolutional neural network or a visual Transformer) extracts high-level features of each image block; finally, a projector maps the image features into the same semantic space as the text, forming image tokens aligned with the text. The advantage of this processing method is that: on the one hand, the image blocking mechanism can effectively capture local details and overall structural information; on the other hand, through encoding and projection operations, images can establish direct contact with text at the semantic level, realizing cross-modal alignment and fusion, which is crucial for improving the performance of the model in multi-modal tasks. Then, the processed text tokens and image tokens are aligned and fused into a unified input sequence. Through the alignment mechanism (such as attention mechanism or multi-modal interaction module), the model can dynamically identify and integrate the relevance between text and image, thereby constructing a more complete and accurate semantic representation. The key role of this step is to promote the deep fusion of cross-modal information, enabling the model to have stronger context perception ability and logical reasoning ability. Overall, the above process realizes the whole process modeling from raw graphic-text data to unified semantic representation, fully exploiting the characteristics and advantages of text and image. Its main advantages include: 1) supporting efficient fusion of multi-modal information, improving the model's understanding ability of complex semantics; 2) through the design of a unified semantic space, enhancing the generalization and accuracy of the model in cross-modal tasks (such as graphic-text retrieval, image description generation, visual question answering, etc.); 3) clear structure and strong modularity, easy to extend and optimize, suitable for various downstream application scenarios. This method not only provides a technical foundation for multi-modal learning, but also provides a feasible path for artificial intelligence systems to realize multi-sensory collaborative understanding closer to humans.

[0029] Step S200, input the input sequence into the visual language large model, calculate the attention distribution through the cross-modal attention mechanism, and obtain the preliminary vocabulary probability distribution of the current time step based on the attention distribution.

[0030] In this embodiment, first, the input sequence is input into the visual language large model. The role of this step is to provide the original information basis for the subsequent cross-modal understanding and generation process. The input sequence usually contains information of both image and text modalities, where the image information is extracted into multiple regional feature vectors through a convolutional neural network or a visual Transformer, and the text information is converted into semantic vector representation through word embedding and other methods. The core advantage of the visual language large model lies in its strong multi-modal fusion capability, which can uniformly process data from different modalities and jointly model in the shared semantic space. The advantages of this step are: on the one hand, the pre-trained visual language model has learned rich cross-modal semantic knowledge and can efficiently encode input information; on the other hand, through the interaction modeling between modalities, the details and context relationships in the input data can be preserved, laying a solid foundation for subsequent attention calculation and vocabulary prediction. Then, the attention distribution is calculated through the cross-modal attention mechanism. This step is the key link of the whole process, which dynamically selects the most relevant part from all input features (including image region features and text history) in the decoding process at the current time step. Specifically, the cross-modal attention mechanism calculates a weight distribution, i.e. the attention distribution, according to the similarity between the current decoder state and all encoder outputs. This distribution reflects the importance of each input feature for the current vocabulary prediction. Its advantages are: first, it realizes fine-grained information selection, enabling the model to accurately identify key visual or language clues in complex scenarios; second, it enhances the model's interpretability, allowing the model to focus on key areas through attention weights; third, it improves the model's ability to capture long-distance dependencies, avoiding the information decay problem in traditional RNN structures; in addition, the attention mechanism also has good generalization ability, which can adapt to different task requirements such as image description generation and visual question answering. Subsequently, the input features are weighted and integrated based on the attention distribution to form a context vector. The role of this step is to weight and sum the input features corresponding to the attention distribution, thereby obtaining a compact context representation. This context vector contains the key information needed at the current time step, including relevant regions in the visual modality and context content in the text modality. Its advantages are: on the one hand, through the weighted fusion of input features, information redundancy is effectively reduced and computational efficiency is improved; on the other hand, this soft attention mechanism allows the model to flexibly adjust the focus at different time steps, thereby realizing more natural and coherent language generation. Finally, based on the obtained context vector, the decoder state is combined to further generate the preliminary vocabulary probability distribution at the current time step through the vocabulary prediction module. The role of this step is to combine the current context information with the historical information of the language model, output a probability distribution for the target vocabulary, and guide the selection of the next vocabulary.Specifically, the vocabulary probability distribution is usually mapped to a vector of the size of the vocabulary table through a fully connected layer and normalized by a softmax function. The advantages are: first, the attention-guided context information makes the vocabulary prediction more targeted and accurate; second, through the end-to-end learning method, the model can automatically optimize the mapping relationship between the attention weight and the vocabulary prediction; third, the form of probability distribution facilitates the subsequent introduction of sampling strategies (such as beam search, Top-k sampling, etc.), which improves the diversity and quality of generated text. In summary, the entire process encodes the input sequence into a visual language large model through multi-modal encoding, dynamically calculates the attention distribution by means of cross-modal attention mechanism, and then generates the context vector by weighting, and finally generates the preliminary vocabulary probability distribution at the current time step, forming a complete and efficient cross-modal generation framework. The overall advantages are: on the one hand, the powerful ability of the visual language large model in multi-modal understanding is fully utilized to realize the deep modeling of complex scenes; on the other hand, through the effective guidance of the attention mechanism, the reasoning ability, expression ability and generation quality of the model in the cross-modal task are significantly improved. In addition, this method has good flexibility and expansibility, and is suitable for various visual language tasks such as image description generation, visual question answering, and image-text retrieval, and has wide application prospects and practical value.

[0031] Specifically, the step S200 comprises the following steps: Step S201, inputting the input sequence into a visual language large model, calculating the standard attention score of each Transformer layer through a cross-modal attention mechanism based on the text token and the image token of the input sequence; Step S202, based on the standard attention score of each Transformer layer, extracting the standard attention score of the middle Transformer layer to obtain the middle attention score; extracting the standard attention score of the deep Transformer layer to obtain the deep attention score; Step S203, weighting and fusing the deep attention score and the middle attention score, and performing visual semantic amplification to obtain a deep recalibration attention score; Step S204, performing normalization operation on the deep recalibration attention score and the standard attention score of the remaining Transformer layers respectively to obtain the attention distribution of each Transformer layer.

[0032] In an implementation manner, as Figure 2As shown, first, the input sequence is input into the visual language large model, and based on the text tokens and image tokens of the input sequence, the standard attention scores of each layer of the Transformer layer are calculated through the cross-modal attention mechanism. Specifically, the text tokens generate query vectors while the image tokens generate key vectors and These vectors are usually obtained from the input features through linear transformation. In the first layer, the standard attention logic score is: wherein , contains the correlation between text and image tokens, is the number of text tokens, is the number of text tokens, is the number of image blocks, is a scaling factor to prevent the gradient from being unstable due to the large dot product result. The role of this step is to establish the initial semantic association between text and image, so that the model can dynamically focus on the key areas in the image according to the context, providing a foundation for subsequent more refined attention optimization. Then, based on the standard attention scores of each layer of the Transformer layer, the standard attention score of the middle layer Transformer layer (e.g., the 14th layer) is extracted, obtaining the middle layer attention score This score is a sub-matrix corresponding to the image token, representing the stable visual semantic association between the image and the text; the standard attention score of the deep layer Transformer layer (e.g., the 28th layer) is extracted, obtaining the deep layer attention score The role of this stage is to obtain the attention signals of the middle layer and the deep layer respectively: the middle layer usually has good feature abstraction ability and can capture more stable visual information; while the deep layer embodies stronger context modeling ability, which is helpful for understanding the global semantics. This hierarchical extraction strategy provides multi-level semantic support for subsequent fusion, enhancing the model's understanding ability of complex scenes. Subsequently, the deep layer attention score and the middle layer attention score are weighted fused and visually semantically amplified, obtaining the deep layer recalibrated attention score The calculation formula is as follows: wherein is the attention fusion weight, which is set to 0.7 by default, indicating the degree of dominance of the deep layer attention score in the fusion process; A visual semantic amplification factor is set to 1.5 by default, which is used to enhance the influence of the image modality in the whole cross-modal decision-making process. The design purpose of this step is: on the one hand, by fusing the middle-level and deep-level attention scores, the possible bias problem of deep-level attention (such as insufficient attention to key visual information) is alleviated, and the robustness and generalization ability of the attention mechanism are improved; on the other hand, the visual semantic amplification factor is introduced to further strengthen the importance of the image modality, so that the model can perform better in image-dominant tasks (such as image description generation, visual question answering, etc.) in terms of reasoning accuracy and expression ability. Then, the deep recalibration attention score and the standard attention score of the remaining Transformer layer are normalized respectively to obtain the attention distribution of each Transformer layer. This normalization operation usually uses the softmax function or other standardization methods to ensure that all attention scores are comparable on the same scale. Finally, based on the updated attention distribution, the preliminary word probability distribution of the current time step is calculated.

[0033] In summary, this embodiment constructs a complete cross-modal attention optimization process from low-level perception to high-level understanding through the extraction, fusion and recalibration of multi-level attention scores. This mechanism not only preserves local detailed information, but also enhances global semantic consistency through the fusion of middle-level and deep-level attention, and further improves the importance of the image modality by combining visual semantic amplification. The whole process realizes the fine modeling of text and image information, significantly improves the expression ability and reasoning accuracy of the visual language large model in complex cross-modal tasks. Compared with the traditional single-layer attention mechanism, this method shows stronger adaptability and robustness in handling multi-modal semantic alignment and context dependency modeling, providing strong support for building high-performance visual language understanding systems.

[0034] Step S300, based on the preliminary word probability distribution and the attention distribution, multi-scale enhancement and fusion reasoning are performed to obtain the target word probability distribution of the current time step.

[0035] In the embodiment, firstly, multi-scale enhancement is performed based on the preliminary vocabulary probability distribution and the attention distribution, which can capture cross-modal semantic association from different levels, accurately locate the correspondence between key vocabulary and image region by using fine-grained scale, and enhance the understanding ability of the model for complex image-text scenes by modeling global semantic consistency through coarse-grained scale; then, in the fusion reasoning stage, the designed multi-scale fusion mechanism is used to dynamically integrate the probability distribution under each scale, so that the model can adaptively allocate the contribution weight of different scales according to the current context, effectively suppress noise interference and highlight key prediction signals, and improve the stability and accuracy of prediction; overall, this step not only strengthens the deep alignment of visual and language modalities in the semantic space, but also significantly improves the semantic coherence, detail richness and context relevance of the generated text through the strategy of divide and conquer, and then aggregation optimization, especially when dealing with ambiguous references, image ambiguity or long sequence generation tasks, it shows stronger reasoning ability and generation robustness.

[0036] Specifically, the step S300 comprises the following steps: Step S301, calculating the information entropy of the preliminary vocabulary probability distribution to determine a multi-scale enlarged region; Step S302, performing multi-scale enlargement on the multi-scale enlarged region through the attention distribution to obtain a multi-scale image set; Step S303, performing multi-scale fusion reasoning on the multi-scale image set to obtain a target vocabulary probability distribution at the current time step.

[0037] In an implementation, generally, first, an information entropy calculation is performed on the preliminary vocabulary probability distribution. The role of this step is to quantify the degree of uncertainty of the current time step vocabulary prediction, so as to determine whether the model has a high degree of confidence in the selection of the target vocabulary in the generation process. The higher the information entropy, the more uncertain the model is about which vocabulary should be selected in the next step; on the contrary, it indicates that the model has a higher certainty. Through this calculation, a basis can be provided for subsequent dynamic adjustment of the visual input region, so that the image focus region can be adaptively adjusted according to the uncertainty of language generation. The advantage lies in introducing the semantic uncertainty in the language generation process as a feedback signal for visual processing, enhancing the interactivity and collaboration between modalities. Then, a multi-scale zoom region is determined based on the information entropy. The role of this step is to locate the key region in the original image that needs further fine-grained analysis according to the uncertainty level of the current language model, and set multiple scale zoom windows. Specifically, when the information entropy is high, it means that the model is not confident enough in vocabulary prediction, so a smaller and more detailed image region should be focused on to obtain more detailed information; when the information entropy is low, it means that the model is more certain, so the focus range can be appropriately expanded to extract more macroscopic context information. The advantage lies in the dynamic adjustment of the visual focus region, which improves the model's ability to capture information at different semantic levels, while avoiding the problem of information omission or redundancy caused by fixed scale focus. Subsequently, a multi-scale zoom operation is performed on the multi-scale zoom region through the attention distribution. The core role of this step is to realize high-resolution reconstruction of the key region in the visual space, thereby providing more rich visual details for the subsequent language generation process. Using the attention distribution calculated earlier, the image region most relevant to the current vocabulary prediction can be identified, and different scale zoom strategies can be applied to these regions to generate a set of image sub-regions with different resolutions and coverage ranges, forming a multi-scale image set. The advantage lies in: first, the zoom operation is guided by the attention mechanism to ensure that the zoomed region is closely related to the current task; second, the multi-scale design allows the model to flexibly switch between local details and global structures, enhancing the depth and breadth of understanding; third, the zoomed image set can be used as new input for the next round of cross-modal fusion, forming a "language-visual-language" closed-loop feedback mechanism to improve the overall generation quality. Then, a multi-scale fusion reasoning is performed on the multi-scale image set. The role of this step is to model and integrate the information of multiple image sub-regions with different resolutions and focus ranges obtained through the multi-scale zoom operation, so as to extract more hierarchical and complementary visual feature representations. The advantage lies in: on the one hand, it can effectively capture multi-level semantic information from local details to global structures, enhancing the model's understanding ability of complex visual content; on the other hand, through the information complementarity between multiple scales, the bias or omission caused by single scale analysis can be reduced, improving the robustness and integrity of visual information.Subsequently, based on the above fusion and reasoning process, the target vocabulary probability distribution of the current time step is obtained. The role of this step is to integrate multi-scale visual features and language context information to output a probability distribution for the target vocabulary to guide the selection and generation of the next vocabulary. The vocabulary probability distribution is usually generated by a classifier (such as a fully connected layer) and normalized by a softmax function to form a probability estimate for all candidate words in the vocabulary. The advantages are: on the one hand, by introducing multi-scale visual information, the vocabulary prediction not only depends on the overall image of coarse granularity, but also combines local features of fine granularity to improve the accuracy of prediction; on the other hand, the probability distribution has good interpretability, which is convenient for subsequent high-quality text generation combined with sampling strategies. In general, through information entropy analysis of the preliminary vocabulary probability distribution, the multi-scale magnification region is dynamically determined, and the multi-scale magnification of the key visual region is realized by combining the attention distribution, and finally a multi-scale image set is generated. The advantage of this method is to construct a visual feedback mechanism based on semantic uncertainty, so that the model can actively adjust the attention range and resolution of the visual input according to the language generation state, thereby improving the consistency and accuracy of cross-modal understanding and generation. Through multi-scale fusion and reasoning of the multi-scale image set, combined with the cross-modal interaction mechanism, the target vocabulary probability distribution of the current time step is finally generated. The overall advantage of this method is to construct a hierarchical and information complementary visual processing framework, which significantly improves the understanding depth and generation quality of the visual language model in cross-modal tasks. On the one hand, the multi-scale fusion mechanism enhances the model's perception ability of complex visual content, making it better serve language generation; on the other hand, through deep fusion of multi-modal information, the model's ability in semantic alignment, context modeling, etc. is improved, further enhancing the system's robustness and generalization performance. This technology is applicable to image description generation, visual question answering, image-text retrieval and other visual language tasks, and has wide application prospects and practical value.

[0038] In an implementation manner, the specific implementation manner of obtaining the multi-scale image set is as shown in Figure 3 First, based on the preliminary vocabulary probability distribution , the probability distribution of the first preset number of text tokens is obtained by sorting from high to low according to the probability. Among them, represents the parameters of the visual language large model, represents the input text information, represents the input image information, is the token sequence up to t-1 time step, t is the time step, and the role of this step is to select a number of candidate text tokens most likely to be generated at the current time step, providing basic data support for subsequent uncertainty evaluation. Then, the information entropy of the preset text token probability distribution is calculated to obtain the preset text token information entropy value , the formula is: wherein, is a preset number, is the first high-probability token. Information entropy measures the confidence of the model on the current output: when the information entropy is high, it means that the probabilities of multiple candidate tokens are close, and the model is uncertain; when the information entropy is low, it means that one or a few tokens dominate, and the model is more certain. The advantage of this step is that it can quantify the level of uncertainty in the generation process, thereby providing a decision basis for whether to trigger the multi-scale amplification mechanism. If the preset text token information entropy value is greater than the entropy value threshold (default 0.5), the multi-scale amplification is triggered, and the multi-scale amplification stage is entered, and the image token corresponding to the preset text token is a multi-scale amplification region. The advantage of this step is that it can dynamically adjust the generation strategy: when the model is not confident enough about the current output, it can further refine the visual information through multi-scale amplification to improve the generation quality. If the information entropy value is less than the entropy value threshold , or reaches the maximum amplification number N (default 3), the amplification is stopped, and the fusion reasoning is entered. This mechanism ensures that the model can gradually refine the visual information in multiple iterations, while avoiding unnecessary computational overhead and overfitting risk. By dynamically adjusting the amplification number, the model can flexibly cope with the complexity requirements in different scenarios, further improving the accuracy and robustness of the generation results. Next, based on the attention distribution, the attention distributions of the middle Transformer layer (e.g., the 14th layer) and the shallow Transformer layer (e.g., the 2nd layer) are selected to obtain the middle-shallow layer attention distribution wherein, represents the number of image blocks. The purpose of this stage is to extract attention signals at different abstraction levels: shallow attention reflects local feature responses, while middle attention embodies higher-level semantic understanding, and the combination of the two helps to identify more representative salient regions. Subsequently, the middle-shallow layer attention distribution is normalized, activated, and weighted to obtain the attention map. First, the middle-shallow layer attention distribution is normalized: wherein, represents the normalized attention distribution, ensuring that all attention scores are comparable on the same scale. Then, the saliency signal is calculated, and the difference between the middle and shallow attention is calculated by ReLU: wherein, denotes the original saliency signal, which effectively highlights the image regions that the mid-level attention responds more strongly than the shallow-level attention, reflecting the potential key semantic positions. Then, the saliency map is re-weighted to maintain the attention energy, and the re-weighted saliency map is obtained as follows: wherein, denotes the final saliency map, i.e., the attention map, which retains the energy distribution of attention through the re-weighting operation, making the saliency map more stable and reliable. The advantage of this step is that it effectively identifies the high semantic value regions in the image through the comparative analysis of multi-level attention, and converts them into guidance signals that can be used for visual enhancement. Then, based on the attention map, the multi-scale enlarged regions are multi-scale enlarged to obtain a multi-scale image set. Specifically, the region with the highest Gaussian response in the saliency map is cropped and enlarged to the original image resolution in turn to generate multiple views , and the multi-scale image set is repeatedly generated: wherein, the enlargement ratios are 1.5x, 2x, 3x, etc. in turn. The design advantage of this step is that it enables the model to focus on image details at different granularities through step-by-step enlargement, thereby improving the cross-modal understanding and generation capability. Especially in the face of complex scenes or ambiguous objects, multi-scale images provide more rich contextual information, which helps to alleviate ambiguity and enhance the accuracy and richness of the generated results.

[0039] In one implementation, based on obtaining the multi-scale image set, the specific implementation of obtaining the target word probability distribution of the current time step is as shown in Figure 4 First, multi-scale prediction is performed on each multi-scale image in the multi-scale image set to obtain a probability distribution. Specifically, for each view , its corresponding probability distribution is calculated as follows: wherein, denotes a function of the visual language large model parameters . This step converts the model output into a probability distribution through the softmax function, ensuring that the sum of probabilities in each dimension is 1. This step utilizes image representations at different scales or angles to enhance the model's perception of image content, while preserving the uniqueness of each view, providing a diversified basis for subsequent fusion. Then, the information entropy is calculated based on the probability distribution to obtain an information entropy value , which is used to measure the uncertainty of the prediction. The greater the information entropy value, the more uncertain the prediction result. This indicator can effectively identify low-quality or unreliable prediction results, thereby suppressing them in subsequent processing. Subsequently, to further enhance the stability and generalization ability of the prediction, consistency prediction is performed on the probability distribution through a cosine similarity matrix to obtain a consistency score, with the specific formula being: wherein, indicates the comparison between different views. Scale The consistency score is: This step evaluates the consistency of the predictions by calculating the cosine similarity of the probability distributions between different views, thereby improving the robustness and stability of the model. Next, based on the consistency score and the information entropy value, the confidence weight is calculated and normalized to obtain the weight coefficient. The specific formula is: wherein, and are the weight coefficients of information entropy and consistency, respectively, with default values of 0.8 and 0.2. This step considers both the uncertainty and consistency of the predictions to determine the confidence weight of each view. Then, the probability distribution is weighted and summed using the weight coefficient to obtain the target word probability distribution at the current time step. The specific formula is: wherein, This step fuses the probability distributions of multiple views into a final target word probability distribution through weighted summation, improving the accuracy and reliability of the predictions. Finally, the maximum probability is sampled from to generate the predicted text token Overall, this method obtains diverse contextual information through multi-scale prediction, measures prediction uncertainty using information entropy, evaluates prediction consistency using cosine similarity, and combines confidence weight normalization and fusion mechanisms to effectively improve the prediction performance and robustness of the model in complex visual scenes.

[0040] This embodiment effectively improves the prediction performance and robustness of the model in complex visual scenes through dynamic adjustment of the generation strategy and multi-scale image processing mechanism. First, information entropy is used to evaluate the uncertainty level of the generation process, providing a decision basis for whether to trigger the amplification mechanism, ensuring that the model can further refine visual information when it is not confident enough about the output, improving the generation quality. Second, by analyzing the attention distribution of the middle and shallow Transformer layers, high semantic value regions in the image are identified and converted into visual enhancement guidance signals, enhancing cross-modal understanding ability. Third, a multi-scale prediction method is used to evaluate the consistency between different views using cosine similarity, improving the stability and generalization ability of the predictions. Finally, through confidence weight normalization and probability distribution fusion mechanism, the accuracy and reliability of the final target word probability distribution are ensured, enabling the model to flexibly cope with the complexity requirements of different scenes, significantly improving the quality and robustness of the generation results.

[0041] In summary, the embodiment first acquires input graphic-text data, processes the graphic-text data to obtain text tokens and image tokens, and aligns the text tokens and the image tokens to obtain an input sequence of the graphic-text data. This step helps to establish a fine-grained correspondence between graphics and texts, and improves the understanding ability of the model for multi-modal information. Then, the input sequence is input into a visual language large model, and an attention distribution is calculated through a cross-modal attention mechanism, so as to dynamically allocate attention weights between different modalities, so that the model can more accurately capture semantic information related to the current task, and then generate a more reasonable preliminary vocabulary probability distribution. Then, based on the preliminary vocabulary probability distribution and the attention distribution, multi-scale enhancement and fusion reasoning are performed to obtain a target vocabulary probability distribution at the current time step. This step enhances the information expression ability of the key visual area, helps to improve the accuracy of subsequent reasoning, and comprehensively considers the complementarity of image features at different scales to obtain the target vocabulary probability distribution at the current time step, so that the finally output vocabulary is more consistent with the overall graphic-text context, and has higher semantic consistency and logical coherence. Since the above process is completely based on the forward output of the model during the reasoning stage (including the attention distribution and the preliminary vocabulary probability distribution) for enhancement operation, all calculations can be completed after a single forward propagation, without the need for gradient back propagation or iterative updating of model parameters. Therefore, the entire method does not need to fine-tune the model parameters, and does not rely on external labeling or data enhancement means, and can be directly applied to the reasoning stage. This effectively avoids the high computational cost and limited generalization ability caused by model retraining, and has good training freedom. At the same time, the method only relies on the intermediate output interface (such as attention weights and logits) of the visual language model, does not involve modification of the internal structure of the model or access to the loss function, and does not need to obtain gradient information, so as to realize dynamic calibration of the generation process. Therefore, it has high modularization characteristics and pluggability, can seamlessly adapt to various existing visual language models, and has wide applicability and deployment flexibility. In addition, the method adopts a single-step, non-iterative reasoning enhancement mechanism, directly completes multi-scale information fusion and probability recalibration on the original generation path, avoids complex reasoning chains such as multi-round self-consistent verification or repeated correction, and fundamentally reduces the error accumulation risk caused by dependence on model self-feedback, significantly improving the stability and reliability of the output results. Thus, the key problems of "high computational cost and difficult to generalize" and "over-reliance on model self-consistency, easy to cause error accumulation, and affect output stability" in the prior art are systematically solved.

[0042] As Figure 5As shown in the middle, the embodiment also provides a hallucination reduction system of a visual language large model, which comprises an input sequence acquisition module 10, a preliminary vocabulary probability distribution acquisition module 20, and a target vocabulary probability distribution module 40. Specifically, the input sequence acquisition module 10 is configured to acquire input graphic-text data, process the graphic-text data to obtain text tokens and image tokens, and align the text tokens and the image tokens to obtain an input sequence of the graphic-text data. The preliminary vocabulary probability distribution acquisition module 20 is configured to input the input sequence into a visual language large model, calculate an attention distribution through a cross-modal attention mechanism, and obtain a preliminary vocabulary probability distribution at a current time step based on the attention distribution. The target vocabulary probability distribution module 30 is configured to perform multi-scale enhancement and fusion reasoning based on the preliminary vocabulary probability distribution and the attention distribution to obtain a target vocabulary probability distribution at the current time step.

[0043] In an implementation manner, the input sequence acquisition module 10 comprises: a text token acquisition unit configured to perform word segmentation, mapping, and embedding processing on text in the graphic-text data to obtain text tokens; an image token acquisition unit configured to perform block division, encoding, and projection processing on images in the graphic-text data to obtain image tokens.

[0044] In an implementation manner, the preliminary vocabulary probability distribution acquisition module 20 comprises: a standard attention score acquisition unit configured to input the input sequence into a visual language large model, calculate a standard attention score of each layer of Transformer based on text tokens and image tokens of the input sequence through a cross-modal attention mechanism; a deep attention score acquisition unit configured to extract a standard attention score of a middle layer of Transformer based on the standard attention score of each layer of Transformer to obtain a middle layer attention score, and extract a standard attention score of a deep layer of Transformer to obtain a deep layer attention score; a deep recalibration attention score acquisition unit configured to perform weighted fusion on the deep layer attention score and the middle layer attention score, and perform visual semantic amplification to obtain a deep recalibration attention score; an attention distribution acquisition unit configured to perform normalization operation on the deep recalibration attention score and the standard attention score of the remaining Transformer layers respectively to obtain an attention distribution of each layer of Transformer.

[0045] In an implementation manner, the target vocabulary probability distribution module 30 comprises: The multi-scale enlarged area determination unit is configured to perform information entropy calculation on the preliminary vocabulary probability distribution to determine a multi-scale enlarged area. The multi-scale image set acquisition unit is configured to perform multi-scale enlargement on the multi-scale enlarged area according to the attention distribution to obtain a multi-scale image set. The target vocabulary probability distribution acquisition unit is configured to perform multi-scale fusion reasoning on the multi-scale image set to obtain a target vocabulary probability distribution of a current time step.

[0046] In an implementation manner, the multi-scale enlarged area determination unit comprises: The preset text token probability distribution acquisition sub-unit is configured to select probability distributions of a preset number of text tokens in a descending order of probability based on the preliminary vocabulary probability distribution to obtain preset text token probability distributions. The preset text token information entropy value acquisition sub-unit is configured to perform information entropy calculation on the preset text token probability distributions to obtain preset text token information entropy values. The multi-scale enlarged area acquisition sub-unit is configured to determine that an image token corresponding to the preset text token is a multi-scale enlarged area if the preset text token information entropy value is greater than an entropy value threshold.

[0047] In an implementation manner, the multi-scale image set acquisition unit comprises: The middle-shallow layer attention distribution acquisition sub-unit is configured to select attention distributions of a middle layer Transformer layer and a shallow layer Transformer layer based on the attention distribution to obtain a middle-shallow layer attention distribution. The attention map acquisition sub-unit is configured to perform normalization, activation and weighting processing on the middle-shallow layer attention distribution to obtain an attention map. The multi-scale image set acquisition sub-unit is configured to perform multi-scale enlargement on the multi-scale enlarged area based on the attention map to obtain a multi-scale image set.

[0048] In an implementation manner, the target vocabulary probability distribution acquisition unit comprises: The probability distribution acquisition sub-unit is configured to perform multi-scale prediction on each multi-scale image in the multi-scale image set to obtain a probability distribution. The weight coefficient acquisition sub-unit is configured to perform processing on the probability distribution through information entropy calculation, consistency prediction and normalization operation to obtain a weight coefficient. The target vocabulary probability distribution acquisition sub-unit is configured to perform weighted summation on the probability distribution by using the weight coefficient to obtain a target vocabulary probability distribution of a current time step.

[0049] In an implementation manner, the weight coefficient acquisition sub-unit comprises: an information entropy value obtaining subunit, configured to calculate information entropy based on the probability distribution, and obtain an information entropy value; a consistency score obtaining subunit, configured to perform consistency prediction on the probability distribution through a cosine similarity matrix, and obtain a consistency score; a weight coefficient obtaining subunit, configured to calculate a confidence weight based on the consistency score and the information entropy value, and perform normalization processing on the confidence weight, and obtain a weight coefficient.

[0050] The working principles of the various modules in the hallucination phenomenon alleviation system of the visual language large model of this embodiment are the same as the principles of the various steps in the method embodiments described above, and will not be repeated here.

[0051] Based on the above embodiments, the present application further provides a terminal, and a principle block diagram of the terminal can be as shown in Figure 6 The terminal can include one or more processors 100 (only one is shown in Figure 6 the memory 101 and the computer program 102 stored in the memory 101 and executable on the one or more processors 100, for example, a hallucination phenomenon alleviation program of a visual language large model. The one or more processors 100 can implement each step in the hallucination phenomenon alleviation method embodiment of the visual language large model when executing the computer program 102. Alternatively, the one or more processors 100 can implement the functions of each module / unit in the hallucination phenomenon alleviation method embodiment of the visual language large model when executing the computer program 102, which is not limited here.

[0052] In one embodiment, the processor 100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0053] In one embodiment, the storage 101 can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The storage 101 can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the storage 101 can include both the internal storage unit and the external storage device of the electronic device. The storage 101 is used to store computer programs and other programs and data required by the terminal. The storage 101 can also be used to temporarily store data that has been output or will be output.

[0054] Those skilled in the art can understand that, Figure 6 The person skilled in the art can understand that the principle block diagram shown in the above-mentioned embodiments is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0055] The person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, operating database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0056] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hallucination alleviation method of a visual language large model, characterized in that, The method comprises: obtaining input image-text data, processing the image-text data to obtain text tokens and image tokens, and aligning the text tokens and the image tokens to obtain an input sequence of the image-text data; inputting the input sequence into a visual language large model, calculating an attention distribution through a cross-modal attention mechanism, and obtaining a preliminary vocabulary probability distribution of a current time step based on the attention distribution; based on the preliminary vocabulary probability distribution and the attention distribution, performing multi-scale enhancement and fusion reasoning to obtain a target vocabulary probability distribution of the current time step. 2.The hallucination alleviation method of a visual language large model according to claim 1, wherein, The inputting the input sequence into a visual language large model, calculating an attention distribution through a cross-modal attention mechanism comprises: inputting the input sequence into a visual language large model, calculating a standard attention score of each Transformer layer through a cross-modal attention mechanism based on the text tokens and the image tokens of the input sequence; based on the standard attention score of each Transformer layer, extracting the standard attention score of a middle Transformer layer to obtain a middle attention score, and extracting the standard attention score of a deep Transformer layer to obtain a deep attention score; performing weighted fusion on the deep attention score and the middle attention score, and performing visual semantic amplification to obtain a deep recalibrated attention score; performing normalization operations on the deep recalibrated attention score and the standard attention scores of the remaining Transformer layers respectively to obtain the attention distribution of each Transformer layer. 3.The hallucination alleviation method of a visual language large model according to claim 2, wherein, The multi-scale enhancement and fusion reasoning based on the preliminary vocabulary probability distribution and the attention distribution to obtain the target vocabulary probability distribution of the current time step comprises: performing information entropy calculation on the preliminary vocabulary probability distribution to determine a multi-scale amplification region; performing multi-scale amplification on the multi-scale amplification region through the attention distribution to obtain a multi-scale image set; performing multi-scale fusion reasoning on the multi-scale image set to obtain the target vocabulary probability distribution of the current time step. 4.The hallucination alleviation method of a visual language large model according to claim 3, wherein, The information entropy calculation on the preliminary vocabulary probability distribution to determine a multi-scale amplification region comprises: based on the preliminary vocabulary probability distribution, sorting the probability from high to low, selecting the probability distribution of a preset number of text tokens to obtain a preset text token probability distribution; performing information entropy calculation on the preset text token probability distribution to obtain a preset text token information entropy value; if the preset text token information entropy value is greater than an entropy value threshold, the image token corresponding to the preset text token is a multi-scale amplification region. 5.The hallucination alleviation method of a visual language large model according to claim 4, wherein, The multi-scale amplification on the multi-scale amplification region through the attention distribution to obtain a multi-scale image set comprises: based on the attention distribution, selecting the attention distribution of a middle Transformer layer and a shallow Transformer layer to obtain a middle-shallow layer attention distribution; performing normalization, activation and weighting processing on the middle-shallow layer attention distribution to obtain an attention map; The multi-scale enlarged regions are multi-scale enlarged based on the attention map, to obtain a multi-scale image set. 6.The hallucination alleviation method of a visual language large model according to claim 5, wherein, The multi-scale image set is subjected to multi-scale fusion reasoning to obtain a target vocabulary probability distribution at a current time step, including: Each multi-scale image in the multi-scale image set is subjected to multi-scale prediction to obtain a probability distribution; The probability distribution is processed through information entropy calculation, consistency prediction, and normalization operation to obtain a weight coefficient; The probability distribution is weighted and summed using the weight coefficient to obtain the target vocabulary probability distribution at the current time step. 7.The hallucination alleviation method of a visual language large model according to claim 6, wherein, The probability distribution is processed through information entropy calculation, consistency prediction, and normalization operation to obtain a weight coefficient, including: Information entropy is calculated based on the probability distribution to obtain an information entropy value; The probability distribution is subjected to consistency prediction through a cosine similarity matrix to obtain a consistency score; Based on the consistency score and the information entropy value, a confidence weight is calculated, and the confidence weight is normalized to obtain a weight coefficient.

8. A hallucination reduction system for a visual language large model, the system comprising: The system includes: An input sequence acquisition module configured to acquire input text data, process the text data to obtain text tokens and image tokens, and align the text tokens and the image tokens to obtain an input sequence of the text data; A preliminary vocabulary probability distribution acquisition module configured to input the input sequence into a visual language large model, calculate an attention distribution through a cross-modal attention mechanism, and obtain a preliminary vocabulary probability distribution at a current time step based on the attention distribution; A target vocabulary probability distribution module configured to perform multi-scale enhancement and fusion reasoning based on the preliminary vocabulary probability distribution and the attention distribution to obtain a target vocabulary probability distribution at the current time step.

9. A terminal, characterized by comprising: The terminal includes a memory, a processor, and a visual language large model hallucination alleviation program stored in the memory and executable on the processor. When the processor executes the visual language large model hallucination alleviation program, the steps of the visual language large model hallucination alleviation method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a visual language large model hallucination alleviation program. When the processor executes the visual language large model hallucination alleviation program, the steps of the visual language large model hallucination alleviation method according to any one of claims 1-7 are implemented.

Images