Description text generation method and device based on attention head dynamic suppression visual language model

Abstract

The invention relates to the technical field of model optimization, financial science and technology and medical health, and discloses a description text generation method and device based on an attention head dynamic suppression visual language model.The method comprises the steps that a to-be-processed image and a text instruction are received, the to-be-processed image is encoded into a plurality of visual tokens, and the visual tokens are stored in the text instruction; encoding the text instruction into an initial text token sequence; based on the multiple visual tokens and the initial text token sequence, sub-texts are generated one by one through a preset visual language model, a description text is generated according to the sub-texts, when each sub-text is generated, for a current layer Transform module, accumulated attention scores of all attention heads in the module to the multiple visual tokens are calculated, and a target suppression head is determined according to the accumulated attention scores; generating a dynamic mask for the target suppression head, and suppressing the output of the target suppression head; and based on the total output of all the attention heads after suppression, obtaining the output feature representation of the current layer.

Description

Technical Field

[0001] This application relates to the fields of model optimization, financial technology, and healthcare technology, and in particular to a method and apparatus, storage medium, and computer device for generating descriptive text based on an attention head dynamic inhibition visual language model. Background Technology

[0002] In recent years, with the rapid development of multimodal artificial intelligence technology, large-scale visual language models have demonstrated outstanding image understanding and content generation capabilities. They can generate coherent and rich text descriptions based on input images and are widely used in various fields such as intelligent question answering, content creation, and assisted diagnosis. These models are typically based on the Transformer architecture, which integrates visual and linguistic information through its core attention mechanism, achieving effective collaboration between visual perception and language generation.

[0003] However, existing visual language models commonly face a serious "illusion" problem in practical applications, where the generated descriptive text contains details that do not match the visual content of the input image or are completely fictitious. For example, when describing a picture containing only pets and furniture, the descriptive text generated by the visual language model might contain the incorrect description "there is a book in the picture." Current mainstream methods to mitigate this problem fall into two categories: one is training-based methods, such as fine-tuning with high-quality data or reinforcement learning alignment, but these are computationally expensive and require repeated training; the other is inference-based intervention methods, such as adjusting the sampling strategy during the decoding stage. While these methods do not require training, they often introduce significant additional computational latency, affecting inference efficiency. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, storage medium, and computer device for generating descriptive text based on a visual language model with dynamic suppression of attention heads. First, it receives an image to be processed and a corresponding text instruction, and encodes them to obtain multiple visual tokens corresponding to the image to be processed and an initial text token sequence corresponding to the text instruction. Next, the encoded visual tokens and the initial text token sequence are input into a preset visual language model, which outputs descriptive text. Specifically, when the preset visual language model generates each word (i.e., "sub-text") of the descriptive text, each Transformer module can be intervened in real time. Specifically, for any Transformer module, the cumulative attention score of each attention head on all visual tokens is first calculated. Based on the calculated cumulative attention scores of each attention head, a target suppression head is identified and determined. After determining the target suppression head, a dynamic mask can be generated for these target suppression heads. Finally, the outputs of all attention heads, weighted and adjusted using the mask, are used for normal subsequent calculations to generate the most likely word for the current step. The embodiments of this application do not require retraining of the preset visual language model. By using lightweight real-time computation and intervention during inference, attention heads that cause hallucinations in the model can be accurately suppressed. This significantly improves the visual fidelity and factual reliability of the generated descriptive text without increasing computational latency.

[0005] According to one aspect of this application, a method for generating descriptive text based on an attention head dynamic suppression visual language model is provided, comprising: Receive an image to be processed and a corresponding text instruction, encode the image to be processed into multiple visual tokens, and encode the text instruction into an initial text token sequence; Based on the multiple visual tokens and the initial text token sequence, sub-texts are generated one by one in an autoregressive manner through a preset visual language model, and descriptive text of the image to be processed is generated based on each sub-text. The preset visual language model includes multiple Transformer modules, which are connected sequentially. Each Transformer module includes multiple attention heads. When the preset visual language model generates sub-texts, the following operations are performed on the current Transformer module that is currently processing data: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

[0006] According to another aspect of this application, a descriptive text generation apparatus based on an attention head dynamic suppression visual language model is provided, comprising: The data receiving module is used to receive the image to be processed and the corresponding text instructions, encode the image to be processed into multiple visual tokens, and encode the text instructions into an initial text token sequence; The descriptive text generation module is used to generate sub-texts one by one autoregressively based on the multiple visual tokens and the initial text token sequence through a preset visual language model, and generate descriptive texts for the image to be processed based on each sub-text; The preset visual language model includes a multi-layer Transformer module, with each layer connected sequentially, and each layer including multiple attention heads; the descriptive text generation module is further used for: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

[0007] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described descriptive text generation method based on an attention head dynamic suppression visual language model.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described descriptive text generation method based on the attention head dynamic suppression visual language model.

[0009] Using the above technical solution, this application provides a method, apparatus, storage medium, and computer device for generating descriptive text based on a dynamic suppression visual language model using attention heads. First, it receives an image to be processed and a corresponding text instruction, and encodes them to obtain multiple visual tokens corresponding to the image to be processed and an initial text token sequence corresponding to the text instruction. Next, the encoded visual tokens and the initial text token sequence are input into a preset visual language model, which outputs descriptive text. Specifically, when the preset visual language model generates each word (i.e., "sub-text") of the descriptive text, real-time intervention can be performed on each Transformer module. For any Transformer module, the cumulative attention score of each attention head for all visual tokens is first calculated. Based on the calculated cumulative attention scores of each attention head, a target suppression head is identified and determined. After determining the target suppression head, a dynamic mask can be generated for these target suppression heads. Finally, the outputs of all attention heads, weighted and adjusted using the mask, are used for normal subsequent calculations to generate the most likely word for the current step. The embodiments of this application do not require retraining of the preset visual language model. By using lightweight real-time computation and intervention during inference, attention heads that cause hallucinations in the model can be accurately suppressed. This significantly improves the visual fidelity and factual reliability of the generated descriptive text without increasing computational latency.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The illustration shows a flowchart of a descriptive text generation method based on an attention head dynamic suppression visual language model provided in an embodiment of this application; Figure 2 This paper shows a schematic diagram of the structure of a preset decision model provided in an embodiment of the present application; Figure 3 This illustration shows a schematic diagram of a descriptive text generation device based on an attention head dynamic suppression visual language model provided in an embodiment of this application; Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a descriptive text generation method based on an attention head dynamic suppression visual language model, such as... Figure 1 As shown, the method includes: Step 101: Receive the image to be processed and the corresponding text instruction, encode the image to be processed into multiple visual tokens, and encode the text instruction into an initial text token sequence.

[0014] Step 102: Based on the multiple visual tokens and the initial text token sequence, sub-texts are generated one by one autoregressively using a preset visual language model, and descriptive text for the image to be processed is generated based on each sub-text.

[0015] The preset visual language model includes multiple Transformer modules, which are connected sequentially. Each Transformer module includes multiple attention heads. When the preset visual language model generates sub-texts, the following operations are performed on the current Transformer module that is currently processing data: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

[0016] This application provides a method for generating descriptive text based on a visual language model with dynamic suppression of attention heads. First, it receives an image to be processed and its corresponding text instruction, and encodes them to obtain multiple visual tokens corresponding to the image to be processed and an initial text token sequence corresponding to the text instruction. The text instruction can be an instruction to generate descriptive text for the image to be processed, such as "Please describe this image in detail." In a specific embodiment, the image to be processed can be segmented into multiple image patches by a visual encoder. Each image patch is converted into a vector representation, called a visual token, through linear projection. These visual tokens together constitute a computable representation of the image to be processed. Simultaneously, the text instruction can be split into basic semantic units by a tokenizer and then mapped into a vector sequence, called a text token, through an embedding layer. These text tokens are then combined to form the initial text token sequence. This method can uniformly transform the unstructured image to be processed and the text instruction into a digital sequence that the model can process.

[0017] Next, the encoded visual tokens and the initial text token sequence are input into a pre-defined visual language model. The core of this model is the Transformer architecture, which progressively fuses visual and textual information through multiple stacked Transformer modules and generates words describing the text one by one in an autoregressive manner. Autoregressive generation means that when the model outputs each word, it can use all previously generated words as context input, iterating in this way until a complete descriptive text is generated. In this way, the model can dynamically combine the content of the image to be processed with the generated text to determine the next most suitable word.

[0018] Specifically, when the preset visual language model generates each word (i.e., "sub-text") of the descriptive text, real-time intervention can be performed on each Transformer module. For any Transformer module, the cumulative attention score of each attention head for all visual tokens is first calculated. Attention heads are parallel sub-modules within each Transformer module, each responsible for calculating the correlation between input visual tokens from different perspectives. The cumulative attention score is obtained by summing the dot product of the query vector corresponding to the current generation position and the visual key vector corresponding to each visual token, quantifying the overall attention the attention head pays to the image being processed when generating the current word.

[0019] Next, based on the calculated cumulative attention scores of each attention head, the target suppression head is identified and determined. The target suppression head's attention score for visual tokens is significantly lower than that of other attention heads in the same layer, indicating that the target suppression head relies excessively on linguistic priors or biases in the text context when generating the current word, while ignoring the actual content in the image to be processed, leading to illusions. For example, when generating the word "chair" which is unrelated to the image to be processed, some attention heads may focus on words such as "background" in the text history, rather than the visual signals in the image to be processed.

[0020] Once the target suppression heads are determined, dynamic masks can be generated for them. The mask is a coefficient multiplied by the output of the attention heads. For the target suppression heads, this coefficient can be set to a suppression factor between 0 and 1 (such as 0 or 0.1), thus partially or completely suppressing their output contribution. For other attention heads that are normally focused on the image being processed, the mask can be kept at 1 to preserve their original output. This allows for real-time reduction of the output contribution of attention heads that might induce hallucinations within the model.

[0021] Finally, the outputs of all attention heads, after being adjusted by masking weighting, are used for normal subsequent calculations (such as stitching, projection, and nonlinear transformations) to generate the output feature representation of the current layer's Transformer module. This corrected output is more likely to rely on attention heads that are faithful to the image being processed, thus significantly reducing the probability of generating words that do not match the visual content in this round.

[0022] In one specific embodiment, when processing skin lesion images and generating diagnostic reports, the system first receives a dermoscopic image and a physician's input instruction: "Please describe the microscopic features of this lesion and assess the risk." The dermoscopic image is segmented and encoded into a series of visual tokens, each token corresponding to a local region of the dermoscopic image, carrying subtle features such as color and texture. Simultaneously, the text instruction is also segmented and converted into an initial text token sequence. A preset visual language model begins generating the report text. Specifically, in a certain Transformer module, the cumulative attention score of each attention head for all visual tokens is calculated in real time. It is found that most attention heads are highly focused on the actual pigment network regions in the dermoscopic image, but a few attention heads severely ignore visual information, with extremely low attention scores. Instead, they focus on previously generated text words, showing a tendency to over-rely on linguistic association rather than visual evidence. Therefore, these attention heads can be identified as target suppression heads, and dynamic masks are immediately generated for these target suppression heads to significantly attenuate their output. After suppression, the attention head that is faithful to the visual information of the dermoscopy image takes the lead in decision-making, thus presupposing that the visual language model can accurately output the description "visible irregular pigment network" based on actual image observation, rather than hallucinating misleading features that do not exist in the image, such as "visible punctate blood vessels".

[0023] By applying the technical solution of this embodiment, firstly, the image to be processed and the corresponding text instruction are received and encoded respectively to obtain multiple visual tokens corresponding to the image to be processed and an initial text token sequence corresponding to the text instruction. Next, the encoded multiple visual tokens and the initial text token sequence are input into a preset visual language model to output descriptive text. Specifically, when the preset visual language model generates each word (i.e., "sub-text") of the descriptive text, real-time intervention can be performed on each Transformer module. Specifically, for any Transformer module, the cumulative attention score of each attention head for all visual tokens is first calculated. Based on the calculated cumulative attention scores of each attention head, target suppression heads are identified and determined. After determining the target suppression heads, dynamic masks can be generated for these target suppression heads. Finally, the outputs of all attention heads, after being weighted and adjusted using the masks, are used for normal subsequent calculations to generate the most likely word for the current step. The embodiments of this application do not require retraining of the preset visual language model. By using lightweight real-time computation and intervention during inference, attention heads that cause hallucinations in the model can be accurately suppressed. This significantly improves the visual fidelity and factual reliability of the generated descriptive text without increasing computational latency.

[0024] Optionally, in this embodiment, the step of "calculating the cumulative attention score of each attention head for the plurality of visual tokens in the current layer Transformer module" includes: for each attention head in the current layer Transformer module, obtaining the query projection matrix and key projection matrix corresponding to the attention head, wherein the query projection matrix and the key projection matrix are obtained during the training of the preset visual language model; generating a current query vector based on the query projection matrix and the input vector of the current layer Transformer module, wherein the input vector is generated based on the plurality of visual tokens and the currently generated description text; generating a visual key vector corresponding to each visual token based on the visual token representation corresponding to each visual token and the key projection matrix; calculating the dot product between the current query vector and each visual key vector to obtain the original attention score of the attention head for each visual token; and summing the original attention scores of all visual tokens to obtain the cumulative attention score of the attention head for the plurality of visual tokens.

[0025] In this embodiment, when calculating the cumulative attention score of each attention head for multiple visual tokens in the current Transformer module, the query projection matrix and key projection matrix corresponding to that attention head are first obtained. These matrices are core parameters of the Transformer attention mechanism, learned during the training of the preset visual language model, and are used to map input vectors to different representation spaces. The query projection matrix is ​​responsible for generating query vectors, while the key projection matrix is ​​used to generate key vectors; together, they determine the behavior pattern of the attention head during information retrieval.

[0026] Next, the current query vector is generated based on the query projection matrix and the input vector from the current Transformer module. Here, the input vector is generated jointly by multiple visual tokens corresponding to the image to be processed and the generated descriptive text, containing all the multimodal information integrated by the preset visual language model in the current generation step. By multiplying the input vector with the query projection matrix, the current query vector is obtained. This vector represents the query intent of the current generation position for the input content, that is, which features the preset visual language model hopes to extract from the input when predicting the next word. This step transforms the abstract generation task into concrete vector computation, laying the foundation for evaluating the attention distribution.

[0027] For each visual token, a corresponding visual key vector can be generated. A visual token is the encoded vector form of the image to be processed, and each visual token corresponds to a local region or global feature of the image. By multiplying the visual token representation corresponding to each visual token with the key projection matrix, visual key vectors can be obtained. These visual key vectors characterize the information features of various parts of the image to be processed in the key space.

[0028] Subsequently, the dot product between the current query vector and each visual key vector is calculated to obtain the raw attention score of the attention head for each visual token. The dot product operation measures the similarity between the current query vector and the visual key vector; a higher score indicates that the visual token is more relevant to the current generation task. The raw attention score reflects the initial attention distribution of the attention head across different parts of the image to be processed, and is fundamental data for understanding the behavior of the attention head, helping to identify which visual information is prioritized or ignored.

[0029] Finally, the original attention scores of all visual tokens are summed to obtain the cumulative attention score of the attention head for multiple visual tokens. This summation operation aggregates the overall attention level of the attention head on the entire image being processed; the score directly reflects the strength of the attention head's dependence on visual content when generating the current word. Attention heads with lower cumulative attention scores tend to rely more on linguistic priors from the textual context than on actual image information, thus potentially becoming a source of hallucinations.

[0030] In a specific embodiment, when generating an analysis summary based on visual charts (such as revenue composition bar charts) in a company's financial statements, a pre-defined visual language model can sequentially perform key calculations in each Transformer module. For each attention head in any Transformer module, it first calls the query projection matrix and key projection matrix that it has learned during training. Then, based on the input vector of the current layer (which is generated by visual tokens encoded by all visual elements in the visual chart, as well as the generated descriptive text such as "this quarter" and "service") and the query projection matrix, a current query vector is generated. At the same time, using the key projection matrix, the visual tokens representing different bar charts such as "product revenue," "service revenue," and "other revenue" in the visual chart are converted into corresponding visual key vectors. Each visual key vector encodes the key information features of a specific component in the visual chart. Next, the dot product of the current query vector and each visual key vector is calculated to obtain the original attention score of the attention head for each visual token. Finally, the original attention scores of this attention head for all visual tokens are summed to obtain its cumulative attention score. This cumulative attention score quantifies the sum of how much an attention head relies on the overall information in the visualization when generating the current key financial terms. If an attention head has an extremely low score, it indicates that it almost ignores the visual information in the visualization when making analysis and judgments, and may over-rely on the inherent patterns in the textual context (e.g., rigidly applying the text template "revenue growth is often accompanied by simultaneous cost growth"), which may induce "illusionary" analytical conclusions that are inconsistent with the facts in the visualization.

[0031] This embodiment achieves fine-grained quantitative analysis of attention head behavior by calculating the cumulative attention score of each attention head, providing a reliable basis for accurately suppressing hallucination sources. This embodiment requires no modification to the model architecture or retraining; it only performs real-time evaluation based on internal signals during inference, balancing efficiency and accuracy.

[0032] Optionally, in this embodiment of the application, the step of "generating a dynamic mask for the target suppression head and suppressing the output of the target suppression head through the dynamic mask" includes: using a preset suppression factor as the dynamic mask of the target suppression head, wherein the preset suppression factor is a real number between 0 and 1; multiplying the output of the target suppression head by the corresponding dynamic mask to obtain the suppressed output of the target suppression head.

[0033] In this embodiment, before generating a dynamic mask for the target suppression head, a preset suppression factor can be set as the specific value of the mask. The dynamic mask is actually an adjustable weighting coefficient, whose value is limited to a real number range between 0 and 1, used to control the contribution of the target suppression head to the attention output. The closer the preset suppression factor is to 0, the stronger the suppression of the attention head; if it is set to 0, it is equivalent to completely blocking the output of the attention head.

[0034] After generating the dynamic mask, the target suppression head's output is multiplied by its corresponding dynamic mask to suppress it. Specifically, in the Transformer module, each attention head can independently compute and output a vector representation. These vectors are typically concatenated and then linearly transformed to form the layer's output feature representation. In this embodiment, for the output vector identified as a target suppression head, it can first be multiplied element-wise with a preset suppression factor to directly reduce its numerical magnitude, obtaining the suppressed output vector. Then, based on the suppressed output vector, it participates in the generation of the layer's final output.

[0035] Optionally, in this embodiment, the step of "determining a target suppression head from the plurality of attention heads based on the cumulative attention scores of each attention head" includes: calculating the statistical dispersion of the cumulative attention scores of all attention heads based on the cumulative attention scores of each attention head; obtaining additional feature information of the current layer Transformer module, wherein the additional feature information includes at least one of the predicted part-of-speech category of the current subtext to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model; outputting suppression head determination parameters through a preset decision model based on the statistical dispersion and the additional feature information, wherein the suppression head determination parameters are the suppression head ratio or the number of suppression heads; and determining a target suppression head from the plurality of attention heads according to the suppression head determination parameters and the cumulative attention scores of each attention head.

[0036] In this embodiment, when determining the target suppression head, firstly, based on the cumulative attention scores already calculated for each attention head, the statistical dispersion of these cumulative attention scores is calculated. Statistical dispersion can be measured using variance, standard deviation, or range, reflecting the consistency of attention given to the image to be processed by all attention heads in the current layer. High statistical dispersion indicates significant differences in the behavior of each attention head, with some attention heads potentially deviating severely from the image information of the image to be processed; low statistical dispersion indicates that the attention given to the image to be processed by each attention head is relatively uniform. This indicator provides a basis for determining whether intervention is needed in the current generation step and the overall intensity of the intervention.

[0037] Simultaneously, additional feature information from the current Transformer module can be obtained to enrich the decision context. Specifically, this additional feature information may include: the predicted part-of-speech category (e.g., noun, verb) of the subtext to be generated, reflecting the degree to which the subtext relies on visual information (nouns describing entities usually require more visual evidence); the prediction confidence of the current Transformer module, reflecting the certainty of generating the subtext; low confidence indicates a conflict between visual and textual signals; and the depth information of this Transformer module within the predefined visual language model (e.g., layer number), as layers of different depths are used to handle information processing tasks at different levels of abstraction. These additional feature information collectively constitute the fine-grained state of the current generation step.

[0038] Furthermore, statistical dispersion and additional feature information can be jointly input into a pre-defined decision model. This model can be a trained small neural network or a rule engine, whose core function is to comprehensively analyze the current context and output a specific suppression head decision parameter. This parameter can be the number of attention heads to be suppressed (e.g., "suppress 2 attention heads in this layer") or the proportion of attention heads to be suppressed (e.g., "suppress 10% of attention heads in this layer"). This pre-defined decision model achieves intelligent mapping from multi-dimensional states to specific operational instructions.

[0039] Finally, based on the suppression head decision parameters output by the preset decision model and the original cumulative attention scores of each attention head, the final screening is performed. For example, if the suppression head decision parameter is the number of suppression heads, and the number of suppression heads is K, then the cumulative attention scores of all attention heads can be sorted, and the K attention heads with the lowest scores can be selected as target suppression heads; if the suppression head decision parameter is the suppression head ratio, and the suppression head ratio is R, then the number of attention heads to be suppressed is calculated, and the corresponding number of attention heads with the lowest scores are suppressed.

[0040] In a specific embodiment, within a medical and health science popularization platform, when a physician uploads a typical fatty liver pathology image and requests a descriptive summary—for example, faced with a slice showing only fat vacuoles—a pre-defined visual language model might instinctively associate it with textual knowledge, attempting to generate common inferences like "potentially developing into cirrhosis," which lack visual evidence in this image. At this point, attention heads that ignore image details and rely solely on textual associations can be identified in real time, and their output dynamically suppressed. This forces the model to anchor its description to visual features, ultimately generating an objective description such as "a large accumulation of fat droplets is visible within liver cells," without adding speculation about disease stages not shown in the image. This ensures the rigor of the popular science information and prevents misleading the public due to the incorporation of general textual knowledge.

[0041] This application embodiment, by fusing the cumulative attention score statistical dispersion and multi-dimensional additional feature information, can not only judge the overall intervention intensity based on the statistical dispersion, but also dynamically output precise suppression ratio or quantity parameters by combining specific contextual information such as part of speech, prediction confidence and network depth. This flexible adjustment of the number of target suppression heads can significantly improve the factual accuracy and reliability of the descriptive text.

[0042] In this embodiment of the application, optionally, when the additional feature information includes the predicted part-of-speech category of the current sub-text to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model, the step of "outputting the suppression head decision parameters based on the statistical dispersion and the additional feature information through the preset decision model" includes: encoding the statistical dispersion, the predicted part-of-speech category, the prediction confidence, and the depth information respectively to obtain multiple encoding vectors; inputting the multiple encoding vectors into the feature fusion module of the preset decision model, and through the attention network in the feature fusion module, dividing... Calculate the dynamic weights corresponding to each encoding vector, and generate a fused feature vector based on the multiple encoding vectors and their respective dynamic weights; input the fused feature vector into the calculation module of the preset decision model, and output the initial decision parameters through the calculation module; input the initial decision parameters into the verification module of the preset decision model, and perform a feasibility verification on the initial decision parameters based on preset boundary constraints and inter-layer smoothing constraints through the verification module; when the verification passes, the initial decision parameters are used as the suppression head decision parameters, wherein the inter-layer smoothing constraints are used to ensure that the rate of change of the suppression head decision parameters between adjacent Transformer modules does not exceed a preset threshold.

[0043] In this embodiment, the additional feature information is first uniformly encoded. The predicted part-of-speech category, as a category variable, can be converted into a dense vector using one-hot encoding or an embedding layer; the predicted confidence score, as a scalar, can be directly used as a one-dimensional numerical feature after normalization; depth information (such as layer numbers) can be normalized or segmented; and the statistical dispersion is itself a numerical scalar. The encoding process transforms features of different types and dimensions into vector representations with unified dimensions, providing standardized input for subsequent fusion calculations and ensuring the comparability and operability of different modal features in the mathematical space.

[0044] The encoded vectors are then fed into the feature fusion module of a pre-defined decision model. This module may contain a lightweight attention network (such as multi-head self-attention or cross-attention) that dynamically evaluates the importance of each dimension in the current decision context. The network calculates a weight coefficient for each encoded vector; a higher weight indicates a greater contribution of that dimension to the current inhibition decision. For example, when generating visually dependent nouns, the weight of the statistical dispersion feature can be automatically increased; conversely, when the model confidence is low, the weight of the confidence dimension can be increased accordingly. Finally, all encoded vectors are weighted and summed to generate a fused feature vector that comprehensively represents the current decision context.

[0045] The fused feature vector is then fed into the computation module of the decision model. This module, which may consist of one or more fully connected layers, is responsible for performing the nonlinear mapping from the fused features to the initial decision. Specifically, the computation module takes the fused feature vector as input, passes it through a series of matrix transformations and activation functions, and outputs an unverified initial decision parameter (e.g., a raw numerical value representing the proposed suppression ratio or number).

[0046] After obtaining the initial decision parameters, a rigorous feasibility check can be performed. This check primarily relies on two constraints: one is a preset boundary constraint, ensuring the initial decision parameters are within a reasonable range (e.g., the suppression ratio must be between 0% and 30%) to prevent extreme or unreasonable decisions; the other is an inter-layer smoothing constraint, ensuring the rate of change between the initial decision parameters output by the current layer and the suppression head decision parameters of its adjacent layers (e.g., the previously processed layer) does not exceed a preset threshold. This constraint forces the model to maintain the consistency and stability of decision-making in the vertical depth, avoiding drastic fluctuations in the suppression strategies of adjacent layers, thereby ensuring the smoothness of information transmission between Transformer modules and maintaining the overall coordination and consistency of the model's processing. If the initial decision parameters pass the feasibility check, they can be used as the final suppression head decision parameters.

[0047] In a specific embodiment, in a fintech scenario, such as when generating credit risk reviews based on corporate financial statement charts and news, when generating the key conclusive phrase (noun phrase) "deterioration of debt repayment ability," the model can determine targeted initial decision parameters based on statistical dispersion, prediction confidence, and depth information. It can also verify the feasibility of the parameters through preset boundary constraints and inter-layer smoothing constraints. This can effectively prevent the model from ignoring the actual cash flow data in the corporate financial statement charts due to excessive focus on negative emotional words in the text, and ultimately generate a risk assessment with more factual basis.

[0048] The embodiments of this application not only consider the statistical dispersion characteristics of the cumulative attention score, but also incorporate multi-dimensional additional feature information, and ensure the rationality and stability of the decision through a constraint verification mechanism, making the suppression of the hallucination source more accurate, adaptive and reliable.

[0049] Optionally, in this embodiment, the computation module includes a three-layer sub-network; the step of "outputting initial decision parameters through the computation module" includes: inputting the fused feature vector into the first-layer sub-network of the computation module, and outputting a basic suppression intensity coefficient through the first-layer sub-network; inputting the fused feature vector and the basic suppression intensity coefficient into the second-layer sub-network of the computation module, and outputting a suppression intensity adjustment amount through the second-layer sub-network; inputting the basic suppression intensity coefficient and the suppression intensity adjustment amount into the third-layer sub-network of the computation module, and adding the basic suppression intensity coefficient and the suppression intensity adjustment amount through the third-layer sub-network to obtain the initial decision parameters.

[0050] In this embodiment, the computing module can consist of three sequentially connected sub-networks. For example... Figure 2 The diagram illustrates the structure of a pre-defined decision model provided in an embodiment of this application. The input to the first sub-network is a fused feature vector from the feature fusion module, which encapsulates the statistical characteristics and contextual information of the current generation step. The first sub-network can consist of a fully connected layer and a non-linear activation function. Its core function is to perform the first abstraction and mapping of the input fused feature vector, outputting a scalar value called the basic suppression strength coefficient. This coefficient reflects the initial judgment of the suppression requirement based on the current input information, forming a baseline for subsequent decisions.

[0051] Next, the fused feature vector and the newly generated baseline suppression coefficients are input together into the second sub-network. This allows the preliminary decision results to also serve as a basis for further analysis. The task of the second sub-network is to evaluate whether and to what extent the current context requires calibration of the baseline suppression coefficients, and output a suppression intensity adjustment variable. This suppression intensity adjustment variable can be positive or negative, and its function is to fine-tune the baseline suppression coefficients.

[0052] Finally, the basic inhibition strength coefficient and the inhibition strength adjustment amount are fed together into the third sub-network. This third sub-network adds these two input values ​​to obtain the initial decision parameters. Specifically, this addition operation can be a simple arithmetic summation, or it can include normalization or mild nonlinear processing of the addition result to ensure the rationality and stability of the output value. In this way, the final initial decision parameters consider both the basic judgment given by the first sub-network and the adjustments made by the second sub-network based on more detailed contextual analysis, forming a decision output that is both robust and adaptive.

[0053] In this embodiment of the application, optionally, the step 101 of "encoding the image to be processed into multiple visual tokens and encoding the text instruction into an initial text token sequence" includes: segmenting the image to be processed into multiple non-overlapping image blocks, and encoding each image block to generate the multiple visual tokens; decomposing the text instruction into multiple sub-word units using a preset text segmenter, and mapping each sub-word unit to a word embedding vector; sorting the word embedding vectors according to the order of each sub-word unit in the text instruction to obtain the initial text token sequence.

[0054] In this embodiment, the input image to be processed is first subjected to structured segmentation and encoding. Specifically, the image to be processed is uniformly divided into multiple non-overlapping image blocks (e.g., a 224×224 pixel image to be processed is divided into 16×16 14×14 pixel image blocks). Each image block serves as a unit carrying local visual information, and then features are extracted through an encoder (e.g., a linear projection layer or a small convolutional network), converting it into a fixed-dimensional vector representation. These vectors are called visual tokens, and these visual tokens together constitute a sequence of digital representations of the image to be processed in a predefined visual language model, enabling continuous image information to be embedded in a Transformer-based architecture in sequence for computation.

[0055] For text instructions, a pre-defined text segmenter is used for processing. The text segmenter can employ a sub-word segmentation algorithm (such as Byte-Pair Encoding, BPE) to decompose the input text instruction into multiple sub-word units. A sub-word unit is a semantic fragment between a whole word and a character, effectively covering common vocabulary while flexibly handling out-of-vocabulary words or specialized terms, balancing vocabulary size with the completeness of semantic representation. Each sub-word unit can then be mapped to a high-dimensional dense vector, i.e., a word embedding vector, by querying a pre-trained word embedding vector table. These vectors not only capture the semantic information of the sub-word unit but also implicitly contain its grammatical and contextual attributes.

[0056] Finally, according to the order of appearance of each sub-word unit in the original text instruction, the corresponding word embedding vectors are arranged in sequence to form an initial text token sequence. After the image to be processed and the text instruction go through the above parallel but structurally similar encoding processes, they are respectively transformed into multiple visual tokens and the initial text token sequence, which are aligned in the vector space dimension, so that they can be input into a preset vision-language model for in-depth cross-modal interaction and fusion.

[0057] Furthermore, as a refinement and extension of the specific implementation manner of the above embodiment, in order to completely illustrate the specific implementation process of this embodiment, another method for generating a description text based on an attention head dynamic suppression vision-language model is provided. This method includes: Receiving an image to be processed and the corresponding text instruction. Here, the content of the image to be processed is "two dogs lying on the bed", and the text instruction is "Please describe this picture in detail".

[0058] The original preset vision-language model (without applying the method of this application embodiment) may output: "The picture shows two dogs lying on the bed.... In addition, there is also a chair in the background." Here, the "chair" is an illusory object.

[0059] In the embodiment of this application, the preset vision-language model starts to autoregressively generate sub-texts that make up the description text, such as "picture", "shows", "two", "dogs".... When the preset vision-language model generates a sub-text each time, the behavior of all attention heads can be analyzed within each Transformer module layer. For example, when generating a certain sub-text, it is found that in some Transformer module layers, a small number of attention heads (for example, the 5th head in the 20th layer and the 8th head in the 22nd layer) have a very low cumulative attention score for all image tokens. On the contrary, the attention of these attention heads is highly concentrated on the previously generated sub-texts such as "in the background". This behavior pattern indicates that these attention heads are fabricating content based on the text associations inherent in the language model ("in the background" usually appears with furniture such as "chair", "table", etc.) rather than the real image to be processed. At this time, the dynamic masks of these identified "problem heads" (the 20-5th head and the 22-8th head, etc.) can be set to α (for example, 0), while the dynamic masks of other normal attention heads that focus on the image to be processed remain 1.

[0060] After that, the preset vision-language model can determine the most likely sub-text at the current step according to the weighted and corrected attention result. Since the wrong guidance of the "problem heads" is suppressed, the preset vision-language model is more likely to generate a sub-text consistent with the content of the image to be processed, or directly end the sentence.

[0061] Ultimately, the preset visual language model output became: "The picture shows two dogs lying comfortably on the bed." The hallucination of the "chair" was successfully avoided.

[0062] In summary, the embodiments of this application, through an innovative mechanism for diagnosis and intervention during the operation of a preset visual language model, clearly locate and suppress the functional units that cause hallucinations in large visual language models, thereby overcoming the problems of high cost and large latency in the prior art, and making the model output results more reliable.

[0063] In one specific embodiment, the method of this application can be encapsulated as a plug-and-play module that runs at inference time without requiring any model retraining and with almost no additional computational or latency overhead. Furthermore, it can be seamlessly integrated with various decoding strategies such as greedy search and bundle search.

[0064] Furthermore, as Figure 1 In terms of specific implementation, this application provides a descriptive text generation device based on an attention head dynamic suppression visual language model, such as... Figure 3 As shown, the device includes: The data receiving module is used to receive the image to be processed and the corresponding text instructions, encode the image to be processed into multiple visual tokens, and encode the text instructions into an initial text token sequence; The descriptive text generation module is used to generate sub-texts one by one autoregressively based on the multiple visual tokens and the initial text token sequence through a preset visual language model, and generate descriptive texts for the image to be processed based on each sub-text; The preset visual language model includes a multi-layer Transformer module, with each layer connected sequentially, and each layer including multiple attention heads; the descriptive text generation module is further used for: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

[0065] Optionally, the descriptive text generation module is further configured to: For each attention head in the current layer Transformer module, obtain the query projection matrix and key projection matrix corresponding to the attention head, wherein the query projection matrix and the key projection matrix are obtained during the training of the preset visual language model; Based on the query projection matrix and the input vector of the current layer Transformer module, a current query vector is generated, wherein the input vector is generated based on the multiple visual tokens and the currently generated descriptive text; Based on the visual token representation corresponding to each visual token and the key projection matrix, a visual key vector corresponding to each visual token is generated. Calculate the dot product between the current query vector and each of the visual key vectors to obtain the original attention score of the attention head for each visual token; The original attention scores of all visual tokens are summed to obtain the cumulative attention score of the attention head for the multiple visual tokens.

[0066] Optionally, the descriptive text generation module is further configured to: A preset suppression factor is used as the dynamic mask of the target suppression head, wherein the preset suppression factor is a real number between 0 and 1; The output of the target suppression head is multiplied by the corresponding dynamic mask to obtain the suppressed output of the target suppression head.

[0067] Optionally, the descriptive text generation module is further configured to: Calculate the statistical dispersion of the cumulative attention scores of all attention heads based on the cumulative attention scores of each attention head; Obtain additional feature information of the current layer Transformer module, wherein the additional feature information includes at least one of the predicted part-of-speech category of the current subtext to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model; Based on the statistical dispersion and the additional feature information, a suppression head determination parameter is output through a preset decision model, wherein the suppression head determination parameter is the suppression head ratio or the number of suppression heads; The target suppression head is determined from the plurality of attention heads based on the suppression head determination parameters and the cumulative attention scores of each attention head.

[0068] Optionally, when the additional feature information includes the predicted part-of-speech category of the current sub-text to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model, the descriptive text generation module is further configured to: The statistical dispersion, the predicted part-of-speech category, the predicted confidence, and the depth information are encoded to obtain multiple encoding vectors; The multiple encoded vectors are input into the feature fusion module of the preset decision model. The dynamic weights corresponding to each encoded vector are calculated through the attention network in the feature fusion module. Based on the multiple encoded vectors and their respective dynamic weights, a fused feature vector is generated. The fused feature vector is input into the calculation module of the preset decision model, and the calculation module outputs the initial decision parameters. The initial decision parameters are input into the verification module of the preset decision model. The verification module performs a feasibility verification on the initial decision parameters based on preset boundary constraints and inter-layer smoothing constraints. When the verification is successful, the initial decision parameters are used as the suppression head decision parameters. The inter-layer smoothing constraints are used to ensure that the rate of change of the suppression head decision parameters between adjacent Transformer modules does not exceed a preset threshold.

[0069] Optionally, the computing module includes a three-layer sub-network; the computing module is used for: The fused feature vector is input into the first sub-network of the computing module, and the basic suppression strength coefficient is output through the first sub-network. The fused feature vector and the basic suppression intensity coefficient are input into the second sub-network of the computing module, and the suppression intensity adjustment amount is output through the second sub-network. The basic inhibition strength coefficient and the inhibition strength adjustment amount are input into the third sub-network of the calculation module. The basic inhibition strength coefficient and the inhibition strength adjustment amount are added through the third sub-network to obtain the initial determination parameters.

[0070] Optionally, the data receiving module is configured to: The image to be processed is segmented into multiple non-overlapping image blocks, and each image block is encoded to generate the multiple visual tokens; The text instruction is decomposed into multiple sub-word units by a preset text segmenter, and each sub-word unit is mapped to a word embedding vector. The word embedding vectors are sorted according to the order of each sub-word unit in the text instruction to obtain the initial text token sequence.

[0071] It should be noted that other corresponding descriptions of the functional units involved in the descriptive text generation device based on the attention head dynamic suppression visual language model provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0072] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0073] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0074] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0075] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0077] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A descriptive text generation method based on an attention-head dynamic suppression visual language model, characterized in that, include: Receive an image to be processed and a corresponding text instruction, encode the image to be processed into multiple visual tokens, and encode the text instruction into an initial text token sequence; Based on the multiple visual tokens and the initial text token sequence, sub-texts are generated one by one in an autoregressive manner through a preset visual language model, and descriptive text of the image to be processed is generated based on each sub-text. The preset visual language model includes multiple Transformer modules, which are connected sequentially. Each Transformer module includes multiple attention heads. When the preset visual language model generates sub-texts, the following operations are performed on the current Transformer module that is currently processing data: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

2. The method according to claim 1, characterized in that, For each attention head in the current layer Transformer module, the cumulative attention score of that attention head for the multiple visual tokens is calculated, including: For each attention head in the current layer Transformer module, obtain the query projection matrix and key projection matrix corresponding to the attention head, wherein the query projection matrix and the key projection matrix are obtained during the training of the preset visual language model; Based on the query projection matrix and the input vector of the current layer Transformer module, a current query vector is generated, wherein the input vector is generated based on the multiple visual tokens and the currently generated descriptive text; Based on the visual token representation corresponding to each visual token and the key projection matrix, a visual key vector corresponding to each visual token is generated. Calculate the dot product between the current query vector and each of the visual key vectors to obtain the original attention score of the attention head for each visual token; The original attention scores of all visual tokens are summed to obtain the cumulative attention score of the attention head for the multiple visual tokens.

3. The method according to claim 1, characterized in that, The step of generating a dynamic mask for the target suppression head and suppressing the output of the target suppression head using the dynamic mask includes: A preset suppression factor is used as the dynamic mask of the target suppression head, wherein the preset suppression factor is a real number between 0 and 1; The output of the target suppression head is multiplied by the corresponding dynamic mask to obtain the suppressed output of the target suppression head.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of the target suppression head from the plurality of attention heads based on the cumulative attention scores of each attention head includes: Calculate the statistical dispersion of the cumulative attention scores of all attention heads based on the cumulative attention scores of each attention head; Obtain additional feature information of the current layer Transformer module, wherein the additional feature information includes at least one of the predicted part-of-speech category of the current subtext to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model; Based on the statistical dispersion and the additional feature information, a suppression head determination parameter is output through a preset decision model, wherein the suppression head determination parameter is the suppression head ratio or the number of suppression heads; The target suppression head is determined from the plurality of attention heads based on the suppression head determination parameters and the cumulative attention scores of each attention head.

5. The method according to claim 4, characterized in that, When the additional feature information includes the predicted part-of-speech category of the current sub-text to be generated, the prediction confidence of the current layer Transformer module, and the depth information of the current layer Transformer module in the preset visual language model, the suppression head decision parameters are output based on the statistical dispersion and the additional feature information through the preset decision model, including: The statistical dispersion, the predicted part-of-speech category, the predicted confidence, and the depth information are encoded to obtain multiple encoding vectors; The multiple encoded vectors are input into the feature fusion module of the preset decision model. The dynamic weights corresponding to each encoded vector are calculated through the attention network in the feature fusion module. Based on the multiple encoded vectors and their respective dynamic weights, a fused feature vector is generated. The fused feature vector is input into the calculation module of the preset decision model, and the calculation module outputs the initial decision parameters. The initial decision parameters are input into the verification module of the preset decision model. The verification module performs a feasibility verification on the initial decision parameters based on preset boundary constraints and inter-layer smoothing constraints. When the verification is successful, the initial decision parameters are used as the suppression head decision parameters. The inter-layer smoothing constraints are used to ensure that the rate of change of the suppression head decision parameters between adjacent Transformer modules does not exceed a preset threshold.

6. The method according to claim 5, characterized in that, The computation module includes a three-layer sub-network; the initial decision parameters output by the computation module include: The fused feature vector is input into the first sub-network of the computing module, and the basic suppression strength coefficient is output through the first sub-network. The fused feature vector and the basic suppression intensity coefficient are input into the second sub-network of the computing module, and the suppression intensity adjustment amount is output through the second sub-network. The basic inhibition strength coefficient and the inhibition strength adjustment amount are input into the third sub-network of the calculation module. The basic inhibition strength coefficient and the inhibition strength adjustment amount are added through the third sub-network to obtain the initial determination parameters.

7. The method according to claim 1, characterized in that, The step of encoding the image to be processed into multiple visual tokens and encoding the text instructions into an initial text token sequence includes: The image to be processed is segmented into multiple non-overlapping image blocks, and each image block is encoded to generate the multiple visual tokens; The text instruction is decomposed into multiple sub-word units by a preset text segmenter, and each sub-word unit is mapped to a word embedding vector. The word embedding vectors are sorted according to the order of each sub-word unit in the text instruction to obtain the initial text token sequence.

8. A descriptive text generation device based on an attention-head dynamic suppression visual language model, characterized in that, include: The data receiving module is used to receive the image to be processed and the corresponding text instructions, encode the image to be processed into multiple visual tokens, and encode the text instructions into an initial text token sequence; The descriptive text generation module is used to generate sub-texts one by one autoregressively based on the multiple visual tokens and the initial text token sequence through a preset visual language model, and generate descriptive texts for the image to be processed based on each sub-text; The preset visual language model includes a multi-layer Transformer module, with each layer connected sequentially, and each layer including multiple attention heads; the descriptive text generation module is further used for: For each attention head in the current layer Transformer module, calculate the cumulative attention score of the attention head for the plurality of visual tokens, and determine the target suppression head from the plurality of attention heads based on the cumulative attention score of each attention head; A dynamic mask is generated for the target suppression head, and the output of the target suppression head is suppressed by the dynamic mask; Based on the total output of all attention heads of the current layer Transformer module after suppression, the output feature representation of the current layer Transformer module is obtained, and the output feature representation is passed to the next layer Transformer module to determine the subtext based on the output feature representation of the last layer Transformer module.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images