Attention entropy-based multimodal large model question and answer method, device, equipment and medium

Abstract

This invention discloses a multimodal large-scale model question answering method, apparatus, device, and medium based on attention entropy, relating to the fields of artificial intelligence and machine learning. It includes: inputting a target image and question text into a multimodal large-scale model to obtain an image tag sequence and a text word sequence; connecting an r-th pruning layer before the r-th decoding layer of the decoder; calculating the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix for the r-th image tag sequence and the r-th text word sequence, and determining the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence; selecting the r-th retained image tag sequence based on the information density weights corresponding to each image tag in the r-th image tag sequence; and inputting the r-th retained image tag sequence and the r-th text word sequence into the r-th decoding layer to obtain the answer text output by the large language model, effectively balancing computational efficiency and semantic integrity without additional training.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning, and in particular to a multimodal large model question answering method, apparatus, device, and medium based on attention entropy. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technologies, multimodal large models have demonstrated powerful comprehensive reasoning capabilities in tasks such as image understanding, visual question answering, cross-modal retrieval, and video semantic analysis. Typical models such as BLIP-2 and LLaVA adopt a "visual encoder + cross-modal alignment + text encoder" structure, which achieves deep fusion of natural language and visual information by mapping visual features and text embeddings to a shared embedding space.

[0003] However, with the rapid growth in model size, these large multimodal models generally face problems such as high computational cost, high memory consumption, and significant latency during the inference phase. Especially in high-resolution image or long video input scenarios, the visual encoder generates thousands to tens of thousands of visual tags. These tags bring extremely high computational and memory burdens to the subsequent cross-modal attention and language decoding processes. This situation is particularly prominent in inference deployments on resource-constrained devices (such as edge devices and embedded platforms). Summary of the Invention

[0004] In view of the above-mentioned technical problems, the present invention provides a multimodal large model question answering method, apparatus, device and medium based on attention entropy, which aims to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of this invention provides a multimodal large-scale model question answering method based on attention entropy, wherein the multimodal large-scale model includes at least: a visual encoder, a text segmenter, and a large language model; the method includes:

[0006] The target image and the question text are input into the visual encoder and the text segmenter, respectively, to obtain the image tag sequence and the text word sequence.

[0007] In the large language model, an r-th pruning layer is connected before the r-th decoding layer of the decoder. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix are calculated. The r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence.

[0008] Based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, determine the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence;

[0009] Based on the information density weights corresponding to each image marker in the r-th image marker sequence, the r-th retained image marker sequence is selected from the r-th image marker sequence;

[0010] The r-th preserved image tag sequence and the r-th text lexical sequence are input into the r-th decoding layer to obtain the answer text output by the large language model.

[0011] A second aspect of the present invention provides a multimodal large-model question answering device based on attention entropy, wherein the multimodal large-model includes at least: a visual encoder, a text segmenter, and a large language model; the device includes:

[0012] The encoding module is used to input the target image and the question text into the visual encoder and the text segmenter respectively to obtain the image tag sequence and the text word sequence;

[0013] An attention calculation module is used to connect an r-th pruning layer before the r-th decoding layer of the decoder in the large language model. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, the module calculates the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix; the r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence.

[0014] The weight determination module is used to determine the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix.

[0015] The label filtering module is used to filter out the r-th retained image label sequence from the r-th image label sequence based on the information density weights corresponding to each image label in the r-th image label sequence;

[0016] The answer output module is used to input the r-th preserved image tag sequence and the r-th text lexical sequence into the r-th decoding layer to obtain the answer text output by the large language model.

[0017] A third aspect of the present invention provides an electronic device comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the attention entropy-based multimodal large model question answering method as described in the first aspect of the present invention.

[0018] A fourth aspect of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the attention entropy-based multimodal large model question answering method described in the first aspect of the present invention.

[0019] The multimodal large-scale language model question answering method proposed in this invention, based on attention entropy, connects an r-th pruning layer before the r-th decoding layer during the decoding process of the multimodal large-scale language model. Through the r-th pruning layer, bidirectional attention is calculated based on the r-th image tag sequence and the r-th text word sequence to obtain the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix. Then, based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence is determined to achieve dynamic evaluation of the importance of the image tag. Based on the information density weights corresponding to each image tag in the r-th image tag sequence, the r-th image tag sequence is filtered to obtain the r-th retained image tag sequence, so as to achieve adaptive pruning of image tags during the inference process, and thus obtain the answer text output by the multimodal large-scale language model. Thus, this invention achieves dynamic evaluation and adaptive pruning of the importance of image tags (i.e., visual tags) by jointly calculating the attention distribution in both the visual-to-text and text-to-visual directions during the inference phase. It constructs an index system that combines semantic consistency evaluation and information density measurement, avoiding the problem of insufficient semantic alignment caused by using a unidirectional attention mechanism. It effectively balances computational efficiency and semantic integrity without additional training, reduces the computational burden faced by multimodal large models during the inference phase, reduces memory consumption and latency, and realizes the inference optimization of multimodal large models and the inference deployment of multimodal large models on resource-constrained devices. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a multimodal large model question answering method based on attention entropy, as shown in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of a classic architecture for a multimodal large model proposed in related technologies;

[0023] Figure 3This is a framework diagram of a multimodal large model inference optimization algorithm based on attention entropy, as shown in an embodiment of the present invention.

[0024] Figure 4 This is a structural block diagram of a multimodal large model question answering device based on attention entropy provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Currently, to alleviate the problems of high computational cost, high memory consumption, and significant latency commonly faced by large multimodal models during the inference stage, various model compression and efficient inference techniques have been proposed, such as visual tag pruning and aggregation techniques. This is a plug-and-play inference acceleration method that optimizes computational efficiency by learning shallow attention patterns and pruning visual tags in deeper layers. However, this invention found that existing pruning methods are based solely on a unidirectional attention matrix from vision to text, ignoring the reverse semantic attention from text to vision, resulting in insufficient semantic alignment. Furthermore, the pruning ratio of this method is a fixed hyperparameter, lacking dynamic adaptive adjustment to input complexity and information density, making it prone to losing key information in complex image scenarios.

[0028] Specifically, this invention has carefully studied current pruning methods for large multimodal models and found at least the following technical defects and the reasons for these defects:

[0029] Current multimodal large-scale model inference optimization techniques mainly focus on model structure-level compression (such as parameter pruning and low-rank decomposition) or visual-side unidirectional attention pruning. Research on the combination of cross-modal consistency and input information entropy remains limited. First, unidirectional attention leads to insufficient semantic alignment: related inference acceleration algorithms are generally based on visual-to-text unidirectional attention matrices for visual tag selection. However, in multimodal interaction, the reverse attention of text terms to visual tags also carries important semantic information. Unidirectional attention pruning can cause decisions to favor the model's visual feature distribution while ignoring the contextual intent of text instructions. For example, when the text description contains multiple targets or implicit semantics (e.g., "Please find the object in the person's hand"), unidirectional attention cannot reflect the true semantic relevance of the targets. In addition, the model may discard semantically critical regions after pruning, leading to a decrease in cross-modal alignment capability. Therefore, unidirectional attention-driven pruning strategies have defects in semantic consistency and cannot guarantee that the compressed model can still accurately capture the semantic correspondence between images and text.

[0030] Second, the statistical information of attention distribution is not utilized, resulting in insufficient information density assessment. In large multimodal models, the attention matrix contains rich semantic and structural information. Related methods often directly use attention values ​​as importance indicators, but ignore the statistical characteristics of attention distribution—such as entropy, variance, or concentration. This approach leads to two problems: First, attention values ​​themselves are highly volatile, sensitive to the number of model layers, the length of the input sequence, and the task type, resulting in insufficient stability. Second, it fails to reflect the degree of information concentration of visual tags, because a single tag may distribute attention evenly across multiple text terms (high entropy), but with lower information value; conversely, low-entropy tags, while attracting attention, may be incorrectly pruned. Therefore, the lack of a global information density assessment mechanism based on attention distribution statistics is a significant weakness of related technologies.

[0031] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes a multimodal large model question answering method based on attention entropy. By jointly calculating the attention distribution in both the visual-to-text and text-to-visual directions during the inference phase of the multimodal large model, dynamic evaluation and adaptive pruning of the importance of image tags (i.e., visual tags) are achieved. An index system that combines semantic consistency evaluation and information density measurement is constructed, avoiding the problem of insufficient semantic alignment caused by using a unidirectional attention mechanism. Adaptive pruning without training is achieved during the inference phase, effectively balancing computational efficiency and model performance. This reduces the computational load faced by the multimodal large model during the inference phase, reduces memory consumption and latency, and realizes the inference optimization of the multimodal large model and the inference deployment of the multimodal large model on resource-constrained devices.

[0032] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal large model question answering method based on attention entropy, as shown in an embodiment of the present invention. Figure 1 As shown, the multimodal large model question answering method based on attention entropy provided in this embodiment includes at least the following steps:

[0033] Step S11: Input the target image and the question text into the visual encoder and the text segmenter respectively to obtain the image tag sequence and the text word sequence.

[0034] In this embodiment, the multimodal large model is a pre-trained multimodal large model, which includes at least a visual encoder, a text segmenter, and a large language model. During the inference phase of this multimodal large model, the target image and question text can be input into the model to obtain the answer text output by the model, which is derived from inference based on the target image and question text. Specifically, the target image and question text are input into the visual encoder and text segmenter of the multimodal large model, respectively, to obtain the image tag sequence output by the visual encoder and the text word sequence output by the text segmenter. The image tag sequence is a sequence composed of multiple image tags (i.e., visual tags) corresponding to the target image, and the text word sequence is a sequence composed of multiple text features (i.e., text words) corresponding to the question text.

[0035] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a classic architecture for a multimodal large model proposed in related technologies. Figure 2 In this multimodal large-scale model, a structure of visual encoder + cross-modal alignment + text encoder is adopted. This multimodal large-scale model includes a visual encoder, a cross-modal alignment module, a text encoder, and a large language model. The target image is encoded by the visual encoder and then cross-modal aligned to obtain an image tag sequence; the question text (e.g., user: What is the number on the player's jersey?) is encoded by a text segmenter to obtain a text word sequence. Then, the image tag sequence and the text word sequence are input into the large language model for processing.

[0036] Step S12: Connect an r-th pruning layer before the r-th decoding layer of the decoder in the large language model. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, calculate the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix.

[0037] In this embodiment, the decoder of the large language model includes multiple decoding layers. The r-th decoding layer is the r-th key decoding layer in the decoder, and the r-th pruning layer is the r-th pruning layer, which is also the pruning layer corresponding to the r-th decoding layer. The r-th pruning layer is connected before the r-th decoding layer. Here, the r-th decoding layer can be any one of the key decoding layers in the decoder. In this embodiment, the key decoding layer is the decoding layer that is previously connected to the pruning layer. In this embodiment, multiple key decoding layers can be selected from multiple decoding layers (they can be freely selected according to needs, such as selecting the 1st, 3rd, and 5th decoding layers as key decoding layers, etc., without any limitation), and the corresponding pruning layer is connected before each key decoding layer. r is an integer greater than 0.

[0038] In this embodiment, in the r-th pruning layer, bidirectional attention can be calculated based on the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer to obtain the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix.

[0039] For example, the r-th image label sequence is The r-th text word sequence is Where v is the image tag in the image tag sequence, and t is the text word in the text word sequence. It can be based on... and The calculated attention matrix from the r-th visual image to the text is: The attention matrix from text to vision for the r-th element. .in, Let i be the element in the i-th row and j-th column of the r-th visual-to-text attention matrix, representing the attention from the i-th image tag to the j-th text word, i∈[1,n], j∈[1,m]; Let be the element in the j-th row and i-th column of the r-th text-to-visual attention matrix, representing the attention from the j-th text term to the i-th image tag, i∈[1,n], j∈[1,m].

[0040] Here, the r-th image tag sequence and the r-th text word sequence are respectively the image tag sequence and text word sequence output by the previous decoding layer adjacent to the r-th decoding layer. It can be understood that the r-th image tag sequence and the r-th text word sequence are respectively the image tag sequence and text word sequence output by the previous decoding layer directly connected to the r-th pruning layer; when the previous decoding layer is a key decoding layer, the r-th image tag sequence and the r-th text word sequence are respectively the image tag sequence and text word sequence output by the (r-1)-th decoding layer. It can be understood that after the image tag sequence and the text word sequence are input into the decoder, they will be processed sequentially by multiple decoding layers (including the key decoding layer) in the decoder. Therefore, the r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence.

[0041] Step S13: Based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, determine the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence.

[0042] In this embodiment, based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, the importance of the i-th image marker in the r-th image marker sequence can be evaluated to determine the i-th information density weight corresponding to the i-th image marker in the r-th image marker sequence, thereby obtaining the information density weight corresponding to each image marker in the r-th image marker sequence. Here, the i-th information density weight is the information density weight corresponding to the i-th image marker in the r-th image marker sequence.

[0043] Step S14: Based on the information density weights corresponding to each image marker in the r-th image marker sequence, select the r-th retained image marker sequence from the r-th image marker sequence.

[0044] In this embodiment, the image markers in the r-th image marker sequence can be filtered based on the information density weights corresponding to each image marker in the r-th image marker sequence. The r-th retained image marker sequence can be selected from the r-th image marker sequence to achieve pruning of the r-th decoding layer, and the r-th retained image marker sequence and the r-th text word sequence output by the r-th pruning layer can be obtained.

[0045] In one optional example, the higher the information density weight of an image tag, the higher the information density of the image tag and its cross-modal consistency, and the more likely it should be retained. For example, in one specific embodiment, an information density threshold can be set for each image tag sequence, and image tags with information density weights exceeding the information density threshold can be retained.

[0046] Step S15: Input the r-th preserved image tag sequence and the r-th text lexical sequence into the r-th decoding layer to obtain the answer text output by the large language model.

[0047] In this embodiment, the r-th retained image tag sequence and the r-th text word sequence output from the r-th pruning layer can be input into the r-th decoding layer for decoding processing (e.g., the r-th retained image tag sequence and the r-th text word sequence can be merged and input into the r-th decoding layer for processing) to obtain the image tag sequence and text word sequence output from the r-th decoding layer, and then perform subsequent processing to obtain the answer text output by the large language model.

[0048] In this embodiment, for each key decoding layer in the decoder, the aforementioned steps S12 to S15 are performed for pruning to obtain the image tag sequence and text word sequence output by the key decoding layer. For example, if the decoding layer adjacent to the r-th decoding layer is not a key decoding layer, then the image tag sequence and text word sequence output by the r-th decoding layer are directly input into that decoding layer for decoding. If the decoding layer adjacent to the r-th decoding layer is a key decoding layer (i.e., the (r+1)-th decoding layer), then the image tag sequence and text word sequence output by the r-th decoding layer are used as the (r+1)-th image tag sequence and the (r+1)-th text word sequence, respectively. The (r+1)-th image tag sequence and the (r+1)-th text word sequence are input into the (r+1)-th pruning layer corresponding to the (r+1)-th decoding layer for pruning until the answer text output by the large language model is obtained. The answer text output by the large language model is the output of the multimodal large model.

[0049] In this embodiment, by jointly calculating the attention distribution in both the visual-to-text and text-to-visual directions during the inference phase, dynamic evaluation and adaptive pruning of the importance of image tags (i.e., visual tags) are achieved. This constructs an index system that combines semantic consistency evaluation and information density measurement, avoiding the semantic alignment problem caused by using a unidirectional attention mechanism. It effectively balances computational efficiency and semantic integrity without additional training, reducing the computational burden, memory consumption, and latency faced by large multimodal models during the inference phase. This enables inference optimization of large multimodal models and their deployment on resource-constrained devices. This embodiment uses bidirectional attention to characterize the bidirectional consistency of cross-modal semantic dependencies, achieving semantically complete tag importance evaluation. Compared to pruning schemes based solely on unidirectional attention matrices, this mechanism reflects the true semantic relevance of the target, more comprehensively reflects cross-modal semantic associations, improves the semantic consistency and robustness of the pruned model, and ensures that the compressed model can still accurately capture the semantic correspondence between images and text.

[0050] In conjunction with the above embodiments, in one implementation, the present invention also provides a multimodal large-scale question-answering method, in which step S13 may specifically include steps S21 to S23:

[0051] Step S21: Perform row normalization processing on each row of the r-th visual-to-text attention matrix to obtain the r-th visual-to-text row normalization matrix; perform column normalization processing on each column of the r-th text-to-visual attention matrix to obtain the r-th text-to-visual column normalization matrix.

[0052] In this embodiment, row normalization processing can be performed on each row of the r-th visual-to-text attention matrix to obtain the r-th visual-to-text row normalization matrix; and column normalization processing can be performed on each column of the r-th text-to-visual attention matrix to obtain the r-th text-to-visual column normalization matrix.

[0053] For example, the attention matrix for the r-th visual-to-text sequence. Perform row normalization to obtain the r-th visual-to-text row normalization matrix. Wherein, the r-th row normalization matrix from visual to text The element in the i-th row and j-th column is:

[0054] .

[0055] Attention matrix for r-th text-to-visual representation Column normalization is performed to obtain the r-th text-to-visual column normalization matrix. Wherein, the r-th text-to-visual column normalization matrix The element in the j-th row and i-th column is:

[0056] .

[0057] It is understandable that for the i-th image tag v in the r-th image tag sequence... i The image is labeled v i The distribution of the text is normalized to The text tags v to this image i The distribution is normalized to .

[0058] Step S22: Based on the i-th row of the r-th visual-to-text row normalization matrix, determine the first entropy of the i-th image tag in the r-th image tag sequence, and determine the maximum first entropy among all image tags in the r-th image tag sequence; based on the i-th column of the r-th text-to-visual column normalization matrix, determine the second entropy of the i-th image tag in the r-th image tag sequence, and determine the maximum second entropy among all image tags in the r-th image tag sequence.

[0059] In this embodiment, the row normalization matrix from the r-th visual image to the text is obtained. Then, the row normalization matrix of the r-th visual image to the text. Each row in the matrix corresponds to one image tag. Therefore, based on the i-th row of the row normalization matrix from the r-th visual to the text, the first entropy corresponding to the i-th image tag in the r-th image tag sequence can be determined, and thus the first entropy corresponding to each image tag in the r-th image tag sequence can be obtained. Here, the first entropy represents the attention entropy from the visual to the text.

[0060] After obtaining the first entropy corresponding to each image marker in the r-th image marker sequence, the largest first entropy can be selected from the first entropies corresponding to each image marker in the r-th image marker sequence to obtain the largest first entropy among all image markers in the r-th image marker sequence. .

[0061] In an optional example, the first entropy corresponding to the i-th image tag It can be calculated using the following formula: ;in, The r-th visual-to-text row normalization matrix The element in the i-th row and j-th column, m is The column number is the number of text words in the r-th text word sequence.

[0062] In this embodiment, the r-th text-to-visual column normalization matrix is ​​obtained. Then, the r-th text-to-visual column normalization matrix Each column in the matrix corresponds to one image tag. Therefore, based on the i-th column of the r-th text-to-visual column normalization matrix, the second entropy corresponding to the i-th image tag in the r-th image tag sequence can be determined, and thus the second entropy corresponding to each image tag in the r-th image tag sequence can be obtained. Here, the second entropy represents the attention entropy from text to vision.

[0063] After obtaining the second entropy corresponding to each image label in the r-th image label sequence, the largest second entropy can be selected from the second entropies corresponding to each image label in the r-th image label sequence to obtain the largest second entropy among all image labels in the r-th image label sequence. .

[0064] In an optional example, the second entropy corresponding to the i-th image tag It can be calculated using the following formula: .in, Let r be the column normalized matrix from text to vision. The element in the j-th row and i-th column, m is The number of rows is the number of text words in the r-th text word sequence.

[0065] Step S22: Obtain the i-th information density weight based on the maximum first entropy, the maximum second entropy, the first entropy and the second entropy of the i-th image tag in the r-th image tag sequence.

[0066] In this embodiment, the information density weight of the i-th image can be calculated based on the obtained maximum first entropy, maximum second entropy, first entropy of the i-th image tag in the r-th image tag sequence, and second entropy of the i-th image tag in the r-th image tag sequence.

[0067] In this embodiment, attention entropy is used as a quantitative indicator of image tag information density, which can reflect the degree of information concentration of visual tags. By jointly utilizing the attention distribution entropy from vision to text and from text to vision, an indicator system that combines semantic consistency assessment and information density measurement is constructed. This solves the problem of insufficient semantic alignment caused by the common use of unidirectional attention mechanism in related inference acceleration algorithms and the problem of inaccurate information density assessment caused by the failure to fully utilize the statistical information of attention distribution. It can effectively balance computational efficiency and semantic integrity without additional training.

[0068] In conjunction with any of the above embodiments, the present invention also provides a multimodal large model question answering method based on attention entropy. In this method, in addition to the steps described above, steps S31 and S32 may be included, and step S14 may specifically include the following step S33:

[0069] Step S31: Based on the maximum first entropy, the maximum second entropy, the first entropy and the second entropy of each image tag in the r-th image tag sequence, and the number of image tags in the r-th image tag sequence, obtain the mean value of the r-th attention entropy.

[0070] In this embodiment, the mean attention entropy of the r-th image can also be determined based on the obtained maximum first entropy, maximum second entropy, the first entropy of each image tag in the r-th image tag sequence, the second entropy of each image tag in the r-th image tag sequence, and the number of image tags in the r-th image tag sequence. The mean attention entropy of the r-th image can be the mean overall attention entropy corresponding to the r-th pruning layer.

[0071] Step S32: Determine the retention rate of the r-th attention based on the mean value of the r-th attention entropy.

[0072] In this embodiment, after obtaining the mean value of the r-th attention entropy, the r-th retention rate can be determined based on the mean value of the r-th attention entropy. The r-th retention rate characterizes the proportion of image tags retained in the r-th pruning layer; that is, the ratio of the number of image tags in the r-th retained image tag sequence to the number of image tags in the r-th image tag sequence is the r-th retention rate.

[0073] In this embodiment, a higher mean value of the r-th attention entropy indicates that the overall information is more dispersed, requiring more information to be retained, resulting in a higher r-th retention rate. Conversely, a lower mean value of the r-th attention entropy indicates that the overall information is more concentrated, allowing for the compression of more information, resulting in a lower r-th retention rate. In other words, the r-th retention rate in this embodiment is directly proportional to the mean value of the r-th attention entropy.

[0074] In an optional example, the retention rate of the r-th digit is... ;in, The preset base retention rate, ; Let be the mean of the attention entropy at the r-th digit.

[0075] Step S33: Based on the information density weights corresponding to each image marker in the r-th image marker sequence, according to the r-th retention rate, select the r-th retained image marker sequence from the r-th image marker sequence.

[0076] In this embodiment, after obtaining the r-th retention rate, the r-th retained image label sequence can be obtained by filtering from the r-th image label sequence according to the information density weights corresponding to each image label in the r-th image label sequence and the r-th retention rate.

[0077] In an optional example, the image tags in the r-th image tag sequence can be sorted in descending order of information density weight. Based on the r-th retention rate and the number of image tags in the r-th image tag sequence, the number a of image tags in the r-th retained image tag sequence is determined. The first a of the r-th image tag sequence are retained to obtain the r-th retained image tag sequence.

[0078] It is understood that in this embodiment, the retention rate of each key decoding layer is related to the average attention entropy of that key decoding layer. The retention rates of each key decoding layer can be the same or different, thereby dynamically adjusting the pruning intensity of each key decoding layer, achieving adaptive fine compression of the input, and improving compression efficiency and robustness. Furthermore, this embodiment can implement different dynamic pruning for different target images, achieving adaptive fine compression of the target image.

[0079] In this embodiment, an adaptive pruning strategy based on entropy distribution is proposed: the pruning threshold and / or the r-th retention rate are dynamically determined based on the overall attention entropy mean of the r-th decoding layer (i.e., the entropy distribution characteristics of the input samples), realizing an adaptive inference optimization mechanism for input complexity, semantic distribution, and task requirements. Thus, this embodiment quantitatively evaluates the information density and redundancy of each visual tag by calculating the entropy of the attention distribution, and dynamically adjusts the pruning intensity (adaptive retention rate) accordingly. This mechanism can adaptively retain or compress tags based on the true information complexity of a single image and text pair. Therefore, more aggressive compression can be achieved in simple scenarios to save computational resources, while pruning is automatically relaxed in complex scenarios to maintain performance, significantly improving overall compression efficiency and robustness, and avoiding performance fluctuations caused by a fixed pruning ratio.

[0080] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal large model question answering method based on attention entropy. In this method, step S22 may specifically include steps S41 to S42, and step S31 may specifically include steps S43 to S44:

[0081] Step S41: Based on the maximum first entropy and the first entropy of the i-th image tag in the r-th image tag sequence, obtain the visual-to-text entropy factor of the i-th image tag; based on the maximum second entropy and the second entropy of the i-th image tag in the r-th image tag sequence, obtain the text-to-visual entropy factor of the i-th image tag.

[0082] In this embodiment, the visual-to-text entropy factor corresponding to the i-th image tag can be obtained based on the maximum first entropy and the first entropy of the i-th image tag in the r-th image tag sequence; and the text-to-visual entropy factor corresponding to the i-th image tag can be obtained based on the maximum second entropy and the second entropy of the i-th image tag in the r-th image tag sequence.

[0083] Step S42: Based on a preset balance factor, fuse the text-to-visual entropy factor and the visual-to-text entropy factor of the i-th image tag to obtain the i-th information density weight.

[0084] In this embodiment, a preset balance factor (which can be freely set) is pre-set to balance the text-to-visual entropy factor and the visual-to-text entropy factor. Based on the preset balance factor, this embodiment can fuse the text-to-visual entropy factor and the visual-to-text entropy factor corresponding to the i-th image tag to obtain the i-th information density weight.

[0085] In one optional example, based on a preset balance factor, the text-to-visual entropy factor corresponding to the i-th image tag and the visual-to-text entropy factor corresponding to the i-th image tag are fused to obtain the bidirectional attention entropy (BAE) of the i-th image tag, and then the information density weight of the i-th image tag is obtained based on the bidirectional attention entropy of the i-th image tag.

[0086] In an optional specific example, the bidirectional attention entropy of the i-th image tag can be obtained by the following formula. :

[0087] ;

[0088] And, the information density weight of the i-th information density is obtained by the following formula. ;

[0089] in, To preset the balance factor, Let be the visual-to-text entropy factor for the i-th image label. The text-to-visual entropy factor for the i-th image label.

[0090] Step S43: The sum of the text-to-visual entropy factor and the visual-to-text entropy factor of the i-th image tag is determined as the bidirectional attention entropy factor of the i-th image tag.

[0091] In this embodiment, the sum of the text-to-visual entropy factor corresponding to the i-th image tag and the visual-to-text entropy factor corresponding to the i-th image tag can be determined as the bidirectional attention entropy factor of the i-th image tag, thereby obtaining the bidirectional attention entropy factors of each image tag in the r-th image tag sequence.

[0092] Step S44: Based on the bidirectional attention entropy factor of each image tag in the r-th image tag sequence and the number of image tags in the r-th image tag sequence, obtain the mean value of the r-th attention entropy.

[0093] In this embodiment, the mean attention entropy of the r-th image tag sequence can be determined based on the bidirectional attention entropy factor of each image tag in the r-th image tag sequence and the number of image tags in the r-th image tag sequence.

[0094] In an optional specific example, the mean attention entropy of the r-th digit can be determined according to the following formula. :

[0095] ;

[0096] Where n is the number of image tags in the r-th image tag sequence. Let i be the visual-to-text entropy factor corresponding to the i-th image label. Let be the text-to-visual entropy factor corresponding to the i-th image label.

[0097] This embodiment simultaneously calculates entropy values ​​using attention distributions in both the visual-to-text and text-to-visual directions, and integrates these two values ​​into a comprehensive index for tag importance assessment. This bidirectional entropy index can characterize the mutual attention relationship and semantic concentration of visual tags between the two modalities, fundamentally avoiding semantic bias caused by relying solely on unidirectional attention. Since pruning decisions are based on bidirectional semantic consistency, visual regions highly relevant to the semantics of text instructions (such as implicit targets or directional regions) can be more reliably preserved, thereby significantly reducing the risk of semantic information loss and task performance degradation due to pruning, improving cross-modal semantic consistency, and reducing mispruning and information loss.

[0098] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal large model question answering method based on attention entropy. In this method, in addition to the steps described above, steps S51 to S52 may be included, and step S33 may specifically include steps S53 to S55:

[0099] Step S51: Restore the r-th image tag sequence to the two-dimensional grid corresponding to the target image.

[0100] In this embodiment, the r-th image marker sequence can be restored to the two-dimensional grid corresponding to the target image. The i-th image marker in the r-th image marker sequence is restored to the i-th grid in the two-dimensional grid to describe the target image content in the i-th grid.

[0101] Step S52: Divide the two-dimensional grid into multiple grid groups.

[0102] In this embodiment, after obtaining the two-dimensional grid, the two-dimensional grid is divided into multiple grid groups of size k. Each grid group includes k grids, corresponding to k image markers, where k is a preset spatial grouping size that can be freely set.

[0103] Step S53: Determine the number of retained image tags λk in each grid group based on the r-th retention rate λ and k.

[0104] In this embodiment, after obtaining the r-th retention rate λ and the number of grids k in each grid group, the number of retained image tags λk in each grid group can be determined based on the r-th retention rate λ and k.

[0105] Step S54: For each grid group, sort the image tags within that grid group in descending order of information density weight.

[0106] In this embodiment, after dividing into multiple grid groups, each image marker in each grid group can be sorted in descending order of its information density weight.

[0107] Step S55: For each grid group, retain the first λk image labels within that grid group.

[0108] In this embodiment, for each grid group, the first λk image tags within that grid group are retained according to their sorting order. These first λk image tags are the retained image tags corresponding to that grid group. In other words, this embodiment retains the first λk image tags within each grid group in descending order of their information density weights, thus obtaining the first λk image tags corresponding to each grid group. In this embodiment, the retained image tags corresponding to multiple grid groups form the r-th retained image tag sequence.

[0109] For example, the r-th image tag sequence has 16 image tags, and k is 4. These 16 image tags are reconstructed into a 4x4 two-dimensional grid. When dividing the two-dimensional grid, it can be divided into four grid groups: grid group 1 (top right, 4 grids, corresponding to the top right tags); grid group 2 (top left, 4 grids, corresponding to the top left tags); grid group 3 (bottom right, 4 grids, corresponding to the bottom right tags); and grid group 4 (bottom left, 4 grids, corresponding to the bottom left tags). For another example, if λ is 0.5, then the number of retained image tags in each grid group is λk = 2. Therefore, for each grid group, the first two image tags in that grid group are retained in descending order of their information density weights, resulting in the r-th retained image tag sequence consisting of 8 retained image tags.

[0110] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal large model question answering method based on attention entropy. In this method, in addition to the steps described above, step S61 may also be included, and step S54 may specifically include step S62:

[0111] Step S61: For the i-th image tag in the r-th image tag sequence, take the other image tags in the grid group corresponding to the i-th image tag as the neighborhood, and perform neighborhood smoothing based on the i-th information density weight and the information density weight of each image tag in the grid group corresponding to the i-th image tag to obtain the smoothed information density weight of the i-th image tag.

[0112] In this embodiment, for the i-th image tag in the r-th image tag sequence, the other image tags in the grid group corresponding to the i-th image tag (other image tags are image tags other than the i-th image tag in the grid group corresponding to the i-th image tag) can be used as the neighborhood. Then, based on the i-th information density weight and the information density weight of each image tag in the grid group corresponding to the i-th image tag, neighborhood smoothing is performed to reduce single-point noise and obtain the smoothed information density weight of the i-th image tag.

[0113] In a specific optional example, the smoothed information density weight of the i-th image label can be determined by the following formula. :

[0114] ;

[0115] in, For the information density weight of the i-th element, This represents the number of image tags within the grid group corresponding to the i-th image tag.

[0116] Step S62: For each grid group, sort the image markers within that grid group according to the smoothed information density weights from largest to smallest.

[0117] In this embodiment, after dividing into multiple grid groups, each image marker in the grid group can be sorted in descending order of the smoothed information density weight of each image marker in the grid group.

[0118] In this embodiment, spatial awareness strategies such as two-dimensional grid grouping and neighborhood smoothing are employed during the image labeling process to ensure that the retained image labels are evenly distributed in space and maintain the integrity of key regions. This design reduces the loss of positional information caused by simply serializing two-dimensional visual features, improves the model's ability to express object boundaries, local context, and spatial relationships, maintains spatial structure and local context information, and reduces structural damage caused by pruning.

[0119] In one implementation, in conjunction with any of the above embodiments, the step may further include: determining multiple key decoding layers in the decoder based on the image scene corresponding to the target image. Specifically, when the image scene is simple, more aggressive compression can be implemented to save computational resources, such as determining each decoding layer in the decoder of the large language model as a key decoding layer; while when the image scene is complex, pruning can be automatically relaxed to maintain performance, such as determining one or more decoding layers in the decoder of the large language model as key decoding layers. Here, a simple scene refers to a target image with relatively little information, and simple content such as objects, attributes, and background, such as an image of a puppy; a complex scene refers to a target image containing rich content, such as an image with foreground objects including a child, a puppy, and a ball, where each visual element is important for the final inference of the multimodal large model. For example, the application scene corresponding to the target image can be determined by the number of objects in the target image, with a preset object threshold. If the number of objects in the target image exceeds the object threshold, the application scene corresponding to the target image is determined to be a complex scene; if the number of objects in the target image does not exceed the object threshold, the application scene corresponding to the target image is determined to be a simple scene. This embodiment can significantly improve overall compression efficiency and robustness, and avoid performance fluctuations caused by a fixed pruning ratio.

[0120] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal large model question answering method based on attention entropy. In this method, r is an integer greater than 1 and less than or equal to R, and R is an integer greater than 1, without limitation; in addition to the above steps, steps S71 to S76 may also be included:

[0121] Step S71: Connect a first pruning layer before the first decoding layer of the decoder in the large language model, and input the image tag sequence and the text lexical sequence into the first pruning layer.

[0122] In this embodiment, each decoding layer in the decoder of the large language model is a key decoding layer. A first pruning layer is connected before the first decoding layer (i.e., the first key decoding layer, also the first decoding layer) of the decoder in the large language model, and the image tag sequence and text lexical sequence are input to the first pruning layer.

[0123] Step S72: Select the first retained image tag sequence from the image tag sequence through the first pruning layer.

[0124] In this embodiment, a first retained image tag sequence can be selected from the image tag sequence through the first pruning layer. Specifically, the method by which the first retained image tag sequence is selected through the first pruning layer is the same as or similar to the method by which the r retained image tag sequence is selected through the r-th pruning layer in the aforementioned embodiment, and can be referred to steps S12 to S14 above.

[0125] Step S73: Input the first preserved image tag sequence and the text word sequence into the first decoding layer to obtain the second image tag sequence and the second text word sequence output by the first decoding layer.

[0126] In this embodiment, the first pruning layer can input the selected first retained image tag sequence and text word sequence into the first decoding layer for decoding processing to obtain the image tag sequence and text word sequence output by the first decoding layer, which are used as the second image tag sequence and the second text word sequence, respectively.

[0127] Step S74: Connect the Rth pruning layer before the Rth decoding layer of the decoder in the large language model.

[0128] In this embodiment, an R-th pruning layer is connected before the R-th decoding layer (i.e., the last key decoding layer, also known as the last decoding layer) in the large language model, and the R-th image tag sequence and the R-th text word sequence output by the R-1 decoding layer are input to the R-th pruning layer.

[0129] Step S75: Select the R-th retained image tag sequence from the R-th image tag sequence output by the R-1 decoding layer through the R-th pruning layer.

[0130] In this embodiment, the R-th retained image tag sequence can be selected from the R-th image tag sequence through the R-th pruning layer. Specifically, the method of selecting the R-th retained image tag sequence through the R-th pruning layer is the same as or similar to the method of selecting the r-th retained image tag sequence through the r-th pruning layer in the previous embodiment, and can be referred to steps S12 to S14 above.

[0131] Step S76: Input the R-th preserved image tag sequence and the R-th text word sequence output by the R-1-th decoding layer into the R-th decoding layer to obtain the answer text.

[0132] In this embodiment, the Rth pruning layer can input the Rth retained image tag sequence and the Rth text word sequence output by the R-1th decoding layer into the Rth decoding layer for decoding processing to obtain the answer text, thereby realizing multimodal reasoning of the multimodal large model.

[0133] In one embodiment, such as Figure 3 As shown, Figure 3This is a framework diagram of a multimodal large model inference optimization algorithm based on attention entropy, as shown in an embodiment of the present invention. Figure 3 In the large language model decoder, there are at least large language model decoding layers l1, l2, ..., ln(l1, l2, l3, l4, l5, l6, l7, l8, l9, l1, l10, l11, l12, l13, l14, l15, l16, l17, l18, l19, l10 ... s Among them, the large language model decoding layer l2 is the key decoding layer, and a pruning layer (i.e., ...) is connected before this large language model decoding layer l2. Figure 3 In the process of pruning, the multimodal large model outputs image tag sequences and text word sequences based on the target image and question text. These sequences are then input into the large language model's decoding layer l1 for decoding, resulting in the l2nd image tag sequence and l2nd text word sequence output by the decoding layer l1. The l2nd image tag sequence and l2nd text word sequence are then input into the pruning layer before the large language model's decoding layer l2 to calculate bidirectional attention entropy, perform spatial awareness and smoothing equalization, and calculate the entropy-driven adaptive retention ratio.

[0134] In this pruning layer, bidirectional attention entropy is first calculated to obtain information density weights. The bidirectional attention entropy is used to characterize the bidirectional consistency of cross-modal semantic dependencies, thereby achieving semantically complete label importance assessment. Compared with pruning schemes that are based solely on unidirectional attention matrices, this mechanism can more comprehensively reflect cross-modal semantic associations and improve the semantic consistency and robustness of the pruned model.

[0135] Then, spatial perception and smoothing are performed to obtain the smoothed information density weights of each image label in each grid group: a two-dimensional spatial neighborhood constraint is introduced during the pruning process to equalize the spatial distribution of image labels; grid aggregation or neighborhood smoothing is used to prevent spatial structure destruction and maintain the integrity of object boundaries and regions; this method can ensure that the pruned model has strong spatial perception and localization capabilities.

[0136] Finally, adaptive retention is performed to filter image labels: attention entropy is used as a quantitative indicator of visual label information density. The pruning threshold and retention ratio (retention rate) are dynamically determined according to the entropy distribution characteristics of the input samples. Image labels are retained based on the retention rate and the smoothed information density weights, realizing an adaptive inference optimization mechanism for input complexity, semantic distribution and task requirements.

[0137] After the l2-th preserved image tag sequence is output from the pruning layer, the l2-th preserved image tag sequence and the l2-th text word sequence are input into the large language model decoding layer l2 for decoding processing, until the large language model decoding layer l2 is reached. s .

[0138] This embodiment introduces bidirectional attention entropy and entropy-based adaptive pruning and compression strategies during the inference stage of a multimodal large model, achieving a comprehensive improvement in the inference efficiency and semantic fidelity of the multimodal large model without additional training. In other words, this invention proposes an attention entropy-based multimodal large model inference optimization algorithm, particularly a method for achieving collaborative compression and efficient inference of visual and textual modalities by calculating bidirectional attention entropy without additional training.

[0139] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0140] Based on the same inventive concept, one embodiment of the present invention provides a multimodal large-model question answering device based on attention entropy, wherein the multimodal large-model includes at least: a visual encoder, a text segmenter, and a large language model. (Reference) Figure 4 , Figure 4 This is a structural block diagram of a multimodal large model question-answering device based on attention entropy provided in an embodiment of the present invention. Figure 4 As shown, the device includes:

[0141] The encoding module is used to input the target image and the question text into the visual encoder and the text segmenter respectively to obtain the image tag sequence and the text word sequence;

[0142] An attention calculation module is used to connect an r-th pruning layer before the r-th decoding layer of the decoder in the large language model. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, the module calculates the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix; the r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence.

[0143] The weight determination module is used to determine the i-th information density weight corresponding to the i-th image tag in the r-th image tag sequence based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix.

[0144] The label filtering module is used to filter out the r-th retained image label sequence from the r-th image label sequence based on the information density weights corresponding to each image label in the r-th image label sequence;

[0145] The answer output module is used to input the r-th preserved image tag sequence and the r-th text lexical sequence into the r-th decoding layer to obtain the answer text output by the large language model.

[0146] Optionally, the weight determination module includes:

[0147] The normalization processing module is used to perform row normalization processing on each row of the r-th visual-to-text attention matrix to obtain the r-th visual-to-text row normalization matrix; and to perform column normalization processing on each column of the r-th text-to-visual attention matrix to obtain the r-th text-to-visual column normalization matrix.

[0148] The entropy determination module is used to determine the first entropy of the i-th image tag in the r-th image tag sequence based on the i-th row of the r-th visual-to-text row normalization matrix, and to determine the maximum first entropy among all image tags in the r-th image tag sequence; and to determine the second entropy of the i-th image tag in the r-th image tag sequence based on the i-th column of the r-th text-to-visual column normalization matrix, and to determine the maximum second entropy among all image tags in the r-th image tag sequence.

[0149] The density weight determination module is used to obtain the i-th information density weight based on the maximum first entropy, the maximum second entropy, and the first and second entropies of the i-th image tag in the r-th image tag sequence.

[0150] Optionally, the device further includes:

[0151] The entropy mean determination module is used to obtain the mean attention entropy of the r-th image based on the maximum first entropy, the maximum second entropy, the first entropy and the second entropy of each image label in the r-th image label sequence, and the number of image labels in the r-th image label sequence;

[0152] The retention rate determination module is used to determine the r-th retention rate based on the mean value of the r-th attention entropy.

[0153] The tagging and filtering module includes:

[0154] The first filtering submodule is used to filter out the r-th retained image tag sequence from the r-th image tag sequence according to the r-th retention rate and based on the information density weight corresponding to each image tag in the r-th image tag sequence; the ratio of the number of image tags in the r-th retained image tag sequence to the number of image tags in the r-th image tag sequence is the r-th retention rate.

[0155] Optionally, the density weight determination module includes:

[0156] The entropy factor determination module is used to obtain the visual-to-text entropy factor of the i-th image tag based on the maximum first entropy and the first entropy of the i-th image tag in the r-th image tag sequence; and to obtain the text-to-visual entropy factor of the i-th image tag based on the maximum second entropy and the second entropy of the i-th image tag in the r-th image tag sequence.

[0157] The first entropy factor fusion module is used to fuse the text-to-visual entropy factor and the visual-to-text entropy factor of the i-th image tag based on a preset balance factor to obtain the i-th information density weight.

[0158] The entropy mean determination module includes:

[0159] The second entropy factor fusion module is used to determine the sum of the text-to-visual entropy factor and the visual-to-text entropy factor of the i-th image tag as the bidirectional attention entropy factor of the i-th image tag.

[0160] The attention entropy mean determination module is used to obtain the r-th attention entropy mean based on the bidirectional attention entropy factor of each image tag in the r-th image tag sequence and the number of image tags in the r-th image tag sequence.

[0161] Optionally, the device further includes:

[0162] The grid restoration module is used to restore the r-th image marker sequence to the two-dimensional grid corresponding to the target image, wherein the i-th image marker in the r-th image marker sequence is restored to the i-th grid in the two-dimensional grid to describe the target image content within the i-th grid;

[0163] The grid division module is used to divide the two-dimensional grid into multiple grid groups, each grid group including k grids to correspond to k image labels;

[0164] The first filtering submodule includes:

[0165] The retention quantity determination module is used to determine the number of retained image tags λk in each grid group based on the r-th retention rate λ and k;

[0166] The first sorting module is used to sort the image tags in each grid group in descending order of information density weight for each grid group.

[0167] The in-group retention module is used to retain the first λk image labels within each grid group;

[0168] The retained image tags corresponding to the multiple grid groups constitute the r-th retained image tag sequence.

[0169] Optionally, the device further includes:

[0170] The smoothing module is used to perform neighborhood smoothing on the i-th image label in the r-th image label sequence, taking other image labels in the grid group corresponding to the i-th image label as the neighborhood, and based on the i-th information density weight and the information density weight of each image label in the grid group corresponding to the i-th image label, to obtain the smoothed information density weight of the i-th image label.

[0171] The first sorting module includes:

[0172] The second sorting module is used to sort the image markers in each grid group according to the smoothed information density weights from largest to smallest.

[0173] Optionally, r is an integer greater than 1 and less than or equal to R; the device further includes:

[0174] The first input module is used to connect a first pruning layer before the first decoding layer of the decoder in the large language model, and input the image tag sequence and the text word sequence into the first pruning layer;

[0175] The first filtering module is used to filter out a first retained image tag sequence from the image tag sequence through the first pruning layer;

[0176] The second input module is used to input the first preserved image tag sequence and the text word sequence into the first decoding layer to obtain the second image tag sequence and the second text word sequence output by the first decoding layer;

[0177] The second filtering module is used to connect the Rth pruning layer before the Rth decoding layer of the decoder in the large language model, and to filter out the Rth retained image tag sequence from the Rth image tag sequence output by the R-1th decoding layer through the Rth pruning layer;

[0178] The third input module is used to input the R-th preserved image tag sequence and the R-th text word sequence output by the R-1 decoding layer into the R-th decoding layer to obtain the answer text.

[0179] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0180] The attention entropy-based multimodal large model question-answering device in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0181] The multimodal large model question answering device based on attention entropy in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0182] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the attention entropy-based multimodal large model question answering method described in any of the above embodiments of the present invention.

[0183] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0184] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the attention entropy-based multimodal large model question answering method described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0185] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0188] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A multimodal large-scale model question answering method based on attention entropy, characterized in that, The multimodal large model includes at least: a visual encoder, a text segmenter, and a large language model; the method includes: The target image and the question text are input into the visual encoder and the text segmenter, respectively, to obtain the image tag sequence and the text word sequence. In the large language model, an r-th pruning layer is connected before the r-th decoding layer of the decoder. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix are calculated. The r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence. Based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, the visual-to-text entropy factor of the i-th image tag is obtained. And, the text-to-visual entropy factor of the i-th image tag. ; Determine the information density weight of the i-th image label in the r-th image label sequence to jointly calculate the attention distribution in both the visual-to-text and text-to-visual directions, and dynamically evaluate the importance of the image labels; including: using the formula The bidirectional attention entropy of the i-th image label is obtained. ; through formula We obtain the information density weight of the i-th information; where, The preset balance factor; according to the formula Determine the mean of the attention entropy at the r-th digit. Where n is the number of image tags in the r-th image tag sequence; Based on the mean of the attention entropy at the rth digit According to the formula Determine the retention rate at the r-th digit. ;in, The preset base retention rate, ; According to the aforementioned retention rate r Based on the information density weights corresponding to each image tag in the r-th image tag sequence, an r-th retained image tag sequence is selected from the r-th image tag sequence to jointly calculate the attention distribution in both the visual-to-text and text-to-visual directions, and to perform adaptive pruning on the image tags; the ratio of the number of image tags in the r-th retained image tag sequence to the number of image tags in the r-th image tag sequence is the r-th retention rate; The r-th preserved image tag sequence and the r-th text lexical sequence are input into the r-th decoding layer to obtain the answer text output by the large language model; Specifically, based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix, the visual-to-text entropy factor of the i-th image tag is obtained. And, the text-to-visual entropy factor of the i-th image tag. ,include: For each row of the r-th visual-to-text attention matrix, row normalization is performed to obtain the r-th visual-to-text row normalization matrix; for each column of the r-th text-to-visual attention matrix, column normalization is performed to obtain the r-th text-to-visual column normalization matrix. Based on the i-th row of the r-th visual-to-text row normalization matrix, determine the first entropy of the i-th image tag in the r-th image tag sequence. And determine the maximum first entropy among all image tags in the r-th image tag sequence. ; Based on the i-th column of the r-th text-to-visual column normalization matrix, determine the second entropy of the i-th image tag in the r-th image tag sequence. And determine the maximum second entropy among all image tags in the r-th image tag sequence. ; According to the maximum first entropy and the first entropy of the i-th image tag in the r-th image tag sequence The visual-to-text entropy factor of the i-th image tag is obtained. ; According to the maximum second entropy and the second entropy of the i-th image tag in the r-th image tag sequence The text-to-visual entropy factor of the i-th image tag is obtained. .

2. The multimodal large model question answering method based on attention entropy according to claim 1, characterized in that, The method further includes: The r-th image tag sequence is restored to the two-dimensional grid corresponding to the target image, and the i-th image tag in the r-th image tag sequence is restored to the i-th grid in the two-dimensional grid to describe the target image content in the i-th grid; The two-dimensional grid is divided into multiple grid groups, each grid group including k grids to correspond to k image labels; Based on the aforementioned retention rate and the information density weights corresponding to each image marker in the r-th image marker sequence, the r-th retained image marker sequence is selected from the r-th image marker sequence, including: Based on the r-th retention rate λ and k, determine the number λk of retained image tags within each grid group; For each grid group, the image tags within that grid group are sorted in descending order of their information density weights; For each grid group, retain the first λk image labels within that grid group; The retained image tags corresponding to the multiple grid groups constitute the r-th retained image tag sequence.

3. The multimodal large model question answering method based on attention entropy according to claim 2, characterized in that, The method further includes: For the i-th image tag in the r-th image tag sequence, other image tags in the grid group corresponding to the i-th image tag are used as the neighborhood. Based on the i-th information density weight and the information density weights of each image tag in the grid group corresponding to the i-th image tag, neighborhood smoothing is performed to obtain the smoothed information density weight of the i-th image tag. For each grid group, the image labels within that grid group are sorted in descending order of their information density weights, including: For each grid group, the image markers within that grid group are sorted in descending order of the smoothed information density weights.

4. The multimodal large model question answering method based on attention entropy according to any one of claims 1 to 3, characterized in that, r is an integer greater than 1 and less than or equal to R; the method further includes: A first pruning layer is connected before the first decoding layer of the decoder in the large language model, and the image tag sequence and the text lexical sequence are input into the first pruning layer; The first pruning layer is used to filter out the first retained image tag sequence from the image tag sequence; The first preserved image tag sequence and the text word sequence are input into the first decoding layer to obtain the second image tag sequence and the second text word sequence output by the first decoding layer; In the large language model, the Rth pruning layer is connected before the Rth decoding layer of the decoder; The R-th retained image tag sequence is selected from the R-th image tag sequence output by the R-1 decoding layer through the R-th pruning layer; The R-th preserved image tag sequence and the R-th text word sequence output by the R-1-th decoding layer are input into the R-th decoding layer to obtain the answer text.

5. A multimodal large-model question-answering device based on attention entropy, characterized in that, The multimodal large model includes at least: a visual encoder, a text segmenter, and a large language model; the device includes: The encoding module is used to input the target image and the question text into the visual encoder and the text segmenter respectively to obtain the image tag sequence and the text word sequence; An attention calculation module is used to connect an r-th pruning layer before the r-th decoding layer of the decoder in the large language model. For the r-th image tag sequence and the r-th text word sequence input to the r-th pruning layer, the module calculates the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix; the r-th image tag sequence and the r-th text word sequence are obtained based on the image tag sequence and the text word sequence. The weight determination module is used to obtain the visual-to-text entropy factor of the i-th image tag based on the r-th visual-to-text attention matrix and the r-th text-to-visual attention matrix. And, the text-to-visual entropy factor of the i-th image tag. ; Determine the information density weight of the i-th image tag in the r-th image tag sequence to jointly calculate the attention distribution in both the visual-to-text and text-to-visual directions, and dynamically evaluate the importance of the image tags; using the formula The bidirectional attention entropy of the i-th image label is obtained. ; through formula We obtain the information density weight of the i-th information; where, The preset balance factor; according to the formula Determine the mean of the attention entropy at the r-th digit. Where n is the number of image tags in the r-th image tag sequence; based on the mean attention entropy of the r-th sequence... According to the formula Determine the retention rate at the r-th digit. ;in, The preset base retention rate, ; The tagging and filtering module is used to select the r-th retention rate. Based on the information density weights corresponding to each image tag in the r-th image tag sequence, an r-th retained image tag sequence is selected from the r-th image tag sequence to jointly calculate the attention distribution in both the visual-to-text and text-to-visual directions, and to perform adaptive pruning on the image tags; the ratio of the number of image tags in the r-th retained image tag sequence to the number of image tags in the r-th image tag sequence is the r-th retention rate; The answer output module is used to input the r-th preserved image tag sequence and the r-th text lexical sequence into the r-th decoding layer to obtain the answer text output by the large language model; The weight determination module is further configured to perform row normalization processing on each row of the r-th visual-to-text attention matrix to obtain the r-th visual-to-text row normalization matrix; perform column normalization processing on each column of the r-th text-to-visual attention matrix to obtain the r-th text-to-visual column normalization matrix; and determine the first entropy of the i-th image tag in the r-th image tag sequence based on the i-th row of the r-th visual-to-text row normalization matrix. And determine the maximum first entropy among all image tags in the r-th image tag sequence. Based on the i-th column of the r-th text-to-visual column normalization matrix, determine the second entropy of the i-th image tag in the r-th image tag sequence. And determine the maximum second entropy among all image tags in the r-th image tag sequence. According to the maximum first entropy and the first entropy of the i-th image tag in the r-th image tag sequence The visual-to-text entropy factor of the i-th image tag is obtained. According to the maximum second entropy and the second entropy of the i-th image tag in the r-th image tag sequence The text-to-visual entropy factor of the i-th image tag is obtained. .

6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the attention entropy-based multimodal large model question answering method as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the attention entropy-based multimodal large model question answering method as described in any one of claims 1 to 4.

Images