Multi-modal large language model visual lexical element efficient pruning system, method and equipment

Abstract

The invention discloses an efficient pruning system, method and equipment for multi-modal large language model visual lexical elements, the system comprises R visual lexical element pruning units, an image encoder, a text encoder and a decoding unit, R is greater than or equal to 2, the R visual lexical element pruning units are sequentially connected and are respectively called as a first-level visual lexical element pruning unit to an Rth-level visual lexical element pruning unit, the image encoder and the text encoder are connected with the first-level visual lexical element pruning unit in parallel, and the Rth-level visual lexical element pruning unit is connected with the decoding unit. The method comprises the following steps: firstly, performing multi-stage screening of spatial perception on visual lexical elements to reserve core visual information; then, similarity evaluation is carried out on the preliminarily pruned lexical units, and lexical units with supplementary information are identified and screened out; and finally, through clustering and reconstruction, the abandoned supplementary information is efficiently compressed and re-integrated into a lexical element sequence, and then the lexical element sequence is sent to a subsequent network layer for processing. According to the method, the information loss is reduced, and the performance of the model under a relatively high pruning rate is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to an efficient visual lexical pruning system, method and device for multimodal large language models. Background Technology

[0002] In recent years, multimodal large language models have made significant progress in fusing visual information with natural language, driving the development of image and video understanding tasks. A typical architecture of these models includes a visual encoder responsible for converting the input image into a large number of visual units, which are then processed by a large language model backbone. However, due to the massive number of visual units to be processed, this architecture incurs huge computational and memory overhead, especially when dealing with high-resolution images or long videos, posing a significant challenge to the deployment of these models on resource-constrained devices.

[0003] To address these challenges, researchers have actively explored alternative methods. For example, expert hybrid methods increase model capacity by adjusting the total number of model parameters while keeping activation parameters constant, thereby enhancing performance without affecting inference speed. Visual expert agents and multi-view input methods, by introducing lightweight structures into the model's input and between modules, have demonstrated some generalization performance in natural language processing. While these methods alleviate the computational and storage costs associated with fine-tuning model parameters to some extent, existing methods generally require an additional training phase, which introduces significant computational and storage overhead.

[0004] Therefore, given the increasingly complex visual language models and diverse downstream tasks, researching a lightweight model simplification method that requires no additional training and adapts to multiple modalities has significant academic value and practical prospects. This method allows the model adaptation process for specific tasks to be freed from high training costs, becoming more flexible and efficient. Summary of the Invention

[0005] This invention provides a system, method, and device for efficient visual lexical pruning of multimodal large language models to solve the technical problems existing in the prior art.

[0006] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0007] A high-efficiency visual lexical pruning system for a multimodal large language model is provided. The system includes R visual lexical pruning units, an image encoder, a text encoder, and a decoding unit, where R is greater than or equal to 2. The R visual lexical pruning units are connected sequentially and are referred to as the first-level to the R-level visual lexical pruning units, respectively. The image encoder and the text encoder are connected in parallel with the first-level visual lexical pruning unit, and the R-level visual lexical pruning unit is connected with the decoding unit.

[0008] Image encoders are used to encode images to generate visual word sequences;

[0009] A text encoder is used to generate a sequence of text terms from a text question;

[0010] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module;

[0011] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance score of the two-dimensional grid structure and the visual word.

[0012] The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set.

[0013] The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set.

[0014] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0015] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it.

[0016] The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it includes a large language model decoding layer;

[0017] The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit.

[0018] The decoding unit inputs and finally merges the visual word sequence and the text word sequence to generate a text answer to the input question.

[0019] Furthermore, the large language model decoding layer module includes a Transformer submodule and a visual lexical grid reconstruction submodule. The Transformer submodule uses an attention mechanism to evaluate the correlation between each visual lexical and the text lexical; the visual lexical grid reconstruction submodule reconstructs the visual lexical sequence... Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: The Transformer submodule, through and The average attention score of each text word is obtained. Importance score The importance score is calculated using the following formula:

[0020] ;

[0021] In the formula:

[0022] express Importance score;

[0023] Represents a two-dimensional grid group The j-th visual word element, ;

[0024] Represents the i-th two-dimensional grid group; ;

[0025] This represents the m-th text word in the text word sequence. ;

[0026] i represents the two-dimensional grid group number;

[0027] j represents the visual lexical index in the two-dimensional grid group;

[0028] m represents the text lexical index in the text lexical sequence;

[0029] A higher importance score indicates a stronger relevance to the text context;

[0030] The key visual lexical generation module is based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

[0031] Furthermore, the supplementary visual lexical generation module includes a maximum cosine similarity calculation submodule and a similarity comparison submodule. The maximum cosine similarity calculation submodule is used to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula is as follows:

[0032] ;

[0033] In the formula:

[0034] This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set;

[0035] Indicates key visual words in a set of visual words;

[0036] This indicates the removal of visual lexical units from the visual lexical unit set;

[0037] Represents a set of key visual terms;

[0038] Visual lexical units representing key visual lexical units The L2 norm;

[0039] Indicates the removal of visual lexical units from the visual lexical set. The L2 norm;

[0040] This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result;

[0041] Indicates the visual lexicon number in the key visual lexicon set;

[0042] Indicates the visual lexical number in the pruning visual lexical set;

[0043] Set a similarity threshold , The value range is 0.70 to 0.85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

[0044] Furthermore, the compression and reconstruction module includes a density peak clustering submodule, which calculates the local density of each visual word in the initial supplementary visual word set according to the following formula:

[0045] ;

[0046] The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula:

[0047] ;

[0048] according to and The product of the two determines the cluster centers; let... ;

[0049] Having the highest The visual lexical unit of the value was selected as the center point;

[0050] In the formula:

[0051] Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set;

[0052] The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density;

[0053] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0054] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0055] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0056] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0057] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication;

[0058] Visual lexical elements The set of k nearest neighbors;

[0059] Visual lexical elements With any visual lexical unit with a higher local density The minimum distance between them.

[0060] This invention also provides an efficient visual lexical pruning method for multimodal large language models, which includes the following steps:

[0061] Set up R visual word pruning units, image encoder, text encoder and decoding unit, R is greater than or equal to 2, so that the R visual word pruning units are connected in sequence, and are called the first level to the Rth level visual word pruning units respectively. The image encoder and text encoder are connected in parallel with the first level visual word pruning unit, and the Rth level visual word pruning unit is connected with the decoding unit.

[0062] Image encoders are used to encode images to generate visual word sequences;

[0063] A text encoder is used to generate a sequence of text terms from a text question;

[0064] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module.

[0065] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance score of the two-dimensional grid structure and the visual word.

[0066] The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set.

[0067] The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set.

[0068] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0069] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it.

[0070] The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it sets up the large language model decoding layer;

[0071] The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit.

[0072] The decoding unit inputs a visual word sequence and a text word sequence, which are then combined to generate a text answer to the input question.

[0073] Furthermore, the large language model decoding layer module employs an attention mechanism to evaluate the relevance between each visual lexical and text lexical, which processes the visual lexical sequence... Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: ,pass and The average attention score of each text word is obtained Importance score The importance score is calculated using the following formula:

[0074] ;

[0075] In the formula:

[0076] express Importance score;

[0077] Represents a two-dimensional grid group The j-th visual word element, ;

[0078] Represents the i-th two-dimensional grid group; ;

[0079] This represents the m-th text word in the text word sequence. ;

[0080] i represents the two-dimensional grid group number;

[0081] j represents the visual lexical index in the two-dimensional grid group;

[0082] m represents the text lexical index in the text lexical sequence;

[0083] A higher importance score indicates a stronger relevance to the text context;

[0084] The key visual lexical generation module is based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

[0085] Furthermore, a visual lexical generation module is added to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula is as follows:

[0086] ;

[0087] In the formula:

[0088] This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set;

[0089] Indicates key visual words in a set of visual words;

[0090] This indicates the removal of visual lexical units from the visual lexical unit set;

[0091] Represents a set of key visual terms;

[0092] Visual lexical units representing key visual lexical units The L2 norm;

[0093] Indicates the removal of visual lexical units from the visual lexical set. The L2 norm;

[0094] This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result;

[0095] Indicates the visual lexicon number in the key visual lexicon set;

[0096] Indicates the visual lexical number in the pruning visual lexical set;

[0097] Set a similarity threshold , The value range is 0.70 to 0.85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

[0098] Furthermore, the compression and reconstruction module employs a density peak clustering algorithm to calculate the local density of each visual word in the initial supplementary visual word set using the following formula:

[0099] ;

[0100] The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula:

[0101] ;

[0102] according to and The product of the two determines the cluster centers; let... ;

[0103] Having the highest The visual lexical unit of the value was selected as the center point;

[0104] In the formula:

[0105] Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set;

[0106] The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density;

[0107] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0108] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0109] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0110] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0111] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication;

[0112] Visual lexical elements The set of k nearest neighbors;

[0113] Visual lexical elements With any visual lexical unit with a higher local density The minimum distance between them.

[0114] Furthermore, R is 2 to 5.

[0115] The present invention also provides an apparatus for an efficient visual lexical pruning method for multimodal large language models, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the efficient visual lexical pruning method for multimodal large language models as described above.

[0116] The advantages and positive effects of this invention are:

[0117] (1) This invention designs a visual lexical pruning framework that does not require additional training. It preserves the original two-dimensional structure of visual information through a spatially aware lexical selection mechanism, which effectively alleviates the inherent positional bias problem in traditional one-dimensional sequence processing methods. At the same time, its unique supplementary lexical compression strategy can recover and integrate supplementary visual information from discarded lexicals, minimize information loss, and significantly improve the performance of the model at a high pruning rate.

[0118] (2) The method proposed in this invention has good versatility and scalability, can be integrated into a variety of mainstream multimodal large language model architectures, and has shown excellent effectiveness in a variety of downstream tasks such as image and video understanding, solving the technical limitations of existing pruning techniques that are usually limited to specific models or task types.

[0119] (3) This invention significantly reduces the number of visual lexical units while maintaining extremely high model performance, achieving an effective balance between computational efficiency and accuracy. This method reduces the computational and memory overhead of multimodal large language models during the inference stage, shortens latency, and provides an efficient and practical solution for deploying and applying advanced multimodal large language models on devices with limited computing resources, thus having significant practical application value.

[0120] This invention is applied to the field of multimodal intelligent sensing and processing technology. Specifically, it can be integrated into physical devices such as portable search devices (e.g., handheld emergency search terminals, small drone search systems), mobile intelligent monitoring devices (e.g., portable video surveillance terminals), and edge computing terminals (e.g., outdoor lightweight AI processing devices). It is used for tasks such as rapid image target recognition, video key information extraction, and cross-modal scene perception. It is adapted to practical application scenarios such as emergency rescue search, outdoor environmental monitoring, and mobile intelligent interaction, providing an efficient multimodal data processing solution for physical devices with limited computing resources. Attached Figure Description

[0121] Figure 1 This is a schematic diagram of the structure of a multimodal large language model visual lexical efficient pruning system according to the present invention.

[0122] Figure 2 This is a schematic diagram illustrating the working principle of a multimodal large language model visual lexical efficient pruning system according to the present invention.

[0123] Figure 3 This is a flowchart of an efficient visual lexical pruning method for a multimodal large language model according to the present invention. Detailed Implementation

[0124] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0125] In the description of this invention, the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," and "bottom," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and do not require the invention to be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on the invention. The terms "connected" and "linked" used in this invention should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; a direct connection or an indirect connection through intermediate components; or an electrical connection or signal transmission. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0126] Please see Figures 1 to 3 A high-efficiency visual lexical pruning system for multimodal large language models is proposed. The system includes R visual lexical pruning units, an image encoder, a text encoder, and a decoding unit, where R is greater than or equal to 2. The R visual lexical pruning units are connected sequentially and are referred to as the first-level to the R-level visual lexical pruning units, respectively. The image encoder and the text encoder are connected in parallel with the first-level visual lexical pruning unit, and the R-level visual lexical pruning unit is connected with the decoding unit.

[0127] Image encoders are used to encode images to generate visual word sequences;

[0128] A text encoder is used to generate a sequence of text terms from a text question;

[0129] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module;

[0130] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance score of the two-dimensional grid structure and the visual word.

[0131] The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set.

[0132] The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set.

[0133] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0134] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it.

[0135] The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it includes a large language model decoding layer;

[0136] The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit.

[0137] The decoding unit inputs and finally merges the visual word sequence and the text word sequence to generate a text answer to the input question.

[0138] Preferably, the large language model decoding layer module may include a Transformer submodule and a visual lexical grid reconstruction submodule. The Transformer submodule uses an attention mechanism to evaluate the correlation between each visual lexical and text lexical; the visual lexical grid reconstruction submodule can reconstruct the visual lexical sequence... Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: The Transformer submodule can be accessed via... and The average attention score of each text word is obtained. Importance score The importance score can be calculated using the following formula:

[0139] ;

[0140] In the formula:

[0141] express Importance score;

[0142] Represents a two-dimensional grid group The j-th visual word element, ;

[0143] Represents the i-th two-dimensional grid group; ;

[0144] This represents the m-th text word in the text word sequence. ;

[0145] i represents the two-dimensional grid group number;

[0146] j represents the visual lexical index in the two-dimensional grid group;

[0147] m represents the text lexical index in the text lexical sequence;

[0148] A higher importance score indicates a stronger relevance to the text context;

[0149] The key visual lexical generation module can generate lexical terms based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

[0150] Preferably, the supplementary visual lexical generation module may include a maximum cosine similarity calculation submodule and a similarity comparison submodule. The maximum cosine similarity calculation submodule is used to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula may be as follows:

[0151] ;

[0152] In the formula:

[0153] This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set;

[0154] Indicates key visual words in a set of visual words;

[0155] This indicates the removal of visual lexical units from the visual lexical unit set;

[0156] Represents a set of key visual terms;

[0157] Visual lexical units representing key visual lexical units The L2 norm;

[0158] Indicates the removal of visual lexical units from the visual lexical set. The L2 norm;

[0159] This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result;

[0160] Indicates the visual lexicon number in the key visual lexicon set;

[0161] Indicates the visual lexical number in the pruning visual lexical set;

[0162] A similarity threshold can be set. , The value range can be 0.70 to 0.85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

[0163] Preferably, the compression and reconstruction module may include a density peak clustering submodule, which can calculate the local density of each visual word in the initial supplementary visual word set according to the following formula:

[0164] ;

[0165] The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula:

[0166] ;

[0167] according to and The product of the two determines the cluster centers; let... ;

[0168] Having the highest The visual lexical unit of the value was selected as the center point;

[0169] In the formula:

[0170] Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set;

[0171] The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density;

[0172] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0173] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0174] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0175] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0176] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication;

[0177] Visual lexical elements The set of k nearest neighbors;

[0178] Visual lexical elements With any visual lexical unit with a higher local density The minimum distance between them.

[0179] This invention also provides an efficient visual lexical pruning method for multimodal large language models, which includes the following steps:

[0180] Set up R visual word pruning units, image encoder, text encoder and decoding unit, R is greater than or equal to 2, so that the R visual word pruning units are connected in sequence, and are called the first level to the Rth level visual word pruning units respectively. The image encoder and text encoder are connected in parallel with the first level visual word pruning unit, and the Rth level visual word pruning unit is connected with the decoding unit.

[0181] Image encoders are used to encode images to generate visual word sequences;

[0182] A text encoder is used to generate a sequence of text terms from a text question;

[0183] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module.

[0184] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance score of the two-dimensional grid structure and the visual word.

[0185] The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set.

[0186] The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set.

[0187] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0188] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it.

[0189] The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it sets up the large language model decoding layer;

[0190] The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit.

[0191] The decoding unit inputs a visual word sequence and a text word sequence, which are then combined to generate a text answer to the input question.

[0192] Preferably, the large language model decoding layer module can employ an attention mechanism to evaluate the correlation between each visual lexical and text lexical, which can convert the visual lexical sequence... Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: It can be accessed through and The average attention score of each text word is obtained Importance score The importance score can be calculated using the following formula:

[0193] ;

[0194] In the formula:

[0195] express Importance score;

[0196] Represents a two-dimensional grid group The j-th visual word element, ;

[0197] Represents the i-th two-dimensional grid group; ;

[0198] This represents the m-th text word in the text word sequence. ;

[0199] i represents the two-dimensional grid group number;

[0200] j represents the visual lexical index in the two-dimensional grid group;

[0201] m represents the lexical index in the text lexical sequence;

[0202] A higher importance score indicates a stronger relevance to the text context;

[0203] The key visual lexical generation module is based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

[0204] Preferably, a supplementary visual lexical generation module is added to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula can be as follows:

[0205] ;

[0206] In the formula:

[0207] This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set;

[0208] Indicates key visual words in a set of visual words;

[0209] This indicates the removal of visual lexical units from the visual lexical unit set;

[0210] Represents a set of key visual terms;

[0211] Visual lexical units representing key visual lexical units The L2 norm;

[0212] Indicates the removal of visual lexical units from the visual lexical set. The L2 norm;

[0213] This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result;

[0214] Indicates the visual lexicon number in the key visual lexicon set;

[0215] Indicates the visual lexical number in the pruning visual lexical set;

[0216] A similarity threshold can be set. , The value range can be 0.70 to 0.85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

[0217] Preferably, the compression and reconstruction module can employ a density peak clustering algorithm, which can calculate the local density of each visual word in the initial supplementary visual word set according to the following formula:

[0218] ;

[0219] The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula:

[0220] ;

[0221] according to and The product of the two determines the cluster centers; let... ;

[0222] Having the highest The visual lexical unit of the value was selected as the center point;

[0223] In the formula:

[0224] Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set;

[0225] The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density;

[0226] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0227] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density;

[0228] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0229] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units;

[0230] The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication;

[0231] Visual lexical elements The set of k nearest neighbors;

[0232] Visual lexical elements With any visual lexical unit with a higher local density The minimum distance between them.

[0233] Preferably, R can be 2 to 5. R is preferably 3.

[0234] The present invention also provides an apparatus embodiment of an efficient visual lexical pruning method for multimodal large language models, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the efficient visual lexical pruning method for multimodal large language models as described above.

[0235] The workflow and working principle of the present invention will be further described below with reference to a preferred embodiment:

[0236] An efficient visual lexical pruning method for multimodal large language models, comprising the following steps:

[0237] Set up R visual word pruning units, an image encoder, a text encoder, and a decoding unit, where R is greater than or equal to 2. Connect the R visual word pruning units sequentially, and call them the first-level to the R-level visual word pruning units respectively. Connect the image encoder and the text encoder in parallel with the first-level visual word pruning unit, and connect the R-level visual word pruning unit with the decoding unit.

[0238] Image encoders are used to encode images to generate visual word sequences.

[0239] A text encoder is used to generate a sequence of text terms from a text question.

[0240] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module.

[0241] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size. Based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance scores of the two-dimensional grid structure and the visual words.

[0242] The key visual lexicon generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram based on the preset retention ratio and the importance score of the visual lexicon. It combines the visual lexicons with higher importance scores to generate a key visual lexicon set and combines the pruned visual lexicons to generate a pruned visual lexicon set.

[0243] The supplementary visual lexical generation module is used to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set, set a similarity threshold, identify visual lexical with similarity scores lower than the similarity threshold as supplementary visual lexical, and combine the supplementary visual lexical into an initial supplementary visual lexical set.

[0244] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0245] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, and output the merged visual lexicon sequence.

[0246] The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it sets up the large language model decoding layer.

[0247] The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence at each level, and output the final merged visual lexical sequence to the decoding unit.

[0248] The decoding unit inputs a visual word sequence and a text word sequence, which are then combined to generate a text answer to the input question.

[0249] An efficient visual lexical pruning method for multimodal large language models, the specific workflow of which is as follows: Figure 3 As shown, this method is executed during the model inference phase and requires no additional training. First, it performs a multi-stage spatial perception screening of visual lexical units to retain core visual information. Then, it evaluates the similarity of the initially pruned lexical units, identifying and selecting those with supplementary information. Finally, through clustering and reconstruction, the discarded supplementary information is efficiently compressed and reintegrated into the lexical sequence before being fed into subsequent network layers for processing. The specific implementation steps are illustrated below using the inference process of the LLaVA large language model in a visual question-answering task as an example.

[0250] An efficient visual lexical pruning method for multimodal large language models includes the following steps:

[0251] S1. Based on the given image-text pairs and the actual requirements of downstream tasks, construct the required network structure using image encoders, text encoders, multilayer perceptrons, and large-scale language models in visual language models; for example... Figure 2As shown, the LLaVA-1.5-7B model was selected as the backbone network. This model consists of a CLIP-ViT-L visual encoder (i.e., an image encoder), a visual projection module, and a language model backbone. When processing visual question answering tasks, the model receives an image. and a text instruction As input, the image is processed by a visual encoder and converted into a set of visual terms. .

[0252] The LLaVA-1.5-7B model is an open-source multimodal language model based on the Transformer architecture. It is trained by fine-tuning the multimodal instruction-following data generated by LLaMA / Vicuna in GPT. This model supports multi-image generation and multi-cue generation, and can convey multiple images in cues.

[0253] Set up R visual word pruning units, an image encoder, a text encoder, and a decoding unit, where R is greater than or equal to 2. Connect the R visual word pruning units sequentially, and call them the first-level to the R-level visual word pruning units respectively. Connect the image encoder and the text encoder in parallel with the first-level visual word pruning unit, and connect the R-level visual word pruning unit with the decoding unit.

[0254] Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module.

[0255] The first to R-level visual lexical pruning units are equivalent to a multilayer perceptron, and the decoding unit is equivalent to the decoding layer of a large-scale language model. The large language model decoding layer module adopts the decoding layer of a large-scale language model.

[0256] S2. Design a spatially aware lexical selection mechanism. To address the positional bias problem, this mechanism divides visual lexical units into a grid and evaluates their importance, preserving the original two-dimensional structure of visual information. This mitigates the bias caused by one-dimensional sequence processing and selects important key lexical units.

[0257] The large language model decoding layer module is used to decode the visual word sequence into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size. Based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs the importance scores of the two-dimensional grid structure and the visual words.

[0258] The key visual lexicon generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram based on the preset retention ratio and the importance score of the visual lexicon. It combines the visual lexicons with higher importance scores to generate a key visual lexicon set and combines the pruned visual lexicons to generate a pruned visual lexicon set.

[0259] S3. Design a supplementary word compression strategy to process the words initially discarded in step S1. First, through similarity calculation, supplementary words with low similarity to the retained keyword word set are selected, and then clustering algorithm is used to further compress these supplementary words.

[0260] The supplementary visual lexical generation module is used to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set, set a similarity threshold, identify visual lexical with similarity scores lower than the similarity threshold as supplementary visual lexical, and combine the supplementary visual lexical into an initial supplementary visual lexical set.

[0261] The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set.

[0262] S4. Reconstruct the final word sequence by merging the key words selected in step S1 with the supplementary words generated by compression in step S2 to form a concise and information-rich visual word sequence, and input it into the subsequent decoder layer for processing.

[0263] The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, and output the merged visual lexicon sequence.

[0264] Step S1, concerning the network structure construction process for applying the visual language model to downstream tasks, includes:

[0265] Based on the given image-text pairs and the actual requirements of the downstream task, the required network structure is built using visual encoders, text encoders, multilayer perceptrons, and large-scale language models in visual language models, such as... Figure 1 As shown.

[0266] The method for spatially aware lexical selection in step S2 specifically includes the following steps:

[0267] First, flatten the visual word sequence Reconstructing it into a two-dimensional mesh and then unioning it can divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: Subsequently, the calculated attention matrix is ​​used to evaluate the relevance between each visual lexical and text lexical. Specifically, for each visual lexical... Importance score through its comparison with all Text words The average attention score is used for calculation, and the formula is as follows: The higher the score, the stronger the relevance to the text context. Finally, a multi-stage selection strategy is adopted to divide the decoder into... Each stage, in each stage At the end, according to the preset retention ratio Determine the number of key visual terms to be retained in the next stage. And based on importance scores and the threshold determined by the number of retentions Visual lexicons are selected from each group to form a new set of key visual lexicons. .

[0268] Step S3, regarding the supplementary visual lexical compression strategy, specifically includes the following steps:

[0269] First, a similarity-based filtering strategy is implemented to identify the parts of the discarded visual lexical units that can supplement visual information. For each pruned visual lexical unit... Calculate the maximum cosine similarity between it and the key visual word set D. The calculation formula is as follows: Set a similarity threshold. Visual words with similarity scores below the threshold will be excluded. The selected words form a supplementary visual lexicon. This ensures that only visual terms that provide new information are included in the subsequent compression process.

[0270] Then, to avoid reintroducing all supplementary visual lexical units into the sequence and offsetting the redundancy caused by pruning, a density peak clustering algorithm is used to compress set C. Specifically, for each supplementary visual lexical unit in set C... The local density is calculated based on its k nearest neighbors. ,in Visual lexical elements The set of k nearest neighbors.

[0271] At the same time, a distance indicator is calculated. This value represents visual lexical units. With any visual lexical unit with a higher local density The minimum distance between them. This is calculated by multiplying the two. To identify cluster centers, it has the highest The visual lexicon of the value was selected as the center point.

[0272] Step S4, concerning the method for reconstructing the final visual word sequence, specifically includes the following steps:

[0273] After determining the cluster centers through density peaks, a visual lexicon set will be added. Each remaining visual lexical unit is assigned to the nearest center based on cosine similarity, thus dividing the supplementary visual lexical set into a predetermined number of clusters H. Finally, a visual lexical reconstruction step is performed to reconstruct the clusters assigned to the nearest centers. All visual morphemes of a cluster It is aggregated into a completely new single visual lexical unit. , , Indicates the first The total number of all visual lexical units in each cluster.

[0274] The calculation formula is as follows: , .

[0275] These newly reconstructed visual lexes encapsulate supplementary and key information. It will be attached to the key visual metaset. Then, the final visual word sequence is formed and passed to subsequent layers of the decoder for processing, such as... Figure 1 As shown.

[0276] Finally, integrate the lexical sequence and complete the reasoning: combine the key visual lexical set. With the generated supplementary visual lexical set { The sequences are merged to form a final visual word sequence with a significantly reduced number of words but retaining key information. This sequence is then fed into subsequent layers of the LLaVA model decoder, where it is processed together with the text words to ultimately generate a textual answer to the input question.

[0277] Please refer to the previous expression definitions for the definitions of the above expressions.

[0278] The aforementioned image encoder, text encoder, decoding unit, large language model decoding layer module, key visual lexical generation module, supplementary visual lexical generation module, compression and reconstruction module, merging module, Transformer submodule, visual lexical grid reconstruction submodule, maximum cosine similarity calculation submodule, similarity comparison submodule, density peak clustering submodule, attention mechanism module, clustering algorithm, density peak clustering algorithm, LLaVA large language model, LLaVA-1.5-7B model and other functional modules, algorithms and models can all adopt existing functional modules, algorithms and models, or adopt existing functional modules, algorithms and models and construct them using conventional technical means.

[0279] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention still fall within the patent scope of the present invention.

Claims

1. A high-efficiency visual lexical pruning system for multimodal large language models, characterized in that, The system includes R visual word pruning units, an image encoder, a text encoder, and a decoding unit, where R is greater than or equal to 2. The R visual word pruning units are connected sequentially and are referred to as the first-level to the R-level visual word pruning units, respectively. The image encoder and the text encoder are connected in parallel with the first-level visual word pruning unit, and the R-level visual word pruning unit is connected with the decoding unit. Image encoders are used to encode images to generate visual word sequences; A text encoder is used to generate a sequence of text terms from a text question; Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module; The large language model decoding layer module is used to decode visual word sequences into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; Based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs a two-dimensional grid structure and the importance score of the visual word; The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set. The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set. The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set. The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it. The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it includes a large language model decoding layer; The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit. The decoding unit inputs and finally merges the visual word sequence and the text word sequence to generate a text answer to the input question.

2. The efficient visual lexical pruning system for multimodal large language models according to claim 1, characterized in that, The large language model decoding layer module includes a Transformer submodule and a visual lexical grid reconstruction submodule. The Transformer submodule uses an attention mechanism to evaluate the correlation between each visual lexical and the text lexical. The visual lexical grid reconstruction submodule reconstructs visual lexical sequences Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: The Transformer submodule, through and The average attention score of each text word is obtained. Importance score The importance score is calculated using the following formula: ； In the formula: express Importance score; Represents a two-dimensional grid group The j-th visual word element, ; Represents the i-th two-dimensional grid group; ; This represents the m-th text word in the text word sequence. ; i represents the two-dimensional grid group number; j represents the visual lexical index in the two-dimensional grid group; m represents the text lexical index in the text lexical sequence; A higher importance score indicates a stronger relevance to the text context; The key visual lexical generation module is based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

3. The efficient visual lexical pruning system for multimodal large language models according to claim 1, characterized in that, The supplementary visual lexical generation module includes a maximum cosine similarity calculation submodule and a similarity comparison submodule. The maximum cosine similarity calculation submodule is used to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula is as follows: ； In the formula: This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set; Indicates key visual words in a set of visual words; This indicates the removal of visual lexical units from the visual lexical unit set; Represents a set of key visual terms; Visual lexical units representing key visual lexical units The L2 norm; This indicates the removal of visual lexical units from the visual lexical set. The L2 norm; This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result; Indicates the visual lexicon number in the key visual lexicon set; Indicates the visual lexical number in the pruning visual lexical set; Set a similarity threshold , The value range is 0.70 to 0.

85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

4. The efficient visual lexical pruning system for multimodal large language models according to claim 1, characterized in that, The compression and reconstruction module includes a density peak clustering submodule, which calculates the local density of each visual word in the initial supplementary visual word set according to the following formula: ； The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula: ； according to and The product of the two determines the cluster centers; let... ; Having the highest The visual lexical unit of the value was selected as the center point; In the formula: Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set; The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication; Visual lexical units The set of k nearest neighbors; Visual lexical units With any visual lexical unit with a higher local density The minimum distance between them.

5. A method for efficient visual lexical pruning in a multimodal large language model, characterized in that, This method includes the following steps: Set up R visual word pruning units, image encoder, text encoder and decoding unit, R is greater than or equal to 2, so that the R visual word pruning units are connected in sequence, and are called the first level to the Rth level visual word pruning units respectively. The image encoder and text encoder are connected in parallel with the first level visual word pruning unit, and the Rth level visual word pruning unit is connected with the decoding unit. Image encoders are used to encode images to generate visual word sequences; A text encoder is used to generate a sequence of text terms from a text question; Each visual lexical pruning unit includes a large language model decoding layer module, a key visual lexical generation module, a supplementary visual lexical generation module, a compression and reconstruction module, and a merging module. The large language model decoding layer module is used to decode visual word sequences into a two-dimensional grid structure that preserves visual information, and divide the two-dimensional grid structure into several equal groups of the same size; Based on the text word sequence, the attention mechanism module in the decoding layer evaluates the importance of each visual word relative to the text word sequence, and outputs a two-dimensional grid structure and the importance score of the visual word; The key visual word generation module is used to perform preliminary pruning on the two-dimensional grid structure diagram according to the preset retention ratio and the importance score of the visual words. The retained visual words with high importance scores are combined to generate a key visual word set, and the pruned visual words are combined to generate a pruned visual word set. The supplementary visual word generation module is used to calculate the maximum cosine similarity between each visual word in the pruned visual word set and all visual words in the key visual word set, set a similarity threshold, identify visual words with similarity scores lower than the similarity threshold as supplementary visual words, and combine each supplementary visual word into an initial supplementary visual word set. The compression and reconstruction module is used to use a clustering algorithm to distribute visual words in the initial supplementary visual word set to the nearest cluster center to form several clusters. All visual words in each cluster are aggregated and reconstructed into a representative word. The representative words of all clusters are combined into a reconstructed supplementary visual word set. The merging module is used to combine the reconstructed supplementary visual lexicon set with the key visual lexicon set, generate a merged visual lexicon sequence, and then output it. The decoding unit is used to decode the visual word sequence into a text answer based on the text word sequence; it sets up the large language model decoding layer; The first-level visual lexical pruning unit takes as input a visual lexical sequence generated by the image encoder and a text lexical sequence generated by the text encoder; the first-level to R-level visual lexical pruning units perform lexical pruning on the input visual lexical sequence according to the text lexical sequence, and output the final merged visual lexical sequence to the decoding unit. The decoding unit inputs a visual word sequence and a text word sequence, which are then combined to generate a text answer to the input question.

6. The efficient visual lexical pruning method for multimodal large language models according to claim 5, characterized in that, The large language model's decoding layer module employs an attention mechanism to evaluate the relevance between each visual lexical and text lexical, which processes the visual lexical sequence... Reconstruct it into a two-dimensional mesh, and divide the two-dimensional mesh into P equal groups of size z. The group's structure is as follows: ,pass and The average attention score of each text word is obtained Importance score The importance score is calculated using the following formula: ； In the formula: express Importance score; Represents a two-dimensional grid group The j-th visual word element, ; Represents the i-th two-dimensional grid group; ; This represents the m-th text word in the text word sequence. ; i represents the two-dimensional grid group number; j represents the visual lexical index in the two-dimensional grid group; m represents the text lexical index in the text lexical sequence; A higher importance score indicates a stronger relevance to the text context; The key visual lexical generation module is based on importance scores. Determine the threshold based on the retention ratio. Visual lexical units are selected from each two-dimensional grid group to obtain a set of key visual lexical units. , .

7. The efficient visual lexical pruning method for multimodal large language models according to claim 5, characterized in that, A supplementary visual lexical generation module is added to calculate the maximum cosine similarity between each visual lexical in the pruned visual lexical set and all visual lexical in the key visual lexical set. The calculation formula is as follows: ； In the formula: This represents the maximum cosine similarity between visual words in the removed visual word set and all visual words in the key visual word set; Indicates key visual words in a set of visual words; This indicates the removal of visual lexical units from the visual lexical unit set; Represents a set of key visual terms; Visual lexical units representing key visual lexical units The L2 norm; This indicates the removal of visual lexical units from the visual lexical set. The L2 norm; This represents all visual lexics in the key visual lexicographic set D. A function that takes the maximum value of the calculated result; Indicates the visual lexicon number in the key visual lexicon set; Indicates the visual lexical number in the pruning visual lexical set; Set a similarity threshold , The value range is 0.70 to 0.

85. The similarity comparison submodule is used to compare similarity scores below a threshold. Visual lexical It was identified as a complementary visual lexicographical unit and categorized into the complementary visual lexicographical unit set. , .

8. The efficient visual lexical pruning method for multimodal large language models according to claim 5, characterized in that, The compression and reconstruction module uses density peak clustering algorithm to calculate the local density of each visual word in the initial supplementary visual word set according to the following formula: ； The density peak clustering submodule calculates the distance indicator between every two visual words in the initial supplementary visual word set using the following formula: ； according to and The product of the two determines the cluster centers; let... ; Having the highest The visual lexical unit of the value was selected as the center point; In the formula: Indicates the index of the visual lexicon currently being calculated in the initial supplementary visual lexicon set; The local density ratio in the initial supplementary visual lexical set is indicated by the index. The index of the visual word with high local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Local density; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical units; The initial supplementary visual lexicon set is indicated by the index number. Visual lexical Distance indication; Visual lexical units The set of k nearest neighbors; Visual lexical units With any visual lexical unit with a higher local density The minimum distance between them.

9. The efficient visual lexical pruning method for multimodal large language models according to claim 5, characterized in that, R is 2 to 5.

10. A device for an efficient visual lexical pruning method for multimodal large language models, comprising a memory and a processor, characterized in that, The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the efficient visual lexical pruning method for multimodal large language models as described in any one of claims 5 to 9.

Images