Visual language model pruning method, electronic device, medium and product

Abstract

The present disclosure provides a visual language model pruning method, an electronic device, a medium and a product, and relates to the technical field of computers. The visual language model pruning method comprises: encoding and converting an input image and a text instruction to determine a visual unit sequence and a text unit sequence; calculating the weight discrete degree of each visual unit in the multi-head attention mechanism for each decoder layer of the visual language model to determine the uncertainty score of each visual unit; pruning the visual units based on the uncertainty score to determine a target visual unit; screening out historical visual units different from the target visual unit from a memory bank; merging the target visual unit and the historical visual units to determine a to-be-recombined visual unit; and performing encoding alignment on the to-be-recombined visual unit, taking the encoding-aligned to-be-recombined visual unit as the input of the next decoder layer until the inference process of all decoder layers is completed.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a visual language model pruning method, electronic device, medium, and product. Background Technology

[0002] Large Visual Language Models (LVLMs), by combining high-capacity visual encoders with large language models to encode input images into long sequences of visual units (tokens), suffer from extremely high computational complexity (O(N²)), leading to a trade-off between efficiency and fidelity. Existing techniques primarily employ pruning-and-forget methods to accelerate inference, including vision-centric pruning relying on visual feature similarity and cross-modal pruning based on the static determination of subsequent pruning by early-layer attention. However, existing pruning methods are unidirectional and irreversible; once a key token is mistakenly deleted, the information is permanently lost. Furthermore, they ignore representation drift, a phenomenon that occurs during deep inference, where the importance of visual units dynamically changes with increasing layer depth. This means that visual units deemed redundant in shallow layers may become crucial in deeper layers, leading to model illusions and a decline in inference ability. Summary of the Invention

[0003] This disclosure provides a visual language model pruning method, electronic device, medium, and product.

[0004] According to one aspect of this disclosure, a visual language model pruning method is provided, comprising: encoding and converting an input image and a text instruction to determine a visual unit sequence and a text unit sequence; for the decoder layer of the visual language model, calculating the weight dispersion of each visual unit in a multi-head attention mechanism based on the cross-attention between each visual unit in the visual unit sequence and the text units in the text unit sequence to determine an uncertainty score for each visual unit; pruning the visual units based on the uncertainty score to determine target visual units whose uncertainty score is greater than or equal to a target threshold; filtering historical visual units different from the target visual units from a memory containing target visual units of a historical decoder layer; merging the target visual units and the historical visual units to determine visual units to be reconstructed; and encoding and aligning the visual units to be reconstructed, using the encoded and aligned visual units to be reconstructed as input to the next decoder layer, until the inference process of all decoder layers is completed.

[0005] According to the visual language model pruning method disclosed herein, the weight dispersion of a multi-head attention mechanism is used as an uncertainty perception index to identify semantically rich visual units. Furthermore, by dynamically backtracking through a memory bank, key historical visual units that were mistakenly pruned due to inter-layer representation drift are effectively corrected for erroneous pruning decisions based on single-layer signals. An inter-layer survivor consensus mechanism is established, significantly suppressing model illusion phenomena. Simultaneously, this visual language model pruning method is a plug-and-play, training-free method. The introduced memory bank operation overhead is only O(N log N), far lower than the O(N²) computational cost saved by the self-attention mechanism. Even with extreme compression, it still performs superiorly in strong reasoning tasks such as scientific question answering and image-text reasoning.

[0006] According to at least one embodiment of the visual language model pruning method of this disclosure, for the decoder layer of the visual language model, based on the cross-attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, the weight dispersion of each visual unit in the multi-head attention mechanism is calculated to determine the uncertainty score of each visual unit, including: for the decoder layer of the visual language model, calculating the attention matrix between each visual unit in the visual unit sequence and the text units in the text unit sequence; extracting the attention weight value of each visual unit in the multi-head attention mechanism in the attention matrix; calculating the variance of the attention weight value of each visual unit on different attention heads to determine the uncertainty score of each visual unit, wherein the larger the variance, the higher the uncertainty score.

[0007] A visual language model pruning method according to at least one embodiment of the present disclosure, which prunes the visual units based on the uncertainty score and determines target visual units whose uncertainty score is greater than or equal to a target threshold, includes: sorting the visual units based on the uncertainty score for the decoder layer of the visual language model; pruning the sorted visual units to determine target visual units whose uncertainty score is greater than or equal to a target threshold, wherein the target threshold is determined based on a target pruning rate.

[0008] According to at least one embodiment of the visual language model pruning method of this disclosure, a memory bank constructed based on a sliding window mechanism is used to filter out historical visual units that are different from the target visual unit. If the same historical visual unit appears in at least two historical decoder layers of the memory bank, the target visual unit in the historical decoder layer closest to the current decoder layer is retained.

[0009] According to at least one embodiment of the visual language model pruning method of this disclosure, the visual unit to be reconstructed is encoded and aligned based on the index value of the image block corresponding to the visual unit in the input image, and the encoded and aligned visual unit to be reconstructed is used as the input of the next decoder layer until the inference process of all decoder layers is completed.

[0010] According to at least one embodiment of the visual language model pruning method of this disclosure, the memory bank is constructed based on a sliding window mechanism.

[0011] According to at least one embodiment of the visual language model pruning method of this disclosure, input images and text instructions are encoded and converted to determine visual unit sequences and text unit sequences, including: segmenting the input image into multiple image blocks, generating a corresponding feature vector for each image block, and determining the visual unit sequence; performing word segmentation on the text instructions, decomposing the text instructions into multiple discrete symbols, generating a corresponding embedding vector for each discrete symbol, and determining the text unit sequence.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs a visual language model pruning method according to any embodiment of this disclosure.

[0013] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the visual language model pruning method of any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a visual language model pruning method according to any embodiment of this disclosure.

[0015] The visual language model pruning method disclosed herein has at least the following beneficial effects: It identifies semantically rich visual units by using the weight dispersion of a multi-head attention mechanism as an uncertainty perception index, and dynamically retrieves key historical visual units that were mistakenly pruned due to inter-layer representation drift using a memory bank. This effectively corrects erroneous pruning decisions based on single-layer signals, establishes an inter-layer survivor consensus mechanism, and significantly suppresses model illusion phenomena. Furthermore, this visual language model pruning method is a plug-and-play, training-free method. The introduced memory bank operation overhead is only O(N log N), far lower than the O(N²) computational cost saved by the self-attention mechanism. Even with extreme compression ratios, it still performs superiorly in strong reasoning tasks such as scientific question answering and graph-text reasoning. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 This is a schematic diagram of the overall process of a visual language model pruning method according to one embodiment of the present disclosure.

[0018] Figure 2 This is a flowchart illustrating the process of determining uncertainty scores in a visual language model pruning method according to one embodiment of the present disclosure.

[0019] Figure 3 This is a flowchart illustrating the process of determining a target visual unit in a visual language model pruning method according to one embodiment of the present disclosure.

[0020] Figure 4 This is a comparison diagram of a visual language model pruning method according to one embodiment of the present disclosure and the prior art.

[0021] Figure 5 This is a schematic diagram of the framework of a visual language model pruning method according to one embodiment of the present disclosure.

[0022] Figure 6 This is a visual schematic diagram of the cross-modal attention landscape in the decoder in a visual language model pruning method according to one embodiment of the present disclosure.

[0023] Figure 7 This is a schematic diagram illustrating the principle of the uncertainty-aware visual unit selection strategy in a visual language model pruning method according to one embodiment of the present disclosure.

[0024] Figure 8 This is a schematic diagram of the memory bank in a visual language model pruning method according to one embodiment of the present disclosure.

[0025] Figure 9 This is a schematic structural block diagram of a visual language model pruning device according to one embodiment of the present disclosure.

[0026] Figure 10 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0027] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0028] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] In complex multimodal reasoning scenarios where large visual language models process high-resolution images, such as question-and-answer tasks like "identify all pedestrians violating traffic rules and explain the reasons" for traffic scene images containing dense details, or diagnostic tasks like "mark lesion areas and explain pathological features" for medical images, existing pruning methods, due to their shallow static decision-making mechanisms, often permanently discard seemingly irrelevant background areas or subtle texture units (such as blurred pedestrian outlines in the distance and faint shadows of early lesions) at early layers. This leads to the model being forced to fabricate answers or produce logical breaks during deep reasoning due to the lack of key visual context. Furthermore, in scenarios requiring cross-layer temporal consistency, such as video frame sequence analysis or long document image-text understanding, existing methods cannot correct erroneous pruning caused by the dynamic drift of visual representations from perceptual embedding to semantic representation between layers, resulting in irreversible breaks in key information paths.

[0030] To address this, this disclosure proposes a visual language model pruning method. In traffic scenarios, it can identify and preserve high-uncertainty visual units such as blurred pedestrian outlines in the distance based on multi-head attention variance. Simultaneously, it retrieves violation targets that were misjudged as background due to early layer representation drift from the memory bank, preventing the model from fabricating reasons for violations due to missing key visual evidence. In medical imaging scenarios, it can capture feature signals with discrete cross-head attention distributions, such as faint shadows of early lesions, and re-inject them from the memory bank into the inference flow after deep semantic alignment, ensuring the contextual integrity of lesion localization and pathological interpretation.

[0031] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0032] A visual language model is an artificial intelligence model that can process and understand both image and text modal data simultaneously. It can consist of a visual encoder, a projection layer, and a language model decoder.

[0033] The decoder layer is composed of multi-head self-attention mechanism and feedforward neural network stacked together. The input features are output to the next layer after nonlinear transformation of each decoder layer. The deep network gradually transforms the visual representation from modality-specific perceptual embedding to language-dominated semantic representation.

[0034] A visual unit is a single basic data unit formed after the input image is segmented and encoded. Each visual unit represents the feature vector of a local region (such as an image patch) in the input image.

[0035] The visual language model pruning method disclosed herein can be deployed on cloud servers, such as online intelligent customer service, large-scale automated medical image screening platforms, or remote visual decision centers for autonomous driving systems. While ensuring diagnostic or judgment accuracy, it can improve the throughput of a single server. When deployed on terminal devices, the lightweight memory injection mechanism is suitable for mobile and edge computing environments with limited computing power. For example, users can run efficient LVLM applications locally on smartphones or tablets to achieve real-time visual assistance for the blind. The device captures the surrounding complex environment (such as traffic intersections) through a camera, and even if most tokens are pruned in low-power mode, key details (such as partially occluded pedestrian outlines) can be dynamically recovered through the memory bank, thus providing accurate and reliable voice navigation. Furthermore, the visual language model pruning method disclosed herein is also suitable for resource-constrained IoT devices, such as drones or robots, enabling them to maintain a high level of visual understanding within a limited energy budget when performing long-duration visual exploration and interaction tasks.

[0036] Figure 1 A schematic diagram illustrating the overall flow of a visual language model pruning method according to one embodiment of this disclosure is shown. Figure 1 The method M100 shown includes steps S110 to S160. This method can be executed by electronic devices such as servers, computers, mobile phones, and tablets.

[0037] In step S110, the input image and text instructions are encoded and converted to determine the visual unit sequence and text unit sequence.

[0038] The input image is decomposed and encoded into a sequence of feature vectors, i.e., a sequence of visual units, using a visual encoder (such as CLIP or SigLIP). The input text instruction is then segmented into sub-word units by a tokenizer matched with the corresponding visual language model, and mapped to corresponding embedding vectors to determine the sequence of text units. The sequence of visual units is then aligned in dimension and space by a projection layer to ensure that it is in a compatible feature distribution space with the sequence of text units.

[0039] Optionally, the input image is segmented into multiple image patches, and a corresponding feature vector is generated for each image patch to determine the visual unit sequence; the text instruction is segmented into multiple discrete symbols, and a corresponding embedding vector is generated for each discrete symbol to determine the text unit sequence.

[0040] In step S120, for the decoder layer of the visual language model, based on the cross attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, the weight dispersion of each visual unit in the multi-head attention mechanism is calculated to determine the uncertainty score of each visual unit.

[0041] By analyzing the response differences of the same visual unit across multiple parallel attention heads, the uncertainty score of each visual unit is determined. A higher uncertainty score indicates that the visual unit plays a key role under different interpretation paths.

[0042] In step S130, visual units are pruned based on uncertainty scores to identify target visual units whose uncertainty scores are greater than or equal to the target threshold.

[0043] Each visual unit is sorted in descending order of its uncertainty score to establish a global importance priority. A target threshold for the number of units to be retained is determined based on a preset target pruning rate parameter. The visual units at the top of the sorted list are selected to form the retention set (i.e., the target visual units) of the current decoder layer, while the remaining low-scoring visual units are temporarily pruned and discarded.

[0044] In step S140, historical visual units that are different from the target visual unit are selected from the memory of the target visual unit containing the historical decoder layer.

[0045] The aforementioned historical visual units are target visual units that are determined to be important and retained in the preceding decoder layers during the model inference process.

[0046] The memory caches all target visual units that have been retained in the most recent decoder layers. The set difference operation is performed between the historically retained target visual units in the memory and the target visual units retained in the current decoder layer to identify target visual units that were historically retained but not selected in the current decoder layer as recall candidates, i.e., historical visual units.

[0047] Preferably, the memory is constructed based on a sliding window mechanism.

[0048] Preferably, a memory bank constructed based on a sliding window mechanism is used to filter out historical visual units that are different from the target visual unit. Specifically, if the same historical visual unit appears in at least two historical decoder layers of the memory bank, the target visual unit in the historical decoder layer closest to the current decoder layer is retained. This leverages the local stability of the residual network to ensure feature compatibility.

[0049] In step S150, the target visual unit and the historical visual unit are merged to determine the visual unit to be recombined.

[0050] Perform a union operation between the target visual unit in the current decoder layer and the historical visual units to determine the visual units to be reconstructed.

[0051] In step S160, the visual unit to be reconstructed is encoded and aligned, and the encoded and aligned visual unit to be reconstructed is used as the input of the next decoder layer until the inference process of all decoder layers is completed.

[0052] Preferably, based on the index value of the image patch corresponding to the visual unit to be reconstructed in the input image, the visual unit to be reconstructed is encoded and aligned. The encoded and aligned visual unit to be reconstructed is then used as the input to the next decoder layer, until the inference process of all decoder layers is completed. This ensures that each visual unit can be correctly encoded and aligned with the next decoder layer of the model, thereby maintaining the geometric consistency of the image.

[0053] Therefore, by using the dispersion of multi-head cross-attention weights as an uncertainty indicator, key target visual units with rich semantics are accurately screened. By maintaining a historical memory bank based on a sliding window mechanism, key information that has been mistakenly deleted by the current decoder layer due to the drift of deep representations in the model is actively retrieved and re-injected. This effectively overcomes the irreversible information loss problem caused by existing pruning-forgetting methods, suppresses model illusions, and enhances the ability to understand complex long-range semantic relationships.

[0054] Regarding step S120, for the decoder layer of the visual language model, based on the cross-attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, the weight dispersion of each visual unit in the multi-head attention mechanism is calculated to determine the uncertainty score of each visual unit. In some embodiments of this disclosure, it may include, for example... Figure 2 Steps S1201 to S1203 are shown.

[0055] In step S1201, for the decoder layer of the visual language model, the attention matrix between each visual unit in the visual unit sequence and the text units in the text unit sequence is calculated.

[0056] By treating the text units of the current decoder layer as queries and the visual units as keys and values, the correlation between them is calculated to generate a two-dimensional matrix, namely the attention matrix. Each element represents the degree of attention a text unit pays to a certain visual unit.

[0057] In step S1202, the attention weight value of each visual unit in the attention matrix in the multi-head attention mechanism is extracted.

[0058] Based on the attention matrix, the specific weight values ​​of each visual unit on all parallel attention heads (Multi-head Attention) are extracted, and the overall attention distribution is decomposed into independent attention signals from multiple expert perspectives.

[0059] In step S1203, the variance of the attention weight values ​​of each visual unit on different attention heads is calculated to determine the uncertainty score of each visual unit. The larger the variance, the higher the uncertainty score.

[0060] The uncertainty of a visual unit is quantified by calculating the variance of the attention weights across multiple parallel attention heads. A larger variance indicates a more significant difference in the level of attention evoked by the visual unit across different attention heads; that is, the visual unit is highly focused on in some heads but ignored in others.

[0061] Therefore, by leveraging the functional differentiation of different attention heads (such as global scanning and local detail focusing), high variance can be used as a reliable indicator of information-rich and semantically complex key visual cues. This is an important objective for effectively distinguishing genuine redundant background from inconsistent attention patterns arising from viewpoint differences.

[0062] Regarding step S130, visual units are pruned based on uncertainty scores to determine target visual units whose uncertainty scores are greater than or equal to a target threshold. In some embodiments of this disclosure, this may include, for example... Figure 3 Steps S1301 to S1302 are shown.

[0063] In step S1301, for the decoder layer of the visual language model, the visual units are sorted based on the uncertainty score.

[0064] Preferably, for the currently processed decoder layer, all visual units are sorted in descending order according to their uncertainty scores.

[0065] In step S1302, the sorted visual units are pruned to identify target visual units whose uncertainty scores are greater than or equal to the target threshold, which is determined based on the target pruning rate.

[0066] Preferably, the target threshold is k represents the target threshold, and r represents the target pruning rate. This indicates the total number of visual units input to the current decoder layer.

[0067] Therefore, this method achieves quantified hierarchical classification and precise selection of visual units based on their importance. Compared to fixed threshold judgment, this dynamic retention mechanism for visual units allows for precise control over the number of units retained, and can flexibly adjust the compression intensity based on the terminal's computing power. This reduces the quadratic computational complexity of the self-attention mechanism while maintaining model inference performance.

[0068] The technical solution of this disclosure will be further explained below with a specific implementation example.

[0069] like Figure 4 As shown, Figure 4 (a) is a vision-centric pruning method. After unidirectional pruning of the visual units at the output of the visual encoder, the data is input into a large language model, and the language commands are input independently. Figure 4 (b) For cross-modal pruning, static pruning is also performed between the visual encoder and the language model, but the pruning basis may involve cross-modal interaction information. Figure 4 (c) is the method of this disclosure, in which a UVMemIn module is embedded between the visual encoder and the language model. Solid arrows represent the input stream of regular visual units or text units, and dashed arrows represent the dynamic retrieval and re-injection path of historical key visual units by the UVMemIn module, forming a closed-loop mechanism of pruning and retrieval, which is different from the irreversible characteristics of pruning and forgetting in the previous two paradigms.

[0070] The specific implementation of this disclosure is training-free, a plug-and-play approach, deployed between the Transformer decoder layers of a large visual language model LVLM, as shown in the structure diagram below. Figure 5 As shown.

[0071] Load a pre-trained large visual language model (such as LLaVA-v1.5), which includes a visual encoder. Projection layer W and language model Input image I Converted into a visual token sequence That is, a sequence of visual units; text instructions. T Converted into a text token sequence That is, a sequence of text units.

[0072] Specifically, for the decoder's first l layer- l Obtain the attention matrix of text tokens (i.e., text units) to visual tokens (i.e., visual units). ,in, Indicates the first l The attention matrix of the layer, Indicates the number of attention heads. Represents the space of real numbers. Indicates the number of text units. Indicates the first l The number of visual units input to the layer.

[0073] like Figure 6 As shown, the functional differentiation exhibited by different attention heads when processing visual units (such as global scanning and specific feature detection) reveals the theoretical basis for the method of this disclosure to use attention variance for uncertainty estimation.

[0074] For each visual unit j Calculate its attention across all heads H The variance is large if a visual unit is not only focused on and varies drastically in some heads (specific feature detectors) but ignored in others, indicating that its semantics are complex and important.

[0075] like Figure 7 As shown, Figure 7 (a) is a heatmap of uncertainty scores, showing that regions of high uncertainty correspond to semantically rich regions; Figure 7 (b) is a distribution of attention scores for representative visual units on different attention heads, illustrating how variance can be used to distinguish between information-rich and redundant visual units.

[0076] According to the uncertainty score Select Top- k Each visual unit is used as the target visual unit. ,in k represents the target threshold, and r represents the target pruning rate. This indicates the total number of visual units input to the current decoder layer.

[0077] like Figure 8 As shown, maintain a sliding window memory. It includes the former Layers (e.g.) The union of all surviving target visual units establishes a "survivor consensus" mechanism. (This involves) calculating historical visual units. This involves identifying target visual units that were historically important but were almost removed from the current layer. Indicates the first l The historical visual unit of the layer, Indicates the first l The layer's memory, with Unique(⋅) representing the deduplication operation. Indicates the first l The target visual unit of the layer.

[0078] Preferably, during retrieval, if the same visual unit appears in multiple decoder layers of the memory, only the feature representation of the most recent decoder layer (the deepest layer) (i.e., the target visual unit) is retained to ensure the freshness and compatibility of the features (utilizing the local stability of the residual network).

[0079] The target visual unit retained in the current decoder layer With retrieved historical visual units Merging. Crucially, the visual units must be re-sorted based on their patch index in the original input image to recover spatial location information and ensure that the positional embeddings of subsequent layers are correctly aligned.

[0080] The sorted and reorganized visual unit sequence to be reorganized Enter the number l The calculation is performed on layer +1, and the above process is repeated until the last layer.

[0081] Therefore, experiments show that, even in the extreme case of retaining 64 visual units (approximately 89% compression ratio), the method disclosed herein achieves an average performance retention rate of 91.17% on LLaVA-v1.5-7B. In comparison, FastV drops to 86.73%, and PDrop drops to 86.28%. The method disclosed herein achieves performance improvements of approximately 4% or more compared to existing technologies at the same compression ratio.

[0082] In the POPE (Phantom Evaluation) test, the method disclosed herein scored 81.80% when retaining 64 tokens, significantly higher than FastV's 75.12%. This indicates that recovering mistakenly deleted visual details effectively suppresses hallucinations.

[0083] In strong reasoning tasks such as SQA (Scientific Question Answering) and GQA (Graphic and Text Reasoning), the method disclosed herein even outperforms the full model after pruning (e.g., on SQA, the score is 70.10% with 128 tokens, compared to 69.56% with the full model), demonstrating the denoising effect brought about by removing visual noise.

[0084] While maintaining high accuracy, the method disclosed in this paper significantly reduces FLOPs. For example, when retaining 128 tokens, FLOPs are reduced from 4.41G to 1.82G (a reduction of approximately 59%).

[0085] The additional overhead introduced by the memory (sorting and deduplication) is: O ( N log N Compared to reduced self-attention O ( The computational load is almost negligible.

[0086] The visualization results show that when dealing with bus detail recognition or baseball scenes with cluttered backgrounds, the method disclosed herein can successfully retrieve key objects (such as wheels and distant players) lost in the intermediate layer from the memory bank, while other methods permanently lose this information, resulting in incorrect answers.

[0087] Based on any of the above embodiments, this disclosure also provides a visual language model pruning device.

[0088] Figure 9 This is a schematic block diagram of the structure of a visual language model pruning device according to one embodiment of the present disclosure.

[0089] like Figure 9 As shown, the visual language model pruning device includes: The encoding conversion module 9002 encodes and converts the input image and text instructions to determine the visual unit sequence and the text unit sequence. The score calculation module 9004, for the decoder layer of the visual language model, calculates the weight dispersion of each visual unit in the multi-head attention mechanism based on the cross attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, and determines the uncertainty score of each visual unit. The model pruning module 9006 prunes visual units based on uncertainty scores and identifies target visual units whose uncertainty scores are greater than or equal to the target threshold. The unit retrieval module 9008 filters out historical visual units that are different from the target visual unit from the memory of the target visual unit containing the historical decoder layer. The unit merging module 9010 merges the target visual unit and the historical visual units to determine the visual units to be recombined. The unit reassembly module 9012 encodes and aligns the visual units to be reassembled, and uses the encoded and aligned visual units to be reassembled as the input of the next decoder layer until the inference process of all decoder layers is completed.

[0090] The aforementioned visual language model pruning device can be in the form of computer software, and each module of the aforementioned visual language model pruning device can be implemented through computer software modules.

[0091] The implementation process of the functions and roles of each module in the above visual language model pruning device is detailed in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0092] The execution subject of the visual language model pruning method in the specific embodiments of this disclosure can be an electronic device such as a server, computer, mobile phone, or tablet computer.

[0093] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the visual language model pruning method of any of the embodiments described above.

[0094] Figure 10 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0095] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0096] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0097] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0098] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. A computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0099] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be magnetic media, such as floppy disks, hard disks, and magnetic tapes; optical media, such as digital video discs; or semiconductor media, such as solid-state drives. The computer-readable storage medium can be volatile or non-volatile, or it can include both types of storage media.

[0100] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A visual language model pruning method, characterized in that, include: The input image and text instructions are encoded and converted to determine the visual unit sequence and the text unit sequence; For the decoder layer of the visual language model, based on the cross attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, the weight dispersion of each visual unit in the multi-head attention mechanism is calculated to determine the uncertainty score of each visual unit. Based on the uncertainty score, the visual units are pruned to determine the target visual units whose uncertainty score is greater than or equal to the target threshold. From the memory of the target visual unit containing the history decoder layer, filter out the historical visual units that are different from the target visual unit; The target visual unit and the historical visual unit are merged to determine the visual unit to be recombined; as well as The visual unit to be reconstructed is encoded and aligned, and the encoded and aligned visual unit to be reconstructed is used as the input of the next decoder layer until the inference process of all decoder layers is completed.

2. The visual language model pruning method as described in claim 1, characterized in that, For the decoder layer of the visual language model, based on the cross-attention between each visual unit in the visual unit sequence and the text units in the text unit sequence, the weight dispersion of each visual unit in the multi-head attention mechanism is calculated to determine the uncertainty score of each visual unit, including: For the decoder layer of the visual language model, calculate the attention matrix between each visual unit in the visual unit sequence and the text units in the text unit sequence; The attention weight value of each visual unit in the attention matrix in the multi-head attention mechanism is extracted; The variance of the attention weight values ​​of each visual unit on different attention heads is calculated to determine the uncertainty score of each visual unit, wherein the larger the variance, the higher the uncertainty score.

3. The visual language model pruning method as described in claim 1, characterized in that, Pruning the visual units based on the uncertainty score, and determining target visual units whose uncertainty score is greater than or equal to a target threshold, includes: For the decoder layer of the visual language model, the visual units are ranked based on the uncertainty score; The sorted visual units are pruned to identify target visual units whose uncertainty scores are greater than or equal to a target threshold, which is determined based on a target pruning rate.

4. The visual language model pruning method as described in claim 1, characterized in that, The memory bank, constructed based on the sliding window mechanism, filters out historical visual units that are different from the target visual unit. If the same historical visual unit appears in at least two historical decoder layers in the memory bank, the target visual unit in the historical decoder layer closest to the current decoder layer is retained.

5. The visual language model pruning method as described in claim 1, characterized in that, Based on the index value of the image block corresponding to the visual unit to be reconstructed in the input image, the visual unit to be reconstructed is encoded and aligned, and the encoded and aligned visual unit to be reconstructed is used as the input of the next decoder layer until the inference process of all decoder layers is completed.

6. The visual language model pruning method as described in claim 1, characterized in that, The memory is constructed based on a sliding window mechanism.

7. The visual language model pruning method as described in claim 1, characterized in that, The input image and text instructions are encoded and converted to determine the visual unit sequence and text unit sequence, including: segmenting the input image into multiple image blocks, generating a corresponding feature vector for each image block, and determining the visual unit sequence; performing word segmentation on the text instructions, decomposing the text instructions into multiple discrete symbols, generating a corresponding embedding vector for each discrete symbol, and determining the text unit sequence.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the visual language model pruning method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the visual language model pruning method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual language model pruning method according to any one of claims 1 to 7.

Images