Image processing method and computing device
By selecting visual tokens based on attention variance and saliency in the visual language model and pruning non-critical tokens, the high computational cost of the visual language model is solved, achieving model acceleration and performance maintenance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-29
Smart Images
Figure CN122116035A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the fields of artificial intelligence and machine learning, and more particularly to image processing methods and corresponding computing devices. Background Technology
[0002] Multimodal Large Language Models (MLLMs) are an important development direction in the field of artificial intelligence, aiming to enable models to simultaneously understand and generate data in multiple modalities, such as text, images, audio, and video. Vision-Language Models (VLMs), also known as image-text large models, are a core branch of multimodal large models, focusing on integrating visual perception and language understanding capabilities to achieve unified representation and interaction of images and text. Image-text large models can be used in various scenarios such as image description generation, visual question answering, image-text retrieval, OCR text recognition, chart understanding, and visual localization.
[0003] To improve image understanding capabilities, visual language models capture finer image details by using higher-resolution inputs, enabling them to achieve superior performance on a wide range of multimodal tasks. However, this performance improvement comes with a significant challenge: the dramatic increase in the number of visual tokens leads to an exponential increase in computational costs and inference latency, which has become a key bottleneck limiting the widespread deployment of visual language models in edge devices and real-time applications.
[0004] We hope to find an improved solution that can reduce the computational cost of visual language models and improve their efficiency without affecting inference performance as much as possible. Summary of the Invention
[0005] This specification provides one or more embodiments of an image processing method that can effectively prune visual tokens, thereby reducing the computational cost of visual language models with minimal impact on inference performance.
[0006] According to the first aspect, an image processing method is provided, comprising:
[0007] The N patches corresponding to the target image are input into the image encoder to obtain the representation set composed of the representation vectors of each patch output by the target attention layer. The target attention layer includes H attention heads.
[0008] For any target patch, the attention variance of the target patch is determined based on the variance between the attention scores calculated by the H attention heads for the target patch respectively.
[0009] Select multiple tiles with large attention variance, and form a first candidate set based on the multiple tiles;
[0010] The set of tiles to be retained is determined based on the first candidate set;
[0011] From the representation set, determine the partial representation vectors corresponding to the retained tile set, and only perform further processing on the partial representation vectors.
[0012] According to a second aspect, an apparatus for performing image processing is provided, comprising:
[0013] The data acquisition unit is configured to input N patches corresponding to the target image into an image encoder to obtain a representation set composed of the representation vectors of each patch output by the target attention layer, wherein the target attention layer includes H attention heads;
[0014] The variance determination unit is configured to determine the attention variance of any target patch based on the variance between the attention scores calculated by the H attention heads for the target patch respectively.
[0015] The first candidate set determination unit is configured to select multiple tiles with large attention variance and form a first candidate set based on the multiple tiles;
[0016] The unit for determining the set of retained tiles is configured to determine the set of retained tiles based on the first candidate set.
[0017] The tile pruning unit is configured to determine a portion of the representation vectors corresponding to the retained tile set from the representation set, and to perform further processing only on the portion of the representation vectors.
[0018] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.
[0019] According to a fourth aspect, a computer system is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method of the first aspect when executing the program.
[0020] In the embodiments of this specification, N patches corresponding to the target image are input into an image encoder to obtain an attention score matrix calculated by H attention heads included in the target attention layer. For any patch, the attention variance of the patch is determined based on the variance between the attention scores calculated by the H attention heads for that patch, and patches with larger attention variances are selected and included in the reserved patch set. Thus, in each embodiment, visual tokens with diversity in different attention heads are determined based on the attention variance, and token pruning is performed based on the diversity of visual tokens, thereby accelerating the model inference speed while retaining diverse key tokens and avoiding degradation of inference performance. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This illustrates the overall architecture of the graphic model;
[0023] Figure 2 A schematic diagram of token pruning is shown in one embodiment;
[0024] Figure 3 This diagram illustrates attention processing using a single attention head.
[0025] Figure 4 A method flow for performing image processing is shown in one embodiment;
[0026] Figure 5 This diagram illustrates a method for fine-tuning a mapping network according to one embodiment.
[0027] Figure 6 A schematic block diagram of an apparatus for image processing is shown;
[0028] Figure 7 shows an example computer system 1000. Detailed Implementation
[0029] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0030] Figure 1 This illustrates the overall structure of the large graphic model. For example... Figure 1 As shown, the typical architecture of a large image-text multimodal model includes an image encoder, a mapping network, and a large language model, which work together to achieve cross-modal understanding.
[0031] Image encoders, also known as visual encoders, typically employ the Vision Transformer (ViT) architecture. An image encoder segments the input image into fixed-size patches, transforms them into vector sequences through linear projection, and then processes them through multiple layers of self-attention mechanisms and feedforward networks to ultimately extract high-dimensional visual features from each patch. Each patch and its corresponding visual feature are referred to as a visual token. The high-dimensional visual features extracted by the image encoder can capture multi-level visual information, from edge texture to object semantics, but their dimensionality and semantic space differ fundamentally from those of language models.
[0032] Mapping networks serve as a crucial bridge connecting visual and textual modalities, undertaking the functions of feature alignment and spatial transformation. Common implementations include simple linear projection layers, multilayer perceptrons (MLPs), or more complex Query Transformer (Q-Former) structures. This network maps visual features output from the image encoder to the input embedding space of the language model, enabling visual information to be processed by the language model in the form of "pseudo-text tokens." Some advanced designs introduce learnable query vectors, extracting key information from dense visual features through cross-attention mechanisms, significantly reducing the number of tokens input to the language model.
[0033] Large language models, serving as the cognitive core of the architecture, are typically based on a decoder-only Transformer architecture. They receive input consisting of a concatenated sequence of mapped visual features (visual tokens) and text tokens. Leveraging their powerful semantic understanding, logical reasoning, and knowledge representation capabilities acquired through pre-training, they jointly model cross-modal information. The model's attention mechanism automatically establishes fine-grained associations between visual regions and text descriptions, enabling complex tasks such as image description and visual question answering.
[0034] As image resolution increases, the number of visual tokens processed by image encoders expands dramatically, causing the computational cost and inference latency of large image-text models to increase exponentially. To address this challenge, visual token pruning techniques have emerged. The core idea of visual token pruning is to first define the importance of each visual token, and then prune unimportant tokens during the subsequent inference stage—that is, to cut off or remove unimportant tokens and not process them further. However, how to accurately and effectively define this set of important visual tokens becomes a more fundamental problem and challenge.
[0035] In some related techniques, the cross-attention between visual and text tokens in a large language model is obtained. Less important visual tokens are then identified based on the cross-attention score and pruned. However, the inventors found that the cross-attention obtained in the large language model exhibits a visual-text bias, causing attention to be biased towards visual tokens closer to the text. Furthermore, this method also prevents the flexible application of more efficient implementations (such as flash attention).
[0036] In other related techniques, the self-attention score of visual tokens in the image encoder is obtained, and tokens with stronger visual saliency or global information tokens are selected for further processing based on the self-attention score. However, the inventors found that this approach ignores the role of token diversity and comprehensiveness in subsequent processing and can also lead to a deterioration in inference accuracy.
[0037] In view of this, this specification proposes an image processing scheme in which token diversity is considered in visual token pruning, so that the retained visual tokens have greater diversity and comprehensiveness, thereby reducing the computational cost of the model while having the least possible impact on inference performance.
[0038] Figure 2 This diagram illustrates token pruning in one embodiment. Generally, an image is divided into multiple tiles and input to an image encoder. For example... Figure 2 As shown, the image encoder adopts a Transformer-based architecture, which includes several attention layers. Generally, an attention layer further includes multiple attention heads. Each attention head can calculate the attention scores among visual tokens corresponding to multiple patches, forming an attention matrix. Based on the attention matrix, the current representation vector of each visual token is further processed to obtain a further fused representation vector.
[0039] Based on the visual token pruning approach proposed in this specification, H attention matrices can be obtained, each calculated by one of the H attention heads in one or more attention layers. For a given visual token, the variance of the attention scores calculated by the H attention heads for that visual token is obtained, and this attention variance is used as a measure of the diversity of that visual token. Therefore, some visual tokens with larger attention variances can be selected as diversity tokens and included in the reserved token set.
[0040] In addition, visual tokens with higher overall attention scores can be identified based on H attention matrices and included in the reserved token set as salient tokens.
[0041] Therefore, in Figure 2The retained token set includes both saliency tokens with high overall attention scores and diversity tokens with large variance in attention scores. Such a retained token set can more comprehensively reflect the information of the original image and is beneficial to maintaining the model's inference performance.
[0042] The token pruning and image processing procedures in the embodiments of this specification are described in detail below.
[0043] To clearly illustrate the calculation of attention scores in token pruning, we first describe the process of attention processing in any attention layer of the Transformer architecture. Figure 3 This diagram illustrates attention processing using a single attention head.
[0044] Assuming the image to be processed is divided into N patches, the input to the image encoder includes the initial representations of N visual tokens corresponding to the N patches. Following the processing approach of the Transformer architecture, a special character token representing global information is typically appended to the input sequence by default. This special character token is often represented as the classification symbol CLS used for classification. Each attention head in each attention layer of the image encoder performs attention processing on the aforementioned N visual tokens and the CLS special token.
[0045] Thus, as Figure 3 As shown, the input to a single attention head includes N visual tokens (represented as T1, T2, ..., T in the figure). N The current representation vectors of the token and the CLS token are respectively, i.e., a vector sequence of N+1 current representation vectors. These N+1 current representation vectors can be represented as the current representation matrix F. in .
[0046] In the attention head, first, the three parameter matrices W corresponding to the attention head are... Q W K and W V Acting on the aforementioned current representation matrix F respectively in A linear transformation is performed on the current representation matrix to obtain the corresponding query matrix Q, key matrix K, and value matrix V. Then, the attention head can obtain the attention matrix A according to the following formula (1):
[0047] (1)
[0048] in, Let Q be the vector dimension in matrices Q and K.
[0049] It can be understood that the attention matrix is an (N+1)*(N+1) dimensional matrix, where the element a in the i-th row and j-th column is... i,jLet represent the attention score between token i and token j.
[0050] Furthermore, based on this attention matrix, the attention head can derive the representation matrix F of N+1 tokens after attention fusion:
[0051] (2)
[0052] The representation matrix F contains N+1 representation vectors, corresponding to the updated representation vectors of the N+1 tokens, i.e. Figure 3 In F-CLS, F1, F2…, F N .
[0053] When an attention layer l includes H attention heads, each attention head processes the data in parallel according to the above process, and each obtains a representation matrix F. In this attention layer l, the H representation matrices obtained by the H attention heads are merged to form the representation set output by this attention layer l, which is used for processing in the next network layer.
[0054] For an image encoder that performs attention processing according to the Transformer architecture described above, the attention-based token pruning and image processing procedures described in the embodiments of this specification can be performed. Figure 4 A method flow for image processing in one embodiment is shown. Figure 4 As shown, the method includes the following steps.
[0055] In step S401, the N patches corresponding to the target image are input into the image encoder to obtain the representation set composed of the representation vectors of each patch output by the target attention layer; the target attention layer includes H attention heads.
[0056] As mentioned earlier, the target image is divided into N patches and input to the image encoder. The N patches correspond to N visual tokens. The image encoder can employ a Transformer-based architecture, which includes several attention layers. The target attention layer can be one or more attention layers from the image encoder. When there are multiple target attention layers, the representation set output by the target attention layer is the representation set output by the last attention layer among them.
[0057] It is understood that the calculation result of the target attention layer will serve as the basis for visual token pruning. In one embodiment, the target attention layer can be the last attention layer in the image encoder. In this case, it is equivalent to pruning the output visual tokens of the image encoder.
[0058] In another embodiment, the target attention layer can be an intermediate attention layer among several attention layers of the image encoder. When visual token pruning is performed based on the attention score of this intermediate attention layer, subsequent attention layers only need to process the pruned visual tokens, which can reduce the computational load of subsequent attention layers.
[0059] Through experiments, in one embodiment with better performance, the penultimate attention layer in the image encoder can be used as the target attention layer. Then, after visual token pruning is performed on this target attention layer, the pruned visual tokens undergo final attention fusion processing in the last attention layer, becoming the output of the image encoder.
[0060] In order to perform visual token pruning, for the target attention layer mentioned above, in addition to obtaining its output representation set, we can also obtain the attention matrix calculated by each of the H attention heads contained in the target attention layer, as the basis for visual token pruning.
[0061] It can be understood that each of the H attention heads calculates the attention matrix according to the aforementioned formula (1), and calculates the representation vector of each tile (or each visual token) according to the aforementioned formula (2).
[0062] Therefore, in step S403, for any target patch, the attention variance of the target patch is determined based on the variance between the attention scores calculated by the H attention heads for the target patch.
[0063] In response, the inventors discovered that if a token exhibits significant differences in attention scores across different attention heads, it indicates that the model is focusing on the token from multiple and diverse semantic perspectives. Such a token is likely an information-rich "hub" feature. Therefore, in the embodiments of this specification, the difference in attention scores of a token across different attention heads, i.e., the attention variance, is used as a measure of the token's "diversity."
[0064] Specifically, for any visual token i corresponding to a target patch, we can obtain H attention heads, each calculating an attention score s for that visual token i. i Calculate H attention scores s i The variance between them is taken as the attention variance of the visual token i.
[0065] In a specific example, any attention head h computes an attention score s for visual token i. i This can be the mean of the elements in the row or column corresponding to the visual token i in the attention matrix calculated by the attention head h. Then, the H values of s obtained from the H attention heads are calculated. i The variance between them is used as the attention variance of token i.
[0066] In another example, for any attention head h, the attention score calculated by that attention head h is obtained between the target tile i (corresponding to visual token i) and the special token CLS representing global information. , where h is the attention score calculated for visual token i. Thus, H attention scores can be obtained: .
[0067] Next, the variance among the H attention scores can be calculated and used as the attention variance of the target patch i. As shown in the following formula (3):
[0068] (3)
[0069] Using the above method, the attention variance corresponding to any target patch i is obtained. Similarly, the attention variance corresponding to each patch, or rather, each visual token, can be obtained.
[0070] Next, in step S405, several tiles with large attention variances are selected, and a first candidate set is formed based on these tiles.
[0071] In one embodiment, the N visual tokens corresponding to the N tiles are sorted according to the magnitude of their attention variance, and the M tiles with larger attention variances are obtained.
[0072] In another embodiment, attention variance-based pruning can be combined with other token pruning methods. Accordingly, in this step, M visual tokens with larger attention variances can be selected from the remaining visual tokens pruned using other token pruning methods for subsequent processing.
[0073] The number K of visual tokens to be retained for diversity token pruning can be preset. div The number of tokens in the first alternative set to be formed.
[0074] Accordingly, in one implementation, when making selections based on the ranking of attention variances, M=K, which has the larger attention variance, is selected. div One tile, as the first candidate set J div .
[0075] In another implementation, when selecting based on the ranking of attention variance, the number of selected tiles M can be greater than K. div This means selecting M tiles as an initial tile pool, and then further filtering K tiles from this pool. div One tile, as the first candidate set J div .
[0076] In a further embodiment, tile selection can be performed through the following process: First, an initial tile pool X is formed based on the M tiles (or their corresponding M visual tokens) with the largest attention variance. pool Select a certain number q tiles from the initial tile pool and move them into the selection set X. div This forms the initial selection set, where q <K div The selection of q tiles can be determined through random sampling.
[0077] Next, several rounds of sampling are performed, in which one tile is removed from the tile pool X in each round of sampling. pool Move to selection set X div Until set X is selected div The number of blocks in the middle map has reached the preset number K. div The first candidate set J was obtained. div Each round of sampling can specifically adopt the "farthest point sampling" approach to further ensure the diversity of visual tokens. That is, in each round of sampling, the tile whose representation vector is farthest from the current selection set is selected from the current tile pool as the sampling tile and moved into the current selection set.
[0078] In a more specific example, the current selection set X can be calculated. div The mean of the representation vectors of each tile in the current tile pool X is used as the center vector; pool For each remaining patch, calculate the distance between its representation vector and the aforementioned center vector, and select the remaining patch with the largest distance as the sampling patch. It should be understood that the representation vector of each patch is the same as the representation vector of each patch in the representation set of the target attention layer output obtained in step S401.
[0079] Thus, through the various specific implementation methods described above, the first candidate set J is determined based on the attention variance. div .
[0080] Then, in step S407, based on the aforementioned first alternative J div Determine the set of tiles to be retained J. Generally, the set of tiles to be retained J includes the first candidate set J. div All tiles in the image.
[0081] When attention-based variance-based pruning is combined with other token pruning methods, the aforementioned set of retained tiles may include the union of the first candidate set and the retained tiles selected by other token pruning methods.
[0082] Thus, in step S409, the partial representation vectors corresponding to the above-mentioned reserved patch set can be determined from the representation set output by the target attention layer, and only this part of the representation vectors is further processed. That is to say, for the N visual token representation vectors corresponding to the N patches output by the target attention layer, the patches that do not fall into the reserved patch set are cropped or removed, and only the partial representation vectors corresponding to the patches in the reserved patch set are further processed. Assuming that the number of patches in the reserved patch set is P, where P < N, then through the above process, N - P visual tokens are cropped, and only P visual tokens are processed subsequently.
[0083] Typically, the further processing of the partial representation vectors corresponding to the reserved patch set includes using the network layers subsequent to the target attention layer in the image encoder to further process the partial representation vectors corresponding to the P visual tokens to obtain an image representation. When the target attention layer is the last network layer of the image encoder, the partial representation vectors corresponding to the above-mentioned P visual tokens are used as the image representation output by the image encoder. Then, the image representation is input into the mapping network to obtain a mapping representation mapped to the text semantic space; and then the above mapping representation is input into the large language model for image-text processing. The architectures and data streams of the image encoder, the mapping network, and the large language model are as Figure 1 shown.
[0084] In this way, visual tokens with diversity in different attention heads are determined based on the attention variance, and token pruning is performed based on the diversity of the visual tokens, so as to accelerate the model inference speed while retaining the diverse key tokens and avoiding the deterioration of the inference performance.
[0085] As mentioned above, the above token pruning based on the attention variance can be combined with other token pruning methods. In one implementation, the above other token pruning methods include token pruning based on saliency.
[0086] In this implementation, Figure 4 the method further includes an optional step S402 shown by the dashed line. In this step S402, according to the attention score matrices calculated by each of the H attention heads, the attention mean of each of the N patches is determined; several patches with larger attention means are selected to form the second alternative set J sal .
[0087] In one embodiment, for any patch i, all the attention scores related to the patch i in the H attention matrices calculated by the H attention heads are extracted, and their mean is calculated as the attention mean of the patch i. The attention means of each patch are sorted, and K sal patches with larger attention means are selected from them to form the above-mentioned second alternative set J sal.
[0088] In another embodiment, the aforementioned average attention value is determined based on the attention scores of patch i and the special token CLS. Specifically, for any attention head h, the global attention vector is obtained from its calculated attention score matrix. The global attention vector It can be the row vector corresponding to the special token CLS representing global information, which includes the attention scores between N tiles and the CLS token.
[0089] Then, the mean vector of the H global attention vectors corresponding to the H attention heads is determined, as shown in the following formula (4):
[0090] (4)
[0091] It is understandable that the mean vector The i-th element in the mean vector indicates the average attention score of the i-th token and the CLS token across the H attention heads. The N elements in the mean vector corresponding to the N tiles represent the average attention score of each of the N tiles. Therefore, we can select K tiles with the largest average attention score from the mean vector. sal Each tile (or visual token) forms the second candidate set J mentioned above. sal It is understandable that the above K... sal The number can be preset.
[0092] The patches with higher mean attention values can be called salient patches; the process of forming the second candidate set can be called salient patch selection.
[0093] Saliency-based tile selection and attention variance-based (i.e., diversity) tile selection can be performed in parallel or in any reasonable order. If saliency-based tile selection yields a second candidate set, then during attention variance-based selection, the second candidate set can be eliminated from the N tiles, resulting in a remaining set of tiles. Then, tiles with higher attention variances are selected from the remaining set to form the aforementioned first candidate set. Similarly, if attention variance-based tile selection yields a first candidate set, then saliency-based selection can eliminate the first candidate set from the N tiles, resulting in a remaining set of tiles. Then, tiles with higher mean attention values are selected from the remaining set to form the aforementioned second candidate set.
[0094] In one embodiment, given that a first set of candidates and a second set of candidates are obtained respectively, in step S407, the union of the first set of candidates and the second set of candidates can be used as the set of reserved tiles.
[0095] Of course, in some implementations, attention variance-based token pruning can be combined with more pruning methods or other token pruning methods different from saliency methods. This is not limited here, nor will it be listed one by one.
[0096] According to one implementation, considering that the data processed by the mapping network and the large language model after visual token pruning is still different compared to the case without pruning, the mapping network can be fine-tuned for the token pruning case in order to reduce the impact of this difference on the inference effect.
[0097] Figure 5 A flowchart illustrating a method for fine-tuning a mapping network according to one embodiment is shown. Figure 5 As shown, the fine-tuning process includes the following steps.
[0098] In step S501, the N sample patches corresponding to the sample image are input into the image encoder to obtain a sample representation set composed of the sample representation vectors of each sample patch output by the target attention layer. As mentioned above, the target attention layer includes H attention heads.
[0099] In step S503, for any sample patch, the attention variance of the sample patch is determined based on the variance between the attention scores calculated by the H attention heads for the sample patch.
[0100] In step S505, select multiple sample patches with large attention variance, and determine the set of retained sample patches based on these multiple sample patches.
[0101] For the specific execution methods of steps S501 to S505 above, please refer to [the relevant documentation / reference]. Figure 4 Steps S401 to S407 are not repeated here.
[0102] Next, in step S507, the partial sample representation vectors corresponding to the above-mentioned retained sample patch set are determined from the sample representation set. The sample image representation obtained based on the partial sample representation vectors is input into the mapping network to obtain the sample mapping representation. The sample mapping representation is then input into the large language model to obtain the prediction result.
[0103] In step S509, the parameters of the mapping network are adjusted based on the prediction results. Specifically, in this step, the prediction loss is determined by comparing the prediction results of the large language model with the ground truth results (or labels) corresponding to the sample images. The model parameters are adjusted with the goal of reducing the prediction loss. When adjusting the model parameters, the parameters of the image encoder and the large language model are frozen, and only the parameters of the mapping network are adjusted.
[0104] Since the mapping network has a relatively small number of parameters, by fine-tuning the mapping network, the pruned visual tokens can be mapped to the text embedding space more effectively with a small training cost, so that visual token pruning has a smaller degrading effect on the inference results of large language models.
[0105] According to another embodiment, an apparatus for performing image processing is provided, which can be deployed in any computing device, platform, or device cluster with computing and processing capabilities. Figure 6 A schematic block diagram of the device is shown. (For example...) Figure 6 As shown, the device 600 includes:
[0106] The data acquisition unit 601 is configured to input N patches corresponding to the target image into an image encoder to obtain a representation set composed of the representation vectors of each patch output by the target attention layer, wherein the target attention layer includes H attention heads;
[0107] The variance determination unit 603 is configured to determine the attention variance of any target patch based on the variance between the attention scores calculated by the H attention heads for the target patch respectively.
[0108] The first candidate set determination unit 605 is configured to select multiple tiles with large attention variance and form a first candidate set based on the multiple tiles;
[0109] The tile set determination unit 607 is configured to determine the tile set to be retained based on the first candidate set.
[0110] The tile pruning unit 609 is configured to determine a portion of the representation vectors corresponding to the retained tile set from the representation set, and to perform further processing only on the portion of the representation vectors.
[0111] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this specification, the functions of each module can be implemented in one or more software and / or hardware.
[0112] This specification also provides a computer-readable non-volatile storage medium storing a computer program that, when executed by a processor, can be used to perform one or more steps of one or more methods described or illustrated herein, or to provide the functionality described or illustrated herein. In this document, a computer-readable non-volatile storage medium or medium may include one or more semiconductor or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-volatile storage medium, or any suitable combination thereof where appropriate. A computer-readable non-volatile storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.
[0113] This specification also provides a computer system. Figure 7 An example computer system 1000 is illustrated. In a particular embodiment, one or more computer systems 1000 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 1000 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. The particular embodiments include one or more portions of one or more computer systems 1000. References to computer systems herein may include computing devices, and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.
[0114] This disclosure contemplates any suitable number of computer systems 1000. This disclosure contemplates computer systems 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 can be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a notebook or laptop system, an interactive kiosk, a mainframe, a grid of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or any suitable combination of these. Where appropriate, computer system 1000 can include one or more computer systems 1000; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in the cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 can perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. For example, and not by way of limitation, one or more computer systems 1000 can perform one or more steps of one or more methods described or illustrated herein in real-time or batch mode. One or more computer systems 1000 may perform one or more steps of the methods described or illustrated herein at different times or in different locations, where appropriate.
[0115] In a particular embodiment, computer system 1000 includes processor 1002, memory 1004, storage 1006, input / output (I / O) interface 1008, communication interface 1010, and bus 1012. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
[0116] In certain embodiments, processor 1002 includes hardware for executing instructions, such as those that constitute a computer program. For example, but not limited to, to execute instructions, processor 1002 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1004, or storage 1006; decode and execute them; and then write one or more results to internal registers, internal caches, memory 1004, or storage 1006. In certain embodiments, processor 1002 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates that processor 1002 may include any suitable number of suitable internal caches where appropriate. For example, but not limited to, processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation back buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 1004 or storage 1006, and the instruction cache may speed up the retrieval of these instructions by processor 1002. The data in the data cache may be a copy of data in memory 1004 or storage 1006 for operation by instructions executed at processor 1002; the result of instructions previously executed by processor 1002 for access or writing to memory 1004 or storage 1006 by subsequent instructions executed at processor 1002; or other suitable data. The data cache can speed up read or write operations on processor 1002. The TLB can speed up virtual address translation on processor 1002. In a particular embodiment, processor 1002 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates that processor 1002 may include any suitable number of suitable internal registers where appropriate. Where appropriate, processor 1002 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1002. Although this disclosure describes and illustrates specific processors, this disclosure contemplates any suitable processor.
[0117] In a particular embodiment, memory 1004 includes main memory for storing instructions to be executed by processor 1002 or data to be operated on by processor 1002. For example, but not limited to, computer system 1000 may load instructions into memory 1004 from storage 1006 or other sources (e.g., another computer system 1000). Processor 1002 may then load instructions from memory 1004 into internal registers or internal caches. To execute instructions, processor 1002 may retrieve and decode instructions from internal registers or internal caches. During or after instruction execution, processor 1002 may write one or more results (potentially intermediate or final results) to internal registers or internal caches. Processor 1002 may then write one or more of these results into memory 1004. In a particular embodiment, processor 1002 executes instructions only in one or more internal registers or internal caches or memory 1004 (not in storage 1006 or elsewhere), and operates on data only in one or more internal registers or internal caches or memory 1004 (not in storage 1006 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 1002 to memory 1004. Bus 1012 may include one or more memory buses as described below. In a particular embodiment, one or more memory management units (MMUs) are located between processor 1002 and memory 1004 and facilitate access to memory 1004 requested by processor 1002. In a particular embodiment, memory 1004 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, this RAM may be single-port or multi-port RAM. This disclosure contemplates any suitable RAM. Memory 1004 may include one or more memory modules, where appropriate. Although this disclosure describes and illustrates specific memory, this disclosure contemplates any suitable memory.
[0118] In certain embodiments, storage 1006 includes mass storage for data or instructions. For example, but not limited to, storage 1006 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more. Where appropriate, storage 1006 may include removable or non-removable (or fixed) media. Where appropriate, storage 1006 may be internal or external to computer system 1000. In certain embodiments, storage 1006 is a non-volatile solid-state memory. In certain embodiments, storage 1006 includes read-only memory (ROM). Where appropriate, this ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically changeable ROM (EAROM), or flash memory, or a combination of two or more. This disclosure assumes that storage 1006 may take any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units to facilitate communication between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more storage units 1006. Although this disclosure describes and illustrates specific storage units, this disclosure considers any suitable storage unit.
[0119] In a particular embodiment, I / O interface 1008 includes hardware, software, or both, providing one or more interfaces between computer system 1000 and one or more I / O devices. Computer system 1000 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices enable communication between a person and computer system 1000. For example, but not limited to, I / O devices may include a keyboard, keypad, microphone, display, mouse, printer, scanner, speaker, still camera, stylus, tablet computer, touchscreen, trackball, camera, other suitable I / O devices, or combinations thereof. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 1008. Where appropriate, I / O interface 1008 may include one or more device or software drivers enabling processor 1002 to drive one or more of these I / O devices. I / O interface 1008 may include one or more I / O interfaces 1008, where appropriate. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure considers any suitable I / O interface.
[0120] In a particular embodiment, communication interface 1010 includes hardware, software, or both (e.g., packet-based communication) that provides one or more communication interfaces between computer system 1000 and one or more other computer systems 1000 or one or more networks. For example, but not limited to, communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 1010. For example, but not limited to, computer system 1000 may communicate with one or more portions of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of both. One or more portions of these networks may be wired or wireless. For example, computer system 1000 may communicate with a wireless personal area network (WPAN) (e.g., Bluetooth WPAN), a Wi-Fi network, a Wi-Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of both. Where appropriate, computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, communication interface 1010 may include one or more communication interfaces 1010. Although specific communication interfaces are described and illustrated in this disclosure, any suitable communication interface is contemplated in this disclosure.
[0121] In a particular embodiment, bus 1012 includes hardware, software, or both that interconnect components of computer system 1000. For example, but not limited to, bus 1012 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations thereof. Where appropriate, bus 1012 may include one or more buses 1012. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.
[0122] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0123] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0124] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.
Claims
1. An image processing method, comprising: The N patches corresponding to the target image are input into the image encoder to obtain the representation set composed of the representation vectors of each patch output by the target attention layer. The target attention layer includes H attention heads. For any target patch, the attention variance of the target patch is determined based on the variance between the attention scores calculated by the H attention heads for the target patch respectively. Select multiple tiles with large attention variance, and form a first candidate set based on the multiple tiles; The set of tiles to be retained is determined based on the first candidate set; From the representation set, determine the partial representation vectors corresponding to the retained tile set, and only perform further processing on the partial representation vectors.
2. The method according to claim 1, further comprising: Based on the attention score matrix calculated by each of the H attention heads, determine the mean attention value of each of the N patches; Select several patches with larger mean attention values to form a second candidate set; The set of tiles to be retained is determined based on the first candidate set, specifically including: The union of the first and second candidate sets is determined as the set of retained tiles.
3. The method according to claim 2, wherein, Based on the attention score matrices calculated by the H attention heads, the mean attention value of each of the N patches is determined, including: For each attention head, a global attention vector is obtained from its calculated attention score matrix. The global attention vector includes the attention scores between the N tiles and the special tokens representing global information. Determine the mean vector of the H global attention vectors corresponding to the H attention heads; this includes the mean attention value of each of the N tiles.
4. The method according to claim 2, wherein, Select multiple tiles with large attention variance, including: The second candidate set is excluded from the N tiles to obtain the remaining tile set; Select several tiles with large attention variance from the remaining tile set.
5. The method according to claim 1, wherein, The attention variance of the target patch is determined based on the variance between the attention scores calculated by the H attention heads for the target patch, including: For each attention head, obtain the attention score between the target patch calculated by that attention head and the special token representing global information, thereby obtaining H attention scores; Calculate the variance among the H attention scores, and use it as the attention variance of the target patch.
6. The method according to claim 1, wherein, A first candidate set is formed based on the multiple tiles, including: Select a first number of tiles from the tile pool consisting of the plurality of tiles and move them into the initial selection set; Perform several rounds of sampling until the number of tiles in the selection set reaches a preset second number, thus obtaining a first candidate set; wherein each round of sampling includes selecting a sampled tile from the current tile pool and moving it into the current selection set, wherein the sampled tile is the tile in the current tile pool whose representation vector is farthest from the current selection set.
7. The method according to claim 6, wherein, Select sampled tiles from the tile pool, including: Calculate the mean of the representation vectors of each tile in the current selection set, and use it as the center vector; For each remaining tile in the current tile pool, calculate the distance between its representation vector and the center vector, and select the remaining tile with the largest distance as the sampling tile.
8. The method according to claim 1, wherein, The further processing includes: The image representation obtained based on the partial representation vectors is input into the mapping network to obtain the mapping representation mapped to the text semantic space; The mapping representation is then input into the large language model.
9. The method according to claim 8, wherein, The mapping network is obtained through fine-tuning the following steps: Input the N sample patches corresponding to the sample image into the image encoder to obtain the sample representation set composed of the sample representation vectors of each sample patch output by the target attention layer. For any sample patch, the attention variance of the sample patch is determined based on the variance between the attention scores calculated by the H attention heads for that sample patch. Select multiple sample patches with large attention variance, and determine a set of retained sample patches based on these multiple sample patches; Determine the partial sample representation vectors corresponding to the retained sample tile set from the sample representation set; input the sample image representations obtained based on the partial sample representation vectors into the mapping network to obtain sample mapping representations; input the sample mapping representations into the large language model to obtain prediction results; The parameters of the mapping network are adjusted based on the prediction results.
10. The method according to claim 1, wherein, The image encoder is ViT, and the target attention layer is the penultimate attention layer.
11. An apparatus for performing image processing, comprising: The data acquisition unit is configured to input N patches corresponding to the target image into an image encoder to obtain a representation set composed of the representation vectors of each patch output by the target attention layer, wherein the target attention layer includes H attention heads; The variance determination unit is configured to determine the attention variance of any target patch based on the variance between the attention scores calculated by the H attention heads for the target patch respectively. The first candidate set determination unit is configured to select multiple tiles with large attention variance and form a first candidate set based on the multiple tiles; The unit for determining the set of retained tiles is configured to determine the set of retained tiles based on the first candidate set. The tile pruning unit is configured to determine a portion of the representation vectors corresponding to the retained tile set from the representation set, and to perform further processing only on the portion of the representation vectors.
12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-10.
13. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-10.