Visual token compression method and device and storage medium

Abstract

The invention provides a visual token compression method and device and a storage medium, and belongs to the technical field of data processing. The method comprises the steps of obtaining all visual tokens and self-attention matrixes of a decoder of a large visual language model in a target network layer in a process of processing a target image by the large visual language model; screening out all key visual tokens and all non-key visual tokens according to the self-attention matrix; and determining compensation information corresponding to each key visual token according to all the non-key visual tokens, and correcting each key visual token according to the compensation information corresponding to each key visual token to obtain each compressed visual token as a visual token input by a next network layer of the target network layer. The invention aims to improve the reasoning efficiency of the large visual language model.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of data processing, and in particular, to a visual token compression method, device, and storage medium. Background Art

[0002] Based on the large language model, the large visual language model (LVLM) has made revolutionary progress by integrating visual and textual modalities and making full use of the powerful text understanding ability of the large language model. Current research mainly uses sequential visual representations. Visual signals such as images or videos are first divided into blocks and then encoded into a series of tokens. These tokens will be projected into the space where the text representation is located. By using technologies such as visual instruction fine-tuning, complex multimodal tasks including image description, visual question answering and video positioning can be completed.

[0003] However, visual tokens usually contribute less to the final prediction results than text tokens, but the number of visual tokens far exceeds that of text tokens. Especially in high-resolution images and videos, the quadratic complexity of the attention mechanism of the large visual language model will lead to huge computational overhead, which significantly slows down the reasoning process and leads to low reasoning efficiency of the large visual language model. Summary of the Invention

[0004] Embodiments of the present application provide a visual token compression method, device, and storage medium, aiming to improve the reasoning efficiency of large visual language models.

[0005] In a first aspect, an embodiment of the present application provides a visual token compression method, the method comprising: During the process of the large visual language model processing the target image, obtaining all visual tokens and self-attention matrices of the decoder of the large visual language model at the target network layer; According to the self-attention matrix, all key visual tokens and all non-key visual tokens are filtered out; Based on all non-critical visual tokens, the compensation information corresponding to each critical visual token is determined, and based on the compensation information corresponding to each critical visual token, each critical visual token is corrected to obtain the compressed visual tokens as the visual tokens input to the next network layer of the target network layer.

[0006] Optionally, all key visual tokens and all non-key visual tokens are screened out according to the self-attention matrix, including: selecting a base visual token among all the visual tokens; Determine, according to the self-attention matrix, the attention weight of each visual token to the benchmark visual token in each attention head, and calculate the average attention score corresponding to each visual token; All the visual tokens are sorted in descending order of the average attention scores, the first τ% of the visual tokens are taken as key visual tokens, and the remaining visual tokens are taken as non-key visual tokens.

[0007] Optionally, calculating the average attention score corresponding to each visual token includes: For any visual token, the average of the attention weights of the visual token to the benchmark visual token in all attention heads is calculated as the average attention score of the visual token.

[0008] Optionally, the formula for calculating the average attention score corresponding to each visual token is:

[0009] in, For the t The average attention score of visual tokens; H is the number of attention heads; For the h In the attention head, t The attention weights of the visual tokens to the reference visual token.

[0010] Optionally, determining compensation information corresponding to each key visual token based on all non-key visual tokens, and correcting each key visual token based on the compensation information corresponding to each key visual token to obtain compressed visual tokens as visual tokens input to the next network layer of the target network layer, including: Performing average pooling on all the non-critical visual tokens to obtain pooled visual tokens; Concatenate each key visual token with the pooled visual token to obtain a target visual token corresponding to each key visual token; Modeling the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determining the compensation information of each target visual token as the compensation information of the key visual token corresponding to the target visual token; Each key visual token is added to its corresponding compensation information to obtain compressed visual tokens as visual tokens input to the next network layer of the target network layer.

[0011] Optionally, the posterior distribution of the compensation information corresponding to each target visual token is d dimensional multivariate Gaussian distribution, modeling the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determining the compensation information of each target visual token, including: The shallow variational encoder predicts the corresponding d The mean and variance of the dimensional multivariate Gaussian distribution are obtained, and the mean is used as the compensation information corresponding to each target visual token.

[0012] Optionally, the method further includes: Determining a target loss function of the large visual language model during a training process according to the original loss function and the KL divergence loss function of the large visual language model; The KL divergence loss function is used to improve the degree of decoupling of the internal dimensions of each compressed visual token.

[0013] Optionally, the objective loss function is:

[0014] in, Loss is the target loss function, is the original loss function; is the KL divergence loss function; is the weight of the KL divergence loss function; For the t target visual tokens; For the t Compensation information for target visual tokens; For the t The posterior distribution of compensation information corresponding to the target visual tokens; For the t The prior distribution of compensation information for each target visual token.

[0015] In a second aspect, an embodiment of the present application provides an electronic device comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the visual token compression method described in the first aspect of the embodiment is executed.

[0016] In a third aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the visual token compression method described in the first aspect of the embodiment is executed.

[0017] Beneficial effects: This method can obtain all visual tokens and self-attention matrices of the decoder of the large visual language model in the target network layer during the process of the large visual language model processing the target image, and then screen out all key visual tokens and all non-key visual tokens based on the self-attention matrix, and determine the compensation information corresponding to each key visual token based on all non-key visual tokens, and then correct each key visual token based on the compensation information corresponding to each key visual token to obtain the compressed visual tokens as the visual tokens input to the next network layer of the target network layer.

[0018] By screening out key visual tokens from all visual tokens and then compensating the detailed visual information of each key visual token with non-key visual tokens, the number of visual tokens can be significantly reduced without losing important and detailed visual information of the target image, thereby reducing the inference overhead of the large visual language model in the visual modality and improving the overall inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a flowchart of the steps of the visual token compression method proposed in one embodiment of the present application; Figure 2 Schematic diagram of the attention distribution of visual tokens in different layers of the decoder of a large visual language model; Figure 3 is a schematic diagram of a visual token compression module provided in one embodiment of the present application; Figure 4 This is a schematic diagram of the training process after loading the visual token compression module proposed in one embodiment of the present application; Figure 5 This is a functional module diagram of a visual token compression device proposed in one embodiment of the present application; Figure 6 is a schematic diagram of an electronic device provided in one embodiment of the present application; Figure 7 is a schematic diagram of a non-volatile readable storage medium proposed in an embodiment of the present application; Figure 8 It is a schematic diagram of a computer program product proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] LLM: Large Language Model, large language model; LVLM: Large Vision Language Model.

[0023] Building on the Large Language Model (LLM), the Large Vision-Language Model (LVLM) has made revolutionary progress by integrating visual and textual modalities and leveraging the powerful text understanding capabilities of the LLM.

[0024] Current research mainly uses sequential visual representation, in which visual signals such as images or videos are first divided into blocks and then encoded into a series of tokens. These tokens will be projected into the space where the text representation is located. By using technologies such as visual instruction fine-tuning, LVLM can complete complex multimodal tasks, including image description, visual question answering, and video positioning.

[0025] However, the number of visual tokens far exceeds that of textual tokens, especially in high-resolution images and videos. The quadratic complexity of the attention mechanism brings huge computational overhead. In addition, the attention scores obtained by visual tokens are also lower than those of textual tokens, so the contribution of visual tokens to the final prediction results is also smaller than that of textual tokens. Moreover, the current LVLM is inefficient in processing visual signals because it does not process the visual signal as a whole, but only focuses on certain sub-regions. The tokens of these sub-regions will be aggregated together in the decoder layer and further processed to a higher level of abstraction that the LVLM can understand.

[0026] Therefore, most visual tokens contribute little to LLM understanding, but the large number of visual tokens significantly slows down the reasoning process, resulting in low reasoning efficiency for large visual-language models.

[0027] Therefore, an embodiment of the present application provides a visual token compression method that can compress visual tokens while retaining important information and details, thereby improving the reasoning efficiency and generalization ability of large visual language models.

[0028] Reference Figure 1 , shows a flowchart of the steps of a visual token compression method in an embodiment of the present application, and the method may specifically include the following steps: S101: During the process of processing a target image with a large visual language model, all visual tokens and self-attention matrices of a decoder of the large visual language model at a target network layer are obtained.

[0029] Specifically, the big visual language model can perform multimodal tasks. For example, the input of the big visual language model includes text input and visual input such as images or videos. For example, if a target image is input and a text instruction is input: "Describe this picture", the big visual language model can generate output results based on the target image and text instruction.

[0030] In the process of large visual language model processing target images, there are a large number of visual tokens, but the contribution levels of different visual tokens are different. In this embodiment, the attention that can be contributed by visual tokens in the self-attention mechanism of the large visual language model is used as an indicator to measure the importance of a visual token. The higher the contributed attention, the greater the influence of the visual token in the subsequent forward calculation process.

[0031] Reference Figure 2 , a schematic diagram of the attention distribution of visual tokens in different layers of the decoder of the large visual language model is shown. After the initial layer of the decoder of the large visual language model, that is, the second layer, the LVLM begins to pay attention to certain sub-areas of visual information, that is, as the number of network layers of the decoder increases, the average contribution attention distribution of a visual token will gradually change from a dispersed state to a concentrated state, which indicates that the LLM selectively focuses on some key visual tokens in the deep network. Therefore, it can screen out key visual tokens and reduce non-key visual tokens, thereby achieving the effect of compressing visual tokens to improve the computational efficiency of the model.

[0032] In the actual implementation process, all visual tokens and self-attention matrices of the decoder of the large visual language model in the target network layer can be obtained to compress the visual tokens and determine the input visual tokens of the next layer of the target network layer.

[0033] During actual implementation, the target network layer may be selected according to actual application requirements, and this embodiment does not impose any limitation thereto. In this embodiment, the target network layer is recorded as the k-1 layer.

[0034] Specifically, in the k-1 layer of the decoder of the large visual language model, the set of all tokens is ,in, Represents a collection of system tokens, represents a collection of visual tokens, represents a collection of text tokens, d is the hidden layer dimension, is the number of system tokens in the system token set; is the number of visual tokens in the visual token set; is the number of text tokens in the text token set.

[0035] In order to avoid the additional computational overhead of the visual token compression process, this embodiment uses the self-attention matrix of the k-1th layer Compression of visual tokens is performed, where Indicates the total number of tokens, H is the number of attention heads.

[0036] S102: Filter out all key visual tokens and all non-key visual tokens according to the self-attention matrix.

[0037] Specifically, in order to quantify the attention that a visual token can contribute to the self-attention mechanism of the large visual language model, we can randomly select a visual token as the benchmark visual token, and then determine the attention weight of a visual token to the benchmark visual token in different attention heads in the self-attention matrix of the k-1 layer, and then calculate the average attention score of the visual token, and classify all the visual tokens of the k-1 layer by the average attention score.

[0038] Specifically, according to the self-attention matrix, screening out all key visual tokens and all non-key visual tokens may include the following steps: A1: Select a base visual token from among all the visual tokens.

[0039] In actual implementation, a visual token may be randomly selected from all visual tokens in the k-1 layer as a benchmark visual token.

[0040] For example, the first visual token among all visual tokens can be selected as the base visual token, or the last visual token can be selected as the base visual token. This can be set according to actual application requirements and is not limited in this embodiment.

[0041] A2: According to the self-attention matrix, determine the attention weight of each visual token to the benchmark visual token in each attention head, and calculate the average attention score corresponding to each visual token.

[0042] Specifically, in the self-attention matrix of the k-1 layer, the attention weights of each visual token to the benchmark visual token in each attention head can be obtained, and then for any visual token, the average value of the attention weights of the visual token to the benchmark visual token in all attention heads is calculated as the average attention score of the visual token.

[0043] For example, the formula for calculating the average attention score corresponding to each visual token is:

[0044] in, For the t The average attention score of visual tokens; H is the number of attention heads; For the h In the attention head, t The attention weights of the visual tokens to the reference visual token.

[0045] A3: Sort all the visual tokens in descending order of the average attention scores, take the first τ% of the visual tokens as key visual tokens, and take the remaining visual tokens as non-key visual tokens.

[0046] After sorting by the average attention score from high to low, the first τ% of visual tokens are taken as key visual tokens ,in , the remaining visual tokens, i.e. 1-τ% of the visual tokens, are used as non-critical visual tokens .

[0047] The value of τ is related to the required compression rate. When τ is smaller, the compression rate is greater and the number of visual tokens after compression is smaller; when τ is larger, the compression rate is smaller and the number of visual tokens after compression is larger. The value of τ can be set according to the needs of actual application, and this embodiment does not impose any restrictions.

[0048] S103: Determine the compensation information corresponding to each key visual token based on all non-key visual tokens, and correct each key visual token based on the compensation information corresponding to each key visual token to obtain compressed visual tokens as visual tokens input to the next network layer of the target network layer.

[0049] In this embodiment, after dividing each visual token into key visual tokens and non-key visual tokens according to the degree of attention contribution of each visual token in the self-attention mechanism, although the key visual token contains the main visual information of the target image, there is still an information gap between the key visual token and the original visual information of the target image. If the non-key visual token is directly discarded and the existing information gap is ignored, it will cause the loss of some detailed visual information of the target image, thereby affecting the fine-grained understanding ability of the large visual language model.

[0050] Therefore, in order to compress visual tokens without losing important visual information and some detailed visual information of the target image, non-critical visual tokens will not be discarded directly. Decoupling coding technology can be used to make up for and narrow the information gap between critical visual tokens and the target image based on non-critical visual tokens.

[0051] Specifically, determining compensation information corresponding to each key visual token based on all non-key visual tokens includes the following steps: B1: Perform average pooling on all the non-critical visual tokens to obtain pooled visual tokens.

[0052] Since non-critical visual tokens usually carry low information density, average pooling can be used to compress all non-critical visual tokens to obtain a compact representation of all non-critical visual tokens. ,Right now:

[0053] By performing average pooling on all non-critical visual tokens to obtain pooled visual tokens, it is possible to retain the potentially useful visual information of all non-critical visual tokens while reducing the computational and memory cost overhead of the compression process, thereby improving the efficiency of the visual token compression process.

[0054] B2: Concatenate each of the key visual tokens with the pooled visual token to obtain a target visual token corresponding to each of the key visual tokens.

[0055] Specifically, for any key visual token , which is first combined with the pooled visual tokens Splice and get a target visual token , whose dimensions range from d Expand to 2 d .

[0056] B3: Modeling the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determining the compensation information of each target visual token as the compensation information of the key visual token corresponding to the target visual token.

[0057] Specifically, for any target visual token , a shallow variational encoder can be used To model the posterior distribution of its compensation information ,in, express Corresponding compensation information, are learnable parameters.

[0058] Assume that the compensation information corresponding to any target visual token The posterior distribution of obey d dimensional multivariate Gaussian distribution, via a shallow variational encoder Can predict the posterior distribution The mean and variance , and sample from this distribution to obtain compensation information .

[0059] Since this multivariate Gaussian distribution is not differentiable, we can use the reparameterization technique to introduce a random variable ,pass To achieve a differentiable sampling process.

[0060] In order to maintain stability during the inference phase, the predicted mean can be used directly As a target visual token Compensation information .

[0061] The whole process can be formally expressed as:

[0062] in, 、 .

[0063] By modeling the posterior distribution rather than a single value, the robustness of the large visual language model can be effectively improved, enabling it to have better generalization capabilities when facing unknown data. In particular, when the input visual signal such as an image or video is interfered with by noise, the large visual language model can still maintain good performance.

[0064] And because the key visual token and pooled visual tokens processed by pooling After concatenation, it is used as the input of the shallow variational encoder and focuses on capturing key visual tokens The information gap between the visual information contained in the target image and the original visual information of the target image is obtained, so the compensation information is determined by the way of modeling the posterior distribution. Ability to efficiently extract pooled visual tokens Central and Key Visual Tokens The most relevant visual information.

[0065] Identify target visual tokens Compensation information After that, the compensation information As a target visual token Corresponding key visual tokens The compensation information is then used for the key visual token Make corrections, for example, to compensate for the information With key visual tokens Add together to get the compressed visual token .

[0066] Furthermore, the method also includes determining a target loss function of the large visual language model during the training process based on the original loss function and the KL (Kullback-Leibler) divergence loss function of the large visual language model; the KL divergence loss function is used to improve the degree of decoupling of the internal dimensions of each compressed visual token.

[0067] Specifically, the KL divergence loss function is:

[0068] in, is a prior distribution that satisfies:

[0069] That is, the prior distribution The different dimensions are independent of each other.

[0070] By optimizing the KL divergence loss function, the posterior distribution can be narrowed to the prior distribution distance, thereby enhancing the compensation information The degree of decoupling of various internal dimensions.

[0071] In practical applications, shallow variational encoders A two-layer perceptron structure can be used, and is a multivariate independent standard Gaussian distribution.

[0072] The KL divergence loss function is then added to the original loss function of the large visual language model to determine the target loss function of the large visual language model during the training process.

[0073] And we can also introduce a hyperparameter To control the weight of the KL divergence loss function, for example, the target loss function is:

[0074] During the training phase of the large visual language model, by training with a target loss function superimposed with the KL divergence loss function, the large visual language model can learn the decoupled compressed visual tokens and reconstruct the original visual information of the target image on the compressed visual tokens to the greatest extent possible. This can reduce the number of visual tokens to improve the model's reasoning efficiency while minimizing the information loss during the visual token compression process.

[0075] During the actual implementation process, a separate visual token compression module can be set up to execute the visual token compression method provided by this embodiment. When it is necessary to improve the reasoning efficiency of any large visual language model, the visual token compression module is loaded on the large visual language model, and then the two are jointly fine-tuned, and the large visual language model is trained based on the target loss function, so that the large visual language model can learn and understand the representation of the compressed visual tokens, so that the reasoning efficiency of the large visual language model can be improved and the reasoning cost can be reduced by calling the visual token compression module in the subsequent multimodal task processing process.

[0076] Reference Figure 3 , shows a schematic diagram of the visual token compression module provided in an embodiment of the present application. The visual token compression module mainly includes two processes: dynamic visual token screening and decoupling compression.

[0077] In dynamic visual token screening, the input token of the k-1th layer of the decoder of the large visual language model is the set of all tokens , To include A system token collection of system tokens, To include A visual token collection of visual tokens, To include A text token collection of text tokens.

[0078] Then, according to the self-attention matrix of the k-1th layer, the average attention score corresponding to each visual token is calculated respectively, and then the visual tokens with the highest average attention score are sorted from high to low, and the first τ% of the visual tokens are taken as the key visual tokens. , the number of key visual tokens is ; The remaining visual tokens are used as non-key visual tokens , the number of non-critical visual tokens is .

[0079] Then, all non-key visual tokens are average pooled to obtain the pooled visual token Then it is spliced with each key visual token to obtain a dimension of 2 d Visual tokens for each target .

[0080] Then, the compensation information corresponding to each target visual token is modeled through a shallow variational encoder The posterior distribution of Can predict the posterior distribution The mean and variance , the mean As a target visual token and key visual tokens Compensation information .

[0081] Then, the compensation information With key visual tokens Add together to get the compressed visual token .

[0082] The compressed visual token is used as the input visual token of the k-th layer, that is, the compressed visual token, the text token set and the system token set constitute the compressed input token of the k-th layer.

[0083] Reference Figure 4 , shows a schematic diagram of the training process after loading the visual token compression module provided by the embodiment of the present application. When the large visual language model inputs the target image and text instructions, the image encoder in the multimodal encoder of the large visual language model generates a set of visual tokens corresponding to the target image. ,include visual tokens, the text encoder generates a set of text tokens corresponding to the text instructions ,include text tokens, plus a set of system tokens , get the set of all tokens , including the number of all tokens .

[0084] During the forward propagation of the large visual language model, the visual token compression module located between the k-1th layer and the kth layer of the decoder will compress the visual token set Compression is performed, and the compressed visual tokens, text token sets and system token sets are combined into the input tokens corresponding to the kth layer. After obtaining the output tokens, fine-tuning is performed through visual instructions and target loss functions to train the large visual language model.

[0085] For example, for the model LLaVA-v1.5, the visual token compression module and the base Vicuna large model provided in this embodiment are fine-tuned at the same time through visual instruction fine-tuning technology. The Vicuna large model can be fine-tuned using LoRA (low-rank matrix decomposition) and then trained based on the target loss function provided in this embodiment.

[0086] The compression rate is controlled by controlling the hyperparameter τ∈(0,1). A larger τ means a smaller compression rate, which allows the model to maintain most of its performance, but at the cost of saving less computational overhead; a smaller τ means a higher compression rate, which can greatly improve the model's inference performance.

[0087] After training a large visual language model with the visual token compression module provided by this embodiment loaded between the first and second layers of the decoder, the model input includes the target image and the text instruction: "Describe this image". The results of experiments with different values of τ are as follows: τ is set to 1, which means that visual tokens are not compressed and the number of visual tokens is 576; When τ is set to 0.16, the number of visual tokens is compressed to 96; When τ is set to 0.11, the number of visual tokens is compressed to 64; When τ is 1, 0.16, and 0.11, the output results of the model are slightly different. The smaller it is, the faster the model outputs results.

[0088] The visual token compression method provided in this embodiment is mainly divided into two parts: key visual token screening and information loss decoupling compression. The visual tokens are then sorted according to their average attention scores, and the top τ% of visual tokens are selected as the most important key visual tokens. However, the key visual tokens only contain the most important part of the original visual information. Furthermore, by utilizing the remaining 1-τ% of non-key visual tokens, a shallow neural network, namely a shallow variational encoder, is used to make up for and narrow the information gap between the key visual tokens and the original target image, and compression is performed in a decoupled variational manner.

[0089] Then, the KL divergence loss function is further adopted to ensure the degree of decoupling between the internal dimensions of the predicted compensation information, thereby ensuring that the compensation information retains most of the original information. It can significantly reduce the number of visual tokens without losing important information and details in the target image, and improve the generalization ability of the large visual language model after compressing the visual tokens through the design of decoupled variation.

[0090] Specifically, this method has at least the following beneficial effects: 1. Adaptively select the top τ% of key visual tokens and compress the remaining non-key tokens in a decoupled and variational manner, reducing the number of visual tokens in the inference process of large visual language models, improving inference efficiency and reducing inference cost.

[0091] 2. While compressing visual tokens, the original visual information in the target image is retained as much as possible, reducing the impact on the fine-grained understanding ability of the large visual language model. The number of visual tokens is reduced without losing important and detailed visual information, and the generalization ability of the LVLM after compressing visual tokens is improved through the design of decoupled variation.

[0092] 3. It does not require a large number of changes to the large visual language model and can be easily applied to the currently common large visual language models, with high applicability.

[0093] Reference Figure 5 , shows a functional module diagram of a visual token compression device provided by an embodiment of the present application, the device comprising: An acquisition module 100 acquires all visual tokens and self-attention matrices of a decoder of the large visual language model at a target network layer during processing of the target image by the large visual language model; A screening module 200 is configured to screen out all key visual tokens and all non-key visual tokens according to the self-attention matrix; The compensation module 300 is used to determine the compensation information corresponding to each key visual token based on all non-key visual tokens, and to correct each key visual token based on the compensation information corresponding to each key visual token, so as to obtain each compressed visual token as the visual token input to the next network layer of the target network layer.

[0094] Optionally, the screening module includes: a selection unit, configured to select a reference visual token from among all the visual tokens; a calculation unit, configured to determine, based on the self-attention matrix, the attention weight of each visual token to the reference visual token in each attention head, and calculate an average attention score corresponding to each visual token; A screening unit is used to sort all the visual tokens in descending order of the average attention scores, take the first τ% of the visual tokens as key visual tokens, and take the remaining visual tokens as non-key visual tokens.

[0095] Optionally, the computing unit is configured to: For any visual token, the average of the attention weights of the visual token to the benchmark visual token in all attention heads is calculated as the average attention score of the visual token.

[0096] Optionally, the formula for calculating the average attention score corresponding to each visual token is:

[0097] in, For the t The average attention score of visual tokens; H is the number of attention heads; For the h In the attention head, t The attention weights of the visual tokens to the reference visual token.

[0098] Optionally, the compensation module includes: a pooling unit, configured to perform average pooling on all the non-key visual tokens to obtain a pooled visual token; a splicing unit, configured to splice each of the key visual tokens with the pooled visual token to obtain a target visual token corresponding to each of the key visual tokens; A compensation unit, configured to model the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determine the compensation information of each target visual token as the compensation information of the key visual token corresponding to the target visual token; The correction unit is configured to add the respective key visual tokens to the corresponding compensation information to obtain compressed visual tokens as visual tokens input to the next network layer of the target network layer.

[0099] Optionally, the posterior distribution of the compensation information corresponding to each target visual token is d dimensional multivariate Gaussian distribution, the compensation unit is used for: The shallow variational encoder predicts the corresponding d The mean and variance of the dimensional multivariate Gaussian distribution are obtained, and the mean is used as the compensation information corresponding to each target visual token.

[0100] Optionally, the device further comprises: a loss determination module, configured to determine a target loss function of the large visual language model during the training process based on the original loss function and the KL divergence loss function of the large visual language model; The KL divergence loss function is used to improve the degree of decoupling of the internal dimensions of each compressed visual token.

[0101] Optionally, the objective loss function is:

[0102] in, Loss is the target loss function, is the original loss function; is the KL divergence loss function; is the weight of the KL divergence loss function; For the t target visual tokens; For the t Compensation information for target visual tokens; For the t The posterior distribution of compensation information corresponding to the target visual tokens; For thet The prior distribution of compensation information for each target visual token.

[0103] Reference Figure 6 , shows a schematic diagram of an electronic device provided in an embodiment of the present application, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the visual token compression method described in the embodiment when executing the computer program.

[0104] Reference Figure 7 , shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, executes the visual token compression method described in the embodiment.

[0105] Reference Figure 8 , shows a schematic diagram of a computer program product provided in an embodiment of the present application, including a computer program / instruction, which, when executed by a processor, implements the visual token compression method described in the embodiment.

[0106] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0107] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0112] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0113] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A visual token compression method, characterized in that: The method comprises: During the process of the large visual language model processing the target image, obtaining all visual tokens and self-attention matrices of the decoder of the large visual language model at the target network layer; According to the self-attention matrix, all key visual tokens and all non-key visual tokens are filtered out; Based on all non-critical visual tokens, the compensation information corresponding to each critical visual token is determined, and based on the compensation information corresponding to each critical visual token, each critical visual token is corrected to obtain the compressed visual tokens as the visual tokens input to the next network layer of the target network layer.

2. The method according to claim 1, characterized in that According to the self-attention matrix, all key visual tokens and all non-key visual tokens are filtered out, including: selecting a base visual token among all the visual tokens; Determine, according to the self-attention matrix, the attention weight of each visual token to the benchmark visual token in each attention head, and calculate the average attention score corresponding to each visual token; All the visual tokens are sorted in descending order of the average attention scores, the first τ% of the visual tokens are taken as key visual tokens, and the remaining visual tokens are taken as non-key visual tokens.

3. The method according to claim 2, characterized in that Calculating the average attention score corresponding to each visual token, including: For any visual token, the average of the attention weights of the visual token to the benchmark visual token in all attention heads is calculated as the average attention score of the visual token.

4. The method according to claim 3, characterized in that The formula for calculating the average attention score corresponding to each visual token is: in, For the t The average attention score of visual tokens; H is the number of attention heads; For the h In the attention head, t The attention weights of the visual tokens to the reference visual token.

5. The method according to claim 1, wherein Determining compensation information corresponding to each key visual token based on all non-key visual tokens, and correcting each key visual token based on the compensation information corresponding to each key visual token to obtain compressed visual tokens as visual tokens input to a next network layer of the target network layer, including: Performing average pooling on all the non-critical visual tokens to obtain pooled visual tokens; Concatenate each key visual token with the pooled visual token to obtain a target visual token corresponding to each key visual token; Modeling the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determining the compensation information of each target visual token as the compensation information of the key visual token corresponding to the target visual token; Each key visual token is added to its corresponding compensation information to obtain compressed visual tokens as visual tokens input to the next network layer of the target network layer.

6. The method according to claim 5, characterized in that The posterior distribution of the compensation information corresponding to each target visual token is: d dimensional multivariate Gaussian distribution, modeling the posterior distribution of compensation information corresponding to each target visual token through a shallow variational encoder, and determining the compensation information of each target visual token, including: The shallow variational encoder predicts the corresponding d The mean and variance of the dimensional multivariate Gaussian distribution are obtained, and the mean is used as the compensation information corresponding to each target visual token.

7. The method according to claim 1, characterized in that The method further comprises: Determining a target loss function of the large visual language model during a training process according to the original loss function and the KL divergence loss function of the large visual language model; The KL divergence loss function is used to improve the degree of decoupling of the internal dimensions of each compressed visual token.

8. The method according to claim 7, characterized in that The objective loss function is: in, Loss is the target loss function, is the original loss function; is the KL divergence loss function; is the weight of the KL divergence loss function; For the t target visual tokens; For the t Compensation information for target visual tokens; For the t The posterior distribution of compensation information corresponding to the target visual tokens; For the t The prior distribution of compensation information for each target visual token.

9. An electronic device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the visual token compression method according to any one of claims 1 to 8 when executing the computer program.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the visual token compression method according to any one of claims 1 to 8.

Images