Data processing method and device based on visual language model and computer device

Abstract

The present disclosure provides a data processing method and device based on a visual language model and a computer device, relating to the field of artificial intelligence, and the implementation scheme is as follows: obtaining information to be processed provided by a user to the visual language model, the information to be processed including at least one of an image or text; tokenizing the information to be processed to obtain a token stream including a plurality of tokens; generating a plurality of activation matrices corresponding to the plurality of tokens based on the token stream; performing sparse processing on at least one of a query tensor, a key tensor and a value tensor in each activation matrix; storing the sparse key tensor and the value tensor in a cache; and when performing attention calculation in the inference process of the visual language model, performing sparse processing on all the key tensors stored in the cache and performing sparse processing on all the value tensors stored in the cache, which can effectively reduce the amount of calculation and the amount of data access, thereby reducing the overall calculation complexity and improving the inference throughput performance.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to attention computing and sparse processing, and especially to data processing methods, apparatus, computer devices, computer-readable storage media and computer program products based on visual language models. Background Technology

[0002] As the scale of Visual Language Models (VLMs) continues to increase, they are widely used in multimodal reasoning tasks such as image understanding, visual question answering, and multimodal dialogue. These models typically need to process both text tokens and a large number of visual tokens generated from high-resolution images simultaneously, leading to a significant increase in the length of the input sequence. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product based on a visual language model.

[0004] According to one aspect of this disclosure, a data processing method based on a visual language model is provided. The method includes: acquiring information to be processed provided by a user to a visual language model, the information to be processed including at least one of an image or text; lexicalizing the information to be processed to obtain a lexical stream including multiple lexical units; generating multiple activation matrices corresponding to the multiple lexical units based on the lexical stream, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; performing sparsification processing on at least one of the query tensor, key tensor, and value tensor in each activation matrix based on a group-library structure for attention calculation in the pre-filling process of the visual language model to obtain an attention score matrix and an attention weight matrix of the information to be processed, wherein the group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, each of the multiple groups being divided into multiple libraries, and each of the multiple libraries including a fixed number of consecutive elements. The number of elements in the library is equal to the size of the library. The sparsified key tensors and value tensors are stored in a cache for storing key and value tensors. During the attention calculation of the visual language model's inference process, the cache is accessed multiple times to obtain the sparsified key and value tensors. When performing attention calculation during the visual language model's inference process, all key tensors stored in the cache are sparsified based on a group-library structure, and all value tensors stored in the cache are also sparsified based on a group-library structure to calculate the attention score matrix and attention weight matrix of the lexical units generated by the visual language model's inference. The group-library structure-based sparsification includes: within each group, filtering and retaining elements with contributions higher than a preset threshold based on the activation value magnitude or equivalent statistics of elements in multiple libraries; and using the retained elements for attention calculation.

[0005] In some embodiments, when the length of the target dimension cannot be divided by the size of the group or the size of the library, the target dimension is padded so that the length of the padded target dimension can be divided by the size of the group or the size of the library.

[0006] In some embodiments, performing group-library structure-based sparsification on at least one of the query tensor, key tensor, and value tensor in each activation matrix includes: performing group-library structure-based sparsification on the channel dimension of the query tensor or key tensor in the activation matrix to select a subset of channels of the query tensor or key tensor in the activation matrix whose contribution is higher than a first preset threshold for attention computation.

[0007] In some embodiments, sparsification of all key tensors stored in the cache based on a group-library structure includes: performing sparsification based on a group-library structure on the channel dimension of all key tensors stored in the cache to select a subset of channels of all key tensors whose contribution is higher than a second preset threshold for attention computation.

[0008] In some embodiments, sparsification of all value tensors stored in the cache based on a group-library structure includes: performing sparsification based on a group-library structure on the sequence dimension of all value tensors stored in the cache to select a subset of all value tensors whose contribution is higher than a third preset threshold for attention computation.

[0009] In some embodiments, the size of the library can be set to 64, 32, or 16 depending on the hardware characteristics.

[0010] According to another aspect of this disclosure, a data processing apparatus based on a visual language model is provided. The apparatus includes: an acquisition module configured to acquire information to be processed provided by a user to a visual language model, the information to be processed including at least one of an image or text; a lexicalization module configured to lexicalize the information to be processed to obtain a lexical stream including multiple lexical units; a matrix generation module configured to generate multiple activation matrices corresponding to the multiple lexical units based on the lexical stream, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; and a first sparsification module configured to perform sparsification processing on at least one of the query tensor, key tensor, and value tensor in each activation matrix based on a group-library structure, for use in attention calculation during the pre-filling process of the visual language model to obtain an attention score matrix and an attention weight matrix for the information to be processed, wherein the group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, each of the multiple groups being divided into multiple libraries, and each of the multiple libraries being... The system includes a fixed number of consecutive elements, the number of elements in the library being equal to the size of the library; a caching module configured to store the sparsified key tensors and value tensors in a cache for storing the key tensors and value tensors, wherein the attention calculation during the inference process of the visual language model is executed by accessing the cache multiple times to obtain the sparsified key tensors and value tensors; and a second sparsification module configured to perform group-library structure-based sparsification on all key tensors stored in the cache and group-library structure-based sparsification on all value tensors stored in the cache during the attention calculation during the inference process of the visual language model, for calculating the attention score matrix and attention weight matrix of the lexical units generated by the visual language model inference, wherein the group-library structure-based sparsification includes: within each group, filtering and retaining elements with a contribution higher than a preset threshold based on the activation value magnitude or equivalent statistics of elements in multiple libraries; and using the retained elements for attention calculation.

[0011] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0012] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0013] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0014] According to one or more embodiments of this disclosure, the computational load and data access load of attention-related matrix multiplication can be effectively reduced without affecting the inference accuracy of the visual language model, thereby reducing the overall computational complexity and improving inference throughput performance.

[0015] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0016] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 This is a flowchart illustrating a data processing method based on a visual language model according to an exemplary embodiment.

[0018] Figure 2 This is a schematic diagram illustrating the group-library structure in a data processing method based on a visual language model according to an exemplary embodiment.

[0019] Figure 3 This is a schematic diagram illustrating sparsity processing and matrix multiplication in a data processing method based on a visual language model according to an exemplary embodiment.

[0020] Figure 4 This is a block diagram illustrating a data processing apparatus based on a visual language model according to an exemplary embodiment.

[0021] Figure 5 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0024] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0025] In existing Visual Language Models (VLMs), self-attention is one of the core computational modules. The self-attention computation process typically involves constructing query, key, and value tensors, and calculating the attention score matrix and the corresponding attention weight matrix. The size of the attention matrix usually grows quadratically with the length of the input sequence. When the number of visual tokens is large or the inference sequence is long, the attention matrix and its intermediate results become enormous.

[0026] Under the above circumstances, existing technologies suffer from at least the following problems: 1) High storage and computational overhead for intermediate attention matrices. During attention calculation, the attention score matrix and attention weight matrix typically need to be explicitly or implicitly written back to GPU memory or system memory, generating a large number of intermediate data read / write operations and significantly consuming storage bandwidth; 2) Continuous expansion of the key-value cache (KVCache) leads to a decrease in decoding efficiency. In the autoregressive decoding stage, the Key / Value Cache grows continuously with the number of decoding steps, further exacerbating storage pressure and access latency, especially on edge devices or devices with limited bandwidth; and 3) High edge-side inference latency and energy consumption. Due to the large amount of multiplication and addition operations and data transfer in attention calculation, edge-side inference latency is high and energy efficiency is low, making it difficult to meet the requirements of low latency or real-time inference.

[0027] To address the aforementioned issues, existing sparse attention methods often employ global Top-K, fixed-block sparseness, or predefined masks. However, these methods typically suffer from drawbacks such as excessively coarse sparse granularity, fixed structure, and difficulty in balancing accuracy and performance. Furthermore, most schemes do not perform collaborative optimization of the attention intermediate matrix and KV cache.

[0028] Therefore, a technical solution is needed that can effectively reduce the overhead of intermediate Attention calculations and KV Cache storage and computation while maintaining the inference accuracy of the VLM model.

[0029] The embodiments of this disclosure provide a data processing method based on a visual language model, which can effectively reduce the computational load and data access load of attention-related matrix multiplication without affecting the inference accuracy of the visual language model, thereby reducing the overall computational complexity and improving inference throughput performance.

[0030] Figure 1 This is a flowchart illustrating a data processing method 100 based on a visual language model according to an exemplary embodiment.

[0031] like Figure 1As shown, this disclosure proposes a data processing method 100 based on a visual language model, including steps S102: obtaining information to be processed provided by a user to a visual language model, the information to be processed including at least one of an image or text; S104: performing lexicalization on the information to be processed to obtain a lexical stream including multiple lexical units; S106: generating multiple activation matrices corresponding to the multiple lexical units based on the lexical stream, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; S108: performing sparsification processing on at least one of the query tensor, key tensor, and value tensor in each activation matrix based on a group-library structure, for use in attention calculation during the pre-filling process of the visual language model to obtain an attention score matrix and an attention weight matrix for the information to be processed, wherein the group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, and each group is divided into multiple libraries, each library including a fixed number of... The elements in the library are consecutive, and the number of elements in the library is the size of the library; S110, the sparsified key tensors and value tensors are stored in a cache for storing key tensors and value tensors, wherein the attention calculation of the inference process of the visual language model is executed by accessing the cache multiple times to obtain the sparsified key tensors and value tensors; and S112, when performing the attention calculation of the inference process of the visual language model, all key tensors stored in the cache are sparsified based on the group-library structure, and all value tensors stored in the cache are sparsified based on the group-library structure, so as to calculate the attention score matrix and attention weight matrix of the lexical units generated by the inference of the visual language model, wherein the sparsification based on the group-library structure includes: within each group, filtering and retaining elements with a contribution higher than a preset threshold based on the activation value magnitude or equivalent statistics of elements in multiple libraries; and using the retained elements for attention calculation.

[0032] In step S102, the user provides the information to be processed to the visual language model, which includes at least one of an image or text.

[0033] A Visual Language Model (VLM) is a multimodal artificial intelligence model that can simultaneously understand and process visual information (such as images and video frames) and textual information. In the example, the Visual Language Model can be a CLIP / ALIGN, BLIP / BLIP-2, Flamingo, GPT-4V / GPT-4o, Gemini, or LLaVA model, etc., without limitation.

[0034] In the example, users can input images, text, or a combination of both into the VLM. The input images can be photos, paintings, screenshots, charts, videos, etc., and there is no limit to the number of images that can be input. The VLM can automatically generate a summary description of the input image. Users can also provide images and input specific text instructions to guide the model to complete a specific task. For example, users can input text such as "Describe this image in detail," "Summarize the core content of this image in one sentence," or "What does the road sign in the lower left corner of the image say?" The VLM can then complete the corresponding task based on the text.

[0035] In step S104, the information to be processed is lexicalized to obtain a lexical stream containing multiple lexical units.

[0036] In the example, when the VLM receives a complete input (e.g., an image and a text prompt), it can tokenize the information to be processed, obtaining a token stream containing multiple tokens. This process is an encoding process. Specifically, the input image can be tokenized into visual feature tokens, and the input text can be tokenized into text tokens. For example, an image can be converted into a sequence of visual feature tokens (also known as visual tokens) using a visual encoder (such as ViT). Similarly, a user-inputted text prompt (such as "describe this image") can be converted into text tokens using a text segmenter.

[0037] The VLM (Virtual Model for Learning) task execution consists of two phases: a prefill phase and a decoding phase. The prefill phase occurs after the model receives the complete input (e.g., an image + text prompt) and before it infers and generates the first lexical unit (this lexical unit is generated by the VLM model's inference, not derived from the user's input). The core task of the VLM's prefill phase is to process all input information in a single, parallel process, performing initial attention calculations on the complete input sequence. This is equivalent to observing the image and understanding the question, preparing for the decoding and generation task. Steps S104-S110 all belong to the VLM's prefill phase.

[0038] In step S106, multiple activation matrices corresponding to multiple words are generated based on the word stream, wherein each activation matrix may include a query tensor, a key tensor, and a value tensor.

[0039] In the example, during the Prefill phase, the VLM performs forward propagation, generating numerous intermediate results, which are the activation matrices. Specifically, for each layer and each attention head of the VLM, the embedding tensor of each token in the word stream undergoes a linear transformation to obtain the activation matrix corresponding to that token. Each activation matrix includes a query tensor (Q), a key tensor (K), and a value tensor (V). Furthermore, the activation matrix may also include other intermediate results, such as the attention weights matrix A.

[0040] The query tensor is mainly used for "asking questions" in the current step; the key tensor represents the "identity identifier" of each token and is used to calculate the attention weight; the value tensor represents the "content information" of each token and is used for weighted summation to generate the output (e.g., attention score).

[0041] In step S108, at least one of the query tensor, key tensor, and value tensor in each activation matrix undergoes sparsification based on a group-library structure. This sparsification is used for attention calculation in the pre-filling process of the visual language model to obtain the attention score matrix and attention weight matrix of the information to be processed. The group-library structure is constructed by dividing the tensor to be sparsified into multiple groups along the target dimension, with each group further divided into multiple libraries. Each library contains a fixed number of consecutive elements, the number of elements in the library being equal to the size of the library. The sparsification based on the group-library structure includes: within each group, filtering and retaining elements with a contribution higher than a preset threshold based on the activation value magnitude or equivalent statistics of the elements in the multiple libraries; and using the retained elements for attention calculation.

[0042] In the Prefill phase, VLM needs to perform initial attention calculations on the complete input sequence. An example attention score calculation formula can be Attention(Q, K, V) = softmax(Q·K^T / sqrt(c)) · V, where Attention is the attention score, Q is the query tensor, K is the key tensor, V is the value tensor, softmax() is the normalized exponential function, sqrt() is the root mean square function, c is the number of channels in K (usually also the number of channels in Q), and softmax(Q·K^T / sqrt(c)) is the attention weight matrix A.

[0043] Since attention score calculation requires the use of Q, K, and V in the activation matrix, and during the attention calculation process, the score matrix and weight matrix usually need to be explicitly or implicitly written back to GPU memory or system memory, generating a large number of intermediate data read and write operations and significantly occupying storage bandwidth, sparse processing based on the group-library structure can effectively reduce the amount of matrix multiplication computation without changing the Attention output, thereby reducing the overall computational overhead of the subsequent inference process.

[0044] Figure 2 This is a schematic diagram illustrating the group-library structure in a visual language model-based data processing method 100 according to an exemplary embodiment.

[0045] In some embodiments, when the length of the target dimension cannot be divided by the size of the group or the size of the library, the target dimension is padded so that the length of the padded target dimension can be divided by the size of the group or the size of the library, thereby ensuring the regularity and alignment of the group-library structure.

[0046] refer to Figure 2 , Figure 2 The left-hand matrix is ​​a 2×6 matrix, with the target dimension being the columns of this matrix. The elements can be divided into groups of 8 columns each, and a library of 4 columns each (i.e., the size of the library is 4). Since 6 is not divisible by 8 and 4, the columns of this matrix are padded to 8 columns. This gives us the right-hand matrix, which contains a 2×2 library.

[0047] In some embodiments, the size of the library can be set to 64, 32, or 16, etc., depending on the hardware characteristics, and is usually set to a power of 2 to facilitate hardware processing.

[0048] Figure 3 This is a schematic diagram illustrating sparsification and matrix multiplication in a visual language model-based data processing method 100 according to an exemplary embodiment.

[0049] refer to Figure 3 , Figure 3 The example illustrates the sparsified multiplication of a 2×8 matrix and an 8×2 matrix. In this sparsified matrix multiplication, only elements with a contribution level above a preset threshold within each library are selected for computation. For example, the element with the highest contribution level in each library can be selected for calculation. Thus, multiplying a 2×8 matrix and an 8×2 matrix is ​​transformed into multiplying two 2×2 matrices, significantly reducing the computational load. In this example, the contribution level of an element can be determined based on the magnitude of its activation value or the magnitude of its equivalent statistic within the library; a larger activation value or a larger equivalent statistic indicates a greater contribution from that element.

[0050] In some embodiments, performing group-library structure-based sparsification on at least one of the query tensor, key tensor, and value tensor in each activation matrix includes: performing group-library structure-based sparsification on the channel dimension of the query tensor or key tensor in the activation matrix to select a subset of channels of the query tensor or key tensor in the activation matrix whose contribution is higher than a first preset threshold for attention computation.

[0051] In the example, Q, K, and V can include two dimensions: sequence (s) and channels (c). The sequence dimension is determined by the number of input tokens; for example, if there are 10 tokens, the sequence dimension is 10. The channel dimension is generally determined by the hardware. After a linear transformation of the token embedding tensor, Q, K, and V have the same number of channels; for example, the number of channels c can be 1024, 2048, or 512. Therefore, Q, K, and V can be represented as s×c matrices.

[0052] In the attention calculation formula Attention(Q, K, V) = softmax(Q·K^T / sqrt(c)) · V, the dot product operation of Q·K^T is performed along the channel dimension. Therefore, Q or K can be sparsified along the channel dimension based on a group-library structure to select a subset of channels of Q or K in the activation matrix whose contribution is higher than a first preset threshold for attention calculation. The first preset threshold can be set according to actual conditions and is not restricted here. Sparsification of Q or K along the channel dimension based on a group-library structure yields the attention weight matrix A. Alternatively, the attention weight matrix A can also be sparsified along the group-library structure to obtain a sparsified attention weight matrix A.

[0053] In step S110, the sparsified key tensor and value tensor are stored in a cache for storing the key tensor and value tensor. The attention calculation of the inference process of the visual language model accesses the cache multiple times to obtain the sparsified key tensor and value tensor when it is executed.

[0054] In the example, the cache used to store key and value tensors is also called a key-value cache. A key-value cache is a high-speed data storage system that stores data as simple key-value pairs. Key-value caches are used to provide extremely fast read speeds and are typically used to reduce access to slow backend data sources (such as databases or APIs), thereby significantly improving application performance and scalability.

[0055] In the example, during the VLM prefill phase, VLM computes the complete Q, K, and V for each token. For instance, when a user inputs the text "The cat is on the lawn," VLM segments "The cat is on the lawn" into five tokens: "cat," "on," "grass," "lawn," and "on." It then computes the corresponding Q, K, and V for each token. The purpose of this computation is to use these Q, K, and V values ​​to calculate the attention output for the current prefill phase (used to calculate the prediction loss for the next token or to generate the representation for the next token). However, after the prefill phase ends, K and V are cached in the KV Cache, while Q is discarded after the attention output for the prefill phase is computed.

[0056] In step S112, during the attention calculation of the inference process of the visual language model, all key tensors stored in the cache are subjected to sparsification based on the group-library structure, and all value tensors stored in the cache are subjected to sparsification based on the group-library structure, so as to calculate the attention score matrix and attention weight matrix of the lexical generated by the visual language model inference.

[0057] In the attention score calculation during the Decoding phase (i.e., the inference phase) of VLM, Q comes from the new tokens generated by VLM inference, not from the tokens of the input information to be processed. Q changes with each iteration (therefore, it doesn't need to be cached during the Prefill phase). K and V, on the other hand, come from all historical tokens, including all K and V cached during the Prefill phase. During the Decoding phase of VLM, since each new token generation requires accessing the KV Cache to retrieve all cached K and V for attention score calculation, sparsification of all cached K and V based on a group-library structure can effectively reduce the computational cost and data access volume of attention-related matrix multiplications during the Decoding phase, thereby reducing overall computational complexity and improving inference throughput performance.

[0058] In some embodiments, sparsification of all key tensors stored in the cache based on a group-library structure includes: performing sparsification based on a group-library structure on the channel dimension of all key tensors stored in the cache to select a subset of channels of all key tensors whose contribution is higher than a second preset threshold for attention computation.

[0059] In the example, the length of s for the current Q in the Decoding stage is 1 (the length of s is determined by the number of tokens, and in the Decoding stage, VLM generates tokens one by one, so the newly generated token is always one), which is relatively small; while the length of s for all cached Ks grows continuously with the decoding process and is larger in scale (K for all tokens must be cached and participate in the attention score calculation for subsequently generated tokens).

[0060] In the attention calculation formula Attention(Q, K, V) = softmax(Q·K^T / sqrt(c)) · V, the dot product operation of Q·K^T is performed in the channel dimension. Therefore, sparsification of the K tensor in the channel dimension based on the group-library structure can achieve a more significant speedup effect, without the need to sparsify the Q tensor based on the group-library structure.

[0061] In some embodiments, sparsification of all value tensors stored in the cache based on a group-library structure includes: performing sparsification based on a group-library structure on the sequence dimension of all value tensors stored in the cache to select a subset of all value tensors whose contribution is higher than a third preset threshold for attention computation.

[0062] In the example, the attention weight matrix softmax(Q·K^T / sqrt(c)) formed between Q and K during the Decoding phase logically weights and aggregates the V tensor along the sequence dimension. For example, in the attention calculation formula Attention(Q, K, V) = softmax(Q·K^T / sqrt(c)) · V, the attention weight matrix softmax(Q·K^T / sqrt(c)) is an s×s matrix, and V is an s×c matrix; the dot product operation is performed along the sequence (s) dimension. Therefore, sparsifying the V tensor along the sequence dimension effectively reduces the overall computational and data access volume.

[0063] It should be noted that the second and third preset thresholds can be set according to the actual situation, and no restrictions are imposed here.

[0064] Figure 4 This is a block diagram illustrating a visual language model-based data processing apparatus 400 according to an exemplary embodiment.

[0065] like Figure 4As shown, in some embodiments, the apparatus 400 includes an acquisition module 410 configured to acquire information to be processed provided by a user to a visual language model, the information to be processed including at least one of an image or text; a lexicalization module 420 configured to lexicalize the information to be processed to obtain a lexical stream including multiple lexical units; a matrix generation module 430 configured to generate multiple activation matrices corresponding to the multiple lexical units based on the lexical stream, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; and a first sparsification module 440 configured to perform sparsification processing on at least one of the query tensor, key tensor, and value tensor in each activation matrix based on a group-library structure, for use in attention calculation during the pre-filling process of the visual language model to obtain an attention score matrix and an attention weight matrix for the information to be processed, wherein the group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, each of the multiple groups being divided into multiple libraries, and each of the multiple libraries including a fixed number of... The library contains a continuous set of elements, the number of which is equal to the size of the library. A cache module 450 is configured to store the sparsified key tensors and value tensors in a cache for storing the key tensors and value tensors. During the attention calculation of the visual language model's inference process, the cache is accessed multiple times to obtain the sparsified key tensors and value tensors. A second sparsification module 460 is configured to perform group-library structure-based sparsification on all key tensors stored in the cache and group-library structure-based sparsification on all value tensors stored in the cache during the attention calculation of the visual language model's inference process, for calculating the attention score matrix and attention weight matrix of the lexical units generated by the visual language model's inference. The group-library structure-based sparsification includes: within each group, filtering and retaining elements with contributions higher than a preset threshold based on the activation value magnitude or equivalent statistics of elements in multiple libraries; and using the retained elements for calculation.

[0066] In some embodiments, the first sparsification module 440 is further configured to perform group-library structure-based sparsification processing on the channel dimension of the query tensor or key tensor in the activation matrix to filter out a subset of channels of the query tensor or key tensor in the activation matrix whose contribution is higher than a first preset threshold for subsequent calculation.

[0067] In some embodiments, the second sparsification module 460 is further configured to perform group-library structure-based sparsification processing on the channel dimension of all key tensors stored in the cache, so as to filter out a subset of channels of all key tensors whose contribution is higher than a second preset threshold for subsequent attention calculation.

[0068] In some embodiments, the second sparsification module 460 is further configured to perform group-library structure-based sparsification on the sequence dimension of all value tensors stored in the cache, in order to select a subset of all value tensors whose contribution is higher than a third preset threshold for subsequent attention calculation.

[0069] The operations of the aforementioned acquisition module 410, lexicalization module 420, matrix generation module 430, first sparsification module 440, caching module 450, and second sparsification module 460 can be combined. Figure 1 The operations of steps S102, S104, S106, S108, S110 and S112 are the same, so the details of each aspect will not be repeated here.

[0070] According to one aspect of this disclosure, a computer device is also provided, comprising: at least one processor; and a memory storing a computer program thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the steps of any of the method embodiments described above.

[0071] According to one aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the steps of any of the method embodiments described above.

[0072] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0073] Figure 5 An example computer device 500 is shown in which any of the embodiments described herein may be implemented. The computer device 500 may be used to implement one or more components of the systems and methods described above. The computer device 500 may include a bus 502 or other communication mechanism for communicating information, and one or more processors 504 coupled to the bus 502 for processing information. The processor 504 may be, for example, one or more general-purpose microprocessors.

[0074] Computer device 500 may also include main memory 506, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, can make computer device 500 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 506 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0075] Computer device 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 500, enable computer device 500 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 508. Executing the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 506. When these instructions are executed by processor 504, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0076] Computer device 500 also includes a network interface 510 coupled to bus 502. Network interface 510 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 510 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0077] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0078] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0079] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0080] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0081] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0082] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0083] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0084] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0085] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0086] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0087] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0088] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A data processing method based on a visual language model, characterized in that, The method includes: Obtain information to be processed provided by the user to the visual language model, wherein the information to be processed includes at least one of an image or text; The information to be processed is lexicalized to obtain a lexical stream containing multiple lexical units; Based on the lexical stream, multiple activation matrices are generated corresponding to the multiple lexical units, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; At least one of the query tensor, the key tensor, and the value tensor in each activation matrix is ​​subjected to sparsification based on a group-library structure for attention calculation in the pre-filling process of the visual language model to obtain the attention score matrix and attention weight matrix of the information to be processed. The group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, and each of the multiple groups is divided into multiple libraries. Each of the multiple libraries includes a fixed number of consecutive elements, and the number of elements in the library is the size of the library. The sparsified key tensor and value tensor are stored in a cache for storing the key tensor and value tensor, wherein the attention computation of the inference process of the visual language model accesses the cache multiple times during execution to obtain the sparsified key tensor and value tensor; and During the attention calculation in the inference process of the visual language model, all key tensors stored in the cache undergo the group-library structure-based sparsification process, and all value tensors stored in the cache undergo the group-library structure-based sparsification process, to calculate the attention score matrix and attention weight matrix of the lexical units generated by the visual language model inference. The sparsity processing based on the group-library structure includes: Within each group, based on the activation value magnitude or equivalent statistics of the elements in the multiple libraries, elements with a contribution higher than a preset threshold are selected and retained; and The retained elements are used for attention calculation.

2. The method according to claim 1, characterized in that, When the length of the target dimension cannot be divided by the size of the group or the size of the library, the target dimension is padded so that the length of the padded target dimension can be divided by the size of the group or the size of the library.

3. The method according to any one of claims 1-2, characterized in that, Performing sparsification based on a group-library structure on at least one of the query tensor, the key tensor, and the value tensor in each activation matrix includes: On the channel dimension of the query tensor or the key tensor in the activation matrix, the sparsification process based on the group-library structure is performed to filter out a subset of channels of the query tensor or the key tensor in the activation matrix whose contribution is higher than a first preset threshold for attention calculation.

4. The method according to any one of claims 1-2, characterized in that, The sparsification process based on the group-library structure for all key tensors already stored in the cache includes: On the channel dimension of all the key tensors already stored in the cache, the sparsification process based on the group-library structure is performed to filter out a subset of channels of all key tensors whose contribution is higher than a second preset threshold for the attention calculation.

5. The method according to any one of claims 1-2, characterized in that, The sparsification process based on the group-library structure for all value tensors already stored in the cache includes: On the sequence dimension of all value tensors already stored in the cache, the sparsification process based on the group-library structure is performed to filter out a subset of all value tensors whose contribution is higher than a third preset threshold for the attention calculation.

6. The method according to any one of claims 1-2, characterized in that, The size of the library can be set to 64, 32, or 16 depending on the hardware characteristics.

7. A data processing device based on a visual language model, characterized in that, The device includes: The acquisition module is configured to acquire information to be processed provided by the user to the visual language model, the information to be processed including at least one of an image or text. The lexicalization module is configured to lexicalize the information to be processed to obtain a lexical stream including multiple lexical units; The matrix generation module is configured to generate multiple activation matrices corresponding to the multiple words based on the word stream, wherein each activation matrix includes a query tensor, a key tensor, and a value tensor; The first sparsification module is configured to perform group-library structure-based sparsification on at least one of the query tensor, the key tensor, and the value tensor in each activation matrix for attention calculation in the pre-filling process of the visual language model to obtain the attention score matrix and attention weight matrix of the information to be processed. The group-library structure is constructed by dividing the tensor to be sparsified into multiple groups in the target dimension, and each of the multiple groups is divided into multiple libraries. Each library includes a fixed number of consecutive elements, and the number of elements in the library is the size of the library. A caching module is configured to store the sparsified key tensor and the value tensor into a cache for storing the key tensor and the value tensor, wherein the attention computation of the inference process of the visual language model accesses the cache multiple times to obtain the sparsified key tensor and the value tensor during execution; and The second sparsification module is configured to perform the group-library structure-based sparsification process on all key tensors stored in the cache and on all value tensors stored in the cache during the attention calculation in the inference process of the visual language model, for the purpose of calculating the attention score matrix and attention weight matrix of the lexical units generated by the visual language model inference. The sparsity processing based on the group-library structure includes: Within each group, based on the activation value magnitude or equivalent statistics of the elements in the multiple libraries, elements with a contribution higher than a preset threshold are selected and retained; and The retained elements are used for attention calculation.

8. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

Images