Inference acceleration method and device for visual language model, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a reasoning acceleration method, device and equipment for a visual language model and a medium, and the method comprises the steps: obtaining a to-be-processed original input image, and carrying out the preprocessing, and generating a plurality of image groups; respectively calculating the semantic similarity of all the image groups, obtaining the image groups of which the semantic similarity meets a first similarity condition, and recording the image groups as redundant image groups; obtaining target image blocks meeting a second similarity condition in the redundant image group, and combining the target image blocks with the image blocks in the non-redundant image group to generate a non-redundant image group set; and arranging the non-redundant image group set to generate a target input image. According to the method, irrelevant and redundant visual information in the image is removed before the word segmentation operation, the local similarity is evaluated by using lightweight statistical measurement, a pruning decision is made earlier, and the reasoning efficiency of a visual language model is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for accelerating reasoning in visual language models. Background Technology

[0002] In the fintech field, Visual Language Models (VLMs) map speech, text, and images into the same semantic space through cross-modal alignment technology. For example, during transactions, the model associates the text "high-risk investment" with abnormal transaction amounts, or uses facial expression analysis (video) to help determine the authenticity of loan applications. In insurance claims, OCR technology can be used to recognize text in photos, combined with speech-to-text and NLP to extract key information (such as accident time and injury location), and finally, cross-modal verification ensures data consistency.

[0003] In the healthcare field, medical diagnosis requires the integration of multimodal data such as images, medical records, and laboratory indicators. Traditional models tend to overlook the correlations between data. Visual Language Models (VLMs) improve diagnostic accuracy by fusing information from multiple sources.

[0004] Visual Language Models (VLMs) excel in tasks such as image captioning, visual question answering, and image-text retrieval. However, the inference phase requires processing a large number of visual tokens (generated from image blocks), leading to high computational overhead, high latency, and high power consumption, posing challenges for practical deployment. As the scale and complexity of VLMs increase, improving inference efficiency without sacrificing performance becomes increasingly critical. To address this, token pruning methods have emerged. Existing token pruning methods involve discarding less important visual tokens based on attention scores.

[0005] The model inference acceleration algorithm based on attention score pruning requires a complete attention calculation for all tags before removing any tags, which will lead to unnecessary overhead and fail to reduce redundant calculations in the early stages; relying on the attention distribution inside the model rather than the image semantics itself can easily cause misjudgment of the more important image regions and ignore semantically important regions. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method, apparatus, device and medium for accelerating inference of visual language models, aiming to solve the problems of existing model inference acceleration algorithms based on attention score pruning, which require full attention calculation of all tags before removing any tags, resulting in unnecessary overhead and failure to reduce redundant calculations in the early stage; relying on the attention distribution inside the model rather than the image semantics itself, which is prone to misjudgment of the more important image regions and ignores semantically important regions.

[0007] The technical solution of the present invention is as follows: The first embodiment of the present invention provides a method for accelerating inference in visual language models, the method comprising: Obtain the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, wherein each image group includes several image blocks; Calculate the semantic similarity of all image groups respectively, and denote the image groups whose semantic similarity satisfies the first similarity condition as redundant image groups; Obtain target image blocks that satisfy the second similarity condition from the redundant image group, and combine the target image blocks with image blocks from the non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group consists of image groups after removing redundant image groups from all image groups; The non-redundant image set is arranged to generate the target input image.

[0008] Another embodiment of the present invention provides an inference acceleration device for visual language models, the device comprising: The image preprocessing module is used to acquire the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, wherein each image group includes several image blocks. The image redundancy calculation module is used to calculate the semantic similarity of all image groups respectively, and the image groups whose semantic similarity satisfies the first similarity condition are recorded as redundant image groups; The image group filtering module is used to obtain target image blocks that meet the second similarity condition in the redundant image group, and combine the target image blocks with image blocks in the non-redundant image group to generate a non-redundant image group set. The non-redundant image group consists of image groups after removing the redundant image groups from all image groups. The target image generation module is used to arrange the non-redundant image set to generate the target input image.

[0009] Another embodiment of the present invention provides a computer device, the computer device including at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the above-described inference acceleration method for visual language models.

[0010] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the above-described inference acceleration method for visual language models.

[0011] Beneficial Effects: The inference acceleration method, apparatus, device, and medium for visual language models according to embodiments of the present invention include: acquiring a raw input image to be processed; preprocessing the raw input image; dividing the preprocessed raw input image into several image groups, wherein each image group includes several image blocks; calculating the semantic similarity of all image groups respectively; and recording the image groups whose semantic similarity satisfies a first similarity condition as redundant image groups; acquiring target image blocks in the redundant image groups that satisfy a second similarity condition; combining the target image blocks with image blocks in the non-redundant image groups to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing redundant image groups from all image groups; and arranging the non-redundant image group set to generate a target input image. Embodiments of the present invention, by removing irrelevant and redundant visual information contained in the image before performing word segmentation, and using lightweight statistical measures such as structural and color-based features to evaluate local similarity, enable earlier and more interpretable pruning decisions, thereby improving the inference efficiency of visual language models. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the application environment of an embodiment of the reasoning acceleration method for visual language models according to the present invention; Figure 2 This is a flowchart of a preferred embodiment of a reasoning acceleration method for visual language models according to the present invention; Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the inference acceleration device for visual language models according to the present invention; Figure 4 This is a schematic diagram of a preferred embodiment of a computer device according to the present invention; Figure 5 This is another structural schematic diagram of a preferred embodiment of a computer device according to the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0015] The embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0016] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Here, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0017] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0018] The inference acceleration method for visual language models provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The client accesses the server's network or business platform, and the server can obtain the original input image to be processed. The server preprocesses the original input image, dividing it into several image groups, each containing several image blocks. Semantic similarity is calculated for all image groups, and image groups satisfying a first similarity condition are designated as redundant image groups. Target image blocks satisfying a second similarity condition are obtained from the redundant image groups and combined with image blocks from non-redundant image groups to generate a non-redundant image group set. The non-redundant image group set consists of image groups after removing redundant image groups from all image groups. The non-redundant image group set is then arranged to generate the target input image. In this invention, by removing irrelevant and redundant visual information from the image before word segmentation, and using lightweight statistical measures such as structural and color-based features to evaluate local similarity, more interpretable pruning decisions can be made earlier, thereby improving the inference efficiency of the visual language model. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0019] To address the above problems, embodiments of the present invention provide a method for accelerating inference in visual language models. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart of a preferred embodiment of a reasoning acceleration method for visual language models according to the present invention. Figure 2 As shown, it includes: Step S100: Obtain the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, wherein each image group includes several image blocks.

[0020] This invention is applied to accelerate inference using visual language models. The visual language model in this invention can be used in the healthcare industry or the fintech industry; it can effectively improve the inference efficiency of visual language models in both the healthcare and fintech sectors.

[0021] Visual language models are a special type of multimodal large model that specifically processes and integrates visual and linguistic information. Their core objective is to establish a deep connection between images (or videos) and text, enabling bidirectional understanding and generation between the two modalities.

[0022] In the healthcare field, visual language models are being applied more and more deeply. By integrating visual perception and natural language understanding capabilities, they enable in-depth analysis and interactive generation of multimodal data such as medical images and clinical texts, bringing revolutionary changes to medical practice. For example, visual language models combine medical images (such as X-rays, CT scans, and MRI scans) with patient medical records to automatically generate structured diagnostic reports, annotate lesion areas, and analyze features (such as size, shape, and density).

[0023] In the fintech field, visual language models can "read" text and "analyze" layout structure. For example, they can extract textual information such as the applicant's name and income from loan application forms, while simultaneously identifying the completeness of the signature and seal areas below. From listed companies' annual reports, they can not only extract textual descriptions but also interpret financial data charts and generate summaries. This replaces a large amount of manual data entry and verification work, improving efficiency and reducing error rates.

[0024] In this embodiment of the invention, when acquiring the original input image, the resolution of the original input image is adjusted, and the processed original image is divided into several image groups, each consisting of several image blocks.

[0025] Step S100 involves acquiring the original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into several image groups, including: Step S101: Obtain the original input image to be processed, scale the original input image to a preset resolution, and generate the first input image; Step S102: Normalize the first input image to generate the second input image; Step S103: Divide the second input image into several non-overlapping image blocks; Step S104: Combine each adjacent N*N image blocks to generate several image groups, where N is an integer greater than or equal to 2.

[0026] The model pre-receives the image as input. First, the image is scaled to a fixed resolution and then standardized (e.g., mean-variance normalization) to fit the input requirements of the visual model. Then, the image is divided into... Each pair of non-overlapping square blocks is denoted as an image patch. Image blocks can be combined into a group, denoted as an image group. A patch is an ordered collection of one or more small, regular (usually square) image blocks after a complete image has been divided into them. A patch can be understood as a "fragment" or "slice" of an image. It retains all the visual information (color, texture, shape, etc.) of a local region of the original image. A set emphasizes that it is a collection of multiple patches, and this set is usually ordered, containing information about the spatial relationships between these patches.

[0027] Scaling input images to a fixed resolution is a common operation in image processing, aiming to standardize image size to meet specific application needs (such as model input, display adaptation, or storage optimization). This can be achieved using traditional interpolation algorithms, deep learning models, or dedicated image processing libraries. Traditional interpolation algorithms include: Nearest Neighbor Interpolation: Directly copies the value of the nearest source pixel at the target pixel location. Bilinear Interpolation: Interpolates based on the weighted average of the target pixel's four nearest neighbors. Bicubic Interpolation: Calculates using a cubic polynomial fit, utilizing the weighted average of the target pixel's 16 nearest neighbors.

[0028] The primary goal of image normalization is to transform the pixel value distribution of an image into a standard normal distribution with a mean of 0 and a standard deviation of 1. In deep learning models, especially those optimized using gradient descent, the distribution of input data has a significant impact on the stability and efficiency of the training process. Image normalization accelerates model convergence: If the scale differences of input features are huge (e.g., the pixel value range of the R channel is [100, 200], while that of the B channel is [10, 50]), the contour lines of the loss function will be elongated, causing the gradient descent process to oscillate repeatedly and converge slowly. After normalization, all features have a consistent scale, the optimization path is more direct, and convergence is faster. Improves model performance: More stable and faster convergence often helps the model find better solutions, potentially improving the final accuracy. Mitigates gradient vanishing / exploding: For deep networks, normalized data helps maintain the stability of gradients during backpropagation, preventing gradients from becoming too large or too small. Unifies data scale: When it is necessary to fuse multi-source or multimodal data, normalization provides a unified scale.

[0029] This invention advances the pruning operation to the image group segmentation stage, that is, before generating visual tokens; avoiding the unnecessary overhead of performing full attention calculations before pruning, and achieving a truly lightweight pre-pruning operation.

[0030] Step S200: Calculate the semantic similarity of all image groups respectively, and record the image groups whose semantic similarity satisfies the first similarity condition as redundant image groups.

[0031] Semantic similarity is calculated for all blocks in an image, comprising two stages: structural similarity calculation and color distribution similarity calculation. Structural similarity is calculated for each block pair (i.e., two image blocks). Depending on application requirements, a threshold for structural similarity can be set to filter out structurally similar block pairs. Color distribution similarity is then calculated for structurally similar block pairs. The final semantic similarity is obtained by combining the structural and color distribution similarities. This can be achieved through weighted combination or other fusion methods. Subsequently, image blocks can be sorted, classified, or retrieved based on the final semantic similarity. Similar block pairs can be visualized to provide users with an intuitive understanding of the similarity calculation results.

[0032] Step S200, which involves calculating the semantic similarity of all image groups and designating image groups whose semantic similarity satisfies the first similarity condition as redundant image groups, includes: Step S201: Calculate the structural similarity for each group of images to obtain the structural similarity. Step S202: Calculate the color distribution similarity for each image group to obtain the color distribution similarity. Step S203: Combine the image groups that simultaneously satisfy the structural similarity condition and the color distribution similarity condition, and denot them as redundant image groups.

[0033] Structural Similarity Index (SSIM) is a metric for measuring the structural similarity between two images, considering information on brightness, contrast, and structure. At the image block level, structural similarity calculation captures the similarity of structural features such as shape, texture, and edges between blocks. The calculation steps are as follows: Brightness Comparison: Calculate the average brightness of the two blocks and compare their differences. Contrast Comparison: Calculate the standard deviation of the two blocks to measure their contrast difference. Structural Comparison: Calculate the covariance of the two blocks to capture their structural correlation. Comprehensive Similarity: Weighted combination of the similarities in brightness, contrast, and structure to obtain the final structural similarity.

[0034] Color distribution similarity calculation measures the similarity of color space distribution between two image blocks. It considers the types, proportions, and distribution of colors within each block. For example, color space conversion can be used: converting image blocks from the RGB color space to other color spaces (such as HSV, Lab, etc.) to better capture color features. Color histogram construction: Constructing a color histogram for each image block and counting the number of pixels in different color intervals. Similarity calculation: Comparing the color histograms of two blocks and calculating their similarity. Commonly used similarity metrics include histogram intersection, chi-square distance, and Bach distance.

[0035] Image groups with structural similarity greater than a preset structural similarity threshold and color distribution similarity greater than a color distribution similarity threshold are acquired and combined, and are denoted as redundant image groups.

[0036] Two types of similarity metrics are based on image content: structural similarity, which calculates the cosine of the angle between the mean vectors in each 2×2 patch group; and color distribution similarity, which calculates the color histogram differences of each patch group. When both structure and color are highly similar, the patch is considered redundant and only the representative patch is retained.

[0037] Groups that simultaneously meet the above conditions of structural and color similarity are considered "redundant groups".

[0038] Step S201, which involves calculating the structural similarity for each image group to obtain the structural similarity, includes: Step S211: Calculate the average color vector of each image patch in the image group; Step S212: Calculate the mean color vector of the image group based on the average color vector of each image block; Step S213: Calculate the cosine similarity between the average color vector of each image block in the image group and the mean color vector to obtain the structural similarity of the current image group.

[0039] First, the average color vector for each patch is calculated. The average color vector is the arithmetic mean of the color values ​​of all pixels in an image region (which could be an entire image, an image patch, or any region of interest). It's a very concise numerical representation, summarizing the overall hue or dominant hue of the entire region in a single vector. In the RGB color space, the image region is considered as a set of N pixels, each with values ​​in the R, G, and B channels. The average values ​​of the R, G, and B channels for all pixels are calculated separately. These three averages are combined into a three-dimensional vector to obtain the average color vector. Simultaneously, an average is performed within each group to obtain the group's average color vector.

[0040] Calculate the cosine similarity between each of the four image groups within a group and the mean of that group:

[0041] If all four similarities within the group are greater than ,Right now If the similarity is high, then the group is considered to have a high degree of structural similarity. This is the preset cosine similarity threshold.

[0042] Step S202, which involves calculating the color distribution similarity for each image group to obtain the color distribution similarity, includes: Step S221: Generate corresponding RGB histograms based on the image blocks in the image group respectively; Step S222: Calculate the chi-square distance between every two image blocks in the image group based on the RGB histogram; Step S223: Based on the chi-square distance, obtain the color distribution similarity of the current image group.

[0043] When calculating the color distribution similarity, it is necessary to draw and obtain an RGB histogram for each image group. An RGB histogram reflects the color characteristics of an image by statistically analyzing the pixel values ​​of its three color channels: red (R), green (G), and blue (B). Each channel's value ranges from 0 to 255 (for an 8-bit image). The horizontal axis of the histogram represents the pixel value, and the vertical axis represents the frequency of that value. The chi-square distance between any two image groups within a group is calculated as follows:

[0044] If for all image groups within a group, the following conditions are met:

[0045] in It is the chi-square distance threshold.

[0046] Calculating the chi-square distance between any two image groups within a group is a commonly used similarity metric, especially suitable for comparing histograms or feature distributions. The chi-square distance measures the difference between two probability distributions (or histograms). When calculating the chi-square distance, ensure all histograms are on the same scale (e.g., [0,1] or probability distribution). Add a small constant (e.g., 1e-10) to avoid division by zero. All histograms must have the same number of bins. If using deep learning features, dimensionality reduction (e.g., PCA) is necessary to reduce computation.

[0047] Step S300: Obtain the target image block that meets the second similarity condition in the redundant image group, and combine the target image block with the image block in the non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of the image group after removing the redundant image group from all image groups.

[0048] The second similarity condition is set by the user. For example, one or more image blocks from a redundant image can be selected and combined with image blocks from a non-redundant image to generate a non-redundant image set. Combining the target image block with image blocks from the non-redundant image set generates a non-redundant image set.

[0049] Step S300, which involves obtaining target image blocks from the redundant image group that satisfy the second similarity condition, and combining the target image blocks with image blocks from the non-redundant image group to generate a non-redundant image group set, includes: Step S301: Obtain the image patch in the redundant image group that has the highest cosine similarity to the mean color vector; and use it as the target image patch; Step S302: Remove non-target image blocks from the redundant image group; Step S303: Combine the target image block with the image blocks in the non-redundant image group to generate a non-redundant image group set.

[0050] In the redundant image group, the image group with the highest cosine similarity to the mean color vector is selected as the second similarity criterion. Only the image group with the highest cosine similarity to the mean color vector within the group is retained, and the remaining image groups are pruned. All retained blocks are combined with image groups from the non-redundant group to form a new patch set.

[0051] The embodiments of this invention are interpretable, lightweight, and training-independent. The pruning process relies solely on observable image statistical features, eliminating the need to train auxiliary networks. The deleted regions are clearly visualized, facilitating manual understanding and debugging, reducing computational complexity, and enabling better deployment of the model on edge or low-power devices.

[0052] Step S400: Arrange the non-redundant image set to generate the target input image.

[0053] Non-redundant image sets can be denoted as new patch sets. The image patches in the patch set are rearranged into compact images to generate the recombined target input image. The target input image is input into the visual encoder of the subsequent visual language model for subsequent model inference.

[0054] Step S400, which involves arranging the non-redundant image set to generate the target input image, includes: Step S401: Obtain the original coordinates of each image patch in the non-redundant image group set; Step S402: Generate the corresponding position embedding vector based on the original coordinates; Step S403: Arrange all image blocks in the non-redundant image group set based on the location embedding vector to generate the target input image.

[0055] Obtain the spatial location of each image patch in the non-redundant image set within its original image. This is typically represented using a simple two-dimensional index (row, col) or a bounding box. For example, cutting a 224x224 image into 16x16 patches results in 14x14=196 patches. The top-left patch has coordinates (0, 0), the next one to its right has coordinate (0, 1), and so on.

[0056] Since models like the Transformer are permutation-invariant (i.e., shuffling the input order does not affect the output), they require additional information to perceive the order or spatial structure of the data. Location embedding is a location-related vector that is added to the feature vector of an image patch, thus telling the model "where this feature comes from."

[0057] Based on the original coordinates, the corresponding position embedding vector is generated. The coordinates representing spatial information obtained in the previous step are converted into a fixed-length, learnable vector.

[0058] A common approach is learnable position embedding, where each possible position (e.g., 0 to 195) is treated as a learnable parameter. The steps are: Create a learnable parameter matrix `Embedding` of size (total number of positions, embedding dimension). Flatten the (row, col) coordinates of each block into a unique 1D index `pos_index`. For example, for a 14x14 grid, the index of the position (row, col) is `pos_index = row * 14 + col`. The position embedding of this block is `Embedding[pos_index]`.

[0059] Sine / Cosine Position Encoding. This uses sine and cosine functions to generate unique positional codes. It can be naturally extended to sequence lengths not encountered during training. Steps: For 2D coordinates (row, col), a common practice is to encode the rows and columns separately in 1D and then sum them, or concatenate them along the channel dimension.

[0060] 2D relative position embedding / attention bias. Instead of directly assigning a vector to each absolute position, a bias term is added to the attention score of two blocks based on their relative positions (Δrow, Δcol) within the self-attention mechanism. Steps: Define a learnable bias matrix B, where the indices are relative positions. When calculating attention, for query block i and key block j, calculate their relative coordinates (Δrow, Δcol). The final attention score is: Attention = Softmax(QK^T / sqrt(d_k) + B[Δrow, Δcol]).

[0061] The blocks in the patch set are rearranged into a compact image, and a position embedding vector corresponding to the original coordinates is generated for each retained block. This ensures that the model can still understand the structural relationships of the original space when processing the compressed image. The recombined image is encoded by a near-visual encoder and fused with the text embedding vector before being passed to the backbone model to complete the subsequent inference and decoding process.

[0062] The spatial rearrangement and position preservation strategy rearranges the remaining patches into a compact image input after pruning, and generates position embeddings using the original coordinates, ensuring that the model can still understand spatial relationships. At the same time, the optimization strategy is decoupled from the model architecture, allowing it to be directly "plug and play" on existing visual language models.

[0063] Compared with the prior art, the embodiments of the present invention have the following technical advantages: It significantly reduces inference overhead, reducing average FLOPs computation by approximately 25–35% without altering the model structure, and significantly lowers actual latency. It is particularly suitable for tasks with a lot of redundant background, such as image question answering, retrieval, and description generation.

[0064] While maintaining or improving accuracy, the pruning results are consistent with human vision, and semantically relevant regions are preserved. In multiple benchmark tests, the task accuracy on multiple datasets remains above 95%, and on some datasets, it even slightly improves compared to the baseline model.

[0065] The algorithm has high versatility and interpretability. The V-PRUNE strategy does not depend on a specific network structure or training process and can be directly embedded into any patch-based visual Transformer model. The pruning decision process is visualized and the adjustable parameters are clearly defined, making it easy for industrial applications.

[0066] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0067] Another embodiment of the present invention provides an inference acceleration device for visual language models, which corresponds one-to-one with the inference acceleration method for visual language models described in the above embodiments. For example... Figure 3 As shown, device 1 includes: The image preprocessing module 100 is used to acquire the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, wherein each image group includes several image blocks. The image redundancy calculation module 200 is used to calculate the semantic similarity of all image groups respectively, and the image groups whose semantic similarity satisfies the first similarity condition are recorded as redundant image groups; The image group filtering module 300 is used to obtain target image blocks that meet the second similarity condition in the redundant image group, and combine the target image blocks with image blocks in the non-redundant image group to generate a non-redundant image group set. The non-redundant image group is composed of image groups after removing redundant image groups from all image groups. The target image generation module 400 is used to arrange the non-redundant image set to generate a target input image.

[0068] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0069] In one embodiment, the image preprocessing module 100 is specifically used for: Obtain the original input image to be processed, scale the original input image to a preset resolution, and generate the first input image; The first input image is normalized to generate the second input image; The second input image is divided into several non-overlapping image blocks; Each pair of adjacent N*N image blocks is combined to generate several image groups, where N is an integer greater than or equal to 2.

[0070] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0071] In one embodiment, the image redundancy calculation module 200 is specifically used for: Structural similarity was calculated for each group of images to obtain the structural similarity. The color distribution similarity is calculated for each group of images to obtain the color distribution similarity. Image groups that simultaneously satisfy both structural similarity and color distribution similarity conditions are combined and denoted as redundant image groups.

[0072] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0073] In one embodiment, the image redundancy calculation module 200 is further configured to: Calculate the average color vector of each image patch in the image group; Calculate the mean color vector of the image group based on the average color vector of each image block; The structural similarity of the current image group is obtained by calculating the cosine similarity between the average color vector of each image block in the image group and the mean color vector.

[0074] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0075] In one embodiment, the image redundancy calculation module 200 is further configured to: Generate corresponding RGB histograms based on the image patches in the image group; Calculate the chi-square distance between every two image blocks in the image group based on the RGB histogram; Based on the chi-square distance, the color distribution similarity of the current image group is obtained.

[0076] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0077] In one embodiment, the image group filtering module 300 is specifically used for: The image patch with the highest cosine similarity to the mean color vector in the redundant image group is selected as the target image patch. Remove non-target image blocks from the redundant image group; The target image block is combined with image blocks in the non-redundant image group to generate a non-redundant image group set.

[0078] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0079] In one embodiment, the target image generation module 400 is specifically used for: Obtain the original coordinates of each image patch in the non-redundant image set; Generate a corresponding position embedding vector based on the original coordinates; Based on the location embedding vector, all image blocks in the non-redundant image set are arranged to generate the target input image.

[0080] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0081] This invention provides an inference acceleration device for visual language models. By removing irrelevant and redundant visual information contained in images before performing word segmentation, and using lightweight statistical measures such as structural and color-based features to evaluate local similarity, more interpretable pruning decisions can be made earlier, thereby improving the inference efficiency of visual language models.

[0082] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements server-side functions or steps of a reasoning acceleration method for visual language models.

[0083] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a reasoning acceleration method for visual language models.

[0084] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, including: Obtain the original input image to be processed, scale the original input image to a preset resolution, and generate the first input image; The first input image is normalized to generate the second input image; The second input image is divided into several non-overlapping image blocks; Each pair of adjacent N*N image blocks is combined to generate several image groups, where N is an integer greater than or equal to 2.

[0085] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the original input image to be processed, preprocess the original input image, and divide the preprocessed original input image into several image groups, wherein each image group includes several image blocks; Calculate the semantic similarity of all image groups respectively, and denote the image groups whose semantic similarity satisfies the first similarity condition as redundant image groups; Obtain target image blocks that satisfy the second similarity condition from the redundant image group, and combine the target image blocks with image blocks from the non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group consists of image groups after removing redundant image groups from all image groups; The non-redundant image set is arranged to generate the target input image.

[0086] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0087] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0088] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can exist in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0090] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0091] Among other things, conditional language such as “can,” “may,” “may,” or “may,” unless otherwise specifically stated or otherwise understood as in the context in which they are used, is generally intended to convey that a particular implementation may include (but not others) certain features, elements, and / or operations. Therefore, such conditional language is also generally intended to imply that features, elements, and / or operations are necessary for one or more implementations in any way, or that one or more implementations must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular implementation.

[0092] The contents already described herein in this specification and accompanying drawings include examples of methods and apparatuses capable of providing reasoning acceleration for visual language models. It is certainly not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of this disclosure, but it will be appreciated that many other combinations and substitutions of the disclosed features are possible. Therefore, it will be apparent that various modifications can be made to this disclosure without departing from the scope or spirit of this disclosure. Furthermore, or in alternatives, other embodiments of this disclosure may become apparent from consideration of this specification and accompanying drawings and from practice of this disclosure as presented herein. It is intended that the examples presented in this specification and accompanying drawings be considered illustrative rather than restrictive in all respects. Although specific terminology is used herein, it is used in a general and descriptive sense and is not intended for limiting purposes.

Claims

1. A method for inference acceleration of visual language models, the method comprising: The method comprises: obtaining an original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; respectively calculating semantic similarities of all image groups, and recording an image group satisfying a first similarity condition as a redundant image group; obtaining a target image block in the redundant image group satisfying a second similarity condition, combining the target image block with image blocks in a non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; arranging the non-redundant image group set to generate a target input image.

2. The inference acceleration method for visual language model according to claim 1, wherein, The method comprises: obtaining an original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; respectively calculating semantic similarities of all image groups, and recording an image group satisfying a first similarity condition as a redundant image group; obtaining a target image block in the redundant image group satisfying a second similarity condition, combining the target image block with image blocks in a non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; arranging the non-redundant image group set to generate a target input image.

3. The inference acceleration method for visual language models of claim 1, wherein, The method comprises: obtaining an original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; respectively calculating semantic similarities of all image groups, and recording an image group satisfying a first similarity condition as a redundant image group; obtaining a target image block in the redundant image group satisfying a second similarity condition, combining the target image block with image blocks in a non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; 4. The inference acceleration method for visual language model according to claim 3, wherein, arranging the non-redundant image group set to generate a target input image. The method comprises: obtaining an original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; respectively calculating semantic similarities of all image groups, and recording an image group satisfying a first similarity condition as a redundant image group; 5. The inference acceleration method for visual language models of claim 3, wherein, obtaining a target image block in the redundant image group satisfying a second similarity condition, combining the target image block with image blocks in a non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; arranging the non-redundant image group set to generate a target input image. The method comprises: obtaining an original input image to be processed, preprocessing the original input image, and dividing the preprocessed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; 6. The inference acceleration method for visual language model according to claim 4, wherein, respectively calculating semantic similarities of all image groups, and recording an image group satisfying a first similarity condition as a redundant image group; obtaining a target image block in the redundant image group satisfying a second similarity condition, combining the target image block with image blocks in a non-redundant image group to generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; arranging the non-redundant image group set to generate a target input image. ​ 7. The inference acceleration method for visual language models of claim 1, wherein, ​ obtaining original coordinates of each image block in the non-redundant image group set; generating a corresponding position embedding vector according to the original coordinates; arranging all image blocks in the non-redundant image group set based on the position embedding vector to generate a target input image.

8. An inference acceleration apparatus for a visual language model, the apparatus comprising: The device comprises: an image preprocessing module configured to obtain an original input image to be processed, pre-process the original input image, and divide the pre-processed original input image into a plurality of image groups, wherein each image group comprises a plurality of image blocks; an image redundancy calculation module configured to calculate semantic similarity of all image groups respectively; an image group screening module configured to obtain target image blocks in a redundant image group that meet a second similarity condition, combine the target image blocks with image blocks in a non-redundant image group, and generate a non-redundant image group set, wherein the non-redundant image group is composed of image groups after removing the redundant image group from all image groups; a target image generation module configured to arrange the non-redundant image group set to generate a target input image.

9. A computer device, comprising: The computer device comprises at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the inference acceleration method for a visual language model according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions, when executed by one or more processors, can cause the one or more processors to perform the steps of the inference acceleration method for a visual language model according to any one of claims 1-7.

Images