Method and device for realizing multi-modal inference acceleration by computing power reuse token through intelligent computing cloud platform

Abstract

The application provides a method and device for realizing multi-modal reasoning acceleration by algorithm power reuse Token through an intelligent computing cloud platform, relates to the technical fields of intelligent computing centers, intelligent computing centers, algorithm power infrastructure and intelligent computing cloud, and the method comprises the following steps: receiving multi-modal visual content for standardized processing to obtain at least one visual content block; generating Token fingerprints from multiple complementary hashes of each visual content block to retrieve candidate caches; based on attribute information associated with the current request for the visual content, if attribute information and metadata of the candidate cache are successfully matched, calling the algorithm power of the intelligent computing cloud platform to obtain Token and / or Token-KV; according to the prompt word of the current request, the position mapping parameter and the bridging Token of the candidate cache, performing position remapping and bridging Token recalculation to reason the current request. The efficiency and reliability of Token reuse can be improved, and the algorithm power consumption can be reduced.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud computing technologies, specifically to a method and apparatus for accelerating multimodal reasoning through computing power reuse tokens in an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] With the widespread adoption of images, Portable Document Format (PDF) pages, webpage screenshots, scanned documents, graphical user interfaces (GUIs), video clips, audio transcriptions, and structured documents in enterprise-level multimodal large-scale model services, the repeated inference of the same visual content across different sessions, questions, text-image sequences, and tenant permission boundaries is becoming increasingly common. If each request requires re-encoding the same or similar visual content, regenerating visual tokens, and recalculating visual key-value pairs, it results in significant waste of computational resources. Therefore, reusing caches to accelerate inference and reduce computational consumption becomes an essential choice.

[0008] However, while intelligent computing centers provide ample computing power for large-scale model inference, existing caching mechanisms have significant flaws. For example, general prefix caching requires strict input consistency. When visual content is scaled, cropped, or shifted in position, the precise prefix condition fails, leading to the inability to reliably identify and reuse repeated visual content across contexts, resulting in wasted computing power. Secondly, existing visual token pruning is mostly based on static saliency, lacking problem awareness. This not only results in coarse computing power allocation and easy deletion of critical task areas, but also lacks a cross-modal position remapping mechanism. Furthermore, cross-user shared KV cache is prone to side-channel attacks. The lack of a mechanism to deeply bind tenant isolation, permission tags, and cache lifecycle leads to unclear boundaries of multi-tenant sharing, creating security risks while pursuing computing power sharing efficiency.

[0009] It is evident that since the emergence of intelligent computing centers, how to stably identify repetitive visual content, realize cross-context reuse of visual tokens and KV caches, avoid cache invalidation due to location changes, and solve the problem of unclear shared boundaries of multi-tenant caches in order to maximize computing power utilization efficiency and achieve secure token reuse has always been a pressing problem in this field. Summary of the Invention

[0010] This invention provides a method and apparatus for accelerating multimodal inference in an intelligent computing cloud platform by reusing computing power through tokens, in order to solve problems such as the inability to reliably identify repetitive visual content, the difficulty in reusing visual tokens and KV caches across contexts, cache invalidation due to location changes, coarse allocation of visual token computing power, and unclear boundaries of multi-tenant cache sharing in existing multimodal inference.

[0011] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform, comprising: Step S1: Receive multimodal visual content, and perform corresponding normalization processing on the visual content according to the type of each visual content to obtain at least one visual content block; Step S2: Generate the Token fingerprint corresponding to the visual content block from at least two of the following: perceptual hash, optical character recognition (OCR) hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features. Step S3: Based on the Token fingerprint, retrieve the candidate cache that matches the Token fingerprint; Step S4: Based on the attribute information associated with the current request for the visual content, if the attribute information successfully matches the metadata of the candidate cache, the reuse permission verification of the candidate cache passes, and the computing power of the intelligent computing cloud platform is invoked to obtain the Token and / or key-value pair Token-KV of the candidate cache. Step S5: Based on the prompt word of the current request, the location mapping parameters of the candidate cache and the bridging token, perform location remapping on the token and / or token-KV, recalculate the bridging token, and use the location remapping results corresponding to all visual content blocks and the recalculated bridging token to infer the current request.

[0012] In one embodiment, step S4 includes: Step S4.1: Calculate the task relevance between the candidate cache token and the current request based on the prompt word and task type of the current request; Step S4.2: Based on the cache confidence, recalculation cost, security constraints, and task relevance of all candidate caches, determine the computing power budget of the intelligent computing cloud platform for acquiring Token and / or Token-KV. Step S4.3: Based on the computing power budget, obtain the Token and / or key-value pair Token-KV of the candidate cache.

[0013] In one embodiment, prior to step S3, the method further includes: Step S6: For the visual content requested for the first time, call the encoder to generate the token sequence of the visual content and determine the intermediate state of the encoder. Step S7: Input the Token sequence and the prompt word of the initial request into the model for pre-filling, and determine the Token-KV cache, location mapping parameters and bridging Token related to the Token sequence; Step S8: Construct and store a candidate cache based on the Token sequence, the intermediate state, the Token-KV cache, the location mapping parameters, the bridging Token, and the metadata corresponding to the visual content.

[0014] In one embodiment, step S8 includes: Step S8.1: Divide the visual content into multiple region blocks, determine the region coordinates and region type of each region block, and the region token corresponding to the region block in the token sequence; Step S8.2: Based on the region type, determine at least two of the following: perception hash, OCR hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features of the region block. Step S8.3: Determine the cache descriptor of the region block from at least two hashes, the region coordinates, the region type, the region token, and the metadata corresponding to the visual content, and store it as a candidate cache.

[0015] In one embodiment, step S3 includes: Step S3.1: Based on the current request for the visual content and the task type of the current request, determine the weight corresponding to each type hash; Step S3.2: Based on the weights, perform a weighted summation of the similarity between each candidate cache and the visual content block across all types of hashes to determine the similarity between the candidate cache and the visual content block; Step S3.3: The candidate caches with a similarity greater than the first similarity threshold are determined as the candidate caches that the Token fingerprint hits.

[0016] In one embodiment, step S8 includes: Step S8.1': Predict the reuse probability, computing power saving and time saving of the candidate cache, and determine the verification confidence, storage consumption, security risk level and life cycle of the candidate cache; Step S8.2': Calculate the retention priority of the candidate cache using the reuse probability, the saved computing power and time, the verification confidence, the storage usage, the security risk level, and the lifecycle; Step S8.3': Select the corresponding caching level strategy according to the retention priority and store the candidate cache.

[0017] In one embodiment, step S6 includes: Step S6.1: If the visual content is video, determine the set of keyframes of the video and the change region between each pair of adjacent frames in the video; Step S6.2: Determine the target token cached for each keyframe and the increment of the changed region between each two adjacent frames; Step S6.3: For each ordinary frame in the video except for the keyframe, determine the target token of the keyframe that is the fewest frames apart from the ordinary frame, and determine the change token of the ordinary frame relative to the keyframe based on the increment of all the change regions between the ordinary frame and the keyframe. Step S6.4: Combine the target token with the change token to obtain the target token of the ordinary frame buffer.

[0018] In one embodiment, step S8 includes at least one of the following: Step S8.1'': When the change area between the normal frame and the adjacent frame is less than the change threshold, extract the digest token from the target token of the normal frame, and store the KV of the target token of the key frame and the digest token of the normal frame. Step S8.2'': If the change region type between the normal frame and the adjacent frame is a person or object region, determine and store the key value of the change token and the key value of the target token of the key frame.

[0019] In one embodiment, step S5 includes: Step S5.1: Determine the attention region of the current request in the visual content, wherein the attention region contains at least one visual content block; Step S5.2: Calculate the regional attention degree of the attention region using the current request question vector, the region vector of the attention region, the historical attention score of the attention region, and the type-corresponding weight of the attention region; Step S5.3: Convert the region attention into an attention mask bias. Based on the recalculated bridging token and the text token of the current request, superimpose the attention mask bias on the position remapping result corresponding to the visual content block contained in the attention region, and infer the current request.

[0020] In one embodiment, after step S3, the method further includes: Step S9: If the similarity corresponding to the candidate cache is less than the second similarity threshold, the candidate cache is verified. The second similarity threshold is greater than the first similarity threshold. The verification includes at least one of the following: whether there is a risk field, whether the embedding distance is greater than the distance threshold, whether the regional OCR is consistent, whether the object set is consistent, and whether the difference between the recalculated part of the token and the cache is higher than the difference threshold. Step S10: If one of the verification results fails, the candidate cache is rolled back.

[0021] Secondly, the present invention also provides an apparatus for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform, comprising: The input processing module is used to receive multimodal visual content, perform corresponding normalization processing on the visual content according to the type of each visual content, and obtain at least one visual content block. The fingerprint generation module is used to generate a Token fingerprint corresponding to the visual content block from at least two of the following: perceptual hash, optical character recognition (OCR) hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features. The cache retrieval module is used to retrieve candidate caches that match the Token fingerprint based on the Token fingerprint. The verification module is used to verify the reuse permission of the candidate cache by using the attribute information associated with the current request for the visual content. If the attribute information matches the metadata of the candidate cache successfully, the module will call the computing power of the intelligent computing cloud platform to obtain the Token and / or key-value pair Token-KV of the candidate cache. The loading module is used to remap the location of the Token and / or Token-KV based on the prompt word of the current request, the location mapping parameters of the candidate cache, and the bridging Token, and recalculate the bridging Token. It then uses the location remapping results corresponding to all visual content blocks and the recalculated bridging Token to reason about the problem.

[0022] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps in the method for accelerating multimodal inference through computing power reuse tokens in the intelligent computing cloud platform described in the first aspect above.

[0023] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the method for accelerating multimodal inference through computing power reuse tokens in the intelligent computing cloud platform described in the first aspect above.

[0024] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the method for accelerating multimodal inference through computing power reuse tokens in the intelligent computing cloud platform described in the first aspect above.

[0025] In this invention, step S1 involves receiving multimodal visual content and performing corresponding normalization processing on the visual content according to the type of each visual content to obtain at least one visual content block; step S2 involves analyzing the perceptual hash and optical character recognition (OCR) of each visual content block. Step S3: Generate a Token fingerprint corresponding to the visual content block based on at least two of the following: hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features; Step S4: Retrieve candidate caches that match the Token fingerprint based on the Token fingerprint; Step S5: Based on the attribute information associated with the current request for the visual content, if the attribute information matches the metadata of the candidate cache, the reuse permission verification of the candidate cache passes, and the computing power of the intelligent computing cloud platform is invoked to obtain the Token and / or key-value pair Token-KV of the candidate cache; Step S6: Based on the prompt words of the current request, as well as the position mapping parameters and bridging Token of the candidate cache, perform position remapping on the Token and / or Token-KV, and recalculate the bridging Token. Use the position remapping results corresponding to all visual content blocks and the recalculated bridging Token to infer the current request. In this way, by receiving and standardizing multimodal visual content, extracting multidimensional content such as perception, OCR, and structure, abstract hashing is used to generate token fingerprints. Candidate caches matching the token fingerprints are retrieved, and permission verification is performed by combining them with request attributes, such as user identifiers. Computing power is then used to obtain reusable cached assets for the current inference scenario. Based on prompt words and location mapping parameters, location remapping and bridging token recalculation are performed, ultimately completing the inference process. This allows the use of multidimensional fingerprints to overcome the strict input consistency restriction of traditional prefix caching, achieving stable reuse of visual token-KV assets across locations and deformations. Combined with permission verification and bridging recalculation, while ensuring multi-tenant data security, it significantly reduces redundant calculations in the visual encoding and prefill stages, significantly reducing prefill overhead in multimodal inference, lowering memory usage and overall computing power consumption, and improving inference throughput. Attached Figure Description

[0026] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a method for accelerating multimodal reasoning through the reuse of computing power tokens in an intelligent computing cloud platform, provided by the present invention. Figure 2 This is a structural diagram of a device provided by the present invention that accelerates multimodal reasoning through the reuse of computing power tokens in an intelligent computing cloud platform; Figure 3 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0028] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0030] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0031] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0032] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), while the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB). Disaster recovery ratio is an important indicator of security and reliability.

[0033] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0034] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0035] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0036] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0037] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0038] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0039] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0040] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0041] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0042] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0043] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0044] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0045] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0046] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0047] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language, and is trained with a large amount of text data. It can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0048] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0049] The “Token” mentioned in this invention refers to the basic unit of measurement when a large language model or multimodal large model processes text, code, image descriptions, document content, or other serialized inputs and outputs.

[0050] It is important to emphasize that different models use different tokenizers. The same text may correspond to different numbers of tokens in different models.

[0051] The "Token fingerprint" described in this invention refers to a fingerprint generated by extracting features from multimodal data (text, images, audio, etc.) to uniquely identify the content. In this invention, it consists of multiple hashes, each of which can serve as a sub-fingerprint of a type.

[0052] The “Token-KV” mentioned in this invention refers to the intermediate state generated by the token after specific mathematical operations within the model, which is used to accelerate subsequent calculations.

[0053] It should be noted that the hash in this invention is calculated using encoding representations such as SimHash, MinHash, Locality Sensitive Hashing, and Binary Semantic Code, or combinations thereof, which can maintain the semantic similarity of the input. Unlike traditional hash algorithms (such as MD5), which allow small changes in the input to lead to huge differences in the output, this hash calculation method can output similar hash values ​​for inputs with similar content.

[0054] Please see Figure 1 , Figure 1 This is a flowchart of a method for accelerating multimodal inference through token reuse in an intelligent computing cloud platform, as provided by the present invention. Figure 1 As shown, the method includes: Step S1: Receive multimodal visual content, and perform corresponding normalization processing on the visual content according to the type of each visual content to obtain at least one visual content block.

[0055] The aforementioned multimodal visual content may include, but is not limited to, at least one of the following: images, PDF pages, scanned documents, webpage screenshots, screenshots of documents such as zither, video clips, audio clips, text, layout structure files, user questions, and business context.

[0056] In one embodiment, in order to ensure that the same visual content remains recognizable after slight changes, the present invention may perform input normalization processing before fingerprint generation. This normalization processing may include, but is not limited to, scaling normalization, orientation correction, color space conversion, border removal, watermark removal and weakening, video frame extraction, page segmentation, region coordinate normalization, and noise estimation of the input visual content.

[0057] In one embodiment, different types of visual content can be processed in different ways; for example, for images, the input image... The image is converted to a uniform color space, and edge detection, tilt angle estimation, scale normalization and effective content box extraction are performed to obtain a normalized image as a visual content block. The normalization process can be represented by the following formula (1).

[0058] (1) In formula (1), Indicates the scale parameter. Indicates rotation or tilt correction parameters. Indicates the valid content bounding box. This indicates parameters for compression noise, brightness, and contrast correction.

[0059] Alternatively, for PDF pages, the mapping relationship between the original page coordinate system and the normalized image coordinate system can be preserved. The normalization process can be represented by the following formula (2).

[0060] (2) In formula (2), These are the coordinates of the original PDF page. To normalize coordinates, Let be the affine transformation matrix. For translation vectors. The mapping relationship in formula (2) can be used for subsequent region-level cache hits and visual token position remapping.

[0061] Alternatively, for video input, video clips can be used. Divided into frame sequences The set of keyframes is selected based on inter-frame differences, object motion, shot transitions, and subtitle changes. As shown in the following formula (3), each keyframe can be identified as a visual content block.

[0062] (3) In formula (3), For visual frame difference function, For the degree of change of the object collection, The degree of change in subtitles or OCR. By standardizing the video using formula (3), static background frames, repeating frames, and key change frames in the video can be distinguished and processed with different caching and reuse methods, thereby reducing computational power consumption.

[0063] It should be noted that, in the embodiments of the present invention, the granularity of the visual content block can be dynamically defined according to actual needs. For example, for an image, it can be the entire image, or it can be a partial area of ​​text or a table in the image.

[0064] The beneficial technical effects achieved by step S1 are: standardizing visual content into content blocks according to type enables more precise and accurate scheduling of computing power and avoids redundant calculations; at the same time, it facilitates the generation of independent token fingerprints, improves the reuse rate of token-KV cache, significantly reduces the computing power overhead in the Prefill stage, and improves inference throughput.

[0065] Step S2: Generate a token fingerprint corresponding to each visual content block from at least two of the following: perceptual hash, optical character recognition (OCR) hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features.

[0066] The aforementioned token fingerprint, also known as a cache descriptor, uniquely identifies its corresponding visual content block and is used to retrieve existing caches with similar content. Each hash that makes up the token fingerprint is a sub-fingerprint, and multiple sub-fingerprints are complementary to each other.

[0067] In one embodiment, for a visual content block B, a token fingerprint can be generated according to the following formula (4).

[0068] (4) In formula (4), This indicates that a fusion function can be used, such as SimHash, MinHash, Locality Sensitive Hashing, Bloom Filter Coding, Vector Quantization Coding, or a combination thereof. This represents a perceptual hash, used to resist scaling, compression, and brightness changes. This represents the OCR text hash. This indicates that the object is detected by hash. Represents the page layout hash. This represents the hash of the table structure. This represents the patch distribution characteristics of a visual image. Representing the region coordinates and geometric structure, This represents the time fingerprint of a video keyframe. This indicates audio transcription or sound event hash.

[0069] It should be noted that for different video content blocks, when generating their corresponding Token fingerprints using formula (4), there may be different types of hash values ​​of 0. For example, if the video content block is a table in an image, then the table structure hash is not 0 when generating the Token fingerprint, and the audio transcription hash value is 0.

[0070] In one possible implementation, the OCR text hash may not directly use the complete plaintext text. Instead, it may first perform field normalization, sensitive field desensitization, and location binding on the OCR result. For example, fields such as amount, date, contract number, invoice number, and name may generate field type, field value summary, and field location summary, respectively. Then, the visual content block can be calculated using the following formula (5). .

[0071] (5) In formula (5), This indicates the words recognized by OCR. Indicates the field category, This indicates the normalized coordinates of the field on the page. Indicates OCR confidence level. This represents the de-identification or digest function executed according to the security policy. Calculating the OCR text hash using the method shown in formula (5) improves fingerprint stability and reduces the risk of sensitive text being leaked from the cache index.

[0072] In one embodiment, the object detection hash can be composed of object category, bounding box, confidence score, key points, and local visual descriptors, and the visual content block can be calculated using the following formula (6). .

[0073] (6) In formula (6), For object categories, For normalized bounding boxes, To test the confidence level, This is a visual prototype of a localized object, avoiding the sensitivity to subtle changes caused by directly using the visual prototype for hashing.

[0074] For example, for contracts, invoices, and PDF pages, the layout hash also includes the relative relationships between the title area, body area, table area, signature area, header and footer area, image area, and QR code area; for GUI interfaces, it also includes the structural relationships between buttons, input boxes, menus, pop-ups, and table controls.

[0075] In one embodiment, multimodal hashing can be used to determine whether a page, table, signature, part of a product image, or keyframe of a video in a PDF is reusable.

[0076] It should be noted that multimodal token fingerprints can be implemented using SimHash, MinHash, locality-sensitive hash, vector database approximation retrieval, binary semantic codes, graph neural network structure encoding, or a combination thereof.

[0077] It should be noted that perceptual hashing can be implemented using average hashing, difference hashing, discrete cosine transform hashing, depth-perceptual hashing, CLIP-type visual embedding hashing, or self-trained visual fingerprinting networks.

[0078] OCR hashes can be implemented using plain text hashes, de-identified hashes, field type hashes, synonym normalized hashes, edit distance signatures, character n-gram signatures, or structured key-value pair hashes.

[0079] Object detection hashes can come from general detection models, domain detection models, segmentation models, key point models, object tracking models, or multi-task visual encoders.

[0080] Page layout hashing can be implemented based on rule-based page layout analysis, document understanding models, Document Object Model (DOM) trees, control trees, PDF internal structure, table recognition models, or graph structure encoding.

[0081] The beneficial technical effects achieved by step S2 are as follows: by converting images, PDF pages, webpage screenshots, GUI interfaces, video keyframes, OCR text, and layout structures into searchable, comparable, and authorizable multimodal token fingerprints, the intelligent computing cloud platform can identify inputs that are different in file but have the same visual semantics or are partially similar, which can significantly expand the multimodal cache hit range.

[0082] Step S3: Based on the Token fingerprint, retrieve the candidate cache that matches the Token fingerprint.

[0083] In this invention, multimodal token fingerprints can be used to retrieve candidate caches that have a complete hit, a perceptual similarity hit, a region hit, a table structure hit, a video keyframe hit, a weakly similar candidate hit, or a multi-level hit in the cache index.

[0084] In one embodiment, different remapping, recalculation, and verification strategies can be adopted for different hit levels, thereby improving the reuse range and controlling reuse errors.

[0085] It should be noted that in this invention, the savings brought by visual token-kV cache hits can be transformed into quantifiable metrics, providing a basis for computing power scheduling, GPU memory tiering, cost optimization, and commercial billing.

[0086] The beneficial technical effects achieved by step S3 are: accurate retrieval of cache based on token fingerprints, direct reuse of historical token-KV states, avoidance of repeated model reasoning, thereby greatly saving computing power consumption and significantly improving token generation efficiency and system response speed.

[0087] Step S4: Based on the attribute information associated with the current request for visual content, if the attribute information matches the metadata of the candidate cache successfully, the reuse permission verification of the candidate cache passes, and the computing power of the intelligent computing cloud platform is invoked to obtain the Token and / or key-value pair Token-KV of the candidate cache.

[0088] The aforementioned attribute information may include, but is not limited to, the user identifier of the user making the current request, the project identifier, the user role, data security tags, etc.

[0089] In one embodiment, in a multi-tenant cloud computing platform, cache reuse must be subject to strict security controls. Therefore, in this invention, the metadata of each candidate cache... Multiple fields can be bound as shown in the following formula (7).

[0090] (7) In formula (7), Indicates tenant identifier, Indicates the project identifier. Indicates data category, Indicates a safety label. Indicate whether it is public material. Indicates the cache expiration period. Indicates the reuse strategy, This indicates the audit link identifier.

[0091] Therefore, when candidate cache C meets the conditions shown in the following formula (8), the reuse permission verification of candidate cache can be passed, and computing power can be called for reuse.

[0092] (8) As can be seen from the above formula (8), similarity hit is only one of the necessary conditions, and the permission policy and leakage risk must also be satisfied.

[0093] In some possible implementations, different reuse modes can be adopted according to the security level to dynamically adjust the metadata of candidate caches. For example, the public mode allows cross-tenant reuse; the tenant mode only allows reuse within the same tenant; the project mode only allows reuse within the same project; the session mode only allows reuse within the same session; the de-identification mode only allows reuse of structure tokens or low-precision visual summaries; and the disabled mode does not write to the cache or only temporarily caches within the request.

[0094] In this invention, in a multi-tenant computing platform, tenants, projects, permissions, security tags, public material identifiers, cache retention policies, and billing identifiers can be bound to cache metadata. Based on tenants, projects, user roles, data security tags, public material identifiers, desensitization status, and cache domain policies, it can determine whether candidate caches are allowed to be loaded, thereby achieving controllable cross-session reuse and auditable cross-tenant reuse boundaries.

[0095] In one embodiment, within an intelligent computing cloud platform, after verifying the reuse permissions of candidate caches, the reused Token and / or Token-KV in each candidate cache can be determined using the allocated computing power budget. When the computing power budget is sufficient, high-precision cross-modal semantic alignment or traversing deeper cache trees can be performed to ensure high recall; conversely, lightweight algorithms or limiting the retrieval depth can be used to quickly return results and avoid exhausting the overall inference computing power during the retrieval phase.

[0096] It should be noted that the visual token-kv cache can store full precision, 8-bit floating-point numbers (FP8), 8-bit integers (INT8), 4-bit integers (INT4), mixed precision, low-rank decomposition, residual quantization, dictionary encoding, or sparse representation.

[0097] The beneficial technical effects achieved in step S4 are as follows: The intelligent computing cloud platform uses multimodal token fingerprinting to identify the same or similar visual regions, page layouts, graphic and text combinations, and cross-session reuse opportunities. It loads the already computed visual token-KV or regional token representation as reusable computing power assets, thereby avoiding repeated visual encoding and repeated prefilling, and realizing the acceleration of multimodal inference by reusing tokens through computing power.

[0098] Step S5: Based on the prompt words of the current request, the location mapping parameters of the candidate cache and the bridging token, perform location remapping on the token and / or token-KV, and recalculate the bridging token. Use the location remapping results corresponding to all visual content blocks and the recalculated bridging token to infer the current request.

[0099] The prompt can be multimodal, and can be composed of system commands, user questions, image placeholders, OCR text, page summaries, search fragments, and tool call results.

[0100] In one embodiment, the multimodal prompt for the current request can be decoupled to determine that the prompt is composed of text blocks. and visual blocks If the words are interleaved, the prompt words can be represented as a set of combinable fragments as shown in formula (9). .

[0101] (9) For each visual block in the prompt words It can independently generate fingerprints, independently retrieve cached data, independently determine permissions, and independently perform position remapping. Thus, even if the position of the same visual block moves from the beginning to the middle in different requests, or the preceding and following text changes (e.g., from "Please summarize this image" to "Please extract the table amount"), the visual token-key value can still be reused as an independent asset. The impact of differences between text tokens and visual tokens is handled by the bridging token recalculation module, rather than requiring complete consistency of the prompt words.

[0102] In one possible embodiment, visual block boundaries can also be utilized. , area boundary token and bridging token This explicitly marks the combination relationships of visual content within the current cue word. Key-value pairs (KV) within a visual block are reusable, and the interaction edges between visual blocks and text blocks are locally recalculated based on the current context.

[0103] In this invention, the position of the same visual content may differ in different prompts, and the attention calculation of Transformer typically includes positional encoding. Directly loading the old key-value pairs may lead to inconsistent positions. Therefore, a corresponding mapping strategy can be adopted based on the encoding method used by the model to complete the positional remapping of tokens and / or token-key-value pairs.

[0104] For example, for a model that uses rotational position encoding, when the position of the visual token in the old prompt corresponding to the candidate cache is... Its position in the current prompt is Then the key vector can be modified as shown in the following formula (10).

[0105] (10) In formula (10), Indicates position The corresponding rotation matrix. For the value vector, if the value in the model structure is not directly affected by the rotation position encoding, it can remain unchanged; if the model uses position-related projection or relative position bias, the system synchronously corrects the relevant position bias table or local attention mask.

[0106] Alternatively, for models that use absolute position embedding, a new position embedding can be calculated based on position interpolation and local reprojection, as shown in the following formula (11).

[0107] (11) In formula (11), This represents the position interpolation function. This represents the position correction term estimated from the current request context.

[0108] For two-dimensional visual positions, the intelligent computing cloud platform can maintain both the visual region coordinate position and the prompt sequence position simultaneously, as shown in the following formula (12). By separating the sequence position, two-dimensional spatial position and video temporal position, it is possible to handle changes in image insertion position, page coordinates, and video frame order.

[0109] (12) It should be noted that position remapping can also be achieved through relative position offset recalculation, two-dimensional coordinate reprojection, time-position recalibration, or model-specific adapters.

[0110] It should be noted that this invention is applicable not only to visual language models, but also to multimodal mixture of experts (MoE) models, document understanding models, video language models, GUI agent models, OCR enhancement models, and industrial visual reasoning models.

[0111] In this invention, when a visual token-key value is loaded from a historical context into the context of the current request, the visual content itself can be reused, but the text question, system instructions, and retrieval context may change. If cross-modal interactions are not recalculated at all, the model may not be able to correctly focus on the area required by the current question; if it is completely recalculated, the caching value is lost. Therefore, a partial bridging token recalculation mechanism can be adopted.

[0112] In one embodiment, the recalculation of the bridging token is used to locally recalculate only the necessary bridging tokens, attention edges, and interaction areas between the visual token and the text token when the text context changes or the image-text interleaving order changes. In one embodiment, the set of reusable visual tokens is: The current text token set is The bridge token set is The local update of the bridging token and the visual region selected by the problem-aware budget can be performed as shown in the following formula (13).

[0113] (13) In formula (13), and This is a subset of key-value pairs (KV) for the question-related visual tokens, with semicolons indicating concatenation. Unselected background visual tokens are still represented using cached representations and are not subject to costly recalculation. The bridging token can be understood as an adapter between the current question and the reused visual content, used to restore the selective focus of the text question on visual regions.

[0114] Understandably, the number of bridging tokens is much smaller than the number of visual tokens. For example, a single PDF page requires 16 to 64 bridging tokens, a product image requires 8 to 32 bridging tokens, and a video requires 4 to 16 bridging tokens per keyframe. Bridging tokens can be dynamically increased or decreased based on the task type: table-based question-and-answer scenarios add table bridging tokens, defect detection scenarios add local edge bridging tokens, and video event detection scenarios add time bridging tokens.

[0115] Therefore, in this disclosure, by decoupling the visual token-KV from the complete prompt, visual content can still be reused in different text contexts, different image-text orders, different prompt positions, and after partial rearrangement. Through the decoupling of multimodal token fingerprints and visual token-KV, the "reusable object" is expanded from the complete prompt prefix to visual content blocks, visual region blocks, table cell blocks, video keyframe blocks, and audio transcription blocks. Even if the visual content is not in the same prefix in subsequent requests, the intelligent computing cloud platform can still achieve reuse through position-independent matching and bridging token recalculation.

[0116] The beneficial technical effects achieved by step S5 are as follows: By remapping the location and recalculating the bridge token, the historical token-KV cache and the current prompt word can be seamlessly spliced ​​together, which greatly reduces redundant calculations, thereby saving computing power and significantly improving the efficiency of token reasoning and generation.

[0117] In this invention, step S1 involves receiving multimodal visual content and performing corresponding normalization processing on the visual content according to the type of each visual content to obtain at least one visual content block; step S2 involves analyzing the perceptual hash and optical character recognition (OCR) of each visual content block. Step S3: Generate a Token fingerprint corresponding to the visual content block based on at least two of the following: hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features; Step S4: Retrieve candidate caches that match the Token fingerprint based on the Token fingerprint; Step S5: Based on the attribute information associated with the current request for the visual content, if the attribute information matches the metadata of the candidate cache, the reuse permission verification of the candidate cache passes, and the computing power of the intelligent computing cloud platform is invoked to obtain the Token and / or key-value pair Token-KV of the candidate cache; Step S6: Based on the prompt words of the current request, as well as the position mapping parameters and bridging Token of the candidate cache, perform position remapping on the Token and / or Token-KV, and recalculate the bridging Token. Use the position remapping results corresponding to all visual content blocks and the recalculated bridging Token to infer the current request. In this way, by receiving and standardizing multimodal visual content, extracting multidimensional content such as perception, OCR, and structure, abstract hashing is used to generate token fingerprints. Candidate caches matching the token fingerprints are retrieved, and permission verification is performed by combining them with request attributes, such as user identifiers. Computing power is then used to obtain reusable cached assets for the current inference scenario. Based on prompt words and location mapping parameters, location remapping and bridging token recalculation are performed, ultimately completing the inference process. This allows the use of multidimensional fingerprints to overcome the strict input consistency restriction of traditional prefix caching, achieving stable reuse of visual token-KV assets across locations and deformations. Combined with permission verification and bridging recalculation, while ensuring multi-tenant data security, it significantly reduces redundant calculations in the visual encoding and prefill stages, significantly reducing prefill overhead in multimodal inference, lowering memory usage and overall computing power consumption, and improving inference throughput.

[0118] It is understood that the method proposed in this invention can be applied to the following scenarios: when the same contract, instruction manual, product image, quality inspection image, expense reimbursement invoice, webpage screenshot, surveillance video clip, training PPT page, or business system interface is repeatedly asked by a large number of users; when the same visual object is scaled, cropped, compressed, slightly rotated, watermarked, or reformatted and then re-entered into the model; when the same PDF page is inserted into different positions in the prompt in different search enhancement contexts; when repeated frames, static background frames, and a few changing frames in the same video are repeatedly analyzed; and when multi-tenant cloud services need to reuse the computing power assets of public visual materials within controllable boundaries while avoiding cross-tenant data leakage.

[0119] It should be noted that the method proposed in this invention for accelerating multimodal inference by reusing tokens with computing power is designed for intelligent computing cloud platforms that support image understanding, document question answering, screenshot analysis, video frame understanding, and multi-tenant knowledge services. This enables the intelligent computing cloud platform to determine which multimodal tokens are worth pre-computing, caching, reusing across requests, or partially recompiling. After privacy permissions, model version, and location mapping verification are passed, the reused tokens are converted into direct first tokens to reduce latency and GPU cost gains.

[0120] It should be noted that in this invention, user questions, task types, OCR fields, object categories, historical attention, regional importance, and cache heat can be used to determine the visual token retention rate, KV accuracy, cache level, and recalculation range.

[0121] It should be noted that in this invention, the computational power allocation for visual tokens is completed through problem perception.

[0122] In one embodiment, step S4 includes: Step S4.1: Calculate the task relevance between the candidate cache token and the current request based on the prompt word and task type of the current request.

[0123] In one embodiment, the task relevance score between the current request prompt and task type can be quantitatively calculated by converting them into an attention query vector and performing cross-modal semantic alignment and feature interaction with the candidate cache token (key).

[0124] Step S4.2: Based on the cache confidence, recalculation cost, security constraints, and task relevance of all candidate caches, determine the computing power budget of the intelligent computing cloud platform for acquiring Tokens and / or Token-KV.

[0125] In one embodiment, for each visual token or region token in all candidate caches Calculate its task relevance Cache confidence Recalculate costs and safety constraints And solve the budget allocation problem using the conditions shown in the following formula (14).

[0126]

[0127] (14) In formula (14), Indicates the number of KV quantization bits. Indicates whether to recalculate or reuse. Indicates the cache level. This indicates the current GPU computing power or memory budget. This indicates the estimated quality contribution under this allocation. This represents the set of security policies that are allowed to be reused. Intelligent computing cloud platforms can approximate this solution using greedy algorithms, dynamic programming, reinforcement learning strategies, small gating networks, or rule tables.

[0128] It should be noted that problem-aware budgeting can be achieved through rule systems, lightweight classifiers, attention prediction networks, reinforcement learning schedulers, Bayesian optimizers, or historical log statistical models.

[0129] Step S4.3: Based on the computing power budget, obtain the Token and / or key-value pair Token-KV of the candidate cache.

[0130] For example, when a user asks "What is the contract amount in the table?", the areas containing the amount field, header, total row, currency, date, and payment terms are highlighted. High-precision key-value pairs are retained and local OCR bridging recalculation is triggered; background watermarks, footer decorations, and text paragraphs unrelated to amount are reduced in precision or only the summary token is retained. When a user asks "Is the device in the image damaged?", device outlines, cracks, abnormal colors, broken edges, and object detection tokens receive higher precision; when a user asks "Is someone entering in the video?", person detection boxes, doorway areas, moving foregrounds, and keyframe tokens are retained in high-level caching, while static backgrounds and consecutive repeating frames use incremental caching.

[0131] The beneficial technical effects achieved by steps S4.1 to S4.3 are as follows: The visual token computing power budget is dynamically allocated based on user questions, business tasks, OCR fields, object categories, visual regions, historical attention, and cache heat. This allows important regions to retain high-precision, high-level caches, while weakly related regions are compressed, have reduced precision, or are discarded. This enables fine-grained scheduling of computing resources in the intelligent computing cloud platform, maximizing computing power utilization and improving system inference throughput.

[0132] It should be noted that in this invention, security constraints are bound throughout the entire process of cache generation, cache indexing, cache hit, cache loading, hit rate statistics, billing, and eviction. For example, public product manual pages can be reused in a public cache domain shared by multiple tenants; customer contract scan pages can only be reused within the same tenant, the same project, or the same authorized session; areas containing personal information, seals, bank account numbers, or sensitive production data cannot be reused across domains by default, or only anonymized structure tokens, layout tokens, and low-precision visual summaries are allowed to be reused. This makes the invention more suitable for commercial deployment on computing cloud platforms.

[0133] In one embodiment, prior to step S3, the method further includes: Step S6: For the visual content requested for the first time, call the encoder to generate the token sequence of the visual content and determine the intermediate state of the encoder.

[0134] Step S7: Input the Token sequence and the prompt words of the initial request into the model for pre-population, and determine the Token-KV cache, location mapping parameters and bridging token related to the Token sequence.

[0135] The aforementioned model can be a large language model, or other multimodal models capable of processing visual content.

[0136] Step S8: Construct and store candidate caches using the Token sequence, intermediate states, Token-KV cache, location mapping parameters, bridging Token, and metadata corresponding to the visual content.

[0137] In one embodiment, reusable visual content obtained through visual encoder and language model prefilling, which includes token sequences, intermediate states, token-KV caches, position mapping parameters, bridging tokens, and metadata corresponding to the visual content, can be saved as a visual token-KV cache asset, i.e., a candidate cache can be constructed. It can be shown in the following formula (15).

[0138] (15) In formula (15), Represents a visual token sequence. This indicates the intermediate state or output of the visual encoder. and The model number is represented by the first... Layered key-value cache related to visual tokens, This represents the original location encoding or location mapping parameters. Indicates a visual token mask. Indicates the cross-modal bridge token or bridge status. This represents the quantization, compression, and verification parameters. Represents tenant and security metadata, It represents statistics on lifecycle, popularity, number of reuses, and computing power savings.

[0139] In one embodiment, visual tokens, visual key values, encoder states, bridging tokens, quantization parameters, location mappings, region masks, permission tags, and lifecycle metadata can be stored as cached assets in GPU memory, CPU memory, NVMe, local object storage, or a distributed cache.

[0140] In one embodiment, the candidate cache does not need to save the original image or document; it only needs to save the token representation, key-value tensor, structural summary, and necessary positional mappings that cannot be directly restored from the original text. For high-security content, the intelligent computing cloud platform can save only the encrypted key-value pairs that can be reused by the same tenant, or only the structural tokens of the locally desensitized areas. For public materials, they can be promoted to a public cache domain for reuse by multiple authorized tenants.

[0141] The beneficial technical effects achieved by steps S6 to S8 are as follows: by converting the visual token representation and visual KV cache already generated in historical requests into reusable computing power assets across requests, the underlying computing power consumption of the intelligent computing cloud platform is greatly reduced, which helps to shorten the response time of token generation and inference and improve the overall concurrent processing capability of the system.

[0142] It should be noted that, in this invention, in addition to caching the entire image or the entire page, the visual content can be further divided into region blocks, and a cache descriptor can be generated for each region block.

[0143] In one embodiment, step S8 may further include: Step S8.1: Divide the visual content into multiple region blocks, determine the region coordinates and region type of each region block, and the region token corresponding to the region block in the token sequence.

[0144] The above-mentioned region types can be determined based on the type of content contained in each region block.

[0145] For example, a PDF page can be divided into areas such as titles, paragraphs, tables, signatures, footers, and attached images; a table can be further divided into table headers, row headers, data cells, total rows, and notes; and for a video, the divided areas can be people, vehicles, equipment, dashboards, abnormal action areas, or subtitle areas in keyframes.

[0146] It should be noted that the granularity of the visual content segmentation blocks can be dynamically set according to the accuracy requirements of the actual reasoning. In this invention, region-level caching enables intelligent computing cloud platforms to reuse visual tokens (KVs) even when the overall page content differs but certain parts are identical. For example, two contract versions may have slightly different main texts, but the signature page, payment form, or attached product images may be the same; the system can still reuse the visual tokens (KVs) for the corresponding regions.

[0147] Step S8.2: Based on the region type, determine at least two of the following: perceptual hash, OCR hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features of the region block.

[0148] Step S8.3: Determine the cache descriptor of the region block from at least two items: hash, region coordinates, region type, region token, and metadata corresponding to the visual content, and store it as a candidate cache.

[0149] In one embodiment, each region block can be determined using the following formula (16). The cache descriptor.

[0150] (16) In formula (16), For normalized region coordinates, For region type, For de-identified or summarized OCR fields, For a collection of objects, For table cell topology information, This is a prototype for a regional visual token. For the region token mask, Metadata for permissions, security, tenants, and lifecycle.

[0151] The beneficial technical effects achieved by steps S8.1 to S8.3 are as follows: by further splitting the visual content into regional blocks and extracting the multi-dimensional hash of the regional blocks to construct a cache descriptor, the token-KV assets can be accurately identified and stored, effectively improving the cross-request cache hit rate, greatly saving redundant computing power overhead, and significantly improving the token inference efficiency.

[0152] In one embodiment, for PDFs and scanned documents, visual page content can be processed together with structured text content. The intelligent computing cloud platform can extract page-level fingerprints, region-level fingerprints, paragraph-level fingerprints, table-level fingerprints, and cell-level fingerprints. The table structure hash includes the number of rows and columns, merged cell relationships, header semantics, field types, cell positions, border structure, and OCR field summaries. Therefore, for a table... Its form fingerprint It can be expressed as the following formula (17).

[0153] (17) In formula (17), Represents a table topology diagram, ( () represents the row and column coordinates of the cell. Indicates a merger relationship. Indicates the field type. This represents the cell value or a de-identified summary. Even if the table is screenshotted, the column width changes, or the page is re-exported, the corresponding table Token-KV can still be matched as long as the table topology and key fields remain consistent.

[0154] For example, in a contract Q&A scenario, this invention can also create field-level visual token indexes for fields such as amount, date, payment terms, liability for breach of contract, signatures, and names of Party A and Party B. When a user's question only involves the amount field, it is not necessary to load all visual tokens on the entire page; instead, the amount cell and its adjacent header tokens, currency tokens, and table bridging tokens are loaded. When the user requests a "summary of the entire page," then the page summary token and paragraph-level tokens are loaded.

[0155] It should be noted that, in this invention, since the importance of different sub-fingerprints varies with the task, weighted similarity can be used instead of a single threshold to determine which candidate cache the token fingerprint hits.

[0156] In one embodiment, step S3 includes: Step S3.1: Based on the current request for visual content and the task type of the current request, determine the weight corresponding to each type of hash.

[0157] In some embodiments, the focus of visual content may differ for different requests or different task types during request inference. Therefore, when selecting candidate caches for reuse, the weights of each type of hash can be adjusted according to their importance in similarity judgment, and weighted similarity can be used to dynamically determine the candidate caches that are hit.

[0158] For example, when a user's current request for visual content is "What is the contract amount?", the weight of the contract field OCR field hash, table structure hash, the position of the amount field, and the visual token of the amount cell is increased; when a user asks "Is the device damaged?", the weight of the object detection hash, the local visual descriptor of the damaged area, and the edge texture token are increased; or, if a user asks "Is someone entering the video?", the weight of the person object hash, the keyframe time fingerprint, and the motion region token are increased. This makes cache hits depend not only on visual similarity but also on whether the current question actually requires reusing the visual region.

[0159] Step S3.2: Based on the weights, perform a weighted summation of the similarity between each candidate cache and the visual content block across all types of hashes to determine the similarity between the candidate cache and the visual content block.

[0160] In one embodiment, the similarity between the candidate cache and the visual content block can be calculated using the following formula (18).

[0161] (18) In formula (18), This represents the set of sub-fingerprint types. Indicates the current problem and task type The weights are determined dynamically. Indicates a penalty for geometric inconsistency; This indicates a penalty for a mismatch between security policies.

[0162] Step S3.3: The candidate caches with a similarity greater than the first similarity threshold are identified as candidate caches for Token fingerprint matching.

[0163] The aforementioned first similarity threshold can be customized based on the actual business scenario and risk tolerance to balance cache recall and cache reuse accuracy.

[0164] In one embodiment, if the similarity of candidate caches If the similarity exceeds the first similarity threshold, the candidate cache can enter the reuse process and be identified as a candidate cache that matches the Token fingerprint.

[0165] It should be noted that, in this invention, a second similarity threshold can also be set that is greater than the first similarity threshold and has a very small difference from the first similarity threshold. The interval between the first similarity threshold and the second similarity threshold is defined as the weak hit interval. In the case of a weak hit zone, the intelligent computing cloud platform can save computing power by reusing only low-level visual encoding, local region tokens, or low-precision visual summaries, and trigger more stringent local recalculation.

[0166] The beneficial technical effects achieved by steps S3.1 to S3.3 are as follows: Dynamically determining the Hash weight of each type based on the current request and task type, and performing weighted similarity calculation accordingly, and filtering candidate caches with high similarity, can improve the accuracy and robustness of Token cache matching. By comprehensively considering the similarity of visual content blocks at different semantic levels, the limitations of single feature matching can be effectively overcome, ensuring the accurate recall of candidate caches with high similarity and guaranteeing the quality of inference.

[0167] In one embodiment, the present invention may also include tiered caching, and step S8 includes: Step S8.1': Predict the reuse probability of candidate caches, save computing power and time, and determine the verification confidence, storage usage, security risk level and lifespan of candidate caches.

[0168] The confidence level of the above test reflects the cache quality of the candidate cache.

[0169] The aforementioned lifecycle may include, but is not limited to, the cache age and expiration level of the candidate cache.

[0170] In this invention, the visual token-KV cache is regarded as a measurable computing power asset, and the computing power saved by candidate cache reuse can be calculated by the following formula (19).

[0171] (19) In formula (19), The visual encoding cost of submodal requests under full recalculation. For the prefill cost of the statement model, For KV storage and loading costs, The cost of bridging is recalculated.

[0172] like Furthermore, if the quality risk of the candidate cache is below a threshold, then reusing the candidate cache will yield positive benefits. The computing power platform can further map this to saving tokens, saving GPU milliseconds, saving video memory usage, reducing queue waiting time and tenant fee discounts. For high-frequency public visual assets, the cache can also be pre-warmed based on the reuse probability, forming a "visual token CDN"-style computing power distribution capability.

[0173] Step S8.2': Calculate the retention priority of candidate caches by utilizing reuse probability, saving computing power and time, verification confidence, storage usage, security risk level and lifespan.

[0174] In one embodiment, the retention priority of the candidate cache can be calculated using the following formula (20).

[0175] (20) In formula (20), Indicates the predicted reuse probability. This indicates the expected savings in GPU computing power and prefill time. Indicates cache quality or check confidence level. Indicates video memory or storage usage. Indicates security risk. This indicates the cache age or expiration level. The intelligent computing cloud platform can... Perform promotion, demotion, compression, migration, warm-up, and elimination.

[0176] Step S8.3': Select the appropriate caching level strategy based on the retention priority and store the candidate cache.

[0177] In this invention, the cache can be divided into multiple levels, such as L0, L1, L2, and L3 from high to low. Based on the retention priority of different caches, the corresponding caching strategy is selected for storage. For example, L0 is the GPU memory hot cache, which stores high-precision visual key-value pairs and bridging tokens that are frequently reused in the short term; L1 is the CPU memory cache, which stores medium-to-high frequency visual tokens, quantization key-value pairs, and page-level caches; L2 is NVMe or local object storage, which stores low-frequency but warm-upable visual token-key-value pairs, video delta, and PDF page caches; and L3 is a distributed metadata index and fingerprint library, which stores cache descriptors, permission tags, heat statistics, and computing power saving records.

[0178] In one embodiment, public manuals, standard product diagrams, template contracts, and fixed GUI interfaces can be pre-warmed to L0 or L1; sensitive contracts temporarily uploaded by users are only retained for a short period within the authorized session or tenant domain; the ordinary frame delta of long videos can be downgraded to L2 or deleted after the session ends; decorative images with low hit rate and low recalculation cost are directly eliminated.

[0179] The beneficial technical effects achieved by steps S8.1' to S8.3' are as follows: By comprehensively evaluating the reuse probability, saving computing power and time, confidence, storage occupation, security risks and life cycle of candidate caches, the candidate caches are stored in a hierarchical manner. This can prioritize the retention of high-value Token-KV assets, avoid invalid caches occupying valuable GPU memory space, and thus concentrate limited computing resources on core inference tasks, maximize resource utilization, and ensure that high-frequency Token assets can be read at high speed. This significantly reduces the repetitive calculation overhead in the Prefill stage and significantly improves the system's concurrent throughput and response time.

[0180] It should be noted that, due to the high temporal redundancy of video input, in order to reduce the waste of cache resources, this invention constructs a multi-level cache system, including key frame token-KV resident, ordinary frame delta cache, weakened storage of repeated frames, and local recalculation of changed frames.

[0181] In one embodiment, step S6 includes: Step S6.1: When the visual content is video, determine the set of keyframes of the video and the change area between each two adjacent frames in the video.

[0182] In this embodiment of the invention, the set of key frames of the video can be determined by formula (3) provided in step S1. By converting two adjacent frames into grayscale images, calculating the absolute difference of the corresponding pixels, and setting a threshold for binarization, the region of significant change can be extracted. Alternatively, other methods can be used to determine the region of change between each two adjacent frames in the video. This invention does not limit this.

[0183] Step S6.2: Determine the target token cached for each keyframe and the increment of the change region between each two adjacent frames.

[0184] Step S6.3: For each ordinary frame in the video other than the keyframe, determine the target token of the keyframe that has the fewest frames between it and the ordinary frame, and determine the change token of the ordinary frame relative to the keyframe based on the increment of all the change regions between the ordinary frame and the keyframe.

[0185] The above-mentioned ordinary frame refers to any frame in the video except for keyframes. According to the size of the difference between the ordinary frame and the adjacent frame or the type of the difference concentration area, the ordinary frame can be further divided into repeating frames and changing frames.

[0186] In one embodiment, the key frame with the fewest frames between each normal frame can be determined based on the frame timing in the video. Then, the increments of all adjacent frames between the normal frame and the key frame are combined to obtain the change token of the normal frame relative to the key frame.

[0187] Step S6.4: Combine the target token with the change token to obtain the target token of the ordinary frame buffer.

[0188] In one embodiment, the visual token for a normal frame of frame t is: The most recent keyframe is Then the target token for visual caching of ordinary frames can be represented by the following formula (21).

[0189] (twenty one) In formula (21), The token represents the change relative to the keyframe.

[0190] In this invention, video visual token-KV can be divided into key frame cache, ordinary frame cache, repeated frame summary cache and variable region incremental cache (delta cache).

[0191] The beneficial technical effects achieved by steps S6.1 to S6.4 are as follows: a multi-level caching system is constructed, which includes key frame token-KV resident, ordinary frame delta cache, weakened storage of repeated frames, and local recalculation of changed frames. By adopting different caching levels, precision formats, and delta cache strategies for key frames, ordinary frames, repeated frames, and changed frames in the video scene, the visual encoding computing power is avoided from being repeatedly consumed by continuous redundant frames.

[0192] In one embodiment, ordinary frames can also be divided into repeating frames and changing frames to further reduce the resources required for buffering. Step S8 further includes at least one of the following steps: S8.1'' and S8.2''. Step S8.1'': When the change area between a normal frame and an adjacent frame is less than the change threshold, extract the digest token from the target token of the normal frame, and store the KV of the target token of the key frame and the digest token of the normal frame.

[0193] In one embodiment, if the difference between a normal frame and an adjacent frame is less than a change threshold, the frame can be identified as a duplicate frame. For duplicate frames with inter-frame differences below the threshold, only the timestamp, inter-frame similarity, and summary token need to be saved.

[0194] The beneficial technical effects achieved by step S8.1 are as follows: For ordinary frames where the change between adjacent frames is less than a threshold, the digest token is extracted and stored, which greatly reduces the storage footprint of token-KV assets, reduces memory bandwidth pressure, and enables the intelligent computing cloud platform to process video at a lower cost.

[0195] Step S8.2'': When the change region type between a normal frame and an adjacent frame is a person or object region, determine and store the key-value pair of the change token and the key-value pair of the target token of the key frame.

[0196] In one embodiment, if the change region type between a normal frame and an adjacent frame is a person or object region, and the frame is determined to be a change frame, then for change frames where the difference is concentrated in the person or object region, only the delta Token-KV of the change region can be saved. The KV stored based on the above processing can be represented by the following formula (22).

[0197] (twenty two) It should be noted that, at the caching level, keyframe tokens (KV) can reside permanently in the GPU or CPU high-level cache, while the delta of ordinary frames can be stored in the CPU or NVMe. Duplicate frames only retain lightweight summaries and indexes. If the user's problem requires precise action recognition, the system will further improve the delta cache accuracy for the relevant time window; if the problem only requires determining whether someone is present, the system will prioritize loading keyframes and motion region tokens related to person and object detection.

[0198] The beneficial technical effect achieved by step S8.2 is that by focusing on the changes in the area of ​​people or objects, the repeated calculation of the entire map is avoided, and the computing power is concentrated on extracting and updating the KV state of the core token, thereby improving the utilization rate of computing power.

[0199] It should be noted that, in order to further reduce reuse errors in this invention, a visual region attention prior can be generated based on the current problem. In one embodiment, step S5 includes: Step S5.1: Determine the attention region of the current request in the visual content, where the attention region contains at least one visual content block.

[0200] In some embodiments, the area in the visual content where the object is located can be determined as the attention region based on which object the current request asks a question about. For example, if the current request is "Is someone entering the video?", then the attention region is the area where the person is located.

[0201] Step S5.2: Calculate the regional attention of the attention region using the current request question vector, the region vector of the attention region, the attention score of the historical attention region, and the weight corresponding to the type of the attention region.

[0202] In some embodiments, the regional attention level of the attention region can be calculated using the following formula (23).

[0203] (twenty three) In formula (23), Let the problem vector be... For attention area The region vector, The attention score for the historical attention region. Weights are assigned to the types of attention regions. This represents the Sigmoid function. Indicates the task type.

[0204] Step S5.3: Convert the region attention into an attention mask bias. Based on the recalculated bridging token and the text token of the current request, overlay the attention mask bias on the remapping results of the positions corresponding to the visual content blocks contained in the attention region, and infer the current request.

[0205] In some embodiments, the region attention obtained from formula (23) can be converted into an attention mask bias using formula (24). This bias can then be superimposed on the attention of the bridging token and the text token to the visual token in the current prefill, guiding the model to prioritize the reused visual regions relevant to the problem. This correction does not change the core parameters of the model, making it easy to deploy on existing multimodal inference frameworks.

[0206] (twenty four) The beneficial technical effects achieved by steps S5.1 to S5.3 are as follows: By dynamically calculating the region attention and superimposing the attention mask bias to guide inference, it can be ensured that the recalculated bridging token and text token can automatically enhance the feature extraction of key visual regions during inference, thereby ensuring the high accuracy and high relevance of the generated results.

[0207] In one embodiment, after step S3, the method further includes: Step S9: If the similarity of the candidate cache is less than the second similarity threshold, the candidate cache is verified.

[0208] The second similarity threshold is greater than the first similarity threshold, and the weak similarity interval is formed by the first similarity threshold and the second similarity threshold.

[0209] The above verification may include at least one of the following: whether a risky field exists, whether the embedding distance is greater than the distance threshold, whether the regional OCR is consistent, whether the object set is consistent, and whether the difference between the recalculated part of the token and the cache is higher than the difference threshold.

[0210] In one embodiment, to ensure correct reuse, the present invention can perform verification in high-risk or weakly similar hit scenarios. Verification includes embedding distance verification, region OCR consistency verification, object set consistency verification, attention bias verification, and small-scale token recalculation verification. For small-scale token recalculation verification, a subset of visual tokens can be selected randomly or according to importance. Perform recalculation to obtain the recalculated representation. and cache representation The recalculated difference is obtained by comparison, as shown in formula (25).

[0211] (25) Step S10: If one of the verification results fails, the candidate cache is rolled back.

[0212] In one embodiment, for small-proportion token recalculation verification, the recalculation difference calculated by the above formula (25) can be compared with the difference threshold. If the recalculation difference is higher than the difference threshold, it is determined that the small-proportion token recalculation verification fails.

[0213] In one embodiment, if the recalculation difference exceeds a threshold, the current issue involves a high-risk field, or OCR consistency is insufficient, the system can automatically revert to full visual encoding or region recalculation. The reverted result can then update the cached confidence, reducing the probability of future hits on erroneous candidates.

[0214] The beneficial technical effects achieved by steps S9 to S10 are: by using the verification rollback mechanism, controllable reliability can be maintained while expanding the scope of reuse.

[0215] The intelligent computing cloud platform proposed in this invention accelerates multimodal reasoning by reusing computing power through tokens. The platform first standardizes the input visual content, extracting global visual features, local region features, OCR text, object detection results, layout structure, table topology, video keyframes, and audio transcription. Then, it generates multimodal token fingerprints and region-level cache descriptors. If a visual token-KV asset that meets the similarity and permission conditions exists in the cache, the asset is loaded, and position remapping, token selection, bridging token recalculation, and attention correction are performed based on the current prompt position, text context, and question intent. If no cached asset meets the conditions, complete visual encoding and prefilling are performed, and the results are written to the cache according to a security policy.

[0216] As described above, the intelligent computing cloud platform proposed in this invention accelerates multimodal inference by reusing computing power tokens. The intelligent computing cloud platform generates multimodal token fingerprints from input images, PDF pages, video frame groups, OCR text, page layout, object detection results, region coordinates, table structures, audio transcription, and visual patch distribution. When permissions, security tags, and similarity conditions are met, the visual encoding results, visual token representations, visual layer KV cache, and necessary cross-modal bridging tokens are cached as reusable computing power assets in a layered manner. When online requests arrive, the system uses position-independent mapping, region-level matching, local bridging token recalculation, cross-modal attention correction, and problem-aware visual token computing power budgeting to enable the reuse of the same or similar visual content in different text contexts, different prompt positions, different image-text interleaving orders, and different business tasks, thereby reducing the prefill overhead, visual encoding overhead, GPU memory usage, and overall computing power consumption of multimodal inference. It can decouple visual content from the complete prompt into addressable, reusable, verifiable, compressible, and billable token-level computing power assets, which are suitable for scenarios such as cloud computing power platforms, multi-tenant model services, document question answering, video understanding, industrial quality inspection, contract review, knowledge base retrieval enhancement, and GUI automation.

[0217] The present invention will be further described below with reference to specific embodiments. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Without departing from the concept of the present invention, those skilled in the art can make equivalent substitutions for the module deployment method, threshold, model structure, caching medium, fingerprint algorithm and security strategy.

[0218] For example, in a corporate contract review system, a user uploads a 120-page PDF purchase contract. This contract will be repeatedly questioned by sales, legal, finance, and audit personnel, asking questions such as "What is the total contract amount?", "How are the payment terms agreed upon?", "Is there a penalty cap?", "Are all signatures and seals complete?", and "What are the product models in the attachments?". Traditional multimodal inference systems need to render the relevant PDF pages as images, generate visual tokens, and calculate the corresponding visual key-value pairs during the language model prefill phase each time a question is asked. If the same page is repeatedly queried, it will generate a large amount of redundant visual computational overhead.

[0219] In this invention, the intelligent computing cloud platform can perform normalized rendering, OCR, layout analysis, and table detection on each page during the initial PDF processing. For example, for the payment table on page 35, it generates page-level fingerprints, table-level fingerprints, and amount cell-level fingerprints, and writes the page's visual token, table area visual token, amount cell key-value pair, OCR field summary, and table bridging token into the tenant domain cache. When the finance personnel ask "What is the contract amount?" for the second time, the intelligent computing cloud platform does not completely recalculate page 35. Instead, it increases the weight of the amount field, currency, total row, and payment header area based on the question's intent, hits the amount cell visual token-key-value pair, and only recalculates a small number of bridging tokens.

[0220] If legal personnel subsequently inquire whether the payment milestone is related to the acceptance result, the intelligent computing cloud platform can still reuse the visual token-key value of the same payment form. However, the problem-aware budget will increase the bridging recalculation of the areas containing fields such as "acceptance," "payment terms," ​​and "mileage date." If the auditor uploads a scanned version of the same contract, the page edges are cropped and the resolution is different, and the file hash cannot be matched. This invention can still identify the same payment form by using the perception hash, OCR field hash, table topology hash, and normalized region coordinates, thereby reusing the original cache or performing a partial recalculation.

[0221] In this embodiment, the intelligent computing cloud platform transforms contract PDF pages from objects that require "re-visual encoding with each request" into token-level computing power assets that are "reusable by page, by table, and by field," significantly reducing the cost of repetitive reasoning in contract review scenarios.

[0222] For example, in e-commerce customer service or industrial product Q&A systems, the same product image might be used for multiple questions, such as "Where are the interfaces on this device?", "Is the casing damaged?", "What is the model number on the nameplate?", and "Can this image be used for a quality inspection report?". Different users might upload different versions of the same product image, including compressed images, screenshots, watermarked images, or partially cropped images. Traditional file hashes are difficult to use, and generic prefix caching will also fail due to differences in the preceding and following text.

[0223] In this invention, the intelligent computing cloud platform generates a global perception hash, an object detection hash, a nameplate OCR hash, a local hash of the interface area, and a visual patch distribution for product images. When a user inquires about the location of an interface, the interface area visual token-KV is reused first, and the interface area bridging token is recalculated. When a user inquires about shell damage, the intelligent computing cloud platform improves the budget of edge texture, abnormal color, crack candidate area, and device outline tokens. When a user inquires about the nameplate model, the nameplate OCR field and local visual token are reused first. For cropped images, as long as the cropped area covers the interface or nameplate, the intelligent computing cloud platform can still achieve region-level hits, without requiring the entire image to be completely identical.

[0224] In one implementation, the intelligent computing cloud platform can establish a public cache domain for public product images. If a product image belongs to a public instruction manual or standard promotional image and is marked as public material by the administrator, multiple tenants can reuse its visual token-key value within the permitted permissions. If the image comes from a customer's on-site quality inspection, reuse is only allowed within the same tenant, the same project, or the same session. This embodiment illustrates that the present invention can function effectively in situations involving "the same visual subject, different image files, and different business problems."

[0225] For example, in enterprise automated operations and maintenance or RPA scenarios, multimodal models need to complete task planning based on GUI screenshots, such as "click the add button," "find the invoice management menu," "check if the pop-up window reports an error," and "read the status of the first row of the table." The same business system interface appears repeatedly in multiple rounds of tasks, but the button positions may change due to browser window size, zoom level, theme color, or pop-up window obstruction.

[0226] In this invention, the intelligent computing cloud platform extracts control detection results, OCR text, layout tree, button icon hash, table structure hash, and region coordinates from GUI screenshots. For stable controls such as the "Add" button, "Search Box," and "Export" button, a control-level visual Token-KV cache is established. When multiple rounds of automated tasks repeatedly enter the same page, the intelligent computing cloud platform only performs partial recalculation on changing areas such as pop-ups, loading states, or newly added rows in tables. The main navigation, menu bar, fixed buttons, and table headers can all reuse the cache.

[0227] If the user's question is "Click the Add button", the question-aware budget will prioritize the button text, button border, adjacent controls, and the current page layout. If the question is "Read the first row of status", the intelligent computing cloud platform will prioritize retaining the table header, the first row of cells, and the status field OCR token. This reduces the inference latency caused by repeated visual encoding of GUI screenshots and improves the consistency of automated operations.

[0228] For example, in video surveillance or industrial inspection scenarios, users may repeatedly request the same video clip, such as "Has anyone entered the warehouse?", "Is the equipment smoking?", "Has the vehicle stopped for more than three minutes?", or "Are there any personnel not wearing safety helmets?" Many frames in the video have the same background, with only a few frames showing changes in the status of people or equipment. Re-extracting frames and re-encoding all frame tokens for each request would result in extremely high computational costs.

[0229] In this invention, when the intelligent computing cloud platform processes video for the first time, it selects keyframes based on shot transitions, inter-frame differences, person detection, vehicle detection, smoke candidate regions, and subtitle changes, and generates visual tokens (KV) for each keyframe. For ordinary frames, only the delta token relative to the most recent keyframe is saved; for repeated frames, only the frame index, timestamp, and summary are saved. If the user asks "Has anyone entered?", the keyframes related to person detection and the delta of the entry area are loaded; if the user asks "Is the device emitting smoke?", the high-precision delta of the device area, smoke candidate region, and relevant time window is loaded.

[0230] In this embodiment, keyframe tokens (KV) can reside permanently in the L1 cache, while entry and device regions can be elevated to L0 based on popularity, and the delta of other ordinary frames is stored in L2. When multiple requests occur around the same video segment, it is unnecessary to repeatedly calculate the visual tokens of all frames, thereby reducing the GPU usage of the video understanding service.

[0231] For example, in a multi-tenant cloud computing platform, some visual content is public material, such as publicly available product manuals, standard training materials, open webpage screenshots, and official template forms; other visual content is tenant-sensitive data, such as customer contracts, financial invoices, production site photos, and personnel information screenshots. Completely prohibiting cross-tenant cache sharing would waste opportunities to reuse public material; without boundary sharing, there is a risk of data leakage.

[0232] In this invention, the intelligent computing cloud platform binds tenant identifiers, project identifiers, security tags, public material identifiers, data categories, validity periods, and audit links to the cache metadata. Public materials, after authorization, are written to the public cache domain, and their visual token-key value can be reused by multiple tenants. Sensitive materials are written to the tenant cache domain or session cache domain, allowing access only from the same tenant, the same project, or the same session. Areas containing personal information, account information, or trade secrets can store only de-identified token structures or not write them to the persistent cache. Each cache hit records the hit source, loading time, tenant policy, and computing power saved, facilitating auditing.

[0233] This embodiment enables the present invention to simultaneously obtain reuse benefits and security compliance capabilities in computing power platform businesses. Especially when public knowledge bases, industry templates, standard teaching materials, and public product diagrams are repeatedly used, the platform can form a public visual token asset pool, significantly reducing the overall reasoning cost.

[0234] For example, in multimodal question answering, the same image might be inserted into the prompt in different ways. For instance, the first request might be "Please see the image below and tell me the device model," with the image after the question; the second request might be "Below is a user-uploaded site image; determine if it's abnormal based on the aforementioned work order description," with the image after the search text; and the third request might be "Please read the image OCR first, then answer whether the device needs maintenance," where the order of the OCR text and image changes again. Traditional prefix caching cannot be reused due to inconsistent prefixes.

[0235] In this invention, visual blocks can be decoupled from the complete prompt, establishing independent fingerprints and visual token-key value assets for images or pages. When a new request arrives, position remapping parameters are calculated based on the visual block's position in the new prompt, and recalculation is only performed on the bridging token and the question-related area. If the visual content is completely identical, the visual encoder state and most of the visual key value are reused; if the visual content has partial changes, only the unchanged areas are reused, and the changed areas are recalculated. Thus, the reuse of the same visual content in different prompt structures is no longer limited by precise prefix conditions.

[0236] For example, in an enterprise knowledge base question-and-answer system, the retrieval module searches for relevant pages from a large number of PDF manuals, contract templates, and training materials, and inserts page images, OCR text, and summaries into a multimodal prompt. Popular manual pages are frequently retrieved repeatedly. In this invention, the cloud platform can predict popular pages based on retrieval logs, generate multimodal token fingerprints and visual token-KVs for these pages in advance, and pre-warm them to L1 or L0 caches.

[0237] When a user's question hits a popular page, the intelligent computing cloud platform can directly load a preheated visual token (KV) and recalculate the bridging token according to the question's intent. If the search results contain multiple pages, the KV can be reused independently for each page, and then a connection between the current question and multiple pages can be established through cross-page bridging tokens. For similar pages, such as different versions of a manual where only the header and footer differ while the main table is the same, the main table token can be reused at the regional level. This embodiment demonstrates that the present invention can be deeply integrated with a search enhancement generation system to form a computing power preheating and distribution capability for visual documents.

[0238] Please see Figure 2 , Figure 2This is a structural diagram of a device for accelerating multimodal inference through the reuse of computing power tokens in an intelligent computing cloud platform, as provided by the present invention. Figure 2 As shown, the device 200 for accelerating multimodal inference through the reuse of computing power tokens in an intelligent computing cloud platform includes: The input processing module 201 is used to receive multimodal visual content, perform corresponding normalization processing on the visual content according to the type of each visual content, and obtain at least one visual content block. The fingerprint generation module 202 is used to generate a Token fingerprint corresponding to the visual content block from at least two of the following: perceptual hash, optical character recognition OCR hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features. The cache retrieval module 203 is used to retrieve candidate caches that match the Token fingerprint based on the Token fingerprint. The verification module 204 is used to verify the reuse permission of the candidate cache by using the attribute information associated with the current request for the visual content. If the attribute information matches the metadata of the candidate cache successfully, the verification is passed. The module then calls the computing power of the intelligent computing cloud platform to obtain the Token and / or key-value pair Token-KV of the candidate cache. The loading module 205 is used to perform position remapping on the Token and / or Token-KV based on the prompt word of the current request, the position mapping parameters of the candidate cache, and the bridging Token, and recalculate the bridging Token. It then uses the position remapping results corresponding to all visual content blocks and the recalculated bridging Token to reason about the problem.

[0239] In one embodiment, the verification module 204 is used for: Based on the prompt word and task type of the current request, calculate the task relevance between the candidate cache token and the current request; Based on the cache confidence, recalculation cost, security constraints, and task relevance of all candidate caches, determine the computing power budget of the intelligent computing cloud platform for acquiring Tokens and / or Token-KV. Based on the computing power budget, obtain the Token and / or key-value pair Token-KV of the candidate cache.

[0240] In one embodiment, the device 200 for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform further includes a storage module for: For the visual content requested for the first time, the encoder is invoked to generate a token sequence of the visual content, and the intermediate state of the encoder is determined. The token sequence and the prompt word of the initial request are input into the model for pre-population to determine the token-KV cache, location mapping parameters and bridging token related to the token sequence; Candidate caches are constructed and stored using the Token sequence, the intermediate state, the Token-KV cache, the location mapping parameters, the bridging Token, and the metadata corresponding to the visual content.

[0241] In one embodiment, the storage module is used for: The visual content is divided into multiple region blocks, and the region coordinates and region type of each region block are determined, as well as the region token corresponding to the region block in the token sequence. Based on the region type, determine at least two of the following: perceptual hash, OCR hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features of the region block. The cache descriptor of the region block is determined by at least two hashes, the region coordinates, the region type, the region token, and the metadata corresponding to the visual content, and is stored as a candidate cache.

[0242] In one embodiment, the cache retrieval module 203 is used for: Based on the current request for the visual content and the task type of the current request, determine the weight corresponding to each type of hash; Based on the weights, the similarity between each candidate cache and the visual content block across all types of hashes is weighted and summed to determine the similarity between the candidate cache and the visual content block. Candidate caches with a similarity greater than the first similarity threshold are identified as candidate caches that match the Token fingerprint.

[0243] In one embodiment, the storage module is used for: Predict the reuse probability, computing power saving, and time saving of the candidate cache, and determine the verification confidence, storage consumption, security risk level, and lifespan of the candidate cache; The retention priority of the candidate cache is calculated using the reuse probability, the saving of computing power and time, the verification confidence, the storage consumption, the degree of security risk, and the life cycle. Based on the retention priority, the corresponding caching level strategy is selected, and the candidate cache is stored.

[0244] In one embodiment, the storage module is used for: When the visual content is video, determine the set of keyframes of the video and the change region between each pair of adjacent frames in the video; Determine the target token cached for each keyframe and the increment of the changed region between each two adjacent frames; For each ordinary frame in the video other than the keyframe, determine the target token of the keyframe that is the fewest frames apart from the ordinary frame, and determine the change token of the ordinary frame relative to the keyframe based on the increment of all the change regions between the ordinary frame and the keyframe. The target token is combined with the change token to obtain the target token of the ordinary frame buffer.

[0245] In one embodiment, the storage module is used for at least one of the following: If the change area between the normal frame and the adjacent frame is less than the change threshold, extract the digest token from the target token of the normal frame, and store the key-value pair of the target token of the key frame and the digest token of the normal frame. When the change region type between the normal frame and the adjacent frame is a person or object region, the key-value pair (KV) of the change token and the key-value pair (KV) of the target token of the key frame are determined and stored.

[0246] In one embodiment, loading module 205 is used for: Determine the attention region of the current request in the visual content, the attention region containing at least one block of visual content; The regional attention score of the attention region is calculated using the question vector of the current request, the region vector of the attention region, the historical attention score of the attention region, and the type-corresponding weight of the attention region. The region attention is converted into an attention mask bias. Based on the recalculated bridging token and the text token of the current request, the position remapping result corresponding to the visual content block contained in the attention region is superimposed with the attention mask bias to infer the current request.

[0247] In one embodiment, the device 200 for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform further includes a verification module for: If the similarity of the candidate cache is less than the second similarity threshold, the candidate cache is verified. The second similarity threshold is greater than the first similarity threshold. The verification includes at least one of the following: whether there is a risk field, whether the embedding distance is greater than the distance threshold, whether the regional OCR is consistent, whether the object set is consistent, and whether the difference between the recalculated token and the cache is higher than the difference threshold. If any verification result fails, the candidate cache is rolled back.

[0248] The device for accelerating multimodal inference through the reuse of computing power tokens in the intelligent computing cloud platform provided by the present invention is capable of implementing the various processes of the above-mentioned method for accelerating multimodal inference through the reuse of computing power tokens in the intelligent computing cloud platform. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0249] It should be noted that the device for accelerating multimodal inference through the reuse of computing power tokens in the intelligent computing cloud platform of this invention can be a device, or a component, integrated circuit, or chip in an electronic device.

[0250] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 301, a processor 302, and a program or instructions stored in the memory 301 that run on the memory. When the program or instructions are executed by the processor 302, they can achieve the following: Figure 1 The corresponding intelligent computing cloud platform achieves the same beneficial effect by reusing computing power tokens to accelerate multimodal reasoning through any step in the method embodiment, and will not be elaborated here.

[0251] The processor 302 can be a CPU, ASIC, FPGA or GPU.

[0252] Those skilled in the art will understand that all or part of the steps of the above-described method embodiment for accelerating multimodal reasoning through the reuse of computing power tokens in the intelligent computing cloud platform can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0253] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1The corresponding intelligent computing cloud platform accelerates multimodal inference through the use of computing power reuse tokens. Any step in the implementation of this method achieves the same technical effect, and to avoid repetition, it will not be elaborated further here. The storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0254] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The corresponding intelligent computing cloud platform achieves the same technical effect by reusing computing power tokens to accelerate multimodal reasoning through various processes in its implementation method, and will not be described in detail here to avoid repetition.

[0255] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0256] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0257] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0258] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for accelerating multimodal reasoning through token reuse in an intelligent computing cloud platform, characterized in that, include: Step S1: Receive multimodal visual content, and perform corresponding normalization processing on the visual content according to the type of each visual content to obtain at least one visual content block; Step S2: Generate the Token fingerprint corresponding to the visual content block from at least two of the following: perceptual hash, optical character recognition (OCR) hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features. Step S3: Based on the Token fingerprint, retrieve the candidate cache that matches the Token fingerprint; Step S4: Based on the attribute information associated with the current request for the visual content, if the attribute information successfully matches the metadata of the candidate cache, the reuse permission verification of the candidate cache passes, and the computing power of the intelligent computing cloud platform is invoked to obtain the Token and / or key-value pair Token-KV of the candidate cache. Step S5: Based on the prompt word of the current request, the location mapping parameters of the candidate cache and the bridging token, perform location remapping on the token and / or token-KV, recalculate the bridging token, and use the location remapping results corresponding to all visual content blocks and the recalculated bridging token to infer the current request.

2. The method as described in claim 1, characterized in that, Step S4 includes: Step S4.1: Calculate the task relevance between the candidate cache token and the current request based on the prompt word and task type of the current request; Step S4.2: Based on the cache confidence, recalculation cost, security constraints, and task relevance of all candidate caches, determine the computing power budget of the intelligent computing cloud platform for acquiring Token and / or Token-KV. Step S4.3: Based on the computing power budget, obtain the Token and / or key-value pair Token-KV of the candidate cache.

3. The method as described in claim 1, characterized in that, Before step S3, the procedure also includes: Step S6: For the visual content requested for the first time, call the encoder to generate the token sequence of the visual content and determine the intermediate state of the encoder. Step S7: Input the Token sequence and the prompt word of the initial request into the model for pre-filling, and determine the Token-KV cache, location mapping parameters and bridging Token related to the Token sequence; Step S8: Construct and store a candidate cache based on the Token sequence, the intermediate state, the Token-KV cache, the location mapping parameters, the bridging Token, and the metadata corresponding to the visual content.

4. The method as described in claim 3, characterized in that, Step S8 includes: Step S8.1: Divide the visual content into multiple region blocks, determine the region coordinates and region type of each region block, and the region token corresponding to the region block in the token sequence; Step S8.2: Based on the region type, determine at least two of the following: perception hash, OCR hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features of the region block. Step S8.3: Determine the cache descriptor of the region block from at least two hashes, the region coordinates, the region type, the region token, and the metadata corresponding to the visual content, and store it as a candidate cache.

5. The method as described in claim 4, characterized in that, Step S3 includes: Step S3.1: Based on the current request for the visual content and the task type of the current request, determine the weight corresponding to each type of hash; Step S3.2: Based on the weights, perform a weighted summation of the similarity between each candidate cache and the visual content block across all types of hashes to determine the similarity between the candidate cache and the visual content block; Step S3.3: The candidate caches with a similarity greater than the first similarity threshold are determined as the candidate caches that the Token fingerprint hits.

6. The method as described in claim 3, characterized in that, Step S8 includes: Step S8.1': Predict the reuse probability, computing power saving and time saving of the candidate cache, and determine the verification confidence, storage consumption, security risk level and life cycle of the candidate cache; Step S8.2': Calculate the retention priority of the candidate cache using the reuse probability, the saved computing power and time, the verification confidence, the storage usage, the security risk level, and the lifecycle; Step S8.3': Select the corresponding caching level strategy according to the retention priority and store the candidate cache.

7. The method as described in claim 3, characterized in that, Step S6 includes: Step S6.1: If the visual content is video, determine the set of keyframes of the video and the change region between each two adjacent frames in the video; Step S6.2: Determine the target token cached for each keyframe and the increment of the changed region between each two adjacent frames; Step S6.3: For each ordinary frame in the video except for the keyframe, determine the target token of the keyframe that is the fewest frames apart from the ordinary frame, and determine the change token of the ordinary frame relative to the keyframe based on the increment of all the change regions between the ordinary frame and the keyframe. Step S6.4: Combine the target token with the change token to obtain the target token of the ordinary frame buffer.

8. The method as described in claim 7, characterized in that, Step S8 includes at least one of the following: Step S8.1'': When the change area between the normal frame and the adjacent frame is less than the change threshold, extract the digest token from the target token of the normal frame, and store the KV of the target token of the key frame and the digest token of the normal frame. Step S8.2'': If the change region type between the normal frame and the adjacent frame is a person or object region, determine and store the key value of the change token and the key value of the target token of the key frame.

9. The method as described in claim 1, characterized in that, Step S5 includes: Step S5.1: Determine the attention region of the current request in the visual content, wherein the attention region contains at least one visual content block; Step S5.2: Calculate the regional attention degree of the attention region using the current request question vector, the region vector of the attention region, the historical attention score of the attention region, and the type-corresponding weight of the attention region; Step S5.3: Convert the region attention into an attention mask bias. Based on the recalculated bridging token and the text token of the current request, superimpose the attention mask bias on the position remapping result corresponding to the visual content block contained in the attention region, and infer the current request.

10. The method as described in claim 1, characterized in that, After step S3, the method further includes: Step S9: If the similarity corresponding to the candidate cache is less than the second similarity threshold, the candidate cache is verified. The second similarity threshold is greater than the first similarity threshold. The verification includes at least one of the following: whether there is a risk field, whether the embedding distance is greater than the distance threshold, whether the regional OCR is consistent, whether the object set is consistent, and whether the difference between the recalculated part of the token and the cache is higher than the difference threshold. Step S10: If one of the verification results fails, the candidate cache is rolled back.

11. A device for accelerating multimodal reasoning through the reuse of computing power tokens in an intelligent computing cloud platform, characterized in that, include: The input processing module is used to receive multimodal visual content, perform corresponding normalization processing on the visual content according to the type of each visual content, and obtain at least one visual content block. The fingerprint generation module is used to generate a Token fingerprint corresponding to the visual content block from at least two of the following: perceptual hash, optical character recognition (OCR) hash, object detection hash, layout structure hash, table structure hash, audio transcription hash, region coordinate hash, video keyframe hash, and image block distribution features. The cache retrieval module is used to retrieve candidate caches that match the Token fingerprint based on the Token fingerprint. The verification module is used to verify the reuse permission of the candidate cache by using the attribute information associated with the current request for the visual content. If the attribute information matches the metadata of the candidate cache successfully, the module will call the computing power of the intelligent computing cloud platform to obtain the Token and / or key-value pair Token-KV of the candidate cache. The loading module is used to perform position remapping on the Token and / or Token-KV based on the prompt word of the current request, the position mapping parameters of the candidate cache, and the bridging Token, and to recalculate the bridging Token. It then uses the position remapping results corresponding to all visual content blocks and the recalculated bridging Token to perform inference on the current request.

12. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when executed by the processor, the program implements the steps of the method for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for accelerating multimodal inference through computing power reuse tokens in the intelligent computing cloud platform as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the method for accelerating multimodal inference through computing power reuse tokens in an intelligent computing cloud platform as described in any one of claims 1 to 10.

Images