Language inference servers, methods for language inference, systems, media, and products for large-scale visual language model inference.

Abstract

This invention generally relates to computer systems based on specific computational models, and more specifically to language inference servers, methods for language inference, systems, media, and products for large-scale visual language model inference. A language inference server includes computational resources, a large language model, a cache pool, and a task scheduling component. In response to detecting that the large language model needs to perform a first part of language inference associated with a first word embedding and language elements, the task scheduling component checks a cached mapping dictionary based on a first request identifier. Based on the result of the check, it selectively loads the first word embedding from the cache pool and selectively instructs the large language model to invoke computational resources to perform the first part of the language inference task. The execution of this first part by the large language model is at least partially concurrent with the visual server performing visual encoding computation. Some embodiments improve system throughput and reduce system latency.

Description

Technical Field

[0001] This invention generally relates to computer systems based on specific computational models, and more specifically to language inference servers, methods for language inference, systems, media, and products for large-scale visual language model inference. Background Technology

[0002] Current mainstream multimodal large language models (MLLMs) still face significant system bottlenecks in the vision-language fusion stage.

[0003] In current vision-language (VL) model designs, most architectures follow a monolithic deployment approach. The visual encoder (e.g., the vision transformer (ViT)) and the language model (e.g., the large language model (LLM)) are loaded, trained, and inferred together as a single, complete model. This approach is adequate in terms of accuracy but still has limitations.

[0004] On the one hand, the large token length puts computational pressure on large language models. Taking the InternVL 2.5 system as an example, the input image is usually divided into 1024 patches, each embedded as a token before being fed into the large language model. Even though current large language models can support context lengths higher than 4K, this approach will significantly reduce the model's response time during the inference phase, which is particularly prominent in tasks that need to process multiple images simultaneously (e.g., VQA, InfographicsQA, etc.).

[0005] On the other hand, the visual encoder and the language model consume the same resources. In the traditional inference process, visual forward computation and the execution of the language model occur on the same computing resources of the same device (usually the same Graphics Processing Unit (GPU)). The visual encoding task performed by the visual encoder is a compute-bound task, while the execution throughput of the language model has high requirements for system bandwidth and context cache, and the resource requirements of the two are significantly mutually exclusive.

[0006] This type of resource coupling limits the system's scalability and high concurrency capabilities.

[0007] To address the aforementioned limitations, some studies have cached the computational results of the visual encoder and then manually injected them into the prompt as special lexical units (e.g.,<image_feat> However, this approach is highly manual and difficult to handle dynamic tasks.

[0008] Some studies have deployed multimodal models on a single machine with multiple GPUs, achieving asynchronous simulation through memory separation or lazy loading. However, communication frequency and bandwidth limitations remain bottlenecks.

[0009] There is a need in this field for solutions to improve any of the above aspects of visual language large model inference systems. Summary of the Invention

[0010] This invention is provided to improve the performance of large-scale visual language reasoning tasks.

[0011] One aspect of the present invention provides a language inference server, comprising: computing resources; a large language model for invoking the computing resources to perform a language inference task associated with a request from a user, the request including multiple images and language elements and associated with a graph processing task; a cache pool for caching word embeddings required by the language inference task, the word embeddings in the cache pool being derived from visual encoding computations associated with the multiple images performed by a visual processing server communicatively coupled to the language inference server, the cache pool including a cache mapping dictionary for associating word embeddings with request identifiers; and a task scheduling component configured to: receive the request; and respond. Upon detecting that the large language model needs to perform a first part of the language inference task associated with the first word embedding and the language elements, the cached mapping dictionary is checked based on a first request identifier associated with the first word embedding; based on the result of the check, the first word embedding is selectively loaded from the cache pool, and the large language model is selectively instructed to invoke the computing resources to perform the first part of the language inference task, wherein the execution of the first part of the language inference task by the large language model and the execution of the visual encoding computation by the visual processing server are at least partially simultaneous, and wherein the first response to the first request is determined at least partially based on the execution result of the first part of the language inference task.

[0012] In the language inference server described above, where the request is a first request and the response is a first response, the task scheduling component is further configured to: send a second request to the vision processing server, the second request including the plurality of images and a request identifier associated with each image, and associated with a visual encoding calculation for each image; receive a second response to the second request from the vision processing server, the second response including a word embedding and an associated request identifier, the word embedding being obtained by the vision processing server performing visual encoding calculations on the plurality of images; write the received word embedding to the cache pool; and update the cache mapping dictionary with the request identifier associated with the received word embedding.

[0013] The language inference server as described in any of the above, wherein the reception of the second response is achieved via a remote direct memory access protocol.

[0014] A language inference server as described in any of the above, wherein the first part of the language inference task performed by the large language model is performed at least partially concurrently with the part of the visual processing server performing visual encoding computation on an image associated with a second request identifier.

[0015] A language inference server as described in any of the preceding embodiments, wherein the task scheduling component is configured to: load the first word embedding from the cache pool in response to a result of the check indicating that the first request identifier exists in the cache mapping dictionary; provide the first word embedding to the large language model; and instruct the large language model to perform a first portion of the language inference task.

[0016] A language inference server as described in any of the above, wherein the task scheduling component is configured to: in response to the result of the detection indicating that the first request identifier does not exist in the cache mapping dictionary, instruct the large language model to wait until the first word is ready to be embedded in the cache pool.

[0017] The language inference server as described in any of the above, wherein the computing resources include a graphics processing unit, the graphics processing unit includes video memory, and the cache pool is associated with the video memory.

[0018] The language inference server as described in any of the above also includes memory, and the cache pool is also associated with the memory.

[0019] The language inference server as described in any of the above, wherein the cache pool includes a hierarchical cache, wherein the hierarchical cache includes a first-level cache and a second-level cache, the first-level cache being used to temporarily store word embeddings, the second-level cache being used to permanently store word embeddings, the first-level cache being associated with the video memory, and the second-level cache being associated with the main memory.

[0020] A language reasoning server as described in any of the above, wherein the request is associated with a text-to-image question answering task, a graph comprehension task, a video comprehension task, or a MathVista logic reasoning task.

[0021] Another aspect of the present invention provides a system for large-scale visual language model inference, comprising: a language inference server as described in any of the preceding claims, wherein the computing resource is a first computing resource and the task scheduling component is a first task scheduling component; and a visual processing server communicatively coupled to the language inference server, the visual processing server comprising: a second computing resource; a visual encoder; and a second task scheduling component configured to: receive a second request from the language inference server, the second request including the plurality of images and a request identifier associated with each image and associated with a visual encoding computation of each image; instruct the visual encoder to invoke the second computing resource to process the plurality of images included in the second request to obtain word embeddings; associate the word embeddings with the request identifier; and send a second response to the language inference server, the second response including the word embeddings and the associated request identifier.

[0022] In the system described above, the second task scheduling component is configured to: merge the plurality of images into a plurality of batches; and instruct the visual encoder to invoke the second computing resources to process the plurality of images in batches to obtain word embeddings.

[0023] The system as described in any of the above embodiments, wherein the visual processing server further includes: a downsampling module, the downsampling module being capable of downsampling word embeddings generated by the visual encoder at multiple downsampling rates, the multiple downsampling rates including a first downsampling rate and a second downsampling rate, the downsampling module being configured to: perform downsampling on the word embeddings at the first downsampling rate or the second downsampling rate based on the result of analysis of the multiple images, wherein the downsampled word embeddings are included in the second response.

[0024] The system as described in any of the above embodiments, wherein the downsampling module comprises: a first downsampling submodule configured to perform grouped average downsampling on the word embeddings generated by the visual encoder at the first downsampling rate; and a second downsampling submodule configured to perform nonlinear mapping compression downsampling on the word embeddings generated by the visual encoder at the second downsampling rate, wherein the first downsampling rate is greater than the second downsampling rate.

[0025] The system as described in any of the above, wherein the parameters of the visual encoder are copied from the teacher visual encoder, the parameters of the downsampling module are copied from the teacher downsampling module, the parameters of the large language model are copied from the teacher large language model, the teacher visual encoder, the teacher downsampling module and the teacher large language model constitute a teacher system for performing visual language large model inference, and the large language model is obtained from the teacher large language model by distillation.

[0026] In any of the systems described above, the loss function used in the distillation includes at least one of classification soft label loss, lexical feature reconstruction and orientation alignment loss, and lexical coverage loss.

[0027] In any of the above systems, the positional encoding information obtained by the visual encoder during the word embedding generation process is preserved and incorporated into the attention masking mechanism of the large language model.

[0028] The system as described in any of the above embodiments, wherein the visual processing server further includes: a routing module configured to: analyze the plurality of images; and, based on the results of the analysis, instruct the downsampling module to perform downsampling on the word embedding at a first downsampling rate or a second downsampling rate.

[0029] Another aspect of the present invention provides a method for language reasoning, comprising: S1: receiving a request from a user at a language reasoning server, the request including a plurality of images and language elements and associated with a text-image processing task, each of the plurality of images being associated with a request identifier; in response to detecting that a large language model needs to perform a first portion of a language reasoning task associated with a first word embedding and the language elements; S2: checking a cache mapping dictionary included in a cache pool based on the first request identifier associated with the first word embedding, the cache pool being used to cache the word embeddings required for the language reasoning task, the word embeddings in the cache pool being the result of a visual encoding computation associated with the plurality of images performed by a visual processing server communicatively coupled to the language reasoning server, the cache mapping dictionary being used to associate the word embeddings with the request identifier; S3: selectively loading the first word embedding from the cache pool based on the result of the check, and selectively instructing the large language model at the language reasoning server to invoke computing resources at the language reasoning server to perform the first portion of the language reasoning task, wherein the large language model performing the first portion of the language reasoning task and the visual processing server performing the visual encoding computation are at least partially simultaneous.

[0030] The method described above, wherein the request is a first request and the response is a first response, further includes: S4: sending a second request to the vision processing server, the second request including the plurality of images and a request identifier associated with each image, and associated with a visual encoding calculation for each image; S5: receiving a second response to the second request from the vision processing server, the second response including a word embedding and an associated request identifier, the word embedding being obtained by the vision processing server performing a visual encoding calculation on the plurality of images; S6: writing the received word embedding into the cache pool; and S7: updating the cache mapping dictionary with the request identifier associated with the received word embedding.

[0031] The method as described in any of the above, wherein the first part of the language inference task performed by the large language model is performed at least partially concurrently with the part of the visual encoding computation performed by the visual processing server on the image associated with the second request identifier.

[0032] The method as described in any of the above, wherein S3 includes: in response to the result of the check indicating that the first request identifier exists in the cache mapping dictionary, S31: loading the first word embedding from the cache pool; S32: providing the first word embedding to the large language model; and S33: instructing the large language model to perform a first part of the language inference task.

[0033] The method as described in any of the above, wherein S3 includes: S34: in response to the result of the detection indicating that the first request identifier does not exist in the cache mapping dictionary, instructing the large language model to wait until the first word is ready to be embedded in the cache pool.

[0034] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0035] Another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0036] The solution according to embodiments of the present invention improves system throughput for performing large-scale visual language inference tasks, reduces system latency, and promotes system scalability. Attached Figure Description

[0037] Various embodiments of the present invention are described in conjunction with the accompanying drawings.

[0038] Figure 1 A block diagram of an existing computational system for performing large-scale visual language reasoning is shown.

[0039] Figure 2 This demonstrates the resource-constrained bottlenecks of existing computing systems in performing large-scale visual language reasoning.

[0040] Figure 3 A block diagram of a language inference server according to some embodiments of the present invention is shown.

[0041] Figure 4 A block diagram of a system for large-scale visual language reasoning according to some embodiments of the present invention is shown.

[0042] Figure 5 A schematic diagram illustrating the interaction process between a visual processing server and a language inference server according to some embodiments of the present invention is shown.

[0043] Figure 6 The diagram illustrates task execution and computational resource allocation in a system for large-scale visual language reasoning according to some embodiments of the present invention.

[0044] Figures 7-9 Details of the separate deployment of visual encoding and language reasoning tasks in a system for large-scale visual language reasoning according to some embodiments of the present invention are shown.

[0045] Figure 10A block diagram of a second system for large-scale visual language reasoning according to some embodiments of the present invention is shown.

[0046] Figure 11 A schematic diagram illustrating how the components of a second system for large-scale visual language reasoning are formed according to some embodiments of the present invention are shown.

[0047] Figure 12 A flowchart illustrating a method for language reasoning according to some embodiments of the present invention is shown.

[0048] Figure 13 A flowchart illustrating the first process of a method for language reasoning according to some embodiments of the present invention is shown.

[0049] Figure 14 A flowchart illustrating a second process of a method for language reasoning according to some embodiments of the present invention is shown.

[0050] Figure 15 A block diagram of a computer-readable storage medium according to some embodiments of the present invention is shown.

[0051] Figure 16 A block diagram of a computer program product according to some embodiments of the present invention is shown.

[0052] Figure 17 A block diagram of a third system for large-scale visual language reasoning according to some embodiments of the present invention is shown. Detailed Implementation

[0053] In this application, the term "patch" refers to a smaller processing unit into which an original image is divided during the execution of an image encoding task. For example, a visual encoder may divide an input image into hundreds or thousands of patches.

[0054] In this application, the term "token" refers to the basic unit by which a large language model performs reasoning tasks. For language input, the input can be a sequence of text, and correspondingly, a token can be a character, word, phrase, or short sentence within the text sequence. For image input, the input can be a series of images, and correspondingly, each image can be divided into patches, and each patch can be used as a token by the large language model to perform reasoning tasks. A token can reference the object it points to through a scalar value.

[0055] In this application, the term "word embedding" refers to a vector obtained by transforming a lexical (or the object indicated by a lexical) into a fixed-length high-dimensional space. Word embeddings are the actual objects executed during the inference task performed by a large language model. There may be a correspondence between word embeddings, lexical units, and patches.

[0056] In this application, ordinal numbers such as "first," "second," and "third" are used to distinguish different instances of objects with the same name. The use of these ordinal numbers does not indicate any actual temporal, spatial, or sequential order.

[0057] Figure 1 A block diagram of an existing computational system 100 for performing large-scale visual language reasoning is shown.

[0058] The computing system 100 may include computing resources 110, a visual encoder 120, and a large language model 130. The computing system 100 may receive user requests, which may be associated with a text-image processing task and may include visual elements (e.g., multiple images) and language elements. The computing resources 110 may be used to perform image processing or parallel computing. As an example, the computing resources 110 may include a graphics processing unit.

[0059] The visual encoder 120 may receive visual elements (e.g., multiple images) from a user request. The visual encoder 120 may invoke computing resources 110 to generate word embeddings based on the received visual elements (e.g., multiple images).

[0060] The large language model 130 can receive language elements from a user request and word embeddings generated by the visual encoder 120. The large language model 130 can invoke computing resources 110 to perform inference tasks based on the received language elements and word embeddings, thereby generating output as a response to the user request.

[0061] Figure 2 This demonstrates the resource-constrained bottlenecks of existing computing systems in performing large-scale visual language reasoning.

[0062] for Figure 1 The computing system shown, in the visual encoding stage 210, has a visual encoder 120 (see...). Figure 1 The image 212 in the visual elements requested by the user can be decomposed into multiple patches 214, and multiple corresponding lexical units (not shown) and multiple word embeddings 216 can be generated based on the multiple patches 214, wherein there may be a correspondence between the patches, lexical units, and word embeddings. In the model inference stage 230, the large language model 130 (see Figure 1 It can receive language elements and multiple word embeddings from a user request, and perform inference based on the language elements and multiple word embeddings to generate output.

[0063] like Figure 2 As shown in the lower middle section, the visual encoding stage 210 and the model inference stage 230 compete for the same computational resource 110 (see below). Figure 1 Therefore, the existing computing system 100 leads to system resource bottlenecks, increases system latency, and affects system scalability.

[0064] According to one aspect of the present invention, a language reasoning server is provided.

[0065] Figure 3 A block diagram of a language reasoning server 300 according to some embodiments of the present invention is shown. The language reasoning server 300 is capable of performing language reasoning tasks in a vision-language separation deployment.

[0066] The language inference server 300 may include computing resources 110, a large language model 130, a cache pool 310, and a task scheduling component 330.

[0067] Computing resource 110 can be used to perform parallel computing. As an example, computing resource 110 may include a graphics processing unit.

[0068] The large language model 130 can be used to invoke computing resources 110 to perform a language reasoning task associated with a first request from a user. This first request may include multiple images and language elements and may be associated with a text-image processing task.

[0069] Cache pool 310 can be used to cache word embeddings required by the large language model 130 to perform language inference tasks. The word embeddings in cache pool 310 may come from the results of visual encoding computations associated with multiple images performed by a vision processing server communicatively coupled to the language inference server 300. Cache pool 310 may include a cache mapping dictionary. This cache mapping dictionary can be used to associate word embeddings with request identifiers. The following will combine... Figures 4-5 , Figures 7-9 The details of the visual processing server performing visual encoding calculations and its interaction with the language inference server 300 are further described below. Figures 7-9 Further details on the maintenance and updating of cache pool 310 and its cache mapping dictionary.

[0070] The task scheduling component 330 can be used to schedule tasks associated with the first request to enable a vision-language separation deployment. In some embodiments, the task scheduling component 330 can be implemented in software.

[0071] Task scheduling component 330 can receive a first request. Task scheduling component 330 continuously detects whether the large language model 130 needs to perform the first part of a language inference task associated with the first word embedding and language elements. In response to detecting that the large language model 130 needs to perform the first part of a language inference task associated with the first word embedding and language elements, task scheduling component 330 can check the cached mapping dictionary based on the first request identifier associated with the first word embedding. Based on the result of the check, task scheduling component 330 can selectively load the first word embedding from the cache pool 310 and selectively instruct the large language model 130 to call computing resources 110 to perform the first part of the language inference task. The following will combine... Figures 7-9 The execution details of the above operations of the task scheduling component 330 are further described.

[0072] The first part of the language inference task performed by the large language model 130 and the visual encoding computation performed by the visual processing server can be performed at least partially concurrently. The following will combine... Figures 5-9 The characteristics of this asynchronous execution of the large language model 130 and the associated visual processing server (the visual encoder) are further described.

[0073] The first response to the first request ultimately generated by the language inference server 300 can be determined at least in part based on the execution result of the first part of the language inference task.

[0074] Some implementations decouple the language inference and visual encoding tasks by assigning separate servers and allocating independent computing resources to each server. This decoupling of the vision-language tasks is achieved through the asynchronous parallel execution of these tasks. By mitigating the computational resource limitations of existing computing systems through the asynchronous parallel execution of language inference and visual encoding tasks, system latency is effectively reduced, and system throughput and response speed are significantly improved. Furthermore, the system can flexibly schedule the vision processing server and language inference server based on cluster resources, no longer constrained by the computational resource limitations imposed by co-GPU deployments, thus enhancing system scalability.

[0075] In some embodiments, the task scheduling component 330 may be configured to send a second request to the vision processing server. The second request may include multiple images and a request identifier associated with each image, and is associated with the visual encoding calculation for each image.

[0076] The task scheduling component 330 can be configured to receive a second response to the second request from the vision processing server. This second response may include a word embedding and an associated request identifier. The word embedding can be obtained by the vision processing server performing visual encoding calculations on multiple images. The process by which the vision processing server performs visual encoding calculations can be combined as described above. Figure 2 As described, and will be combined below Figures 4-5 , Figures 7-10Further description.

[0077] Task scheduling component 330 can be configured to write the received word embeddings to cache pool 310. Task scheduling component 330 can also be configured to update the cache mapping dictionary with the request identifier associated with the received word embeddings. The following will combine... Figures 7-9 The above operations of the task scheduling component 330 are further described.

[0078] Some implementations support system interrupt recovery by associating request identifiers with word embeddings, and leverage request identifier-based cache update mechanisms to ensure the robustness of asynchronous processes.

[0079] In some embodiments, the reception of the second response can be achieved via the Remote Direct Memory Access (RDMA) protocol. Existing solutions using Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol (TCP) are prone to congestion during high-frequency requests between the visual processing module and the language inference module, resulting in word embedding transmission latency as high as 200-300 ms. This severely restricts end-to-end response speed, especially when the batch size of images is large or the word embedding dimension is high.

[0080] Some implementations use the RDMA protocol instead of HTTP or TCP, which reduces the transmission bandwidth cost of word embedding by about 70% and the average latency of a single word embedding to 70 ms.

[0081] Compared to the traditional TCP / gRPC mechanism, some implementations use RDMA + asynchronous listener mechanism, reducing the transmission time of single-graph word embeddings from 0.23 s to 0.07 s at 896×896 resolution. Simultaneously, under concurrent requests, the RDMA scheme demonstrates superior non-blocking performance compared to the TCP scheme, resulting in lower system jitter. Furthermore, the failure rate of word embedding container transmission is reduced to below 0.5%.

[0082] In some embodiments, the first part of the language reasoning task performed by the large language model 130 may be performed at least partially concurrently with the part of the visual encoding computation performed by the visual processing server. This visual encoding computation is performed on an image associated with the second request identifier. The following will combine... Figures 7-9 Further details are provided. Some embodiments enable asynchronous execution of both the large language model 130 of the language inference server 300 and the vision processing server (e.g., its visual encoder) in a vision-language separation deployment by having them simultaneously process images associated with different request identifiers and their associated word embeddings.

[0083] As described above, in some embodiments, when the task scheduling component 330 detects that the large language model 130 needs to perform the first part of a language inference task associated with the first word embedding and language elements, the task scheduling component 330 may check the cached mapping dictionary based on the first request identifier associated with the first word embedding.

[0084] In some embodiments, the task scheduling component 330 may be configured to, in response to a check indicating that a first request identifier exists in the cache mapping dictionary, load a first word embedding from the cache pool 310, provide the first word embedding to the large language model 130, and instruct the large language model 130 to perform a first part of the language inference task. In some embodiments, the task scheduling component 330 may be configured to, in response to a check indicating that a first request identifier does not exist in the cache mapping dictionary, instruct the large language model 130 to wait until the first word embedding is ready in the cache pool 310. The following will combine... Figures 7-9 The above operations of the task scheduling component 330 are further described.

[0085] In some embodiments, computing resource 110 may include a graphics processing unit, which may include video memory. A cache pool 310 may be associated with the video memory.

[0086] Optionally, in some embodiments, the language inference server 300 may further include memory 314. In some embodiments, a cache pool 310 may also be associated with memory 314.

[0087] In some embodiments, cache pool 310 may include a hierarchical cache. In some embodiments, the hierarchical cache may include a first-level cache and a second-level cache, wherein the first-level cache can be used to temporarily store word embeddings, and the second-level cache can be used to permanently store word embeddings. In some embodiments, the first-level cache may be associated with video memory 312, and the second-level cache may be associated with memory 314. As an example, the results of word embedding may support disk caching mechanisms (e.g., Redis) or checkpoint reuse to reduce redundant computations.

[0088] Compared to existing system frameworks that fail to fully utilize the video memory of graphics processing units, some embodiments utilize video memory for short-term caching of word embeddings, reducing the repeated computation or transmission of the same word embeddings, avoiding unnecessary resource waste, and the multi-layer caching mechanism ensures that word embeddings with high and low usage frequencies are always available and reusable.

[0089] In some embodiments, the system including the language reasoning server 300 can be used for various video understanding tasks. In some embodiments, the first request may be associated with tasks such as text-to-image question answering, graph understanding, video understanding, or MathVista logic reasoning.

[0090] In some embodiments, the system including the language inference server 300 can also be used in a speech-language system. For example, audio can be converted into a spectrogram, processed using a visual lexical route, and then the word embeddings obtained from the visual encoding can be sent to the language module.

[0091] In some embodiments, the first request may be associated with tasks such as medical image-assisted diagnosis, document optical character recognition (OCR) tasks, or complex table parsing tasks.

[0092] According to another aspect of the present invention, a system for large-scale reasoning of visual language is provided.

[0093] Figure 4 A block diagram of a system 400 for large model reasoning of visual language is shown according to some embodiments of the present invention.

[0094] In some embodiments, system 400 may include a language inference server 300 and a vision processing server 410. The vision processing server 410 may be communicatively coupled to the language inference server 300.

[0095] In some embodiments, the language inference server 300 may include computing resources 110, a large language model 130, a cache pool 310, and a task scheduling component 330. Various aspects of the language inference server 300 and its components can be combined as described above. Figure 3 As described.

[0096] In some embodiments, the vision processing server 410 may include a second computing resource 420, a vision encoder 120, and a second task scheduling component 430.

[0097] The second computing resource 420 can be used to perform parallel computing. The second computing resource 420 may include a second graphics processing unit.

[0098] The various aspects of the visual encoder 120 can be combined as described above. Figure 1 As described above, the visual encoder 120 can call upon the second computational resource 420 to divide the image into multiple patches and perform a visual encoding task on each patch to generate a corresponding word embedding, as described above. Figure 2 As described.

[0099] The second task scheduling component 430 can be configured to receive a second request from the language inference server 300. The second request may include multiple images and a request identifier associated with each image, and is associated with the visual encoding calculation of each image.

[0100] The second task scheduling component 430 can be configured to instruct the visual encoder 120 to invoke the second computing resource 420 to process the multiple images included in the second request to obtain word embeddings.

[0101] The second task scheduling component 430 can be configured to associate word embeddings with request identifiers.

[0102] The second task scheduling component 430 can be configured to send a second response to the language inference server 300. The second response may include word embeddings and associated request identifiers.

[0103] The following will combine Figure 5 The interaction process between the language reasoning server 300 and the visual processing server 410 of system 400 is further described.

[0104] According to some embodiments, a system 400 for large-scale visual-language model inference sets up a separate language inference server 300 and a visual processing server 410, allocating independent computing resources to them, thus decoupling the visual-language tasks. By asynchronously and concurrently executing the language inference and visual encoding tasks, the limitations of computing resources in existing computing systems are alleviated, effectively reducing system latency and significantly improving system throughput and response speed. Simultaneously, the system can flexibly schedule the visual processing server and language inference server based on cluster resources, no longer limited by the computing resource constraints of co-GPU deployment, which is beneficial to system scalability.

[0105] In the experiments, the visual encoder 120 was deployed on a separate computing resource (e.g., an image processing unit) independent of the language inference server 300, and the system 400 was tested for processing 1000 image-text pair requests. Results showed that, according to some embodiments, the average processing time of the system 400 decreased from 1080 ms required by the conventional architecture to 620 ms. The large language model 130 of the language inference server 300 can continuously process the queue of language elements (text) without waiting for word embeddings, resulting in an approximately 2.3x increase in the throughput of the system 400. The word embedding cache hit rate on the visual processing server 410 reached 94.5%, and the utilization of the second computing resource 420 (e.g., the graphics processing unit) was improved by 37%.

[0106] In some embodiments, the second task scheduling component 430 may be configured to merge multiple received images into multiple batches. The second task scheduling component 430 may be configured to instruct the visual encoder 120 to call upon the second computing resource 420 to process multiple images in batches to obtain word embeddings. In some embodiments, the visual processing server 410 supports merging multiple images for processing in a short period of time, thereby reducing context switching and resource fragmentation and improving the utilization of the second computing resource 420 (e.g., a graphics processing unit).

[0107] Optionally, in some embodiments, the visual processing server 410 may further include a downsampling module 440. The downsampling module 440 is capable of downsampling the word embeddings obtained by the visual processing server 410 at multiple downsampling rates. These multiple downsampling rates may include a first downsampling rate and a second downsampling rate. In some embodiments, the downsampling module 440 may be configured to perform downsampling of the word embeddings at either a first downsampling rate or a second downsampling rate based on the results of analyzing multiple images. In such embodiments, the downsampled word embeddings are included in a second response to be returned to the language inference server 300 to perform a language inference task. As an example, the downsampling module 440 may be implemented using a Multi-Layer Perception (MLP) module.

[0108] According to some embodiments, the system 400 compresses and downsamples (or compresses lexical units) the word embeddings generated by the visual encoding task while employing a vision-language separation deployment architecture, thereby reducing the overall inference latency of the system by 30-60%.

[0109] Further details regarding the downsampling module 440 of the vision processing server 410 of system 400 will be provided below. Figures 10-11 To describe.

[0110] Figure 5 A schematic diagram illustrating the interaction process between a visual processing server and a language inference server according to some embodiments of the present invention is shown. Figure 5 The interaction process can occur in Figure 4 The system 400 is connected between the vision processing server 410 and the language reasoning server 300.

[0111] The language inference server 300 can receive a first request from a user. This first request may include multiple image and language elements and may be associated with a text-image processing task.

[0112] At step 510, the language inference server 300 can separate the visual elements (e.g., multiple images) and language elements from the first request.

[0113] The language inference server 300 may send a second request to the vision processing server 410. The second request may include multiple images and a request identifier associated with each image, and is associated with the visual encoding calculation of each image.

[0114] At step 520, the vision processing server 410 may merge the images to be processed contained in the second request into a batch.

[0115] In step 530, the visual processing server 410 can perform visual encoding calculations in batches to obtain word embeddings.

[0116] At step 540, the visual processing server 410 may associate the word embedding with the request identifier.

[0117] The visual processing server 410 may return a second response to the language inference server 300. This second response may include word embeddings and associated request identifiers.

[0118] At step 550, the language inference server 300 may write the received word embeddings into its cache pool.

[0119] At step 560, the language inference server 300 can update the cache mapping dictionary using the received request identifier.

[0120] At step 570, the language inference server 300 may check the cache pool in response to detecting that its large language model needs to perform the first part of a language inference task associated with the first word embedding and language elements.

[0121] At step 580, the language inference server 300 may perform language inference computation based on the first word embedding and language elements in response to a result of the check indicating that the first word embedding associated with the first request identifier exists in its cache pool.

[0122] The language inference server 300 may return a first response to the first request to the user, which is determined at least in part based on the result of the execution of the first part of the language inference task.

[0123] Although the above is combined Figure 5 The interaction process between the language inference server 300 and the visual processing server 410 is described sequentially, but the interaction order and situation shown are merely exemplary. Some embodiments of the present invention may omit some of the above steps, some embodiments may change the execution order of some of the above steps, and some embodiments may adaptively execute different steps based on different situations.

[0124] Figure 6 The diagram illustrates task execution and computational resource allocation in a system for large-scale visual language reasoning according to some embodiments of the present invention.

[0125] Figure 6 The middle shows with Figure 5 The operation stages corresponding to steps 530 and 580 in the diagram are also shown. Figure 4 The utilization of computing resources 110 and 420 in the system. Since the system (e.g., system 400) according to some embodiments deploys computing resources 110 and 420 separately, computing resources 110 and 420 can be used simultaneously for steps 530 and 580, respectively, thereby reducing congestion caused by insufficient computing resources in the system 400 during inference, increasing the throughput of the system 400, and reducing the system response time.

[0126] Figures 7-9 Details of the separate deployment of visual encoding and language reasoning tasks in a system for large-scale visual language reasoning according to some embodiments of the present invention are shown. Figures 7-9 Combining the above text Figure 4 The system 400 is described by its language reasoning server 300 and visual processing server 410.

[0127] Figure 7 The execution details of the language inference server 300 and the vision processing server 410 at time t1 are shown. Figure 8 The execution details of the language inference server 300 and the vision processing server 410 at time t2 are shown. Figure 9 The execution details of the language inference server 300 and the vision processing server 410 at time t3 are shown.

[0128] like Figure 7 As shown, the language inference server 300 can receive a first request. This first request may include visual elements and linguistic elements, and may be associated with a text-image processing task. As an example, visual elements may include image 1, ..., image i, image j, ... In the example, indices 1, ..., i, j, ... are used to represent request identifiers associated with images.

[0129] The language inference server 300 can send multiple images to the visual processing server 410.

[0130] At time t1, the large language model 130 needs to execute the first part of the language inference task associated with the request identifier i. The large language model 130 can check the cache mapping dictionary in the cache pool 310 based on the request identifier i. In this example, since the cache mapping dictionary shows that the current request identifier i exists in the request mapping dictionary, indicating that the corresponding word embedding i exists in the cache pool 310, the large language model 130 can load the word embedding i from the cache pool 310. As an example, the word embedding i could have been generated by the visual processing server 410 at a previous time, and could have been cached in the cache pool 310 at a previous time and associated with the request identifier i to be maintained in the cache mapping dictionary. At this moment, since the large language model 130 can directly load the word embedding i required for the first part of the language inference task from the cache pool 310, it does not need to wait for the time required for the word embedding i to be generated, and can directly execute the first part of the language inference task based on the language elements and the loaded word embedding i, thereby reducing system latency.

[0131] Also at time t1, the visual encoder 120 of the visual processing server 410 can generate corresponding word embeddings l, m, and n based on images l, m, and n. The generated word embeddings l, m, and n, along with their associated request identifiers, can be sent to the language inference server 300. The word embeddings l, m, and n can be cached in the cache pool 310, and their request identifiers can be used to update the cache mapping dictionary.

[0132] like Figure 8 As shown, at time t2, the large language model 130 needs to execute the second part of the language inference task associated with the request identifier s. The large language model 130 can check the cache mapping dictionary in the cache pool 310 based on the request identifier s. In this example, since the cache mapping dictionary shows that the current request identifier s does not exist in the request mapping dictionary, it indicates that the corresponding word embedding does not exist in the cache pool 310. At this moment, since the large language model 130 cannot directly load the word embeddings required for the second part of the language inference task from the cache pool 310, the task scheduling component 330 can instruct the large language model to wait for the word embeddings to be ready (e.g., generated by the vision processing server 410 and cached in the cache pool 310).

[0133] Also at time t2, the visual encoder 120 of the visual processing server 410 can generate corresponding word embeddings s and word embeddings t based on images s and t. The generated word embeddings s and word embeddings t can be sent to the language inference server 300 along with the associated request identifier. The word embeddings s and word embeddings t can be cached in the cache pool 310, and their request identifiers can be used to update the cache mapping dictionary.

[0134] like Figure 9 As shown, at time t3, the large language model 130 still needs to execute the second part of the language inference task associated with the request identifier s. The large language model 130 can check the cache mapping dictionary in the cache pool 310 based on the request identifier s. In this example, since the word embeddings were generated by the visual processing server 410 at time t2 and cached in the cache pool 310, the cache mapping dictionary shows that the current request identifier s exists in the request mapping dictionary, indicating that the corresponding word embeddings exist in the cache pool 310. Therefore, the large language model 130 can load the word embeddings from the cache pool 310. At this moment, since the large language model 130 can directly load the word embeddings required for the second part of the language inference task from the cache pool 310, it no longer needs to wait for the word embeddings to be generated and can directly execute the second part of the language inference task based on the language elements and the loaded word embeddings, thereby reducing system latency.

[0135] Also at time t3, the visual encoder 120 of the visual processing server 410 can generate corresponding word embeddings o, p, and q based on images o, p, and q. The generated word embeddings o, p, and q, along with their associated request identifiers, can be sent to the language inference server 300. The word embeddings o, p, and q can be cached in the cache pool 310, and their request identifiers can be used to update the cache mapping dictionary.

[0136] Figure 10 A block diagram of a second system 1000 for large-scale visual language reasoning according to some embodiments of the present invention is shown.

[0137] The second system 1000 may include a language inference server 300 and a vision processing server 410. The language inference server 300 and its components can be combined with those described above. Figure 3 , Figure 4 The description is similar, and for clarity, only the large language model 130 is shown as one of its components. The visual processing server 410 and its components can be combined with the above. Figure 3 , Figure 4 The description is similar.

[0138] In some embodiments, the visual processing server 410 may include a visual encoder 120 and a downsampling module 440. The visual encoder 120 may perform a visual encoding task based on multiple received images to generate multiple word embeddings.

[0139] In some embodiments, the downsampling module 440 may include a first downsampling submodule 1010 and a second downsampling submodule 1020. The first downsampling submodule 1010 may be configured to perform grouped average downsampling on the word embeddings generated by the visual encoder at a first downsampling rate. The first downsampling submodule 1010 may also be configured to perform nonlinear mapping compression downsampling on the word embeddings generated by the visual encoder at a second downsampling rate. The first downsampling rate is greater than the second downsampling rate (the magnitude of the downsampling rate is measured by the number of samples after downsampling). As an example, the first downsampling submodule 1010 and the second downsampling submodule 1020 may each be implemented using an MLP module. As an example, the first downsampling rate may be 4x and the second downsampling rate may be 16x. As an example, the first downsampling submodule 1010 may downsample the received word embeddings into word embeddings of size (1024, D), where 1024 may correspond to the number of patches and D may correspond to the dimension of each word embedding. As an example, the first downsampling submodule 1010 can downsample the received word embeddings into word embeddings of size (64, d), where 64 can correspond to the number of patches and d can correspond to the dimension of each word embedding. As an example, the downsampling path of the first downsampling submodule 1010 can correspond to a high-performance mode, and the downsampling path of the second downsampling submodule 1020 can correspond to a high-efficiency mode.

[0140] In some embodiments, during use, a first downsampling submodule 1010 or a second downsampling submodule 1020 is selected to perform downsampling on the word embeddings at a corresponding downsampling rate. In some embodiments, this selection may be performed manually based on the results of analysis of the image to be processed by the visual encoder 120.

[0141] In some embodiments, the vision processing server 410 may further include a routing module 1030. The routing module 1030 may be coupled to the vision encoder 120 and to the first downsampling submodule 1010 and the second downsampling submodule 1020.

[0142] The routing module 1030 can be configured to analyze multiple images. Based on the analysis results, the routing module 1030 can instruct the downsampling module 440 to perform downsampling on word embeddings at a first downsampling rate or a second downsampling rate. As an example, the routing module 1030 can adaptively allocate the optimal downsampling strategy for different tasks by analyzing metrics such as complexity and semantic density of multiple images in real time. For example, when handling simple tasks with concise graphs and clear semantics, the routing module 1030 can instruct the downsampling module 440 to enable a low downsampling rate path, thereby efficiently completing the inference task with fewer word embeddings. When facing complex tasks with complex scenes and rich details, the routing module 1030 can instruct the downsampling module 440 to enable a high downsampling rate path, thereby allocating more computing resources for deep modeling.

[0143] Some implementations achieve dynamic resource allocation by dynamically adjusting the downsampling rate, which significantly improves inference efficiency while ensuring model performance, thus providing a more flexible and intelligent solution for multi-scenario applications.

[0144] In some embodiments, the positional encoding information obtained by the visual encoder 120 during the generation of word embeddings is preserved and incorporated into the attention masking mechanism of the large language model 130. As an example, the visual encoder 120 may be a Transformer-based encoder, which inherently generates positional encoding information during the generation of lexical units and word embeddings based on multiple input images.

[0145] Some embodiments retain the positional encoding information generated by the visual encoder 120 during the execution of the visual encoding task and add it to the attention masking mechanism of the large language model 130. This enables the large language model 130 to correctly align the spatial structure of lexical units. Compared with directly downsampling the lexical units and word embeddings generated by the visual encoder, this enhances the semantic expression of the obtained lexical units and word embeddings in the large language model, thereby improving model performance.

[0146] The following will combine Figure 11 Describes how the visual encoder 120, downsampling module 440, and large language model 130 of the second system 1000 are formed.

[0147] Figure 11 A schematic diagram illustrating how the components of a second system 1000 for large-scale visual language reasoning according to some embodiments of the present invention are formed.

[0148] In some embodiments, the visual encoder 120, downsampling module 440, and large language model 130 of the second system 1000 can be formed by a distillation mechanism.

[0149] like Figure 11As shown, the teacher visual encoder 1111, teacher downsampling module 1112, and teacher large language model 1113 are first trained. They are trained to perform visual language large model reasoning tasks.

[0150] In some embodiments, to distill the corresponding student module from the teacher module, the parameters of the teacher visual encoder 1111 can be copied to the student visual encoder 1151, the parameters of the teacher downsampling module 1112 can be copied to the student downsampling module 1152 (e.g., the first student downsampling submodule 1153 and the second student downsampling submodule 1154), and the parameters of the teacher large language model 1113 can be copied to the student large language model 1155. This achieves parameter initialization from the teacher module to the corresponding student module.

[0151] In some embodiments, the student large language model 1155 can be distilled from the teacher large language model. As an example, the parameters of the student large language model can be adjusted based on the output of the teacher large language model by calculating a loss function.

[0152] Existing technologies only use soft labels for distillation (e.g., logits distillation), without feature-level alignment of the intermediate representations of lexical units, lacking finer-grained supervision. This leads to inconsistencies between the compressed lexical units and word embeddings and the interface with the large language model, thus affecting the model's generalization performance. Some embodiments, through task-driven soft label distillation and feature distillation combined with joint optimization at the decoding end of the large language model, ensure that the compressed lexical units and word embeddings maintain semantic consistency with the large language model, achieving multi-view consistency learning of low-dimensional lexical unit vectors.

[0153] In some embodiments, the loss function may include at least one of classification soft label loss (e.g., KL divergence), lexical feature reconstruction and orientation alignment loss (e.g., cosine similarity), and lexical cover loss. Some embodiments incorporate lexical cover loss into the combination of loss functions to prevent lexical “collapse” (i.e., over-compression leading to the loss of expression in some regions).

[0154] In some embodiments, the student visual encoder 1151 obtained through the distillation process described above may be Figure 10 The visual encoder 120 and student downsampling module 1152 in the text can be... Figure 10 The downsampling module 440 in the middle, the first student downsampling submodule 1153 can be Figure 10 The first downsampling submodule 1010 and the second student downsampling submodule 1154 can be... Figure 10 The second downsampling submodule 1020 and the student large language model 1155 can be... Figure 10 The large language model 130 in the middle.

[0155] In the experiments, a visual-language separation deployment scheme with lexical compression (i.e., word embedding downsampling) based on some embodiments was adopted. The scheme was tested on InternVL2.5-2B, covering tasks such as text-to-image question answering, graph understanding, and MathVista logical reasoning. The results showed that the accuracy of the Academic subset decreased by only 1.2%, but the inference speed increased by approximately 1.8 times; the accuracy in benchmark tasks such as MMMU and MMVet remained almost unchanged, but the latency decreased by 45%; the accuracy of OCRbench remained at 80.4%, consistent with the original model, indicating that the compressible expressive power of the text region was not compromised.

[0156] According to another aspect of the present invention, a method for language reasoning is provided.

[0157] Figure 12 A flowchart illustrating a method for language reasoning according to some embodiments of the present invention is shown. This method can be combined with the above. Figure 3 The language inference server described is executed by 300, or it can be combined with the above. Figure 5 The description corresponds to a portion of the steps performed on one side of the language inference server 300, but the scope of the invention is not limited thereto.

[0158] In some embodiments, the method may include step S1: receiving a request from a user at a language inference server. The request may include multiple images and language elements and may be associated with a text-to-image processing task. Each of the multiple images may be associated with a request identifier.

[0159] In some embodiments, the method may include a determination step: determining whether the large language model needs to perform a first part of a language reasoning task. This determination step can be performed by continuously monitoring the large language model.

[0160] In some embodiments, if it is determined in the above determination step that the large language model needs to perform a first part of a language inference task associated with the first word embedding and language elements, the method may include step S2: checking a cache mapping dictionary included in a cache pool based on a first request identifier associated with the first word embedding. The cache pool is used to cache the word embeddings required for the language inference task, the word embeddings in the cache pool being derived from the results of visual encoding computations associated with multiple images performed by a vision processing server communicatively coupled to the language inference server. The cache mapping dictionary is used to associate the word embeddings with the request identifier.

[0161] In some embodiments, the method may include step S3: based on the results of the inspection, selectively loading the first word embedding from the cache pool, and selectively instructing the large language model at the language inference server to invoke the computational resources at the language inference server to perform a first part of the language inference task. The large language model performing the first part of the language inference task is at least partially concurrent with the visual encoding computation performed by the visual processing server.

[0162] In some embodiments, if it is determined in the above determination step that the large language model does not need to perform the first part of language inference associated with the first word embedding and language elements, the detection can be performed on the large language model continuously.

[0163] The following will combine Figure 13 Describe the additional procedures of this method, and combine them. Figure 14 Describe the specific implementation method of step S3.

[0164] Figure 13 A flowchart illustrating the first process of a method for language reasoning according to some embodiments of the present invention is shown.

[0165] This first process can be summarized from the above. Figure 3 The language inference server described is executed by 300, or it can be combined with the above. Figure 5 The description corresponds to a portion of the steps performed on one side of the language inference server 300, but the scope of the invention is not limited thereto. This first process can be a combination of the above. Figure 12 The methods described are additional steps, but the scope of the invention is not limited thereto.

[0166] In some embodiments, the first process may include step S4: sending a second request to a vision processing server. The second request may include multiple images and a request identifier associated with each image, and is associated with a visual encoding calculation for each image.

[0167] In some embodiments, the first process may include step S5: receiving a response to the second request from the vision processing server. The second response includes a word embedding and an associated request identifier. The word embedding is obtained by the vision processing server performing visual encoding calculations on multiple images.

[0168] In some embodiments, the first process may include step S6: writing the received word embeddings into a cache pool.

[0169] In some embodiments, the first process may include step S7: updating the cache mapping dictionary with a request identifier associated with the received word embedding.

[0170] Steps S4, S5, S6, and S7 described above can be added to... Figure 12Steps S1, S2, and S3 are executed, and the execution order of steps S4, S5, S6, and S7 relative to steps S1, S2, and S3 is not restricted.

[0171] Figure 14 A flowchart illustrating a second process of a method for language reasoning according to some embodiments of the present invention is shown.

[0172] This second process can be summarized from the above. Figure 3 The language inference server described is executed by 300, or it can be combined with the above. Figure 5 The description corresponds to a portion of the steps performed on one side of the language inference server 300, but the scope of the invention is not limited thereto. This second process can be a combination of the above. Figure 12 The specific implementation of step S3 of the described method is not limited thereto.

[0173] In some embodiments, the first process may include step S30: determining whether the first request identifier exists in the cache mapping dictionary.

[0174] In some embodiments, if it is determined in step S30 that the first request exists in the cache mapping dictionary, the first process may include step S31: loading the first word embedding from the cache pool.

[0175] In some embodiments, the first process may include step S32: providing the first word embedding to the large language model.

[0176] In some embodiments, the first process may include step S33: instructing the large language model to perform a first part of a language reasoning task.

[0177] The execution of steps S31, S32, and S32 above can be combined with the above text. Figure 7 The process described is similar.

[0178] In some embodiments, if it is determined in step S30 that the first request does not exist in the cache mapping dictionary, the first process may include step S34: instructing the large language model to wait until the first word embedding is ready in the cache pool.

[0179] The execution of step S34 above can be combined with the above text. Figure 8 The process described is similar.

[0180] According to another aspect of the present invention, a computer-readable storage medium is provided.

[0181] Figure 15 A block diagram of a computer-readable storage medium 1500 according to some embodiments of the present invention is shown.

[0182] Computer-readable storage medium 1500 may include a computer program 1550 stored thereon. When executed by a processor, the computer program 1550 implements the above-described combination. Figure 12 The method of description, combination Figure 13 The first process described, combined Figure 14 The second process described.

[0183] According to another aspect of the present invention, a computer program product is provided.

[0184] Figure 16 A block diagram of a computer program product 1600 according to some embodiments of the present invention is shown.

[0185] Computer program product 1600 may include computer program 1550. This computer program 1550, when executed by a processor, implements the above-described combination. Figure 12 The method of description, combination Figure 13 The first process described, combined Figure 14 The second process described.

[0186] In some existing systems for large-scale visual language inference, various methods are sometimes employed to reduce the number and / or size of word embeddings provided to the large language model for language inference tasks. Some solutions use simple MLP downsampling modules (e.g., QFormer-like structures) combined with soft-label distillation, which are convergent but have low compression ratios, typically within 4x. Other solutions directly reduce the dimensionality of word embeddings associated with patches through convolutional operations or linear transformations, without guidance from a teacher model or task labels, resulting in weakened semantic representation of lexical units and a significant drop in model performance. Furthermore, most current distillation schemes only use soft labels (logits distillation), which do not perform feature-level alignment of intermediate representations at the lexical level and lack fine-grained supervision. This leads to inconsistencies between the compressed lexical units (i.e., downsampling of word embeddings) and the semantics of the large language model, affecting the model's generalization performance.

[0187] Some embodiments of the present invention also provide solutions that improve upon at least one of the above aspects.

[0188] According to another aspect of the present invention, a system for large-scale reasoning of visual language is provided.

[0189] Figure 17 A block diagram of a third system 1700 for large-scale visual language reasoning according to some embodiments of the present invention is shown.

[0190] In some embodiments, the third system 1700 may include a third computing resource 1710, a second visual encoder 1720, a second downsampling module 1730, and a second large language model 1740.

[0191] In some embodiments, the third system 1700 may be combined with the above. Figure 10 The second system 1000 is similar to the one described, except that the second visual encoder 1720 and the second large language model 1740 of the third system 1700 can call the same third computing resource 1710.

[0192] In some embodiments, the third computing resource 1710 can be connected with Figure 1 The computing resources 110 in the second visual encoder 1720 are similar to those in the third visual encoder 110. Figure 10 Similar to the visual encoder 120 in [the text]. The second downsampling module 1730 can be used with [the text]. Figure 10 The downsampling module 440 in the second largest language model 1740 is similar to the downsampling module 440 in the third largest language model 1740. Figure 10 It is similar to the large language model 130 in China.

[0193] In some embodiments, the second downsampling module 1730 may include a third downsampling submodule 1732 and a fourth downsampling submodule 1734. In some embodiments, the third downsampling submodule 1732 may be connected to... Figure 10 The first downsampling submodule 1010 is similar to this. In some embodiments, the fourth downsampling submodule 1734 can be similar to... Figure 10 The second downsampling submodule 1020 is similar.

[0194] The operation and other aspects of the third system 1700 and its components can be related to Figure 10 The second system 1000 is similar.

[0195] Furthermore, in some embodiments, the above is combined with Figure 10 The second system 1000 described can also be used to improve at least one of the above aspects, wherein computing resources can be deployed on the vision processing server 410 and the language reasoning server 300, respectively.

[0196] Embodiments of the invention have been described with reference to the accompanying drawings. These embodiments are illustrative and not restrictive.

Claims

1. A language inference server, characterized in that, include: Computing resources; A large language model is used to invoke the computing resources to perform language reasoning tasks associated with a request from a user, the request including multiple images and language elements and associated with a text-image processing task; A cache pool is used to cache word embeddings required for the language inference task. The word embeddings in the cache pool are derived from visual encoding calculations performed by a visual processing server communicatively coupled to the language inference server, which are associated with the plurality of images. The cache pool includes a cache mapping dictionary for associating word embeddings with request identifiers. as well as A task scheduling component, wherein the task scheduling component is configured to: Receive the request; In response to the detection that the large language model needs to perform the first part of the language inference task associated with the first word embedding and the language elements, The cache mapping dictionary is examined based on a first request identifier associated with the first word embedding; Based on the results of the inspection, the first word embedding is selectively loaded from the cache pool, and the large language model is selectively instructed to invoke the computing resources to perform the first part of the language inference task. Specifically, the execution of the first part of the language reasoning task by the large language model and the execution of the visual encoding computation by the visual processing server are performed at least partially simultaneously. The first response to the first request is determined at least in part based on the execution result of the first part of the language reasoning task.

2. The language reasoning server as described in claim 1, characterized in that, The request is the first request, and the response is the first response. The task scheduling component is also configured to: A second request is sent to the vision processing server. The second request includes the plurality of images and a request identifier associated with each image, and is associated with the visual encoding calculation of each image. The visual processing server receives a second response to the second request, the second response including a word embedding and an associated request identifier, the word embedding being obtained by the visual processing server performing visual encoding calculations on the plurality of images; The received words are embedded and written into the cache pool; as well as Update the cache mapping dictionary with the request identifier associated with the received word embedding.

3. The language reasoning server as described in claim 2, characterized in that, The second response is received via the Remote Direct Memory Access Protocol.

4. The language reasoning server as described in claim 1, characterized in that, The first part of the language reasoning task performed by the large language model is at least partially concurrent with the part of the visual encoding computation performed by the visual processing server on the image associated with the second request identifier.

5. The language reasoning server as described in claim 1, characterized in that, The task scheduling component is configured to: In response to the result of the check indicating that the first request identifier exists in the cache mapping dictionary, Load the first word embedding from the cache pool; The first word embedding is provided to the large language model; and The large language model is instructed to perform the first part of the language reasoning task.

6. The language reasoning server as described in claim 1, characterized in that, The task scheduling component is configured to: In response to the result of the detection indicating that the first request identifier does not exist in the cache mapping dictionary, the large language model is instructed to wait until the first word is ready to be embedded in the cache pool.

7. The language reasoning server as described in claim 1, characterized in that, The computing resources include a graphics processing unit, the graphics processing unit includes video memory, and the cache pool is associated with the video memory.

8. The language reasoning server as described in claim 7, characterized in that, It also includes memory, and the cache pool is also associated with the memory.

9. The language reasoning server as described in claim 8, characterized in that, The cache pool includes a hierarchical cache, which comprises a first-level cache and a second-level cache. The first-level cache is used to temporarily store word embeddings, and the second-level cache is used to permanently store word embeddings. The first-level cache is associated with the video memory, and the second-level cache is associated with the main memory.

10. The language reasoning server as described in claim 1, characterized in that, The request is associated with a text-based question-and-answer task, a graph comprehension task, a video comprehension task, or a MathVista logical reasoning task.

11. A system for large-scale visual language model reasoning, characterized in that, include: The language inference server as described in any one of claims 1-10, wherein the computing resource is a first computing resource, and the task scheduling component is a first task scheduling component; and A visual processing server, communicatively coupled to the language inference server, the visual processing server comprising: Second computing resource; Visual encoder; and The second task scheduling component is configured as follows: A second request is received from the language inference server. The second request includes the plurality of images and a request identifier associated with each image, and is associated with the visual encoding calculation of each image. The visual encoder is instructed to invoke the second computing resources to process the plurality of images included in the second request to obtain word embeddings; Associate the word embedding with the request identifier; and A second response is sent to the language inference server, the second response including the word embedding and the associated request identifier.

12. The system for large-scale visual language reasoning as described in claim 11, characterized in that, The second task scheduling component is configured as follows: Merge the multiple images into multiple batches; and The visual encoder is instructed to call the second computing resource to process the multiple images in batches to obtain word embeddings.

13. The system as described in claim 11, characterized in that, The vision processing server also includes: A downsampling module is configured to downsample the word embeddings generated by the visual encoder at multiple downsampling rates, including a first downsampling rate and a second downsampling rate. The downsampling module is configured to perform downsampling on the word embeddings at the first downsampling rate or the second downsampling rate based on the results of analysis of the multiple images, wherein the downsampled word embeddings are included in the second response.

14. The system as described in claim 13, characterized in that, The downsampling module includes: The first downsampling submodule is configured to perform grouped average downsampling on the word embeddings generated by the visual encoder at the first downsampling rate; and The second downsampling submodule is configured to perform nonlinear mapping compression downsampling on the word embeddings generated by the visual encoder at the second downsampling rate. Wherein, the first downsampling rate is greater than the second downsampling rate.

15. The system as described in claim 14, characterized in that, The parameters of the visual encoder are copied from the teacher's visual encoder. The parameters of the downsampling module are copied from the teacher downsampling module. The parameters of the large language model are copied from the teacher's large language model. The teacher visual encoder, the teacher downsampling module, and the teacher large language model constitute a teacher system for performing visual language large model inference. The large language model is obtained from the teacher's large language model through distillation.

16. The system as described in claim 15, characterized in that, The loss function used in the distillation includes at least one of classification soft label loss, lexical feature reconstruction and orientation alignment loss, and lexical coverage loss.

17. The system as claimed in claim 15, characterized in that, The positional encoding information obtained by the visual encoder during the word embedding generation process is retained and added to the attention masking mechanism of the large language model.

18. The system as claimed in claim 13, characterized in that, The vision processing server also includes: The routing module is configured as follows: Analyze the multiple images; and Based on the results of the analysis, the downsampling module is instructed to perform downsampling on the word embedding at either the first downsampling rate or the second downsampling rate.

19. A method for linguistic reasoning, characterized in that, include: S1: Receive a request from a user at the language inference server. The request includes multiple images and language elements and is associated with a text-image processing task. Each of the multiple images is associated with a request identifier. In response to the detection that the large language model needs to perform the first part of a language inference task associated with the first word embedding and the language elements, S2: Check the cache mapping dictionary included in the cache pool based on the first request identifier associated with the first word embedding, the cache pool is used to cache the word embeddings required by the language inference task, the word embeddings in the cache pool are from the result of visual encoding calculations associated with the plurality of images performed by a visual processing server communicatively coupled to the language inference server, and the cache mapping dictionary is used to associate the word embeddings with the request identifier; S3: Based on the results of the inspection, selectively load the first word embedding from the cache pool, and selectively instruct the large language model at the language inference server to call the computing resources at the language inference server to execute the first part of the language inference task. The first part of the language reasoning task performed by the large language model is at least partially concurrent with the visual encoding computation performed by the visual processing server.

20. The method as described in claim 19, characterized in that, The request is the first request, and the response is the first response. The method further includes: S4: Send a second request to the vision processing server, the second request including the plurality of images and a request identifier associated with each image, and associated with the visual encoding calculation of each image; S5: Receive a second response to the second request from the visual processing server, the second response including a word embedding and an associated request identifier, the word embedding being obtained by the visual processing server performing visual encoding calculations on the plurality of images; S6: Write the received word embeddings into the cache pool; and S7: Update the cache mapping dictionary with the request identifier associated with the received word embedding.

21. The method as described in claim 19, characterized in that, The first part of the language reasoning task performed by the large language model is performed at least partially concurrently with the part of the visual processing server performing visual encoding computation on an image associated with a second request identifier.

22. The method as described in claim 19, characterized in that, S3 includes: In response to the result of the check indicating that the first request identifier exists in the cache mapping dictionary, S31: Load the first word embedding from the cache pool; S32: Provide the first word embedding to the large language model; and S33: Instruct the large language model to perform the first part of the language reasoning task.

23. The method as described in claim 19, characterized in that, S3 includes: S34: In response to the result of the detection indicating that the first request identifier does not exist in the cache mapping dictionary, instruct the large language model to wait until the first word is ready to be embedded in the cache pool.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 19-23.

25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 19-23.

Images