Split multi-modal large language model service system and first word element generation method

Abstract

The application relates to the technical field of artificial intelligence, and discloses a separated multi-modal large language model service system and a first word element generation method. The system comprises a preprocessor, an encoder instance, an instance interaction layer and a prefilling instance. The preprocessor is configured to analyze a question and answer request to obtain original data of multiple modes. The encoder instance is used for parallel coding of the original data of each mode to generate sub-sequences of the corresponding modes. The instance interaction layer is configured to send the sub-sequences of each mode generated by the encoder instance to the prefilling instance. The prefilling instance comprises a large language model trunk configured to asynchronously prefill each sub-sequence of each mode in the granularity of the sub-sequences to generate a first word element of an answer text corresponding to the question and answer request. The system can improve the response speed of an online question and answer service system and improve user experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a separate multimodal large language model service system and a method for generating initial words. Background Technology

[0002] In recent years, advancements in generative artificial intelligence have driven the development and integration of Large Language Models (LLMs). In online service scenarios, LLMs have demonstrated enormous potential for intelligent reasoning and question answering. Time to First Token (TTFT) is a key metric for measuring the service-level goals of autoregressive inference systems, measuring the latency required for the model to generate the first token of the answer text. In online question-answering service systems deploying LLMs, to ensure service response speed and user experience, the model is often required to provide services to users with low TTFT generation latency.

[0003] Unlike traditional large language models, multimodal large language models (MLLMs) require processing data from multiple modalities, making the process of generating first-word units (TFUs) more complex. Therefore, MLLMs generally suffer from excessively long time-to-time (TTFTs). For example, the TTFT of MLLMs can reach tens of seconds. Based on this, shortening the first-word unit generation latency and improving the response speed of online question-answering service systems deployed with MLLMs is a problem that needs to be solved. Summary of the Invention

[0004] In view of this, this application aims to propose a separate multimodal large language model service system and a first word generation method to shorten the first word generation latency of the multimodal large language model in the online question answering service system and improve the response speed of the online question answering service system.

[0005] To achieve the above objectives, the technical solution of this application is as follows:

[0006] The first aspect of this application provides a separate multimodal large language model service system, the system comprising:

[0007] Preprocessor, encoder instance, instance interaction layer, and pre-filled instance;

[0008] The preprocessor is configured to parse question-and-answer requests to obtain raw data in multiple modalities; the question-and-answer requests contain raw data in at least two of the following modalities: text, images, and audio;

[0009] The encoder instance includes encoders of multiple modes and corresponding encoding schedulers; each encoding scheduler is used to schedule the raw data of the corresponding mode to the corresponding encoder, so that the encoders of each mode encode in parallel to generate the subsequence of the corresponding mode.

[0010] The instance interaction layer is configured to send the subsequences of each modality generated by the encoder instance to the pre-filled instance;

[0011] The pre-filled instance includes a large language model backbone; the large language model backbone is configured to asynchronously pre-fill each subsequence of each modality at the subsequence granularity to generate the first word of the answer text corresponding to the question-answering request.

[0012] Optionally, the instance interaction layer includes: a message control module running on the central processing unit; the pre-filled instance further includes: a sub-sequence level scheduler;

[0013] Each encoding scheduler is also configured to generate a completion message corresponding to the subsequence after the subsequence of the corresponding modality is generated;

[0014] The message control module is configured to send the completion message to the sub-sequence level scheduler via a cross-process message queue.

[0015] The subsequence-level scheduler is configured to allocate the required key-value cache resources for pre-filling the corresponding subsequence based on the completion message.

[0016] Optionally, the preprocessor is further configured to generate a global data structure based on the data organization method defined in the question-and-answer request; the global data structure is used to record the interleaving order between the subsequences of each modality generated based on the question-and-answer request;

[0017] The subsequence-level scheduler is also configured to pre-fill each subsequence according to the interleaving order between them.

[0018] Optionally, the system further includes a performance model analyzer;

[0019] The performance model analyzer is configured to calculate the arrival time of subsequences in the encoders to the large language model backbone based on the scheduling algorithm used by the encoders of each modality, and send it to the subsequence-level scheduler.

[0020] The subsequence-level scheduler is further configured to add the arrival time of each subsequence to the global data structure; and to determine a pre-filled scheduling strategy based on the arrival time and interleaving order of each subsequence, and to schedule the pre-filled subsequences according to the scheduling strategy.

[0021] Optionally, the preprocessor is further configured to generate a request queue based on the arrival time of each question-and-answer request in the case of multiple received question-and-answer requests.

[0022] The subsequence-level scheduler is configured to calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request, specifically including:

[0023] If the first question-and-answer request contains k subsequences, and the first m subsequences have been pre-padded, then iterate through all n∈[m+1,k];

[0024] Based on the arrival time of each subsequence, calculate the earliest completion time d[n] of the nth subsequence. Specifically, this includes: traversing all p∈[m,n-1] based on the current value of n; calculating the total time required to merge the remaining np subsequences for pre-filling after completing the pre-filling of p subsequences, according to the arrival time of each subsequence; and determining the minimum term among the total times corresponding to all p as the earliest completion time d[n] of the nth subsequence.

[0025] Based on the earliest completion time d[n] of the nth subsequence, calculate the earliest completion time d[n+1] of the (n+1)th subsequence;

[0026] After traversing all possible values ​​of n, the earliest completion time of the kth subsequence is determined as the earliest completion time of the first question-and-answer request.

[0027] The pre-filling scheduling method of each subsequence corresponding to the earliest completion time is determined as the scheduling strategy of the subsequence in the first question and answer request.

[0028] Optionally, the sub-sequence-level scheduler is configured to determine a pre-filled scheduling strategy for multi-request scenarios based on the real-time state and interleaving order of the sub-sequences of each question-and-answer request, specifically including:

[0029] Calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request;

[0030] Iterate through the subsequences of other question-and-answer requests in the request queue, excluding the first question-and-answer request, and determine whether the earliest completion time of the first question-and-answer request increases when the subsequences of other question-and-answer requests are pre-filled together with the subsequence of the first question-and-answer request.

[0031] Without increasing the earliest completion time of the first question and answer request, the subsequences of other question and answer requests are pre-filled together with the subsequence of the first question and answer request.

[0032] Optionally, the subsequence-level scheduler is configured to schedule the pre-filling of each subsequence according to the scheduling policy, specifically including:

[0033] Add all subsequences that arrive at the pre-filled instance to the waiting queue;

[0034] Traverse each subsequence in the waiting queue, and determine the preceding subsequence of the subsequence based on the interleaving order of the subsequences recorded in the global data structure;

[0035] Determine whether all preceding subsequences of the given subsequence have been pre-padded or can be pre-padded.

[0036] If all preceding subsequences of the given subsequence have been pre-filled or are ready to be pre-filled, key-value cache resources are allocated to the subsequence, and the subsequence is scheduled to the large language model backbone based on the scheduling strategy, so that the large language model backbone pre-fills the subsequence.

[0037] Optionally, the preprocessor is further configured to perform the following steps:

[0038] Based on the amount of raw data for each modality in the question-and-answer request, determine the first target number of the encoder for the corresponding modality and the second target number of pre-filled instances;

[0039] Based on the first target quantity, the type and quantity of encoders for each mode in the encoder instance are adjusted;

[0040] The number of pre-filled instances is adjusted based on the second target number.

[0041] Optionally, the instance interaction layer further includes: a transmission module deployed on the graphics processor of at least one server;

[0042] The instance interaction layer is configured to send the subsequences of each modality generated by the encoder instance to the pre-fill instance, specifically including: sending the subsequences of each modality to the pre-fill instance via the graphics processor channel through the transmission module using remote direct subsequence transmission.

[0043] According to a second aspect of the embodiments of this application, a method for generating initial words is provided, applied to the system of the method provided in the first aspect of the embodiments of this application, the method comprising:

[0044] Parse the question-and-answer request to obtain raw data in multiple modalities; the question-and-answer request contains raw data in at least two of the following modalities: text, image, and audio;

[0045] The original data of the multiple modalities are processed in parallel by encoders of different modalities to generate subsequences of the corresponding modalities;

[0046] The generated subsequences of each modality are sent to the pre-filled instance;

[0047] Asynchronous pre-filling is performed on each subsequence of each modality at the subsequence granularity to generate the first word of the answer text corresponding to the question-and-answer request.

[0048] The separate multimodal large language model service system provided in this application decouples the two stages (encoding stage and pre-filling stage) of generating first-word units in the multimodal large language model, thereby achieving asynchronous pre-filling of subsequences of different modalities. Specifically, firstly, a preprocessor parses the user's question-and-answer request to obtain raw data of at least two modalities, such as text, images, and audio. Then, the encoders of each modality in the encoder instance encode the raw data of the corresponding modality, generating subsequences of the corresponding modality, and send the generated subsequences to the pre-filling instance through the instance interaction layer, without waiting for the raw data of all modalities to be encoded before sending them to the pre-filling instance. Furthermore, the pre-filling instance performs asynchronous pre-filling of the received subsequences of each modality at the subsequence granularity.

[0049] Compared to the traditional two-stage serial pre-filling scheme, this application decouples the two stages of generating the first word unit in a multimodal large language model, realizing parallel processing of data encoding and data pre-filling. This greatly shortens the latency of generating the first word unit for question-and-answer requests, improves the response speed of the online question-and-answer service system, and enhances the user experience. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an architecture diagram of a split multimodal large language model service system proposed in one embodiment of this application;

[0052] Figure 2 This is a flowchart of the generation of the first word element of a question-and-answer request in one embodiment of this application;

[0053] Figure 3 This is a flowchart of a method for generating initial words according to an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0056] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects as detailed in this application.

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0059] Current methods for optimizing the first-word generation latency in LLM models cannot be directly applied to MLLMs models.

[0060] Because MLLMs need to process data from multiple modalities, the process of generating the first token in MLLMs is more complex than that of traditional LLMs. Unlike traditional LLM inference, which typically involves two stages—pre-filling (processing text input and generating the first token) and decoding (autoregression to generate the remaining tokens)—MLLMs inference differs in its first token generation process.

[0061] Existing MLLMs (Multi-Level Multi-Level Messaging) question-answering systems generally employ a coupled encoding and pre-filling phase, co-scheduling the encoder module with the LLM backbone. This leads to unnecessary data dependencies within individual requests and between multiple requests, resulting in excessively long first-token generation latency. Specifically, the coupled design forces the system to execute serially during the encoding phase, resulting in poor batch processing capabilities and significantly extending the time-to-threshold (TTFT). Furthermore, in the pre-filling phase, the coupled design ignores the asynchronous arrival of encoded tokens; the system only initiates pre-filling after all multi-modal inputs have been encoded, severely slowing down the first-token generation latency. Compared to the LLM model, the first-token generation latency of MLLMs includes not only the encoding phase delay but also the pre-filling phase delay, thus resulting in a significantly longer overall TTFT than the traditional LLM model.

[0062] This application decouples the encoding and pre-filling phases of the MLLMs question-answering service system, thereby eliminating unnecessary data dependencies in the process, making full use of the data dependency characteristics of the LLM backbone itself, and enabling the encoding and pre-filling phases to be pipelined in parallel.

[0063] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0064] Figure 1 This is an architecture diagram of a split multimodal large language model service system 100 proposed in one embodiment of this application. Figure 1 As shown, the system includes: a preprocessor 101, an encoder instance 102, an instance interaction layer 103, and a pre-filled instance 104;

[0065] The preprocessor 101 is configured to parse question-and-answer requests to obtain raw data in multiple modalities; the question-and-answer requests include raw data in at least two of the following modalities: text, images, and audio;

[0066] The encoder instance 102 includes encoders of multiple modes and corresponding encoding schedulers; each encoding scheduler is used to schedule the raw data of the corresponding mode to the corresponding encoder, so that the encoders of each mode encode in parallel to generate the subsequence of the corresponding mode.

[0067] The instance interaction layer 103 is configured to send the subsequences of each modality generated by the encoder instance 102 to the pre-fill instance 104;

[0068] The pre-filling instance 104 includes a large language model backbone; the large language model backbone is configured to asynchronously pre-fill each subsequence of each modality at the subsequence granularity to generate the first word of the answer text corresponding to the question-answering request.

[0069] In this embodiment, the split multimodal large language model service system 100 is an online question-and-answer service system capable of supporting high-concurrency question-and-answer requests and suitable for multi-user online question-and-answer service scenarios. The split multimodal large language model service system 100 is a split system, including: a preprocessor 101, an encoder instance 102, an instance interaction layer 103, and a pre-filling instance 104. The system's processing of question-and-answer requests can be divided into two stages: a decoupling encoding stage and an asynchronous pre-filling stage. The two stages generally satisfy the order of performing the decoupling encoding stage first and then the asynchronous pre-filling stage, but there is pipeline parallelism in the process, meaning there is overlap in time slices.

[0070] In this embodiment, a corresponding encoder is set up for each modality of data, and an isolated execution environment is provided for each modality encoder to ensure that each modality encoding is executed independently and in parallel. In practical applications, a dedicated process can be set up to allocate hardware resources to each encoder to provide an isolated execution environment for each encoder, and an independent scheduler can be provided for each modality encoder to avoid data dependencies caused by batch processing requests in the pre-filling stage. For example, in the traditional serial encoding method, the data of modality A must wait for the data of modality B to be encoded before it can be encoded, resulting in unnecessary data dependencies. In this embodiment, the encoding scheduler of each modality schedules the original data of the corresponding modality to the corresponding encoder for processing according to its own scheduling algorithm and allocated hardware resources.

[0071] When the service system receives a user's question-and-answer request, it first parses the request through the preprocessor 101 to obtain raw data in multiple modalities. In this embodiment, the question-and-answer request includes raw data in at least two different modalities, such as text data, image data, and audio data. The multimodal data is classified and added to the waiting queue of the encoder of the corresponding modality. Subsequently, according to the scheduling algorithm of the encoder scheduler and the allocated hardware resources, the encoders of each modality execute independent encoding in parallel in an isolated environment, generating corresponding subsequences, and sending them to the pre-filled instance 104 through the instance interaction layer 103.

[0072] Optionally, different modal schedulers employ customized encoding strategies for data scheduling. For example, depending on actual needs, the image scheduler schedules multiple images to the image encoder, which performs batch encoding on the multiple images; or, the image scheduler schedules a single image to the image encoder, which directly encodes the single image.

[0073] In the asynchronous pre-filling phase, the large language model backbone in pre-filling instance 104 performs asynchronous segmented pre-filling of received subsequences of different modalities at the sub-sequence granularity. Simultaneously, encoder instance 102 can encode the remaining data in parallel, achieving pipelined parallelism between the encoding and pre-filling phases. For example, in a question-and-answer request containing both text and image data, since text data is encoded faster, it can be encoded before image data is fully encoded. In this case, pre-filling instance 104 can prioritize segmented pre-filling of the already encoded text data without waiting for all data to be encoded. After pre-filling all subsequences of a question-and-answer request, pre-filling instance 104 generates the first token (i.e., the first word) of the answer text corresponding to that request. Compared to traditional coupled inference service systems, where there is only one LLM-centric scheduler and the pre-filling phase must wait until the encoding phase is completely finished, this application significantly improves TTFT and shortens user waiting time.

[0074] In this embodiment, unnecessary data dependencies are eliminated through decoupled encoding and segmented pre-filling, enabling pipelined parallel processing of the two stages. This improves system response speed, shortens first-word generation latency, and enhances user experience while ensuring correct inference. Especially in high-concurrency scenarios (e.g., batch processing of multi-user question-and-answer scenarios), the system can respond quickly and improve batch processing efficiency.

[0075] As one embodiment of this application, the instance interaction layer includes: a message control module running on a central processing unit; the pre-filled instance further includes: a sub-sequence level scheduler;

[0076] Each encoding scheduler is also configured to generate a completion message corresponding to the subsequence after the subsequence of the corresponding modality is generated;

[0077] The message control module is configured to send the completion message to the sub-sequence level scheduler via a cross-process message queue.

[0078] The subsequence-level scheduler is configured to allocate the required key-value cache resources for pre-filling the corresponding subsequence based on the completion message.

[0079] In one embodiment, since the encoder instance and the pre-fill instance have different execution contexts, to ensure close cooperation between the decoupled encoder instance and the pre-fill instance, a corresponding cross-process message queue is allocated for each modality encoder to transmit subsequence encoding completion messages. A subsequence-level scheduler is deployed in the pre-fill instance to schedule the segmented pre-fill process of the subsequences based on the completion messages of each subsequence.

[0080] Specifically, a message control module is deployed in the instance interaction layer, running on the server's CPU (Central Processing Unit). In the control plane, the message control module sends the completed sub-sequence encoding messages (i.e., completion messages) for each modality to the sub-sequence-level scheduler in the pre-fill instance via cross-process message queues corresponding to different modalities. The sub-sequence-level scheduler dynamically schedules the segmented pre-filling process based on the completion messages of each modality's sub-sequences, determines the pre-filling strategy, and allocates the necessary key-value cache resources to the sub-sequences to be pre-filled.

[0081] As one embodiment of this application, the instance interaction layer further includes: a transmission module deployed on the graphics processor of at least one server;

[0082] The instance interaction layer is configured to send the subsequences of each modality generated by the encoder instance to the pre-fill instance, specifically including: sending the subsequences of each modality to the pre-fill instance via the graphics processor channel through the transmission module using remote direct subsequence transmission.

[0083] The dedicated hardware resources of encoder instances create data transfer requirements between different accelerators. Since subsequences can be quite large (the length is related to the hidden layer dimension of the LLM backbone), transmitting data through the control plane results in high latency. In traditional encoding data transfer methods, encoded data must first be transmitted from the data plane (e.g., the GPU channel) to the control plane of the server where the encoder instance resides, then to the control plane of the server where the pre-filled instance resides, and finally from the control plane of the server where the pre-filled instance resides to the data plane of that server. This process is cumbersome and resource-intensive.

[0084] Based on this, in one embodiment, the instance interaction layer separates and transmits the encoder instance's messages and data through different channels. Specifically, in addition to the message control module, the instance interaction layer also deploys a transmission module. The transmission module runs on the server's GPU (Graphics Processing Unit) and utilizes high-speed RDMA (Remote Direct Memory Access) communication between GPUs. RDMA technology allows data to be transferred directly from the memory of one computer to another without the intervention of the CPUs or host memory of either computer, thereby reducing CPU involvement and transmission latency. In this embodiment, the instance interaction layer utilizes RDMA technology to remotely and directly send the subsequences of each modality to the pre-filled instance via the GPU channel through remote direct subsequence transmission, thereby improving data transmission efficiency and reducing transmission overhead.

[0085] In one embodiment, the instance interaction layer interacts with the pre-fill instance through the control plane to transmit completion messages, enabling pre-fill scheduling and resource management. Simultaneously, it interacts with the pre-fill instance through the data plane to transmit corresponding sub-sequences, enabling the actual execution of pre-filling. The two channels process asynchronously; completion messages for each sub-sequence are transmitted to the pre-fill instance for pre-fill scheduling. Separating the control plane and data plane facilitates efficient pre-filling in a decoupled state between the encoder instance and the pre-fill instance. When an encoder generates a sub-sequence, the instance interaction layer sends a completion signal to the pre-fill instance through the message queue of the control plane. Simultaneously, the sub-sequence arrives asynchronously at the data plane of the pre-fill instance via remote direct sub-sequence transmission. The pre-fill instance performs sub-sequence synchronization on the data plane to ensure transmission completion; the entire data transmission process is completely transparent to the control plane.

[0086] Compared to traditional data transmission methods, this embodiment utilizes GPU high-speed RDMA technology and adopts a remote subsequence transmission method to transmit subsequences directly from the data plane to the data plane of the pre-filled instance via the GPU channel, without the need for relaying through the control plane (such as the CPU channel). This achieves one-time transmission of subsequences, reduces data transmission overhead, improves transmission efficiency, and further shortens the model's TTFT.

[0087] Alternatively, in practical applications, the interaction between the control plane and the data plane can be achieved in addition to message queues, using various methods such as shared memory, semaphores, WebWorker communication, and WebSocket, to adapt to different Web operating environments and security requirements.

[0088] In one embodiment of this application, the preprocessor is further configured to generate a global data structure according to the data organization method defined in the question-and-answer request; the global data structure is used to record the interleaving order between the sub-sequences of each modality generated based on the question-and-answer request;

[0089] The subsequence-level scheduler is also configured to pre-fill each subsequence according to the interleaving order between them.

[0090] Considering the data dependencies in the pre-filling stage, namely that the pre-filling order of each subsequence in the pre-filling stage has certain requirements, the subsequence-level scheduler needs to accurately know the interleaving order between each subsequence in order to perform pre-filling scheduling.

[0091] Based on this, in one embodiment, when the preprocessor parses the user's question-and-answer request, it generates a global data structure MMSequence according to the data organization method defined in the question-and-answer request. This global data structure is used to record the real-time status of the subsequence corresponding to each question-and-answer request to be processed, as well as the interleaving order of the subsequences. When the subsequence scheduler in the pre-fill instance performs pre-fill scheduling on each subsequence arriving at the pre-fill instance, it schedules the pre-filling order of the subsequences based on the interleaving order between the subsequences recorded in the global data structure MMSequence, thereby ensuring correct inference of the model.

[0092] Specifically, in the decoupled encoding phase, the system receives multimodal question-and-answer requests from users, parses the requests to obtain raw data for different modalities, and generates a global data structure MMSequence based on the data organization method defined in the question-and-answer requests. Next, the encoders for each modality independently encode the raw data for their respective modalities, and the encoding scheduler generates corresponding completion messages. The instance interaction layer transmits the completion messages and corresponding subsequences to the pre-filling instance through the control plane and data plane, respectively. In the asynchronous pre-filling phase, the subsequence-level scheduler makes scheduling decisions based on the global data structure MMSequence to complete the segmented pre-filling of each subsequence. After completing the pre-filling of all subsequences corresponding to the question-and-answer request, the pre-filling instance manages and reclaims resources such as memory and communication events, and updates the information in the global data structure MMSequence.

[0093] As one embodiment of this application, the system further includes a performance model analyzer;

[0094] The performance model analyzer is configured to calculate the arrival time of subsequences in the encoders to the large language model backbone based on the scheduling algorithm used by the encoders of each modality, and send it to the subsequence-level scheduler.

[0095] The subsequence-level scheduler is further configured to add the arrival time of each subsequence to the global data structure; and to determine a pre-filled scheduling strategy based on the arrival time and interleaving order of each subsequence, and to schedule the pre-filled subsequences according to the scheduling strategy.

[0096] In one embodiment, a performance model analyzer is deployed in the system to estimate the encoding completion time of each modality's subsequence. The performance model analyzer estimates the processing time for the encoder of each modality and the main trunk of the large language model. Specifically, based on the scheduling algorithm used by each encoding scheduler and the size of the original data, the performance model analyzer estimates the encoding completion time of the corresponding modality's subsequence and determines it as the arrival time of the subsequence to the main trunk of the large language model.

[0097] Optionally, when the encoder uses a Transformer architecture, the performance model analyzer predicts the subsequence processing time for a specific batch by parsing the model architecture. By simulating execution and recording times under different batches, combined with regression analysis, model parameters can be determined and an accurate performance model can be established. Based on the performance model, the arrival time of the encoder-generated subsequences can be predicted. Combined with the encoding scheduler algorithm, the encoding and completion times of multimodal inputs are highly predictable. Through advance prediction and real-time adjustment, the time for subsequences to arrive at the pre-filled instance can be accurately estimated.

[0098] After calculating the arrival times of subsequences to the backbone of the large language model, the performance model analyzer sends these arrival times to the subsequence-level scheduler. This allows the subsequence-level scheduler to perform pre-filling scheduling based on the arrival times and interleaving order of each subsequence. Upon receiving the estimated arrival times of each subsequence from the performance model analyzer, the subsequence-level scheduler updates the arrival times of all subsequences in the global data structure using these estimates. Then, based on the arrival times of each subsequence and the interleaving order between them, it determines the current pre-filling scheduling strategy and performs pre-filling scheduling on each subsequence.

[0099] Figure 2 This is a flowchart illustrating the generation of the first word element of a question-and-answer request in one embodiment of this application. For example... Figure 2 As shown, the preprocessor parses the received multimodal question-and-answer requests, obtaining raw data for multiple modalities, and sends the raw data for each modality to the encoder instance. In the encoder instance, the encoding scheduler for each modality receives the raw data for its corresponding modality and adds it to the waiting queue. Then, the encoder for each modality encodes the raw data for a single modality based on its corresponding scheduling algorithm, generating corresponding subsequences. The encoding scheduler then generates completion messages for the corresponding subsequences. In the instance interaction layer, the completion messages of the subsequences are transmitted to the pre-filling instance via a cross-process queue in the control plane. In the data plane, high-speed RDMA technology is used to asynchronously transmit the subsequences corresponding to the completion messages to the pre-filling instance via remote subsequence transmission. The subsequence-level scheduler in the pre-filling instance determines the optimal pre-filling scheduling strategy based on the arrival time of each subsequence and the interleaving order between subsequences, and the large language model backbone executes the pre-filling of the subsequences. When all subsequences of a question-and-answer request have been pre-filled, the large language model backbone generates the first token (i.e., the first word) of the answer text corresponding to that question-and-answer request.

[0100] As one embodiment of this application, the preprocessor is further configured to generate a request queue based on the arrival time of each question-and-answer request in the case of multiple received question-and-answer requests.

[0101] The subsequence-level scheduler is configured to calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request, specifically including:

[0102] If the first question-and-answer request contains k subsequences, and the first m subsequences have been pre-padded, then iterate through all n∈[m+1,k];

[0103] Based on the arrival time of each subsequence, calculate the earliest completion time d[n] of the nth subsequence. Specifically, this includes: traversing all p∈[m,n-1] based on the current value of n; calculating the total time required to merge the remaining np subsequences for pre-filling after completing the pre-filling of p subsequences, according to the arrival time of each subsequence; and determining the minimum term among the total times corresponding to all p as the earliest completion time d[n] of the nth subsequence.

[0104] Based on the earliest completion time d[n] of the nth subsequence, calculate the earliest completion time d[n+1] of the (n+1)th subsequence;

[0105] After traversing all possible values ​​of n, the earliest completion time of the kth subsequence is determined as the earliest completion time of the first question-and-answer request.

[0106] The pre-filling scheduling method of each subsequence corresponding to the earliest completion time is determined as the scheduling strategy of the subsequence in the first question and answer request.

[0107] In one embodiment, in a multi-request batch processing scenario, the preprocessor adds all pending question-and-answer requests to the request queue sequentially according to their request times. The subsequence-level scheduler determines the current pre-filling scheduling strategy based on the arrival times of each subsequence to the large language model backbone estimated by the performance model analyzer, thereby dynamically optimizing the time-to-time (TTFT), fully releasing hardware resource potential, and improving system response speed. Furthermore, using the same model architecture as the parsing encoder, the performance model analyzer parses the model architecture of the LLM backbone, thus estimating the subsequence processing time of the backbone LLM.

[0108] Specifically, the subsequence-level scheduling algorithm optimizes each question-and-answer request by prioritizing the fastest completion of the first question-and-answer request in the request queue. In this embodiment, the subsequence-level scheduling algorithm's flow is as follows:

[0109] (1) Based on the global data structure MMSequence, the arrival times of all k subsequences of the first question-and-answer request in the request queue are obtained. If the first to m subsequences corresponding to the question-and-answer request have been pre-filled (if m=0, then no subsequences have been pre-filled yet), the remaining (m+1) to k subsequences are to be pre-filled. Then, all remaining subsequences n∈[m+1,k] are traversed.

[0110] (2) Based on the arrival time of each subsequence, recursively calculate the earliest completion timestamp of the nth subsequence (i.e., calculate the earliest completion duration of the nth subsequence) d[n].

[0111] During the recursive process, based on the current value of n, all p∈[m,n-1] are traversed, and the total time required to merge (p+1) to n (i.e. the remaining np) subsequences for pre-filling after the pre-filling of p subsequences is completed is calculated, which is the earliest completion time d[n] of the nth subsequence.

[0112] (3) Based on the earliest completion time d[n] of the nth subsequence, calculate the earliest completion time d[n+1] of the (n+1)th subsequence;

[0113] (4) After traversing all values ​​of n, that is, after traversing (m+1) to k subsequences, the earliest completion time of the kth subsequence is determined as the earliest completion time of the first question-and-answer request. This scheduling algorithm aims to recursively compare the pre-filling methods of each subsequence in the remaining subsequences to determine which pre-filling method can complete the generation of the first token the fastest. After determining the earliest completion time of the first question-and-answer request, the pre-filling scheduling method of each subsequence corresponding to the earliest completion time of the first question-and-answer request is determined as the optimal scheduling strategy for pre-filling the subsequences of the question-and-answer request.

[0114] Optionally, in addition to the subsequence-level scheduling algorithm provided in this embodiment, the subsequence-level scheduler may also use various scheduling algorithms such as priority scheduling, shortest remaining time priority, and machine learning-based adaptive scheduling to determine the pre-filling scheduling strategy based on the needs of practical applications.

[0115] As one embodiment of this application, the sub-sequence-level scheduler is configured to determine a pre-filled scheduling strategy for multi-request scenarios based on the real-time state and interleaving order of the sub-sequences of each question-and-answer request, specifically including:

[0116] Calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request;

[0117] Iterate through the subsequences of other question-and-answer requests in the request queue, excluding the first question-and-answer request, and determine whether the earliest completion time of the first question-and-answer request increases when the subsequences of other question-and-answer requests are pre-filled together with the subsequence of the first question-and-answer request.

[0118] Without increasing the earliest completion time of the first question and answer request, the subsequences of other question and answer requests are pre-filled together with the subsequence of the first question and answer request.

[0119] In one embodiment, in order to optimize the latency of first word generation in a batch processing scenario with multiple requests, in addition to using a subsequence-level scheduling algorithm to schedule a single question-and-answer request, a subsequence batch processing algorithm is also used to determine the scheduling strategy between each question-and-answer request.

[0120] The subsequence batch processing algorithm aims to schedule as many subsequences of other question-and-answer requests in the request queue as possible, while ensuring that the TTFT of the first question-and-answer request in the request queue remains the earliest completion time. In this embodiment, the flow of the subsequence batch processing algorithm is as follows:

[0121] (1) Using a subsequence-level scheduling algorithm, calculate the earliest completion time of the first question-and-answer request in the request queue and determine the scheduling strategy for each subsequence in the question-and-answer request;

[0122] (2) Traverse all subsequences corresponding to other question-and-answer requests in the request queue except for the first question-and-answer request, and determine whether the earliest completion time of the first question-and-answer request increases when the subsequences of other question-and-answer requests are pre-filled in parallel with the subsequence corresponding to the first question-and-answer request. If the earliest completion time of the first question-and-answer request is not increased when the subsequences of other question-and-answer requests are pre-filled in parallel with the subsequence corresponding to the first question-and-answer request, then the subsequences of other question-and-answer requests are pre-filled together with the subsequences of the first question-and-answer request, thereby further improving the batch processing efficiency of the system.

[0123] In one embodiment, the subsequence candidate set is initialized as an empty set. The request queue is treated as a subsequence matrix (each row includes all subsequences corresponding to a question-and-answer request). Each row of the subsequence matrix, except the first row, is traversed. For each subsequence in each row, it is assumed that it is pre-filled along with the subsequences in the first row. The earliest completion time of the first question-and-answer request is calculated again using the subsequence-level scheduling algorithm. If the earliest completion time of the first request remains unchanged, then the subsequence is determined to be pre-filled along with the subsequence corresponding to the first question-and-answer request, and the subsequence is added to the subsequence candidate set. If, during the traversal of a row, the calculated completion time of the first question-and-answer request is extended, or if all subsequences in the current batch exist in the subsequence candidate set, then the traversal is terminated.

[0124] Furthermore, if there are subsequences in the first question-and-answer request that cannot be pre-filled at present, the earliest completion algorithm must be reapplied to the first question-and-answer request, assuming that these subsequences that cannot be pre-filled at present will be pre-filled after the subsequence candidate set. Only if the new earliest completion time of the first question-and-answer request remains unchanged after recalculation can the remaining subsequences in the subsequence candidate set be determined to be pre-filled together with the subsequences of the first question-and-answer request; otherwise, the subsequence candidate set is empty.

[0125] In this embodiment, the pre-filling method between each question and answer request is scheduled using a subsequence batch processing algorithm. Combined with the subsequence-level scheduling algorithm, the generation latency of the first word of the remaining question and answer requests in the request queue is reduced while ensuring that the earliest completion time of the first question and answer request remains unchanged. This improves the overall throughput and response efficiency of the system.

[0126] As one embodiment of this application, the sub-sequence-level scheduler is configured to schedule the pre-filling of each sub-sequence according to the scheduling policy, specifically including:

[0127] Add all subsequences that arrive at the pre-filled instance to the waiting queue;

[0128] Traverse each subsequence in the waiting queue, and determine the preceding subsequence of the subsequence based on the interleaving order of the subsequences recorded in the global data structure;

[0129] Determine whether all preceding subsequences of the given subsequence have been pre-padded or can be pre-padded.

[0130] If all preceding subsequences of the given subsequence have been pre-filled or are ready to be pre-filled, key-value cache resources are allocated to the subsequence, and the subsequence is scheduled to the large language model backbone based on the scheduling strategy, so that the large language model backbone pre-fills the subsequence.

[0131] In one embodiment, the subsequence-level scheduler schedules the pre-filling of each subsequence arriving at the pre-fill instance based on the current scheduling policy, specifically including the following steps:

[0132] (1) Based on the global data structure MMSequence, add all subsequences that arrive at the pre-filled instance to the pre-filled waiting queue;

[0133] (2) Traverse each subsequence in the waiting queue and determine whether a subsequence can be pre-filled based on the data dependencies of the pre-filling. It is worth noting that in batch processing scenarios, the pre-filling scheduling of each subsequence is no longer based on question-and-answer requests, but rather on all subsequences that meet the conditions for entering the pre-filling stage, further improving the system response speed. For example, if some subsequences corresponding to user A's question-and-answer request have not been pre-filled, the subsequences corresponding to user B's question-and-answer request can be pre-filled. The steps to determine whether a subsequence can be pre-filled at present are as follows:

[0134] 1) Read the interleaving order of subsequences recorded in the global data structure to determine the preceding subsequence of the given subsequence;

[0135] 2) Determine whether the preceding subsequences of the given subsequence have been pre-padded, or whether pre-padded is currently possible. Also, determine whether the given subsequence can currently be pre-padded. If the given subsequence is currently available for pre-padded, and all its preceding subsequences have been pre-padded or are capable of being pre-padded, then the given subsequence can be pre-padded.

[0136] (3) Allocate key-value cache resources required for pre-filling to the subsequences that can be pre-filled at present, and schedule the pre-filling of the subsequences based on the scheduling strategy, so that the pre-filling of each subsequence is executed by the main trunk of the large language model;

[0137] (4) After the subsequence prefilling is completed, resources such as memory and communication events are managed and reclaimed. For example, the KV-cache (i.e., key-value cache resources) occupied by the subsequence is released after prefilling, while the memory occupied by the subsequence itself is reclaimed after the kernel starts. In addition, after the subsequence prefilling is completed, the subsequence information in the global data structure MMSequence is updated to mark that prefilling has been completed, so as to facilitate the scheduling of prefilling of subsequent subsequences.

[0138] In one embodiment of this application, the preprocessor is further configured to perform the following steps:

[0139] Based on the amount of raw data for each modality in the question-and-answer request, determine the first target number of the encoder for the corresponding modality and the second target number of pre-filled instances;

[0140] Based on the first target quantity, the type and quantity of encoders for each mode in the encoder instance are adjusted;

[0141] The number of pre-filled instances is adjusted based on the second target number.

[0142] In this embodiment, based on the needs of actual applications, the preprocessor also dynamically adjusts the system architecture according to the type and data volume of the question-and-answer request to better match the service scenario. Specifically, the preprocessor determines the first target number of encoders for each modality required in the current scenario based on the data volume of the original data for each modality in the question-and-answer request, and determines the second target number of pre-filled instances required based on the data volume of the original data. Furthermore, based on the first target number, the number of encoders for each modality in the encoder instances is adjusted, and based on the second target number, the number of pre-filled instances is adjusted, dynamically allocating resources to match the current application scenario.

[0143] For example, if the question-answering request only involves images and text, an audio encoder does not need to be deployed in the encoder instance. Therefore, the hardware resources of the audio encoder can be reclaimed, and the idle resources can be allocated to the image encoder and text encoder, thereby improving encoding efficiency. In multi-user batch processing scenarios, if the data volume is large, the number of encoders and pre-filled instances can be adjusted accordingly. Specifically, the data volume of the original data for each modality is compared with a first threshold, and the data volume of all original data is compared with a second threshold. If the data volume of the original data for any modality is greater than the first threshold, the number of encoders for that modality is increased accordingly; if the total data volume of all original data is greater than the second threshold, the number of pre-filled instances is increased accordingly.

[0144] In this embodiment, the modules in the system can be deployed on different servers or on the same server. Each instance (including encoder instances, pre-filling instances, etc.) can independently adopt different data parallelism and model parallelism strategies according to its task load and hardware conditions. For example, encoder instances can adopt data parallelism, increasing the number of encoders for each modality based on the amount of data to be processed to improve the processing efficiency of batch input. Pre-filling instances can adopt model parallelism, splitting the main language model in the pre-filling instance and deploying it on multiple servers, or increasing the number of pre-filling instances and deploying them on multiple servers to achieve parallel processing in the pre-filling stage and accelerate the generation of first-word units.

[0145] Furthermore, to address the unique sequential parallelism requirements of pre-filled instances, micro-batch processing, asynchronous pipelines, and segmented parallelism can be employed to satisfy data dependencies. For example, the input subsequence can be split into fine-grained subtasks, and each subtask can be assigned to different computational units, allowing for asynchronous pre-filling while satisfying data dependency constraints between subsequences.

[0146] In one embodiment, the system also includes a global resource pool for unified management and scheduling of resources across different instances, enabling dynamic allocation and reuse across instances to improve the overall system's resource utilization and scalability.

[0147] Based on the same inventive concept, one embodiment of this application provides a method for generating initial words. (Reference) Figure 3 , Figure 3 This is a flowchart of a method for generating initial morphemes according to an embodiment of this application. For example... Figure 3 As shown, this method is applied to the split multimodal large language model service system in the above embodiments, including:

[0148] S1: The question-and-answer request is parsed by a preprocessor to obtain raw data in multiple modalities; the question-and-answer request contains raw data in at least two of the following modalities: text, image, and audio;

[0149] S2: The encoders of different modes in the encoder instance process the raw data of the multiple modes in parallel to generate the subsequence of the corresponding mode;

[0150] S3: Send the generated sub-sequences of each modality to the pre-filled instance through the instance interaction layer;

[0151] S4: Using the pre-filled instance as the granularity, asynchronous pre-fill each sub-sequence of each modality is performed to generate the first word of the answer text corresponding to the question-and-answer request.

[0152] As one embodiment of this application, the method further includes:

[0153] After generating the subsequence of the corresponding modality, the completion message corresponding to the subsequence is generated by the encoding scheduler;

[0154] The completion message is sent to the sub-sequence-level scheduler via the message control module using a cross-process message queue;

[0155] Based on the completion message, the subsequence-level scheduler allocates the necessary key-value cache resources for pre-filling to the corresponding subsequence.

[0156] As one embodiment of this application, the method further includes:

[0157] A global data structure is generated based on the data organization method defined in the question-and-answer request; the global data structure is used to record the interleaving order between the sub-sequences of each modality generated based on the question-and-answer request;

[0158] The subsequence-level scheduler pre-fills each subsequence according to the interleaving order between them.

[0159] As one embodiment of this application, the method further includes:

[0160] The performance model analyzer calculates the arrival time of subsequences in the encoders to the main language model based on the scheduling algorithm used by the encoders of each modality, and sends it to the subsequence-level scheduler.

[0161] The subsequence-level scheduler adds the arrival times of each subsequence to the global data structure; and, based on the arrival times and interleaving order of each subsequence, the subsequence-level scheduler determines a pre-filling scheduling strategy, and schedules the pre-filling of each subsequence according to the scheduling strategy.

[0162] As one embodiment of this application, the method further includes;

[0163] In the case of receiving multiple question-and-answer requests, the preprocessor generates a request queue based on the arrival time of each question-and-answer request.

[0164] The subsequence-level scheduler determines a pre-filled scheduling strategy based on the arrival time and interleaving order of each subsequence, specifically including:

[0165] The following steps are followed to calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and to determine the scheduling strategy for the subsequence in the first question-and-answer request:

[0166] If the first question-and-answer request contains k subsequences, and the first m subsequences have been pre-padded, then iterate through all n∈[m+1,k];

[0167] Based on the arrival time of each subsequence, calculate the earliest completion time d[n] of the nth subsequence. Specifically, this includes: traversing all p∈[m,n-1] based on the current value of n; calculating the total time required to merge the remaining np subsequences for pre-filling after completing the pre-filling of p subsequences, according to the arrival time of each subsequence; and determining the minimum term among the total times corresponding to all p as the earliest completion time d[n] of the nth subsequence.

[0168] Based on the earliest completion time d[n] of the nth subsequence, calculate the earliest completion time d[n+1] of the (n+1)th subsequence;

[0169] After traversing all possible values ​​of n, the earliest completion time of the kth subsequence is determined as the earliest completion time of the first question-and-answer request.

[0170] The pre-filling scheduling method of each subsequence corresponding to the earliest completion time is determined as the scheduling strategy of the subsequence in the first question and answer request.

[0171] As one embodiment of this application, the method further includes:

[0172] The sub-sequence-level scheduler determines a pre-filling scheduling strategy for multi-request scenarios based on the real-time status and interleaving order of each question-and-answer request sub-sequence, specifically including:

[0173] Calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request;

[0174] Iterate through the subsequences of other question-and-answer requests in the request queue, excluding the first question-and-answer request, and determine whether the earliest completion time of the first question-and-answer request increases when the subsequences of other question-and-answer requests are pre-filled together with the subsequence of the first question-and-answer request.

[0175] Without increasing the earliest completion time of the first question and answer request, the subsequences of other question and answer requests are pre-filled together with the subsequence of the first question and answer request.

[0176] As one embodiment of this application, scheduling the pre-filling of each sub-sequence according to the scheduling strategy includes:

[0177] Add all subsequences that arrive at the pre-filled instance to the waiting queue;

[0178] Traverse each subsequence in the waiting queue, and determine the preceding subsequence of the subsequence based on the interleaving order of the subsequences recorded in the global data structure;

[0179] Determine whether all preceding subsequences of the given subsequence have been pre-padded or can be pre-padded.

[0180] If all preceding subsequences of the given subsequence have been pre-filled or are ready to be pre-filled, key-value cache resources are allocated to the subsequence, and the subsequence is scheduled to the large language model backbone based on the scheduling strategy, so that the large language model backbone pre-fills the subsequence.

[0181] As one embodiment of this application, the method further includes:

[0182] The preprocessor determines the first target number of encoders for each modality and the second target number of pre-filled instances based on the amount of raw data for each modality in the question-and-answer request. Based on the first target number, the type and number of encoders for each modality in the encoder instances are adjusted. Based on the second target number, the number of pre-filled instances is adjusted.

[0183] As one implementation of this application, sending the generated sub-sequences of each modality to the pre-filled instance through the instance interaction layer includes:

[0184] The transmission module utilizes remote direct subsequence transmission to send subsequences of each modality to the pre-filled instance via the graphics processor channel.

[0185] Based on the same inventive concept, one embodiment of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the acronym generation method as described in any of the above embodiments of this application.

[0186] Based on the same inventive concept, one embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps in the initial metaword generation method as described in any of the above embodiments of this application.

[0187] The specific operation of the methods in the above embodiments has been described in detail in the embodiments of the relevant system, and will not be elaborated here.

[0188] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0189] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and components involved are not necessarily essential to this application.

[0190] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0194] Although preferred embodiments of the embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, this application is to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0195] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0196] The above provides a detailed description of the separate multimodal large language model service system and the initial word generation method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A separate multimodal large language model service system, characterized in that, include: Preprocessor, encoder instance, instance interaction layer, and pre-filled instance; The preprocessor is configured to parse question-and-answer requests to obtain raw data in multiple modalities; the question-and-answer requests contain raw data in at least two of the following modalities: text, images, and audio; The encoder instance includes encoders of multiple modes and corresponding encoding schedulers; each encoding scheduler is used to schedule the raw data of the corresponding mode to the corresponding encoder, so that the encoders of each mode encode in parallel to generate the subsequence of the corresponding mode. The instance interaction layer is configured to send the subsequences of each modality generated by the encoder instance to the pre-filled instance; The pre-filled instance includes a large language model backbone; the large language model backbone is configured to asynchronously pre-fill each sub-sequence of each modality at the sub-sequence granularity, and after pre-filling all sub-sequences of the question-and-answer request, generate the first word unit of the answer text corresponding to the question-and-answer request; If the question-and-answer request contains first modal data and second modal data, and the second modal data is still in the encoding stage while the first modal data has been encoded, the pre-filling instance will preferentially pre-fill the already encoded first modal data in segments, without waiting for the first modal data and the second modal data to be encoded before performing the pre-filling operation.

2. The separate multimodal large language model service system according to claim 1, characterized in that, The instance interaction layer includes a message control module running on the central processing unit; the pre-filled instance also includes a sub-sequence level scheduler. Each encoding scheduler is also configured to generate a completion message corresponding to the subsequence after the subsequence of the corresponding modality is generated; The message control module is configured to send the completion message to the sub-sequence level scheduler via a cross-process message queue. The subsequence-level scheduler is configured to allocate the required key-value cache resources for pre-filling the corresponding subsequence based on the completion message.

3. The separate multimodal large language model service system according to claim 2, characterized in that, The preprocessor is further configured to generate a global data structure based on the data organization method defined in the question-and-answer request; the global data structure is used to record the interleaving order between the subsequences of each modality generated based on the question-and-answer request. The subsequence-level scheduler is also configured to pre-fill each subsequence according to the interleaving order between them.

4. The split multimodal large language model service system according to claim 3, characterized in that, The system also includes a performance model analyzer; The performance model analyzer is configured to calculate the arrival time of subsequences in the encoders to the large language model backbone based on the scheduling algorithm used by the encoders of each modality, and send it to the subsequence-level scheduler. The subsequence-level scheduler is further configured to add the arrival time of each subsequence to the global data structure; and to determine a pre-filled scheduling strategy based on the arrival time and interleaving order of each subsequence, and to schedule the pre-filled subsequences according to the scheduling strategy.

5. The split multimodal large language model service system according to claim 4, characterized in that, The preprocessor is also configured to generate a request queue based on the arrival time of each question-and-answer request in the case of multiple received question-and-answer requests. The subsequence-level scheduler is configured to calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request, specifically including: If the first question-and-answer request contains k subsequences, and the first m subsequences have been pre-padded, then iterate through all n∈[m+1,k]; Based on the arrival time of each subsequence, calculate the earliest completion time d[n] of the nth subsequence. Specifically, this includes: traversing all p∈[m,n-1] based on the current value of n; calculating the total time required to merge the remaining np subsequences for pre-filling after completing the pre-filling of p subsequences, according to the arrival time of each subsequence; and determining the minimum term among the total times corresponding to all p as the earliest completion time d[n] of the nth subsequence. Based on the earliest completion time d[n] of the nth subsequence, calculate the earliest completion time d[n+1] of the (n+1)th subsequence; After traversing all possible values ​​of n, the earliest completion time of the kth subsequence is determined as the earliest completion time of the first question-and-answer request. The pre-filling scheduling method of each subsequence corresponding to the earliest completion time is determined as the scheduling strategy of the subsequence in the first question and answer request.

6. The split multimodal large language model service system according to claim 5, characterized in that, The sub-sequence-level scheduler is configured to determine a pre-filled scheduling strategy for multi-request scenarios based on the real-time state and interleaving order of the sub-sequences of each question-and-answer request, specifically including: Calculate the earliest completion time corresponding to the first question-and-answer request in the request queue, and determine the scheduling strategy for the subsequence in the first question-and-answer request; Iterate through the subsequences of other question-and-answer requests in the request queue, excluding the first question-and-answer request, and determine whether the earliest completion time of the first question-and-answer request increases when the subsequences of other question-and-answer requests are pre-filled together with the subsequence of the first question-and-answer request. Without increasing the earliest completion time of the first question and answer request, the subsequences of other question and answer requests are pre-filled together with the subsequence of the first question and answer request.

7. The split multimodal large language model service system according to claim 4, characterized in that, The subsequence-level scheduler is configured to schedule the pre-filled subsequences according to the scheduling policy, specifically including: Add all subsequences that arrive at the pre-filled instance to the waiting queue; Traverse each subsequence in the waiting queue, and determine the preceding subsequence of the subsequence based on the interleaving order of the subsequences recorded in the global data structure; Determine whether all preceding subsequences of the given subsequence have been pre-padded or can be pre-padded. If all preceding subsequences of the given subsequence have been pre-filled or are ready to be pre-filled, key-value cache resources are allocated to the subsequence, and the subsequence is scheduled to the large language model backbone based on the scheduling strategy, so that the large language model backbone pre-fills the subsequence.

8. The split multimodal large language model service system according to claim 1, characterized in that, The preprocessor is also configured to perform the following steps: Based on the amount of raw data for each modality in the question-and-answer request, determine the first target number of the encoder for the corresponding modality and the second target number of pre-filled instances; Based on the first target quantity, the type and quantity of encoders for each mode in the encoder instance are adjusted; The number of pre-filled instances is adjusted based on the second target number.

9. The split multimodal large language model service system according to claim 1, characterized in that, The instance interaction layer also includes: a transmission module deployed on the graphics processor of at least one server; The instance interaction layer is configured to send the subsequences of each modality generated by the encoder instance to the pre-fill instance, specifically including: sending the subsequences of each modality to the pre-fill instance via the graphics processor channel through the transmission module using remote direct subsequence transmission.

10. A method for generating initial words, characterized in that, Applied to the system as described in any one of claims 1-9, comprising: The question-and-answer request is parsed by a preprocessor to obtain raw data in multiple modalities; the question-and-answer request contains raw data in at least two of the following modalities: text, image, and audio; The original data of the multiple modes are processed in parallel by encoders of different modes in the encoder instance to generate subsequences of the corresponding modes; The generated subsequences of each modality are sent to the pre-filled instance through the instance interaction layer; The pre-filling instance asynchronously pre-fills each sub-sequence of each modality at the granularity of the sub-sequence. After pre-filling all sub-sequences of the question-and-answer request, the first word unit of the answer text corresponding to the question-and-answer request is generated. If the question-and-answer request contains first modal data and second modal data, and the second modal data is still in the encoding stage while the first modal data has been encoded, the pre-filling instance will preferentially pre-fill the already encoded first modal data in segments, without waiting for the first modal data and the second modal data to be encoded before performing the pre-filling operation.

Images