Methods and apparatus, constellations, products for model inference

Abstract

Embodiments of the present disclosure relate to a method and apparatus, cluster, product for model inference. The method includes receiving a user input and retrieving a target template corresponding to the user input from a template library, the target template comprising a target template sequence, wherein the template library comprises a plurality of template sequences output by a language model in response to historical user inputs. The method further includes segmenting a plurality of token segments from the target template sequence based on a segmentation window. The method further includes determining a hit target token segment by comparing a first portion of each token segment of the plurality of token segments to a second portion of the user input. The method further includes concatenating the user input with the target token segment and feeding into the language model for inference to obtain a model output. In this way, the accuracy of the predicted token can be improved, and its parallel processing and response time can also be improved, thereby alleviating the performance degradation of decoding by reducing the reloading of weights.

Description

Technical Field

[0001] The embodiments of this application generally relate to the field of computers, and more specifically, to methods and apparatus for model reasoning, computing device clusters, and computer program products. Background Technology

[0002] Model reasoning based on language models (such as Large Language Models (LLMs)) refers to the powerful reasoning ability of language models to complete complex tasks (e.g., answering questions, responding to queries) by relying on existing knowledge and context, through logical deduction, association analysis, and knowledge integration. This capability enables language models to excel in cross-domain tasks, providing new possibilities for content creation, data-driven question answering, and decision support. Currently, the reasoning capabilities of language models are widely used in various fields such as intelligent customer service, educational tutoring, and medical consultation. Summary of the Invention

[0003] This application provides a scheme for model inference. The scheme offers an accompanying input strategy for model inference that improves the inference accuracy of accompanying input terms and enhances the parallelism and speed of model inference, thereby mitigating the impact of reloading weights on decoding performance by reducing decoding steps.

[0004] In a first aspect, a method for model inference is provided. The method includes: receiving user input and retrieving a target template corresponding to the user input from a template library, the target template including a sequence of target templates, wherein the template library includes multiple template sequences output by a language model in response to historical user input, the multiple template sequences including the target template sequence; dividing the target template sequence into multiple lexical segments based on a segmentation window; determining a hit target lexical segment by comparing a first predetermined portion of each of the multiple lexical segments with a second predetermined portion of the user input; and concatenating the user input with the target lexical segments and feeding them into the language model for inference to obtain model output. This method provides high-quality accompanying input to the user input based on the historical context of the language model, improving the inference accuracy of accompanying input lexical units, i.e., improving the quality of accompanying input lexical units. It also improves the parallel processing and response time of model inference, thereby mitigating the performance degradation caused by reloading weights by reducing the number of decoding steps during decoding.

[0005] In some embodiments of the first aspect, the method may further include: sequentially matching the model output with the target word segment word by word; and stopping the matching between the model output and the target word segment in response to a word in the model output not matching a corresponding word in the target word segment, wherein the word in the model output that has matched the target word segment and the word that has not matched can be retained for the next iteration. Thus, for a decoding iteration, the model output of the current iteration can be verified by comparing the model output of the current iteration with the target word segment of that iteration. A highly reliable input can be provided for the next iteration by selecting a reliable portion of the model output (i.e., including the matched word and the word that has not matched).

[0006] In some embodiments of the first aspect, the template library may include a plurality of templates categorized based on predetermined attributes, and the plurality of templates may include the target template. Retrieving the target template from the template library may further include: determining a first category to which the user input belongs under the predetermined attributes; and identifying, among the plurality of templates, a template having the first category or a second category similar to the first category as the target template. Thus, in a template library constructed based on the historical output of a language model, appropriate associated templates can be selected for user input to improve the model inference process.

[0007] In some embodiments of the first aspect, each of the plurality of templates may include a plurality of template sequences, and the method may further include: determining a template sequence that meets a similarity threshold as the target template sequence based on the similarity between the user-input lexical sequence and the plurality of template sequences in the determined target templates. Thus, among target templates of the same category as the user input, a target template sequence that meets the similarity requirement is selected from the plurality of template sequences based on lexical similarity to improve the accuracy and parallelism of inferred lexical units.

[0008] In some embodiments of the first aspect, each of the plurality of templates may include a template sequence, and the method may further include: determining the template sequence included in the determined target templates as the target template sequence, wherein the template sequence included in each template may be an updated template sequence under the corresponding category. Thus, among the target templates of the same category as the user input, the included template sequence is the most usable template sequence under that category. Selecting it as the target template sequence facilitates the inference of lexical terms.

[0009] In some embodiments of the first aspect, the method may further include updating the template sequence included in each template by performing pre-inference using user context associated with the corresponding template among the plurality of templates. Thus, by performing offline pre-inference using user-related context, the template sequence in the target template can be updated, thereby expanding and enriching the template library to further improve the accuracy and quality of inferred lexical terms.

[0010] In some embodiments of the first aspect, the target lexical segment can be a first target lexical segment, the model output can be a first model output, and the model output can be determined as an intermediate model output in response to the absence of a terminal symbol. The method may further include: determining a hit second target lexical segment by comparing a first predetermined portion of each of the plurality of lexical segments with a second predetermined portion of a lexical sequence retained in the first model output, wherein the lexical sequence retained in the intermediate model output can be used as the model input for the current iteration; and concatenating the model input with the second target lexical segment and feeding it into the language model for inference to obtain a second model output. Thus, in each decoding iteration, a lexical segment corresponding to a reliable lexical sequence in the model output of the previous iteration can be determined as an auxiliary input for the current iteration, thereby improving the decoding process round by round.

[0011] In some embodiments of the first aspect, the method may further include: sequentially matching the second model output with the second target word segment word by word; and stopping the matching between the second model output and the second target word segment in response to a word in the second model output not matching a corresponding word in the second target word segment, wherein the word in the second model output that has matched the second target word segment and the word that has not matched can be retained for the next iteration.

[0012] In some embodiments of the first aspect, the method may further include: in response to the detection of a terminator, determining the model output of the current iteration as the target output, and providing the target output to a user.

[0013] In some embodiments of the first aspect, after providing the target output to the user, the method may further include: calculating the reuse rate of lexical units in the target template sequence for the target template to obtain reuse rate information for the target template; and saving the reuse rate information for the target template. Thus, after each round of decoding iteration, reuse rate information for the template can be obtained by calculating the lexical reuse rate of the template sequence for the template.

[0014] In some embodiments of the first aspect, the method may further include: determining the window size of the segmented window based on reuse rate information for the target template. This provides a dynamic determination strategy for the segmented window size based on template reuse rate.

[0015] In some embodiments of the first aspect, determining the window size of the segmented window may include: determining the segmented window to have a large window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is greater than a reuse rate threshold; and determining the segmented window to have a small window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is less than a reuse rate threshold. Thus, for the selected target template, a large segmented window is set when its historical validation pass rate or acceptance rate is high, to reduce decoding steps. Furthermore, a small segmented window is set when its historical validation pass rate or acceptance rate is low, to reduce validation latency. This enables faster model validation and reduces end-to-end latency in model inference.

[0016] In some embodiments of the first aspect, wherein: the first predetermined portion may be a start portion, the start portion may include the target number of lexical units at the beginning of the corresponding lexical unit segment, and the second predetermined portion may be a tail portion, the tail portion may include the target number of lexical units at the end of the user input.

[0017] In some embodiments of the first aspect, determining the target lexical segment may include: identifying lexical segments in the plurality of lexical segments whose first part is equal to their last part as the hit target lexical segment, and wherein the method may further include: in response to more than one of the plurality of lexical segments being hit, concatenating the user input with each of the more than one lexical segment for inference and verification. Thus, when more than one lexical segment is hit, all of the lexical segments are loaded for verification simultaneously.

[0018] In some embodiments of the first aspect, the user input may be a medical query based on a natural language description by the user, and the type of the medical query may include at least one of the following: image, audio / video, or text.

[0019] In a second aspect, an apparatus for model inference is provided. The apparatus includes: an input receiving module configured to receive user input and retrieve a target template corresponding to the user input from a template library, the target template including a target template sequence, wherein the template library includes multiple template sequences output by a language model in response to historical user input, the multiple template sequences including the target template sequence; a segmentation module configured to segment multiple lexical segments from the target template sequence based on a segmentation window; a lexical comparison module configured to determine a matched target lexical segment by comparing a first predetermined portion of each of the multiple lexical segments with a second predetermined portion of the user input; and an output acquisition module configured to concatenate the user input and the target lexical segments and feed them into the language model for inference to obtain model output. The device can provide high-quality accompanying input for user input based on the historical context of the language model, which can improve the inference accuracy of accompanying input words, i.e., improve the quality of accompanying input words. At the same time, it can also improve the parallel processing and response time of model inference, thereby alleviating the degradation of decoding performance caused by reloading weights by reducing the number of decoding steps in the decoding process.

[0020] In some embodiments of the second aspect, the apparatus may further include a matching verification module configured to: sequentially match the model output with the target word segment word by word; and stop matching between the model output and the target word segment in response to a word in the model output not matching a corresponding word in the target word segment, wherein the word in the model output that has matched the target word segment and the word that has not matched can be retained for the next iteration. Thus, for a decoding iteration, the model output of the current iteration can be verified by comparing the model output of the current iteration with the target word segment of that iteration. A highly reliable input can be provided for the next iteration by selecting a reliable portion of the model output (i.e., including the matched word and the word that has not matched).

[0021] In some embodiments of the second aspect, the template library may include a plurality of templates categorized based on predetermined attributes, and the plurality of templates may include the target template. The input receiving module may be further configured to: determine a first category to which the user input belongs under the predetermined attributes; and among the plurality of templates, determine a template having the first category or a second category similar to the first category as the target template. Thus, in a template library constructed based on the historical output of a language model, appropriate associated templates can be selected for user input to improve the model inference process.

[0022] In some embodiments of the second aspect, each of the plurality of templates may include a plurality of template sequences, and the apparatus may further include a target sequence determination module configured to: determine a template sequence that meets a similarity threshold as the target template sequence based on the similarity between the user-input lexical sequence and the plurality of template sequences in the determined target templates. Thus, among target templates of the same category as the user input, a target template sequence that meets the similarity requirement is selected from the plurality of template sequences based on lexical similarity to improve the accuracy and parallelism of inferred lexical units.

[0023] In some embodiments of the second aspect, each of the plurality of templates may include a template sequence, and the apparatus may further include a target sequence determination module configured to: determine the template sequence included in the determined target templates as the target template sequence, wherein the template sequence included in each template may be an updated template sequence under the corresponding category. Thus, among the target templates of the same category as the user input, the included template sequence is the most usable template sequence under that category. Selecting it as the target template sequence improves the accuracy, parallelism, etc., of inferred lexical units.

[0024] In some embodiments of the second aspect, the apparatus may further include a pre-inference module configured to update the template sequence included in each template by performing pre-inference using user context associated with the corresponding template among the plurality of templates. Thus, by performing offline pre-inference using user-related context, the template sequence in the target template can be updated, thereby expanding and enriching the template library to further improve the accuracy and quality of inferred lexical terms.

[0025] In some embodiments of the second aspect, the target lexical segment can be a first target lexical segment, the model output can be a first model output, and the model output can be determined as an intermediate model output in response to the absence of a terminal symbol. The lexical comparison module can be further configured to: determine a hit second target lexical segment by comparing the first predetermined portion of each of the plurality of lexical segments with the second predetermined portion of the lexical sequence retained in the first model output, wherein the lexical sequence retained in the intermediate model output can be used as the model input for the current iteration; and concatenate the model input with the second target lexical segment and feed it into the language model for inference to obtain a second model output. Thus, in each decoding iteration, a lexical segment corresponding to a reliable lexical sequence in the model output of the previous iteration can be determined as an auxiliary input for the current iteration, thereby improving the decoding process round by round.

[0026] In some embodiments of the second aspect, the matching verification module may be further configured to: match the second model output with the second target word segment sequentially word by word; and stop the matching between the second model output and the second target word segment in response to a word in the second model output not matching a corresponding word in the second target word segment, wherein the word in the second model output that has matched the second target word segment and the word that has not matched can be retained for the next iteration.

[0027] In some embodiments of the second aspect, the apparatus may further include a termination module configured to: determine the model output of the current iteration as the target output in response to the detection of a termination symbol, and provide the target output to the user.

[0028] In some embodiments of the second aspect, after providing the target output to the user, the apparatus may further include a reuse rate statistics module configured to: count the reuse rate of words in the target template sequence for the target template to obtain reuse rate information for the target template; and save the reuse rate information for the target template. Thus, after each round of decoding iteration, reuse rate information for the template can be obtained by counting the word reuse rate of the template sequence for the template.

[0029] In some embodiments of the second aspect, the apparatus may further include a window size determination module configured to determine the window size of the segmented window based on reuse rate information for the target template. This provides a dynamic determination strategy for the segmented window size based on template reuse rate.

[0030] In some embodiments of the second aspect, the window size determination module can be further configured to: determine the segmented window to have a large window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is greater than a reuse rate threshold; and determine the segmented window to have a small window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is less than a reuse rate threshold. Thus, for the selected target template, a large segmented window is set when its historical validation pass rate or acceptance rate is high to reduce decoding steps. Furthermore, a small segmented window is set when its historical validation pass rate or acceptance rate is low to reduce validation latency. This accelerates model validation and reduces end-to-end latency in model inference.

[0031] In some embodiments of the second aspect, wherein: the first predetermined portion may be a start portion, the start portion may include the target number of lexical units at the beginning of the corresponding lexical unit segment, and the second predetermined portion may be a tail portion, the tail portion may include the target number of lexical units at the end of the user input.

[0032] In some embodiments of the second aspect, wherein: the lexical comparison module may be further configured to: determine the lexical segment in the plurality of lexical segments whose first part is equal to the last part as the hit target lexical segment, and wherein the device may further include a sequence concatenation module configured to: in response to more than one of the plurality of lexical segments being hit, concatenate the user input with each of the more than one lexical segment for inference and verification. Thus, when more than one lexical segment is hit, all of the lexical segments are loaded for verification simultaneously.

[0033] In some embodiments of the second aspect, the user input may be a medical query based on a user's natural language description, and the type of the medical query may include at least one of the following: image, audio / video, or text.

[0034] According to a third aspect of this disclosure, a computing device is provided. The computing device includes a processor and a memory storing instructions that, when executed by the processor, cause the computing device to perform the method described according to a first aspect of this disclosure.

[0035] According to a fourth aspect of this disclosure, a computing device cluster is provided. The computing device cluster includes at least one computing device, each computing device including a processor and memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method described in the first aspect of this disclosure. In some embodiments, the computing device cluster includes a single computing device. In other embodiments, the computing device cluster includes multiple computing devices. In some embodiments, the computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0036] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described according to a first aspect of this disclosure.

[0037] According to a sixth aspect of this disclosure, a computer program product is provided. The computer program product includes instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described in accordance with a first aspect of this disclosure. In some embodiments, the computer program product may include one or more software installation packages that can be downloaded or copied and executed on the computing device or cluster of computing devices when the method provided in the first aspect or its possible variations is required.

[0038] Please note that the Summary of the Invention is provided to introduce a series of concepts in a simplified form, which will be further described below in the Detailed Description. The Summary of the Invention is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0039] The above and other objects, features, and advantages of this disclosure will become clearer through a more detailed description of the embodiments thereof in conjunction with the accompanying drawings, in which:

[0040] Figure 1 This is a schematic diagram illustrating an exemplary inference environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0041] Figure 2 This is a schematic illustration of a flowchart of a method for model reasoning according to an embodiment of the present disclosure;

[0042] Figure 3 The illustration shows an example difference between historical inputs and outputs and new inputs and outputs for model inference according to embodiments of the present disclosure;

[0043] Figure 4 The illustration shows a matching comparison process according to an embodiment of the present disclosure;

[0044] Figure 5 The illustration shows a process for segmenting words based on a segmented window according to an embodiment of the present disclosure;

[0045] Figure 6 The illustration shows a diagram of a model reasoning framework according to an embodiment of the present disclosure;

[0046] Figure 7A The illustration shows a diagram of a template sequence arrangement within a template according to an embodiment of the present disclosure;

[0047] Figure 7B The illustration shows another arrangement of template sequences within a template according to an embodiment of the present disclosure;

[0048] Figure 8The illustration shows a schematic diagram of a model reasoning process according to an embodiment of the present disclosure;

[0049] Figure 9 A schematic block diagram of a cache management apparatus according to an embodiment of the present disclosure is shown.

[0050] Figure 10 A schematic block diagram of an example device is shown, which can be used to implement exemplary implementations of this disclosure.

[0051] Figure 11 A schematic block diagram of an example device cluster that can be used to implement exemplary implementations of this disclosure is shown; and

[0052] Figure 12 A schematic block diagram of another example device is shown, which can be used to implement exemplary implementations of this disclosure.

[0053] In all the accompanying drawings, the same or similar reference numerals usually indicate the same or similar elements. Detailed Implementation

[0054] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0055] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0056] As mentioned above, language model-based reasoning refers to the powerful reasoning ability of a language model to complete complex tasks (e.g., answering questions, responding to queries) by leveraging existing knowledge and context, and employing logical deduction, association analysis, and knowledge integration. Language models, especially the Large Language Model (LLM) that has emerged in recent years, have become an influential and widely used technology. One of its core functions is to greatly simplify the previously complex and cumbersome knowledge acquisition process. Users can overcome previously high knowledge barriers to achieve accurate and rapid information retrieval, data question answering, and other functions, allowing ordinary users to easily enjoy the convenience brought by technological advancements.

[0057] A token can be the basic computational unit for reasoning in a large language model; examples can include words, numbers, or symbols. The reasoning process of a large language model includes a prefilling phase and a decoding phase. The reasoning process that generates the first token is called the prefilling phase. After the first token is generated, it is fed back into the large language model as input to predict the next token, and this process is repeated recursively to generate each token in the output sequence until the final word is generated. This continuous process of using previously generated tokens to produce new tokens is the decoding phase.

[0058] There are still many problems and challenges in scheduling the inference process. For example, with the increasingly widespread application of large-scale model inference, the length of the input text is getting longer and longer, and the content of the output is also increasing. The rapid increase in sequence length causes the computational cost of large-scale model inference to rise exponentially. Due to the constraints of the autoregressive decoding mechanism of large-scale model inference, the multiple decoding steps requested in a single inference (also referred to as decoding steps in this paper) cannot be directly parallelized, resulting in a disproportionately high proportion of decoding latency in the overall inference time in long sequence output scenarios. In addition, during the decoding process, each iteration requires reloading the weight data, which greatly degrades decoding performance. If we can overcome the existing technical limitations and generate multiple tokens through lossless decoding in one go, and develop a more efficient decoding algorithm, it can propel large-scale model inference to a new stage of development.

[0059] A general approach is to accelerate the inference process by minimizing the total number of decoding steps by increasing the parallelism of LLM computation in each decoding step, thereby effectively avoiding repeated read-and-write operations on LLM parameters. More specifically, in each decoding step, the system first infers the possible terms that the LLM to be accelerated (also referred to as the target LLM in this paper) might generate in multiple future decoding steps (i.e., batch drafting terms), and then simultaneously verifies these terms using the target LLM. The verified terms serve as the decoding result for the current decoding step. If the inference is accurate enough, speculative decoding can generate multiple terms in parallel within a single decoding step, thus accelerating LLM inference. Moreover, the verification process using the target LLM theoretically guarantees the consistency between the decoding result and the autoregressive decoding result of the target LLM.

[0060] In some related approaches, a smaller LLM (e.g., similar to the target LLM in model structure, word segmentation method, training corpus, and process) but with a much smaller number of parameters is used for inference. This results in inferred lexical units that show some similarity to those decoded autoregressively by the target LLM model, thus improving the lexical unit hit rate. In other related approaches, natural language processing methods can be used to train an N-gram model based on existing training corpus for inference and lexical unit generation.

[0061] However, this requires additional training or fine-tuning of a reliable, approximate small LLM or n-gram model based on business data to output usable inferred terms, and it is difficult to guarantee the quality of the generated terms. The quality of term generation is not constant across multiple iterations and fluctuates. Furthermore, the current validation strategy uses a fixed term validation window, which easily leads to high model validation time and low returns when the window is large and the term adoption rate is low. Moreover, it fails to address the issue of reducing decoding steps when the window is small and the term reuse rate is high.

[0062] To address at least some of the aforementioned problems and other potential issues, a scheme for model inference according to embodiments of this disclosure is proposed. The scheme includes receiving user input and retrieving a target template corresponding to the user input from a template library, the target template comprising a sequence of target templates, wherein the template library includes multiple template sequences output by a language model in response to historical user input, the multiple template sequences including the target template sequence. The scheme further includes segmenting the target template sequence into multiple lexical segments based on a segmentation window. The scheme further includes determining the matched target lexical segment by comparing a first predetermined portion of each of the multiple lexical segments with a second predetermined portion of the user input. The scheme further includes concatenating the user input with the target lexical segments and feeding them into a language model for inference to obtain model output.

[0063] The scheme for model inference according to embodiments of the present disclosure can provide high-quality accompanying input for user input based on the historical context of the language model, improve the inference accuracy of accompanying input words, that is, improve the quality of accompanying input words, and at the same time improve the parallel processing and response time of model inference, thereby alleviating the degradation of decoding performance caused by reloading weights by reducing the number of decoding steps in the decoding process.

[0064] The following is for reference. Figures 1 to 12 The present disclosure is provided to illustrate its basic principles and several exemplary implementations. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the disclosure in any way.

[0065] Figure 1 This is a schematic diagram illustrating an exemplary environment 100 for model inference in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown, the exemplary environment 100 for model inference includes an inference system 120 and a user interacting with it (e.g., Figure 1 User 110 is shown in the image. Figure 1As shown, the inference system 120, also known as the inference resource set, schematically includes inference resources 121, 122, and 123. These inference resources are deployed in the cloud to be dynamically accessed or allocated in response to requests from user 110. Figure 1 The implementation environment of embodiments of this disclosure is illustrated using a distributed inference system (i.e., the corresponding physical resources for model inference may not be geographically located, but are linked together through cloud computing or distributed technologies) as an example. It should be understood that this is merely exemplary and not a limiting description; other different types of inference systems are also feasible, such as centralized inference systems (i.e., the corresponding physical resources for model inference are uniformly deployed locally). A suitable inference environment configuration can be selected according to actual usage needs.

[0066] like Figure 1 As exemplarily illustrated, user 110 can submit a user request (e.g., a query, a question, etc.) to the inference system, and such user request can be based on a natural language description. Examples of inference system 120 may include a cluster of computing devices or a cluster of servers, and inference resources may correspond to computing devices or servers within the cluster. In this example, a language model used for inference (e.g., a large language model (LLM)) can be deployed on at least one computing device or server to handle the user request. Appropriate inference resources in inference system 120 can be assigned to respond to the user request, such as retrieving data corresponding to the query or generating an answer corresponding to the question. It should be understood that this is for illustrative and diagrammatic purposes only. Figure 1 Only one user (i.e., user 110) is shown in the example. However, this is only an example, and more than one user can interact with the inference system 120 concurrently, enabling the inference system 120 to respond to multiple users at the same time, thereby meeting the diverse needs of different users in various scenarios.

[0067] According to embodiments of this disclosure, the inference system 120 can refer to a cloud-based computing and service model that couples inference resources such as computing resources and memory access resources together through networking and other coupling methods to provide inference services to users. The inference system 120 is not merely a simple collection of resources, but a highly integrated, intelligently scheduled, and flexibly scalable inference environment. In this environment, key elements such as various resources and applications are tightly coupled, allowing users to dynamically acquire, configure, and use these resources as needed without having to own or manage them locally.

[0068] Within the framework of Inference System 120, high-performance computing systems such as supercomputers and large-scale computing clusters are not only deployed efficiently, but also achieve a high level of flexibility and scalability. These high-performance computing resources, which were originally limited by high deployment costs and complex operation and maintenance management, are now seamlessly integrated into the cloud through cloud computing technology and elastic scheduling technology, forming a powerful and flexible inference ecosystem.

[0069] exist Figure 1 In the example, the inference resources 121, 122, and 123 included in the inference system 120 can be coupled to each other via networks, etc. Taking a server cluster as an example, the servers can interconnect to communicate, for example, to achieve data synchronization or sharing between servers. In this configuration scenario, the servers can process tasks in parallel through distributed computing. This parallel computing capability enables the cloud computing platform to handle more complex and massive tasks. Furthermore, each server in the cluster can have redundancy and fault tolerance mechanisms to ensure reliable task execution. In some embodiments, the servers in the cluster can be configured with storage resources (e.g., solid-state drives (SSDs), hard disk drives (HDDs), or a combination of both) for storing data metadata, backups, etc. When a node fails, the system can automatically migrate the job to other normally functioning nodes, ensuring task continuity and availability.

[0070] In the example configuration of the inference system 120 as a server cluster, examples of inference resources or resource nodes may include a central server, an edge server, or a local server in a local data center. Besides the servers mentioned above, other computing devices can serve as resource nodes in the inference system 120, such as supercomputers, personal computers, laptops, in-vehicle computing devices, mobile devices (such as smartphones, tablets, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), or combinations of any one or more of the above devices. It should be understood that the resource nodes described herein are merely exemplary and not limiting; for example, other different types of resource nodes may exist as technology evolves.

[0071] It should be understood that, for the purpose of ease of explanation and illustration, in Figure 1 Only one inference system (i.e., inference system 120) and three inference resources (i.e., inference resources 121, 122, and 123) are shown in the diagram. However, this is merely an example, and other inference systems and resources with different configurations, numbers, etc., can exist. They can be customized according to specific use cases and needs, as will be discussed below. The above, combined with... Figure 1An exemplary environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented is described below, in conjunction with... Figure 2 To describe a method 200 for model reasoning according to embodiments of the present disclosure.

[0072] Figure 2 This is a schematic illustration of a flowchart of a method 200 for model inference according to an embodiment of the present disclosure. According to the combination... Figure 2 The described method 200 provides an accompanying input strategy for model inference that improves the accuracy of inferred terms and also improves the parallelism and speed of model inference, thereby mitigating the impact of reloading weights on decoding performance by reducing decoding steps.

[0073] At position 210, user input is received, and a target template corresponding to the user input is retrieved from the template library. This target template includes a sequence of target templates, where the template library comprises multiple template sequences output by the language model in response to historical user input, each containing a target template sequence. An accompanying input strategy for model inference is provided. Based on this strategy, the language model can compute multiple words in parallel per iteration. Compared to generating a single lexical unit per iteration, the additional computational time introduced by parallel computation is less than, and in some cases negligible, the time required to reload weights per iteration.

[0074] According to embodiments of this disclosure, a template library for assisting model inference can be constructed based on the historical context of a language model. This template library can include multiple templates, each containing one or more corresponding template (lexical) sequences. These lexical sequences in the template library are the actual outputs (raw or adjusted) of the language model in past inferences. They possess high credibility and stability under testing in model training and practical applications, and can be considered reliable. Retrieving lexical sequences generated by inference from associated historical requests for new requests can improve the similarity of inferred lexical sequences. The more accurate the input and the higher the reuse rate, the better the acceleration effect. This improves the performance of inference decoding. Thus, it is possible to quickly and accurately infer the generation results of multiple future decoding steps of the language model while reducing the number of decoding steps.

[0075] At position 220, multiple lexical segments are divided from the target template sequence based on a segmentation window. According to embodiments of this disclosure, for each lexical in the target template sequence, several lexical units are sequentially shifted forward from the corresponding lexical unit (using the corresponding lexical unit as the starting point). The resulting sequence of lexical units (formed by the corresponding lexical unit and the subsequent several lexical units) is called a lexical segment. The multiple lexical segments can have the same size, and the size of the lexical segment can be determined based on the segmentation window size. The selection of the segmentation window and the division of lexical segments will be further described in detail below.

[0076] At point 230, the target word segment is determined by comparing a first predetermined portion of each of the multiple word segments with a second predetermined portion of the user input. After obtaining a certain number of word segments for the target template sequence, a target word segment needs to be selected from them for the user input. According to embodiments of this disclosure, a predetermined portion of each of these word segments is compared with another predetermined portion of the user input. The predetermined portion and the other predetermined portion may have the same number of words. If the predetermined portion of the corresponding word segment is exactly the same as the other predetermined portion of the user input or meets the similarity requirement, then the corresponding word segment is determined as the expected target word segment. The determination of the target word segment will be further described in detail below.

[0077] At position 240, the user input and target word segments are concatenated and fed into the language model for inference to obtain the model output. The user input is concatenated with the determined target word segments, and the resulting input is fed into the language model. Because the inferred target word segments correspond to the actual historical model responses, they possess high inference usability and high similarity to the user input after matching comparison. This allows speculative decoding to generate multiple words in parallel within a single decoding step.

[0078] The method 200 for model inference according to embodiments of this disclosure solves the degradation defect in decoding performance caused by repeatedly loading weight data in multiple decoding iterations. It can provide high-quality accompanying input for user input based on the historical context of the language model, improving the inference accuracy of accompanying input lexical units, i.e., improving the quality of accompanying input lexical units. Simultaneously, it can improve the parallel processing and response time of model inference, thereby mitigating the degradation of decoding performance caused by reloading weights by reducing the number of decoding steps in the decoding process.

[0079] Figure 3 The illustration shows an example difference 300 between historical input / output and new input / output in model inference according to an embodiment of the present disclosure. When the contextual information changes little between the two inferences, the lexical sequence generated based on historical request inference is highly similar to the lexical sequence generated based on new request inference. Figure 3As shown, historical requests may include prompts 311 and historical context information (hereinafter also referred to as historical information) 312, while new requests may include new context information (hereinafter also referred to as new information) 313 in addition to prompts 311 and historical context information 312. New context information 313 typically constitutes a small percentage of the total input. In other words, historical requests and new requests share a certain degree of similarity. Furthermore, the historical question-and-answer results 321 for historical requests and the new question-and-answer results 322 for new requests also share a certain degree of similarity. According to embodiments of this disclosure, the lexical sequences generated by common request reasoning can be cached for later use. How to fully utilize the historical output of the language model is a core focus explored and addressed by embodiments of this disclosure.

[0080] Figure 4 The illustration shows a matching comparison process 400 according to an embodiment of the present disclosure. As described above, multiple word segments are divided from the target template sequence, and the matching word segments are determined as inference or auxiliary input. After the user input is concatenated with the target word segments and fed into the language model for inference, the obtained model output needs to be matched and verified.

[0081] like Figure 4 As shown, user input 401 can be Xiaoming's medical record, which can be a long sequence about Xiaoming's condition. Target lexical segment 410 can be "Xiaoming is sick," which is selected from multiple lexical segments targeting the corresponding target template sequence. Specifically, a predetermined portion (also referred to as the first predetermined portion) of each of the multiple lexical segments is compared with another predetermined portion (also referred to as the second predetermined portion) of user input 401. If the first predetermined portion of the corresponding lexical segment is exactly the same as the second predetermined portion of user input 401 or meets the similarity requirement, then the corresponding lexical segment is determined as the hit target lexical segment 410. In some embodiments, the first portion can be the beginning portion of each lexical segment, which includes the target number of lexical units at the beginning of the corresponding lexical segment. Furthermore, the second portion can be the end portion of user input 401, which includes the target number of lexical units at the end of user input 401. In other words, the beginning portion and the end portion have the same number of lexical units.

[0082] According to the embodiments of the present disclosure, a token segment whose head part is equal to the tail part of the user input 401 among the multiple token segments corresponding to the target template sequence may be determined as the hit target token segment 410. By way of example and not limitation, assuming that the target number is 2, and the 2 tokens at the tail part of the user input 401 (i.e., Xiaoming's case) are "Xiaoming", if the 2 tokens at the head part of one token segment among the multiple token segments are "Xiaoming", then this token segment (i.e., Xiaoming...) is determined as the hit target token segment 410. In some embodiments, in response to more than one token segment among the multiple token segments being hit (i.e., there being more than one target token segment 410), the user input 401 is spliced with each of the more than one token segment respectively for inference and verification.

[0083] According to the embodiments of the present disclosure, the model output 420 and the target token segment 410 may be matched token by token in sequence. With continued reference to Figure 4 , after the user input 401 and the target token segment 410 are spliced and fed into the language model for inference, the model output 420 may be obtained. In the sequential token-by-token matching between the model output 420 and the target token segment 410, "Xiao" corresponding to the arrow 411 is compared with "Xiao" corresponding to the arrow 421, and if they are equal, the verification is considered to be passed. In the case where the verification of "Xiao" corresponding to the arrow 421 is passed, "Ming" corresponding to the arrow 412 is continuously compared with "Ming" corresponding to the arrow 422, and if they are equal, the verification is considered to be passed.

[0084] In addition, in response to a token in the model output 420 not matching the corresponding token in the target token segment 410, the matching between the model output 420 and the target token segment 410 may be stopped, wherein the tokens in the model output 420 that have been matched with the target token segment 410 and the one unmatched token may be retained for the next iteration. For example, assume that the token corresponding to the arrow 413 is "shen" (not shown). When "shen" corresponding to the arrow 413 is not equal to "sheng" corresponding to the arrow 423, the verification is considered to be failed. In the model output 420, the matched tokens (for example, "Xiao" and "Ming") and the one unmatched token (i.e., "sheng") may be retained, and this token sequence "Xiao Ming sheng" will be used as the model input for the next iteration. For the purpose of clear reference, the target token segment 410 is referred to as a first target token segment, and the model output 420 is referred to as a first model output. In the following, the iterative nature of the decoding process will be described by way of example.

[0085] According to an embodiment of the present disclosure, when an end token is not detected, the first model output may be identified as an intermediate model output. A hit second target token segment may be determined by comparing a first predetermined portion (e.g., the leading portion) of each of the multiple token segments for the corresponding target template sequence with a second predetermined portion (e.g., the trailing portion) of the preserved token sequence in the first model output (e.g., "Xiao Mingsheng" in the above example). The preserved token sequence in the model output of the previous iteration may be used as the model input of the current iteration. After the second target token segment is determined, the model input may be concatenated with the second target token segment and fed into the language model for inference to obtain a second model output.

[0086] Similarly, the second model output and the second target token segment may be matched token by token in sequence. In this matching process, in response to a token in the second model output not matching the corresponding token in the second target token segment, the matching between the second model output and the second target token segment is stopped. The tokens in the second model output that have been matched with the second target token segment and the one unmatched token are preserved for the next iteration.

[0087] Figure 5 The figure illustrates a diagram of a segment window-based token segment partitioning process 500 according to an embodiment of the present disclosure. Figure 5 The left side of the figure shows an example of a static window size. In this example, it is assumed that the fixed window size is 4, that is, the segment window 502 with a window size of "4". For the target template sequence (such as Figure 5 as shown in, S1-S8), for each token, for example, starting from token S1, 4 tokens equal to the window size (i.e., S1-S4) are sequentially determined, then starting from token S2, 4 tokens equal to the window size (i.e., S2-S5) are sequentially determined, and so on, to finally form a plurality of token segments 510 for the target template sequence.

[0088] Figure 5 The right side of the figure shows an example of a dynamic window size. As described above, when an end token is not detected, the preserved token sequence in the model output of the current iteration may be used as the model input of the next iteration. According to an embodiment of the present disclosure, when an end token has been detected, the model output of the current iteration may be determined as a target output, and the target output is provided to the user. After the inference is completed, the reuse rate of the template sequence for the template selected in this inference may be counted. That is, the reuse rate of the target template sequence is counted for the target template to obtain reuse rate information for the target template. Such reuse rate information may be stored for determining the window size of the segment window in subsequent inferences.

[0089] According to embodiments of this disclosure, the window size of the segmented window can be determined based on reuse rate information for the target template. In response to the reuse rate information indicating that the reuse rate of words in the target template sequence is greater than a reuse rate threshold, the segmented window can be determined to have a larger window size. Conversely, in response to the reuse rate information indicating that the reuse rate of words in the target template sequence is less than a reuse rate threshold, the segmented window can be determined to have a smaller window size. Thus, when the historical verification pass rate or acceptance rate is high, a large segmented window is selected to reduce the number of decoding steps. Conversely, when the historical verification pass rate or acceptance rate is low, a small segmented window is set to reduce verification latency.

[0090] exist Figure 5 In this context, shaded morphemes can indicate that, as indicated by historical reuse rate information, these morphemes have not been reused in the target template sequence. For example... Figure 5 As shown in the upper right corner, when the word reuse rate in the target template sequence is not high, a smaller window size (e.g., window size 2) can be selected for the segmentation window 504. Furthermore, in Figure 5 In the lower right corner, when the word reuse rate in the target template sequence is high, a larger window size (e.g., window size 3) can be selected for the segment window 506. In this way, while ensuring quality, as many speculative words as possible can pass the verification, thereby improving decoding parallelism.

[0091] The dynamic segmented window size strategy disclosed herein adaptively adjusts the segmented window size during each inference iteration. By adaptively adjusting the segmented window size based on historical adoption rate fluctuations, the lexical sequence can be divided into appropriately sized lexical segments for verification. This achieves the goal of expanding the verification window and reducing decoding steps when the adoption rate is high, and shrinking the verification window and reducing verification latency when the adoption rate is low. Thus, model verification can be accelerated, and end-to-end latency for large model inference can be reduced.

[0092] Figure 6 A diagram illustrating a model inference framework 600 according to an embodiment of the present disclosure is shown. Figure 6 As shown, a template library 610 can be constructed based on the historical context 605 of the language model. The historical context 605 can be the user's past real interactions with the language model. By systematically integrating these usable interaction information systems, various templates tailored to different needs can be injected into the template library 610, improving its comprehensiveness and usability.

[0093] According to embodiments of this disclosure, template library 610 may include multiple templates categorized based on predetermined attributes. For example, templates in the template library may be categorized based on audience (e.g., minors, adults, the elderly, etc.), or based on field (e.g., medicine, education, etc.), and so on. After receiving input 601, a first category to which input 601 belongs under the predetermined attributes can be determined. Then, among the multiple templates included in template library 610, templates having the first category or a second category similar to the first category can be identified as target templates (e.g., ...). Figure 6 Template 611 shown.

[0094] After obtaining the target template 611, multiple lexical segments can be divided from the target template sequence it includes, and the matched target segments can be selected from these segments. Input 601 can be combined with the matched target segments and fed into the large model 620 for inference and verification 625 to obtain output 631. Inference and verification 625 can have iterative characteristics; if no end-of-phase character is detected, the input 601 for the next iteration can be obtained based on the output 631 of the current iteration to continue the execution of inference and verification 625. When an end-of-phase character is detected, the output 631 of the current iteration can be provided to the user as the final output. Since reusable lexical drafts are provided in the historical lexical sequences, the decoding steps of the large model inference are reduced, and the model's response time is further reduced.

[0095] Figure 7A The illustration shows a template sequence arrangement 700A within a template according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, each of a plurality of templates in a template library may include a plurality of template sequences. For example... Figure 7A As shown, the selected target template 711 may include multiple template sequences (e.g., template sequence 1, template sequence 2, template sequence 3, etc. shown in Figure 7). In determining the target template sequence, based on the similarity between the input word sequence and multiple template sequences in the determined target template 711, template sequences that meet the similarity threshold are determined as target template sequences. For example, if the input word sequence has the highest similarity to template sequence 2 in the target template 711, then template sequence 2 can be determined as the target template sequence.

[0096] Figure 7BThe illustration shows another template sequence arrangement 700B within a template according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, each template in a plurality of templates in a template library may include a template sequence. In other words, each template may correspond to a template sequence. If a corresponding template is determined to be a target template, then the template sequence included in that corresponding template can be directly determined as the target template sequence. In the case of template sequence arrangement 700B, the template sequence included in each template may be an updated template sequence 721 under the corresponding category. In some embodiments, the template sequence included in each template may be the latest (i.e., a token sequence generated based on the latest historical request reasoning), and the updated template sequence 721 will replace the historically cached template sequence.

[0097] Figure 8 The diagram illustrates a model inference process 800 according to an embodiment of the present disclosure. At block 801, idle load can be determined. Based on system load, idle cluster computing power can be used in a time-sharing manner to enable offline pre-inference. At block 802, the offline pre-inference can be performed based on user context for template updates. According to embodiments of the present disclosure, by utilizing user context associated with a corresponding template among multiple templates for pre-inference, the template sequence included in each template can be updated. For example, such user context may include updated personal information of the user. Thus, the corresponding template in the template library can receive content updates. At block 803, the template updated based on pre-inference can be cached in the template library.

[0098] At box 804, after receiving a user request, the online inference system can classify and match templates in the template library based on the question-and-answer information of the current request, retrieve templates for similar historical requests, and then load the corresponding inferred lexical sequence. At box 805, the window size of the segmented window can be dynamically adjusted based on historical reuse rate information, with an initial value of 5. Depending on the reuse rate, it can be increased when it exceeds a threshold and decreased when it is below a threshold. At box 806, the inferred lexical sequence can be divided into multiple lexical segments of equal length according to the window size to form an n-gram lexicon, which is then cached. A header lexical index can be built based on the lexical segments, thereby accelerating matching performance.

[0099] At box 807, query matching for lexical segments can be performed. In one implementation, the first (or more) lexical segments can be matched against the last (or more) lexical segments of the valid lexical sequence output by the model in the previous iteration (the lexical sequence requested by the user in the first iteration). If a match is found, the segments are loaded and used. If multiple lexical segments match simultaneously, they are loaded and validated at the same time.

[0100] At box 808, large model inference verification can be performed. If the result at box 809 is "No", the inference is incomplete, and boxes 7-9 are repeated. If the result at box 809 is "Yes", proceed to box 810, where the final result can be output, along with statistical data, such as reuse rate information. At box 811, the statistical data can be cached (e.g., cached in a template library) for later reference.

[0101] Figure 9 An apparatus 900 for model reasoning according to some embodiments of the present disclosure is shown. Each module in the apparatus 900 can be configured to perform corresponding steps in a method for model reasoning according to embodiments of the present disclosure. Figure 9 As shown, the apparatus 900 for model inference includes an input receiving module 910, a segmentation module 920, a lexical comparison module 930, and an output acquisition module 940. According to embodiments of this disclosure, the apparatus 900 includes an input receiving module 910 configured to receive user input and retrieve a target template corresponding to the user input from a template library, the target template including a target template sequence, wherein the template library includes multiple template sequences output by a language model in response to historical user input, the multiple template sequences including the target template sequence. The apparatus 900 also includes a segmentation module 920 configured to segment multiple lexical segments from the target template sequence based on a segmentation window. The apparatus 900 further includes a lexical comparison module 930 configured to determine a hit target lexical segment by comparing a first predetermined portion of each of the multiple lexical segments with a second predetermined portion of the user input. The apparatus 900 also includes an output acquisition module 940 configured to concatenate the user input with the target lexical segment and feed it into the language model for inference to obtain model output. In this way, high-quality accompanying input can be provided for user input based on the historical context of the language model, which can improve the inference accuracy of accompanying input words, that is, improve the quality of accompanying input words. At the same time, it can also improve the parallel processing and response time of model inference, thereby alleviating the degradation of decoding performance caused by reloading weights by reducing the number of decoding steps in the decoding process.

[0102] In some embodiments, the apparatus 900 may further include a matching verification module configured to: sequentially match the model output with the target word segment word by word; and stop matching between the model output and the target word segment in response to a word in the model output not matching a corresponding word in the target word segment, wherein the word in the model output that has matched the target word segment and the word that has not matched can be retained for the next iteration. Thus, for a decoding iteration, the model output of the current iteration can be verified by comparing the model output of the current iteration with the target word segment of that iteration. A highly reliable input can be provided for the next iteration by selecting a reliable portion of the model output (i.e., including the matched word and the word that has not matched).

[0103] In some embodiments, the template library may include a plurality of templates categorized based on predetermined attributes, and the plurality of templates may include the target template. The input receiving module 910 may be further configured to: determine a first category to which the user input belongs under the predetermined attributes; and among the plurality of templates, determine a template having the first category or a second category similar to the first category as the target template. Thus, in a template library constructed based on the historical output of a language model, appropriate associated templates can be selected for user input to improve the model inference process.

[0104] In some embodiments, each of the plurality of templates may include a plurality of template sequences, and the apparatus 900 may further include a target sequence determination module configured to: determine a template sequence that meets a similarity threshold as the target template sequence based on the similarity between the user-input lexical sequence and the plurality of template sequences in the determined target templates. Thus, among target templates of the same category as the user input, a target template sequence that meets the similarity requirement is selected from the plurality of template sequences based on lexical similarity to facilitate word inference.

[0105] In some embodiments, each of the plurality of templates may include a template sequence, and the apparatus 900 may further include a target sequence determination module configured to determine the template sequence included in the determined target templates as the target template sequence, wherein the template sequence included in each template may be an updated template sequence under the corresponding category. Thus, among the target templates of the same category as the user input, the included template sequence is the most usable template sequence under that category. Selecting it as the target template sequence improves the accuracy and parallelism of inferred lexical units.

[0106] In some embodiments, the apparatus 900 may further include a pre-inference module configured to update the template sequence included in each template by performing pre-inference using user context associated with the corresponding template among the plurality of templates. Thus, by performing offline pre-inference using user-related context, the template sequence in the target template can be updated, thereby expanding and enriching the template library to further improve the accuracy and quality of inferred lexical terms.

[0107] In some embodiments, the target lexical segment can be a first target lexical segment, the model output can be a first model output, and the model output can be determined as an intermediate model output in response to the absence of a terminal symbol. The lexical comparison module 930 can be further configured to: determine a hit second target lexical segment by comparing the first predetermined portion of each of the plurality of lexical segments with the second predetermined portion of the lexical sequence retained in the first model output, wherein the lexical sequence retained in the intermediate model output can be used as the model input for the current iteration; and concatenate the model input with the second target lexical segment and feed it into the language model for inference to obtain a second model output. Thus, in each decoding iteration, a lexical segment corresponding to a reliable lexical sequence in the model output of the previous iteration can be determined as an auxiliary input for the current iteration, thereby improving the decoding process round by round.

[0108] In some embodiments, the matching verification module may be further configured to: match the second model output with the second target word segment sequentially word by word; and stop the matching between the second model output and the second target word segment in response to a word in the second model output not matching a corresponding word in the second target word segment, wherein the word in the second model output that has matched the second target word segment and the word that has not matched can be retained for the next iteration.

[0109] In some embodiments, the apparatus 900 may further include a termination module configured to: determine the model output of the current iteration as the target output in response to the detection of a termination symbol, and provide the target output to the user.

[0110] In some embodiments, after providing the target output to the user, the device 900 may further include a reuse rate statistics module, configured to: count the reuse rate of words in the target template sequence for the target template to obtain reuse rate information for the target template; and save the reuse rate information for the target template. Thus, after each round of decoding iteration, reuse rate information for the template can be obtained by counting the word reuse rate of the template sequence for the template.

[0111] In some embodiments, the apparatus 900 may further include a window size determination module configured to determine the window size of the segmented window based on reuse rate information for the target template. This provides a dynamic determination strategy for the segmented window size based on template reuse rate.

[0112] In some embodiments, the window size determination module can be further configured to: determine the segmented window to have a large window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is greater than a reuse rate threshold; and determine the segmented window to have a small window size in response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is less than a reuse rate threshold. Thus, for the selected target template, a large segmented window is set when its historical validation pass rate or acceptance rate is high, to reduce decoding steps. Furthermore, a small segmented window is set when its historical validation pass rate or acceptance rate is low, to reduce validation latency. This accelerates model validation and reduces end-to-end latency in model inference.

[0113] In some embodiments, the first predetermined portion may be a start portion, which may include the target number of lexical units at the beginning of the corresponding lexical unit segment, and the second predetermined portion may be a tail portion, which may include the target number of lexical units at the end of the user input.

[0114] In some embodiments, the lexical comparison module 930 may be further configured to determine the lexical segment in the plurality of lexical segments whose first part is equal to its last part as the hit target lexical segment, and the device 900 may further include a sequence concatenation module configured to, in response to more than one of the plurality of lexical segments being hit, concatenate the user input with each of the more than one lexical segment for inference and verification. Thus, when more than one lexical segment is hit, all of the lexical segments are loaded for verification simultaneously.

[0115] In some embodiments, the user input may be a medical query based on a natural language description, and the type of the medical query may include at least one of the following: image, audio / video, or text.

[0116] As an example of a software functional unit, the input receiving module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the input receiving module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0117] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0118] As an example of a hardware functional unit, the input receiving module 910 may include at least one computing device, such as a server. Alternatively, the input receiving module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0119] The input receiving module 910 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the input receiving module 910 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the input receiving module 910 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0120] It should be noted that, in other embodiments, the input receiving module 910 can be used to perform any step in the method for model inference according to embodiments of the present disclosure, the segmentation module 920 can be used to perform any step in the method for model inference according to embodiments of the present disclosure, the lexical comparison module 930 can be used to perform any step in the method for model inference according to embodiments of the present disclosure, and the output acquisition module 940 can be used to perform any step in the method for model inference according to embodiments of the present disclosure. Furthermore, the aforementioned matching verification module, target sequence determination module, pre-inference module, input receiving module, matching verification module, termination module, reuse rate statistics module, window size determination module, and sequence splicing module can also be used to perform any step in the method for model inference according to embodiments of the present disclosure.

[0121] The steps implemented by the input receiving module 910, segmentation module 920, word comparison module 930, output acquisition module 940, matching verification module, target sequence determination module, pre-inference module, input receiving module, matching verification module, end module, reuse rate statistics module, window size determination module, and sequence splicing module can be specified as needed. The matching verification module, target sequence determination module, pre-inference module, input receiving module, matching verification module, end module, reuse rate statistics module, window size determination module, and sequence splicing module respectively implement different steps in the method for model inference according to the embodiments of this disclosure to realize all the functions of the apparatus 900 for model inference.

[0122] This disclosure also provides a computing device 1000. Figure 10 A schematic block diagram of an example computing device 1000, which can be used to implement exemplary implementations of the present disclosure, is illustrated. The computing device 1000 can be used to perform methods for model inference according to embodiments of the present disclosure. Figure 10As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 1000.

[0123] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0124] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0125] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0126] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned input receiving module 910, segmentation module 920, word comparison module 930, output acquisition module 940, etc., thereby realizing the method for model reasoning according to the embodiments of this disclosure. That is, the memory 1006 stores instructions for executing the method for model reasoning according to the embodiments of this disclosure.

[0127] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the aforementioned apparatus 1100 for model inference, thereby implementing the method for model inference according to embodiments of the present disclosure. That is, the memory 1006 stores instructions for executing the method for model inference according to embodiments of the present disclosure.

[0128] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0129] This disclosure also provides a computing device cluster. Figure 11 A schematic block diagram of an example computing device cluster 1100 that can be used to implement exemplary implementations of the present disclosure is shown. The computing device cluster 1100 enables the model inference methods provided in this disclosure to be executed in a distributed manner by multiple servers within the cluster. The computing device cluster 1100 includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0130] like Figure 11 As shown, the computing device cluster 1100 includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may store the same instructions for performing a method for model inference according to embodiments of this disclosure.

[0131] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store partial instructions for executing the method for model inference according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the method for model inference according to embodiments of the present disclosure.

[0132] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster 1100 can store different instructions, each used to execute a portion of the functions of the device 1100 for model inference. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules, such as the input receiving module 910, the segmentation module 920, the word comparison module 930, and the output acquisition module 940.

[0133] In some possible implementations, one or more computing devices in the computing device cluster 1100 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for performing the functions of the input receiving module 910 and the segmentation module 920. Simultaneously, the memory 1006 in computing device 1000B stores instructions for performing the functions of the word comparison module 930 and the output acquisition module 940.

[0134] Figure 12 The connection method between the computing device cluster 1100 shown may be that, considering that the model inference method provided in this disclosure may be for distributed storage devices, the functions implemented by the lexical comparison module 930 and the output acquisition module 940 are to be performed by the computing device 1000B.

[0135] It should be understood that, Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0136] This disclosure also provides another computing device cluster 1100. The connection relationships between the computing devices in this computing device cluster 1100 can be similarly referred to... Figure 12 and Figure 12 The computing device cluster 1100 is connected in a manner that differs from the others. The memory 1006 of one or more computing devices 1000 within the computing device cluster 1100 may store the same instructions for executing the method for model inference according to embodiments of this disclosure.

[0137] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store partial instructions for executing the method for model inference according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the method for model inference according to embodiments of the present disclosure.

[0138] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a method for model inference according to embodiments of this disclosure.

[0139] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a method for model inference according to embodiments of this disclosure.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for model inference, comprising: The system receives user input and retrieves a target template corresponding to the user input from a template library, the target template including a sequence of target templates, wherein the template library includes multiple template sequences output by the language model in response to historical user input, the multiple template sequences including the target template sequence; Based on the segmentation window, the target template sequence is divided into multiple word segments; The target word segment is determined by comparing a first predetermined portion of each of the plurality of word segments with a second predetermined portion of the user input. as well as The user input and the target word are segmented and concatenated, and then fed into the language model for inference to obtain the model output.

2. The method according to claim 1, further comprising: The model output is matched sequentially with the target word segment by word. as well as In response to a mismatch between a word in the model output and a corresponding word in the target word segment, the matching between the model output and the target word segment is stopped. The word segments that have been matched with the target word segment and the word segments that have not been matched in the model output are retained for the next iteration.

3. The method of claim 1, wherein the template library comprises a plurality of templates categorized based on predetermined attributes, and the plurality of templates includes the target template, and wherein retrieving the target template from the template library further comprises: Determine the first category to which the user input belongs under the predetermined attribute; as well as Among the multiple templates, the template that has the first category or a second category similar to the first category is determined as the target template.

4. The method of claim 3, wherein each of the plurality of templates comprises a plurality of template sequences, and the method further comprises: Based on the similarity between the user-input lexical sequence and multiple template sequences in the determined target template, the template sequence that meets the similarity threshold is determined as the target template sequence.

5. The method of claim 3, wherein each of the plurality of templates comprises a template sequence, and the method further comprises: The template sequence included in the determined target template is defined as the target template sequence. Each template includes a sequence of templates that is an updated sequence of templates for the corresponding category.

6. The method according to claim 5, further comprising: The sequence of templates included in each template is updated by pre-inference using the user context associated with the corresponding template among the plurality of templates.

7. The method according to claim 2, wherein the target word segmentation is a first target word segmentation, the model output is a first model output, and the model output is determined as an intermediate model output in response to the absence of a terminal character, and wherein the method further comprises: The second target word segment is determined by comparing the first predetermined portion of each word segment in the plurality of word segments with the second predetermined portion of the word sequence retained in the first model output, wherein the word sequence retained in the first model output is used as the model input for the current iteration. as well as The model input is concatenated with the second target word segment and fed into the language model for inference to obtain the second model output.

8. The method according to claim 7, further comprising: The output of the second model is matched sequentially with the second target word segment by word. as well as In response to a mismatch between a word in the output of the second model and a corresponding word in the second target word segment, the matching between the output of the second model and the second target word segment is stopped. In the second model output, the segments that have been matched with the second target lexical segment, as well as the unmatched lexical segment, are retained for use in the next iteration.

9. The method according to claim 1, further comprising: In response to the detection of the end-of-phase marker, the model output of the current iteration is determined as the target output, and the target output is provided to the user.

10. The method of claim 9, wherein after providing the target output to the user, the method further comprises: The reuse rate of words in the target template sequence is calculated for the target template to obtain reuse rate information for the target template; as well as Save the reuse rate information for the target template.

11. The method according to claim 1, further comprising: The window size of the segmented window is determined based on the reuse rate information of the target template.

12. The method of claim 11, wherein determining the segment size of the segmented window comprises: In response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is greater than the reuse rate threshold, the segmentation window is determined to have a large window size; as well as In response to the reuse rate information for the target template indicating that the reuse rate of the lexical units in the target template sequence is less than the reuse rate threshold, the segmentation window is determined to have a small window size.

13. The method according to claim 1, wherein: The first predetermined portion is the beginning portion, which includes the target number of lexical units at the beginning of the corresponding lexical unit segment, and The second predetermined portion is the tail portion, which includes the target number of words at the end of the user input.

14. The method of claim 13, wherein determining the target word segmentation comprises: The segment whose first part is equal to the last part among the plurality of lexical segments is determined as the target lexical segment, and The method further includes: In response to a hit in more than one of the plurality of lexical segments, the user input is concatenated with each of the more than one lexical segments for inference and verification.

15. The method according to claim 1, wherein: The user input is a medical query based on natural language description, and The types of medical queries include at least one of the following: images, audio / video, or text.

16. An apparatus for model reasoning, comprising: An input receiving module is configured to receive user input and retrieve a target template corresponding to the user input from a template library, the target template including a target template sequence, wherein the template library includes multiple template sequences output by a language model in response to historical user input, the multiple template sequences including the target template sequence; The segmentation module is configured to divide the target template sequence into multiple word segments based on a segmentation window; The lexical comparison module is configured to determine the target lexical segment by comparing a first predetermined portion of each of the plurality of lexical segments with a second predetermined portion of the user input. as well as The output acquisition module is configured to concatenate the user input and the target word segment into segments and feed them into the language model for inference to obtain the model output.

17. A computing device cluster comprising at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method according to any one of claims 1 to 15.

18. A computer program product comprising instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 15.

Images