Method and device for generating text

Abstract

The embodiment of the invention provides a text generation method and device, and the method comprises the steps: routing a current user request to a first quantitative model instance in a thinking link CoT generation stage, and generating a thinking text of the stage; in an answer generation stage, the current user request and the thinking text are routed to a first non-quantitative model instance, and an answer text of the stage is generated; wherein the parameter precision of the first quantitative model instance is lower than that of the first non-quantitative model instance. And the model performance, the resource efficiency and the response speed can be effectively balanced.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to methods and apparatus for generating text. Background Technology

[0002] With the rapid development of artificial intelligence technology, generative AI, represented by large language models (LLMs), has demonstrated powerful capabilities in the field of natural language processing. Large language models can understand complex instructions, generate high-quality text, conduct multi-turn dialogues, and play a crucial role in various application scenarios such as question answering, content creation, and code generation. However, the powerful capabilities of large language models are also accompanied by significant computational resource consumption and inference latency challenges.

[0003] In existing technologies, the reasoning process for large-scale model-generated text typically involves a large number of floating-point operations and memory accesses. Especially when dealing with complex tasks, the use of Chain-of-Thought (CoT) hints requires the model to generate intermediate reasoning steps, which significantly increases computational load and generation length, resulting in longer response times and higher resource overhead. Traditional solutions often tend to deploy high-precision models to ensure model performance and thus output quality, but this contradicts the requirements for resource efficiency and response speed.

[0004] Therefore, an improved solution is needed to balance model performance, resource efficiency, and response speed. Summary of the Invention

[0005] This specification describes one or more embodiments of a method and apparatus for generating text that effectively balances model performance, resource efficiency, and response speed.

[0006] Firstly, a method for generating text is provided, including:

[0007] During the CoT generation phase of the Mind Chain, the current user request is routed to the first quantization model instance to generate the thought text for this phase.

[0008] During the answer generation phase, the current user request and the thought text are routed to the first non-quantized model instance to generate the answer text for this phase; wherein, the parameter precision of the first quantized model instance is lower than that of the first non-quantized model instance.

[0009] In one possible implementation, the first quantized model instance is one of a first number of quantized model instances deployed, and the first non-quantized model instance is one of a second number of non-quantized model instances deployed.

[0010] Furthermore, the first number is determined based on the expected length of the thought text; the second number is determined based on the expected length of the answer text.

[0011] In one possible implementation, the current user request includes a user input statement and a pre-defined prompt statement, the prompt statement including knowledge text in the target domain to which the user input statement belongs.

[0012] In one possible implementation, the method further includes:

[0013] During the CoT generation phase, the current user request is routed to the first non-quantized model instance for first processing. The first processing includes calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on the attention mechanism, and caching the key matrix and value matrix.

[0014] Furthermore, the first quantization model instance generates the thought text, which is then processed in parallel with the first non-quantization model instance.

[0015] Further, routing the current user request to the first non-quantized model instance for first processing includes:

[0016] The current user request is input into the first non-quantized model instance, and it is set to generate only one output word.

[0017] Furthermore, generating the answer text for this stage includes:

[0018] The key matrix and value matrix corresponding to the second word sequence in the thought text are calculated based on the attention mechanism.

[0019] Using the key and value matrices corresponding to the first word sequence and the key and value matrices corresponding to the second word sequence in the cache, an attention-based calculation is performed to obtain the relevance score between the current user request and each input word in the thought text.

[0020] Based on the relevance scores between each input word, the output words are obtained to form the answer text.

[0021] Furthermore, the ratio between the first number and the second number is a preset value.

[0022] Secondly, an apparatus for generating text is provided, comprising:

[0023] The first generation unit is used to route the current user request to the first quantification model instance and generate the thinking text for this stage during the CoT generation phase of the thinking chain.

[0024] The second generation unit is used to route the current user request and the thought text generated by the first generation unit to the first non-quantitative model instance during the answer generation stage, and generate the answer text for this stage; wherein the parameter precision of the first quantized model instance is lower than that of the first non-quantitative model instance.

[0025] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0026] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0027] The method and apparatus provided in the embodiments of this specification divide the text generation reasoning process into two stages: a CoT generation stage and an answer generation stage. In the CoT generation stage, where accuracy requirements are relatively low, quantized model instances are used to save resources and accelerate the process; while in the answer generation stage, where accuracy requirements are extremely high, non-quantized model instances are used to ensure high-quality output answers. This staged, precision-differentiated processing effectively avoids the impact of accuracy loss caused by quantization on the output answer, and effectively balances model performance, resource efficiency, and response speed. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0030] Figure 2 A flowchart illustrating a method for generating text according to one embodiment is shown;

[0031] Figure 3 A flowchart illustrating a method for generating text according to another embodiment is shown;

[0032] Figure 4 A schematic block diagram of an apparatus for generating text according to one embodiment is shown;

[0033] Figure 5 A schematic block diagram of an apparatus for generating text according to another embodiment is shown. Detailed Implementation

[0034] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves text generation, specifically, text generation using a large language model under the CoT mechanism. (Refer to...) Figure 1 Under the CoT (Copyright Context) mechanism, the Large Language Model (LLM) does not directly generate the answer to the current user request. Instead, it first decomposes the complex problem corresponding to the current user request into subtasks and generates text for the intermediate reasoning process, and then generates the text corresponding to the answer, which can improve problem-solving capabilities. The entire text generation process can be divided into two stages: the CoT generation stage and the answer generation stage. In the CoT generation stage, the LLM generates the text for the intermediate reasoning process; in the answer generation stage, the LLM generates the text corresponding to the answer. These two stages are executed sequentially, with the CoT generation stage preceding the answer generation stage. During the CoT generation stage, the generated text is typically very long, leading to a significant increase in computational cost and response latency in the LLM inference process.

[0036] This specification's embodiments optimize the text generation process under the COT mechanism. By hybrid deployment of quantized and non-quantized model instances, resource consumption is saved while optimizing the response time of large language models. Specifically, the use of quantized or non-quantized model instances is dynamically selected based on the different accuracy requirements during the inference stage, maximizing resource utilization efficiency while ensuring output quality, demonstrating promising application prospects.

[0037] Among them, the Large Language Model (LLM) refers to a deep learning model with hundreds of millions to hundreds of billions or even more parameters, which can understand and generate human language and perform various complex natural language processing tasks.

[0038] Quantization is a model compression technique that reduces model size, memory usage, and computational cost by decreasing the numerical precision of model parameters and / or activation values. This accelerates the inference process and reduces deployment costs; for example, it reduces numerical precision from 32-bit floating-point numbers to 8-bit integers.

[0039] For quantized model instances, the hardware acceleration advantages of low-precision computation can be leveraged to improve inference speed. Quantization techniques can be divided into Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Model quantization has many advantages: it can significantly reduce resource consumption, greatly reduce model size and memory footprint, and lower storage and transmission costs; it can improve inference speed, as low-precision computation is generally faster and can better utilize the acceleration capabilities of dedicated hardware (such as TPUs and NPUs); and it offers greater deployment flexibility, suitable for edge devices and resource-constrained environments. Model quantization also has some disadvantages: there is a loss of accuracy, and the quantization process may lead to a decrease in model performance (such as accuracy and generation quality), especially with extremely low-precision quantization.

[0040] Chain-of-Thought (CoT) is a prompt engineering technique that improves the performance of large language models on complex reasoning tasks by guiding them to generate a series of intermediate reasoning steps, enabling them to solve problems progressively, similar to human thought processes. For example, when answering math questions, the model might first list the solution steps before providing the final answer. This improves the accuracy and interpretability of the model on complex reasoning tasks.

[0041] The CoT mechanism offers several advantages, including enhanced reasoning capabilities, significantly improving the performance of large language models in complex logical reasoning and mathematical computation tasks; improved interpretability, as the generation of intermediate steps makes the model's decision-making process more transparent; and the elimination of model fine-tuning, requiring only adjustments to input prompts without modifying or retraining the model itself. However, the CoT mechanism also has some drawbacks, such as increased generation length. Generating intermediate reasoning steps significantly increases the model's output length, leading to longer inference times and higher computational costs.

[0042] While conventional model quantization schemes and the CoT mechanism have achieved significant results in their respective fields, they still have limitations. Model quantization, although reducing resource consumption and improving speed, may sacrifice accuracy; CoT improves model capabilities but at the cost of increased computational load and response time. The embodiments in this specification aim to combine the advantages of these technologies and propose a smarter and more efficient hybrid deployment strategy to achieve a balance between performance and efficiency.

[0043] Figure 2 This diagram illustrates a method for generating text according to one embodiment, which can be based on... Figure 1 The implementation scenario shown employs a hybrid deployment of quantized model instances and non-quantized model instances. For example... Figure 2As shown, the method for generating text in this embodiment includes the following steps: Step 21, in the CoT generation stage, the current user request is routed to the first quantized model instance to generate the thinking text for this stage; Step 22, in the answer generation stage, the current user request and the thinking text are routed to the first non-quantized model instance to generate the answer text for this stage; wherein, the parameter precision of the first quantized model instance is lower than that of the first non-quantized model instance. The specific execution method of each of the above steps is described below.

[0044] First, in step 21, during the CoT generation phase of the thought chain, the current user request is routed to the first quantization model instance, generating the thought text for this phase. It can be understood that under the COT mechanism, the reasoning process of generating text from a large language model is divided into two phases: the CoT generation phase and the answer generation phase.

[0045] The embodiments described in this specification allow for the simultaneous deployment of several quantized model instances and several non-quantized model instances. For example, multiple quantized model instances can be deployed to generate thought text during the CoT generation phase, while a small number of non-quantized model instances can be deployed to generate answer text during the answer generation phase. This hybrid deployment strategy can be dynamically adjusted based on actual load and resource availability.

[0046] In one example, the first quantization model instance is one of a first number of quantization model instances deployed, and the first non-quantization model instance is one of a second number of non-quantization model instances deployed.

[0047] In this example, multiple quantized model instances and non-quantized model instances can be deployed to handle multiple user requests simultaneously.

[0048] Depending on the specific application scenario, the first number may be greater than the second number; or the first number may be equal to the second number; or the first number may be less than the second number.

[0049] In this example, the size relationship between the first number and the second number can be determined based on the relationship between the length of the thought text and the length of the answer text. For example, in a scenario where the length of the thought text is greater than the length of the answer text, a larger number of quantization model instances can be configured to generate the thought text, that is, the first number is greater than the second number.

[0050] Furthermore, the first number is determined based on the expected length of the thought text; the second number is determined based on the expected length of the answer text.

[0051] In this example, different expected lengths of thought text and answer text are used in different application scenarios, so that the number of quantized model instances and non-quantized model instances can be configured to suit the scenario.

[0052] In one example, the current user request includes a user input statement and a pre-defined prompt statement, the prompt statement including knowledge text in the target domain to which the user input statement belongs.

[0053] For example, in the context of insurance and financial planning, the aforementioned knowledge texts can be laws and regulations.

[0054] Furthermore, the ratio between the first number and the second number is a preset value.

[0055] In this example, by appropriately allocating the number of quantized model instances and non-quantized model instances, system throughput can be improved, resource consumption reduced, and response time shortened. This allows a single server or cluster to handle more concurrent requests, thereby increasing the overall throughput of the large model service. This makes the architecture flexible and scalable; the hybrid deployment architecture allows for flexible adjustment of the deployment ratio of quantized and non-quantized models according to actual business needs and resource conditions, making it easy to expand and maintain.

[0056] Then, in step 22, during the answer generation phase, the current user request and the thought text are routed to the first non-quantized model instance to generate the answer text for this phase; wherein, the parameter precision of the first quantized model instance is lower than that of the first non-quantized model instance. It is understood that using a non-quantized model instance ensures the accuracy of the answer text.

[0057] This specification divides the reasoning process of the large language model into two main stages: the CoT generation stage and the answer generation stage. In the CoT generation stage, the model is primarily responsible for generating intermediate reasoning steps, i.e., the thought text. This text generation process consumes significant computational resources, but the accuracy requirements for the answer text are relatively low, focusing more on logical coherence and the correctness of the steps. In the answer generation stage, the model generates the answer text based on the thought text generated in the CoT stage. This stage has extremely high requirements for the accuracy and quality of the output, requiring the model to possess the highest possible precision.

[0058] Based on the above division, the embodiments of this specification propose the following optimization strategies: In the CoT generation stage, a model with quantized weights is used for reasoning. Quantized models can significantly reduce computational load and memory consumption, thereby saving resources and accelerating the generation of the thought process. Since the output of the CoT stage is an intermediate step, even if there is a slight loss of accuracy, it will not have a significant impact on the answer text, and can be corrected or improved by subsequent non-quantized models. In the answer generation stage, a model without quantization is used for reasoning. Non-quantized models can provide the highest output accuracy and quality, ensuring the accuracy of the answer text. By using the output of the CoT stage as input to the non-quantized model, its powerful reasoning ability can be fully utilized to generate high-quality answer text.

[0059] In one example, the method further includes:

[0060] During the CoT generation phase, the current user request is routed to the first non-quantized model instance for first processing. The first processing includes calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on the attention mechanism, and caching the key matrix and value matrix.

[0061] In this example, a key-value (KV) caching mechanism is used for non-quantized model instances, and the KV caching mechanism is performed in advance during the CoT generation stage, which optimizes the utilization efficiency of the KV caching mechanism and further shortens the response time.

[0062] The KV caching mechanism, applied during the inference process of large language models, is a technique used to cache the key (K) and value (V) matrices calculated in the attention mechanism. By caching these intermediate results, repeated calculations during the autoregressive generation process can be avoided, significantly improving inference speed and efficiency.

[0063] While the key-value (KV) caching mechanism effectively reduces redundant computations during autoregressive generation, its memory consumption increases linearly with the generation length, further exacerbating resource pressure. How to effectively reduce the inference cost and shorten response time of large models while ensuring model output quality, and optimize the utilization efficiency of the KV cache, has become a critical issue that urgently needs to be addressed in the application and deployment of large models. The embodiments in this specification combine model quantization technology and KV cache pre-filling optimization to intelligently balance model performance and resource efficiency, thereby achieving more economical and faster large model services in practical applications.

[0064] Furthermore, the first quantization model instance generates the thought text, which is then processed in parallel with the first non-quantization model instance.

[0065] This example employs a pre-filling strategy. In generative large-scale model inference, this refers to the model processing the input prompt and calculating its corresponding key-value cache before generating the first token. This stage is typically computationally intensive, and parallel processing can further shorten the model's response time.

[0066] Further, routing the current user request to the first non-quantized model instance for first processing includes:

[0067] The current user request is input into the first non-quantized model instance, and it is set to generate only one output word.

[0068] In this example, since only one output word is generated, the computational cost and time consumption of the first processing are very small, effectively saving computing resources.

[0069] Furthermore, generating the answer text for this stage includes:

[0070] The key matrix and value matrix corresponding to the second word sequence in the thought text are calculated based on the attention mechanism.

[0071] Using the key and value matrices corresponding to the first word sequence and the key and value matrices corresponding to the second word sequence in the cache, an attention-based calculation is performed to obtain the relevance score between the current user request and each input word in the thought text.

[0072] Based on the relevance scores between each input word, the output words are obtained to form the answer text.

[0073] In this example, the specific process is decoding. In generative large-scale model inference, this refers to the process by which the model generates output tokens one by one after pre-filling. The generation of each token depends on the previously generated tokens and the cached key and value matrices.

[0074] The method provided in the embodiments of this specification divides the text generation reasoning process into two stages: the CoT generation stage and the answer generation stage. In the CoT generation stage, where accuracy requirements are relatively low, quantized model instances are used to save resources and accelerate the process; while in the answer generation stage, where accuracy requirements are extremely high, non-quantized model instances are used to ensure high-quality output answers. This staged, precision-differentiated processing effectively avoids the impact of accuracy loss caused by quantization on the output answer, and effectively balances model performance, resource efficiency, and response speed.

[0075] Figure 3 This diagram illustrates a method for generating text according to another embodiment, which can be based on... Figure 1 The implementation scenario shown employs not only a hybrid deployment of quantized and non-quantized model instances, but also a key-value cache pre-filling optimization strategy. For example... Figure 3 As shown, the method for generating text in this embodiment includes the following steps: Step 31, in the CoT generation stage, the current user request is routed to the first quantized model instance to generate the thinking text for this stage; Step 32, in the CoT generation stage, the current user request is routed to the first non-quantized model instance for first processing, the first processing including calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on an attention mechanism, and caching the key matrix and value matrix; Step 33, in the answer generation stage, the current user request and the thinking text are routed to the first non-quantized model instance, and the key matrix and value matrix corresponding to the second word sequence in the thinking text are calculated based on an attention mechanism; Step 34, using the cached key matrix and value matrix corresponding to the first word sequence and the key matrix and value matrix corresponding to the second word sequence, an attention mechanism-based calculation is performed to obtain the relevance score between each input word in the current user request and the thinking text; Step 35, based on the relevance score between each input word, each output word is obtained to constitute the answer text. The specific execution method of each of the above steps is described below.

[0076] First, in step 31, during the CoT generation phase of the thought chain, the current user request is routed to the first quantification model instance, generating the thought text for this phase. In essence, when a user request arrives, it is first routed to the quantification model instance. The quantification model instance receives the user request and generates the thought text, which represents a thought process with chain-like thinking.

[0077] The embodiments described in this specification can significantly save computing resources. By using a quantized model in the computationally intensive and precision-tolerant CoT generation stage, GPU memory usage and computational load can be greatly reduced, thereby lowering inference costs and improving hardware utilization.

[0078] Then, in step 32, during the CoT generation phase, the current user request is routed to the first non-quantized model instance for first processing. The first processing includes calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on an attention mechanism, and caching the key matrix and value matrix. The parameter precision of the first quantized model instance is lower than that of the first non-quantized model instance.

[0079] In this embodiment, to further optimize response time, a parallel optimization strategy of KV cache pre-filling is introduced. After a user request arrives, in addition to sending the user request to the quantization model for generating the thought text, a user request can also be sent simultaneously to the non-quantization model. The purpose of this parallel request is to allow the non-quantization model to pre-fill the KV cache in advance.

[0080] Specifically, when requesting the non-quantized model in parallel, its maximum output length (max_output_length) is set to 1. This means that the non-quantized model will only generate one token, but in the process, it will complete the calculation and caching of the key matrix and value matrix of the user request, including the prompt word. Since only one token is generated, the computational cost and time consumption of the entire operation are very small.

[0081] Next, in step 33, during the answer generation phase, the current user request and the thought text are routed to the first non-quantized model instance, and the key matrix and value matrix corresponding to the second word sequence in the thought text are calculated based on the attention mechanism. It can be understood that during the CoT generation phase, the key matrix and value matrix corresponding to the first word sequence in the current user request are generated, reducing the time spent calculating the key matrix and value matrix during the answer generation phase.

[0082] In this embodiment of the specification, after generating the thought text during the CoT generation phase, it is sent as new input, along with the current user request (or its key information), to the non-quantitative model instance. The non-quantitative model instance then performs reasoning based on the thought text to generate the answer text.

[0083] This embodiment of the specification demonstrates how a key-value (KV) cache pre-filling optimization strategy can improve cache hit rate. When the thought text generated during the CoT generation phase is completed and sent to the non-quantized model instance for answer generation, the non-quantized model instance has already pre-filled its KV cache for the user request. At this point, its KV cache already contains most of the required data. This significantly improves the KV cache hit rate, reduces the pre-filling time during the answer generation phase, and thus accelerates the overall decoding process.

[0084] This parallel pre-filling strategy also allows for the priority invocation of non-quantized model instances that have already undergone KV cache pre-filling after the CoT generation phase. This not only improves cache hit rate but also avoids re-initializing or pre-filling new non-quantized model instances during the answer generation phase, further optimizing resource utilization and response speed.

[0085] In step 34, using the cached key and value matrices corresponding to the first and second word sequences, an attention-based calculation is performed to obtain the relevance score between the current user request and each input word in the thought text. It can be understood that by determining the relevance score, the model can selectively focus on key parts of the input.

[0086] Finally, in step 35, based on the relevance scores between each input word, the output words are obtained to form the answer text. It is understandable that using relevance scores allows for more accurate generation of output words.

[0087] The embodiments described in this specification optimize response time. By combining the fast inference capabilities of the quantized model with parallel optimization of KV cache prefilling, the end-to-end response time of large models can be significantly shortened, improving user experience. It also improves the KV cache hit rate. The innovative parallel prefilling strategy ensures that the KV cache of the non-quantized model is largely ready when receiving the output of the CoT generation stage, thereby significantly improving the cache hit rate and reducing redundant calculations. Furthermore, it balances performance and accuracy. While ensuring high-quality output of the answer text, it achieves a balance between performance (i.e., speed and resources) and accuracy (i.e., output quality) by intelligently selecting between quantized and non-quantized models.

[0088] According to another embodiment, an apparatus for generating text is also provided, which is used to perform the methods provided in the embodiments of this specification. Figure 4 A schematic block diagram of an apparatus for generating text according to one embodiment is shown. Figure 4 As shown, the device 400 includes:

[0089] The first generation unit 41 is used to route the current user request to the first quantization model instance and generate the thinking text for this stage during the CoT generation stage of the thinking chain.

[0090] The second generation unit 42 is used to route the current user request and the thought text generated by the first generation unit 41 to the first non-quantitative model instance during the answer generation stage, and generate the answer text for this stage; wherein the parameter precision of the first quantized model instance is lower than the parameter precision of the first non-quantitative model instance.

[0091] Optionally, as an embodiment, the first quantization model instance is one of a first number of quantization model instances deployed, and the first non-quantization model instance is one of a second number of non-quantization model instances deployed.

[0092] Furthermore, the first number is determined based on the expected length of the thought text; the second number is determined based on the expected length of the answer text.

[0093] Optionally, as an embodiment, the current user request includes a user input statement and a pre-set prompt statement, wherein the prompt statement includes knowledge text in the target domain to which the user input statement belongs.

[0094] Alternatively, as an example, Figure 5 A schematic block diagram of an apparatus for generating text according to another embodiment is shown. Figure 5 As shown, in addition to the aforementioned first generating unit 41 and second generating unit 42, the device 500 also includes:

[0095] The caching unit 43 is used to route the current user request to the first non-quantized model instance during the CoT generation stage so that it can undergo first processing. The first processing includes calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on the attention mechanism, and caching the key matrix and value matrix.

[0096] Furthermore, the first quantization model instance generates the thought text, which is then processed in parallel with the first non-quantization model instance.

[0097] Furthermore, the caching unit 43 is specifically used to input the current user request into the first non-quantized model instance and set it to generate only one output term.

[0098] Furthermore, the second generation unit 42 includes:

[0099] The first computational subunit is used to calculate the key matrix and value matrix corresponding to the second word sequence in the thought text based on the attention mechanism;

[0100] The second calculation subunit is used to perform attention-based calculations using the key and value matrices corresponding to the cached first word sequence and the key and value matrices corresponding to the second word sequence obtained by the first calculation subunit, to obtain the relevance score between the current user request and each input word in the thought text.

[0101] The output subunit is used to obtain each output word based on the relevance score between each input word obtained from the second calculation subunit, so as to form the answer text.

[0102] Furthermore, the ratio between the first number and the second number is a preset value.

[0103] In this embodiment, the reasoning process is divided into two stages: the CoT generation stage and the answer generation stage. A quantized model is used in the CoT generation stage, where accuracy requirements are relatively low, to save resources and accelerate the process; while a non-quantized model is used in the answer generation stage, where accuracy requirements are extremely high, to ensure the highest quality of the final output. This staged, precision-based approach effectively avoids the impact of accuracy loss caused by quantization on the final result.

[0104] Furthermore, a parallel optimization strategy of KV cache pre-filling was introduced. While generating the thought text during the CoT generation phase, a request with a maximum output length of 1 was sent in parallel to the non-quantized model to pre-fill its KV cache. This significantly improves the KV cache hit rate when the non-quantized model generates the answer text after the CoT generation phase, reducing the pre-filling time in the answer generation phase and thus significantly shortening the overall response time. By using a less resource-intensive quantized model in the CoT generation phase and accelerating text generation in the answer generation phase through KV cache pre-filling optimization, the additional overhead introduced by the CoT generation phase is effectively offset. While ensuring the improved reasoning capabilities brought by the CoT mechanism, the overall computational cost and response latency are significantly reduced.

[0105] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 or Figure 3 The method described.

[0106] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 or Figure 3 The method described.

[0107] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0108] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating text, comprising: During the CoT generation phase of the Mind Chain, the current user request is routed to the first quantization model instance to generate the thought text for this phase. During the answer generation phase, the current user request and the thought text are routed to the first non-quantized model instance to generate the answer text for this phase; wherein, the parameter precision of the first quantized model instance is lower than that of the first non-quantized model instance.

2. The method as described in claim 1, wherein, The first quantization model instance is one of a first number of quantization model instances deployed, and the first non-quantization model instance is one of a second number of non-quantization model instances deployed.

3. The method as described in claim 2, wherein, The first number is determined based on the expected length of the thought text; the second number is determined based on the expected length of the answer text.

4. The method of claim 1, wherein, The current user request includes a user input statement and a pre-defined prompt statement, wherein the prompt statement includes knowledge text in the target domain to which the user input statement belongs.

5. The method of claim 1, wherein, The method further includes: During the CoT generation phase, the current user request is routed to the first non-quantized model instance for first processing. The first processing includes calculating the key matrix and value matrix corresponding to the first word sequence in the current user request based on the attention mechanism, and caching the key matrix and value matrix.

6. The method of claim 5, wherein, The first quantization model instance generates the thought text, which is then processed in parallel with the first non-quantization model instance.

7. The method of claim 5, wherein, The current user request is routed to the first non-quantized model instance for first processing, including: The current user request is input into the first non-quantized model instance, and it is set to generate only one output word.

8. The method of claim 5, wherein, The generation of the answer text for this stage includes: The key matrix and value matrix corresponding to the second word sequence in the thought text are calculated based on the attention mechanism. Using the key and value matrices corresponding to the first word sequence and the key and value matrices corresponding to the second word sequence in the cache, an attention-based calculation is performed to obtain the relevance score between the current user request and each input word in the thought text. Based on the relevance scores between each input word, the output words are obtained to form the answer text.

9. The method of claim 2, wherein, The ratio between the first number and the second number is a preset value.

10. An apparatus for generating text, comprising: The first generation unit is used to route the current user request to the first quantification model instance and generate the thinking text for this stage during the CoT generation phase of the thinking chain. The second generation unit is used to route the current user request and the thought text generated by the first generation unit to the first non-quantitative model instance during the answer generation stage, and generate the answer text for this stage; wherein the parameter precision of the first quantized model instance is lower than that of the first non-quantitative model instance.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images