Cross-model collaborative inference method, apparatus, device and storage medium

Abstract

Embodiments of the present application provide a cross-model collaborative reasoning method and device, equipment and a storage medium, relating to the technical field of intelligent computing equipment. The input prompt is sent to the first model for reasoning to generate a probability value corresponding to each word element at the current time step. The highest probability value is determined from the probability value, and the marginal benefit of the current time step is determined according to the highest probability value. The current benefit threshold corresponding to the current time step is obtained, and the decision variable is determined according to the marginal benefit and the current benefit threshold. The target word of the current time step is obtained based on the decision variable, and the output sequence corresponding to the input prompt is obtained according to the target word sequence before the current time step and the target word. The on-demand switching of the double model is realized by the dynamic determination of the marginal benefit and the benefit threshold. The first model is used to realize low-energy-consumption reasoning at the time step with high reasoning confidence. The second model is used to ensure accuracy at the key time step with low reasoning confidence. The reasoning accuracy is ensured while reducing the reasoning energy consumption.

Description

Technical Field

[0001] This application relates to the field of intelligent computing device technology, and in particular to cross-model collaborative reasoning methods, apparatus, devices and storage media. Background Technology

[0002] The application of large language models in industry and scientific research continues to accelerate, but the energy consumption and carbon emission pressures during the inference phase are also rising sharply. The inference load itself is characterized by its unpredictability and long-term nature, which requires data centers to reserve higher power supply and heat dissipation redundancy. At the same time, the marginal improvement rate of hardware energy efficiency is gradually slowing down, and the benefits of operator and graph-level optimization are also showing a diminishing trend. Therefore, it is necessary to reduce the energy consumption per unit of computing power while ensuring the quality of model inference.

[0003] Most inference optimization schemes for large language models in related technologies focus on optimizing the internal structure of a single model. These schemes shorten inference time and reduce energy consumption through acceleration and approximation techniques, and include two main categories: speculative sampling and beam search. Speculative sampling uses a lightweight proxy model to pre-generate candidate content, which is then batch-verified by the target model, thus reducing the number of forward computations in the target model. However, the energy efficiency gains of this method are highly sensitive to the load pattern and the hit rate of candidate content; if verification fails, it degenerates into a conventional decoding mode. Beam search, on the other hand, expands multiple generation paths in parallel and retains high-probability prefixes to improve the global optimality of the generated results. However, its computational overhead increases rapidly with the beam width, resulting in persistently high energy costs. Summary of the Invention

[0004] The main objective of this application is to propose a cross-model collaborative reasoning method, apparatus, device, and storage medium that reduces reasoning energy consumption while ensuring reasoning accuracy.

[0005] To achieve the above objectives, a first aspect of this application proposes a cross-model collaborative reasoning method, comprising:

[0006] Get input prompts, send the input prompts into the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal benefit of the current time step based on the highest probability value; Obtain the current revenue threshold corresponding to the current time step, and determine decision variables based on the marginal revenue and the current revenue threshold. The decision variables include a first decision variable for indicating the reasoning of the current time step by the first model and a second decision variable for indicating the reasoning of the current time step by the second model. The model size of the first model is smaller than that of the second model. Based on the decision variables, the target word for the current time step is obtained, and the output sequence corresponding to the input prompt is obtained based on the target word sequence before the current time step and the target word.

[0007] In some embodiments, obtaining the target word for the current time step based on the decision variable includes: If the decision variable is the first decision variable, the corresponding word element is determined as the target word based on the highest probability value; If the decision variable is the second decision variable, the target word sequence before the current time step is input into the second model for inference generation to obtain the decision probability value corresponding to each word element. Based on the maximum value of the decision probability value, the corresponding word element is determined as the target word, and the decision variable corresponding to the preset number of steps after the current time step is set as the second decision variable.

[0008] In some embodiments, determining the decision variable based on the marginal benefit and the current benefit threshold includes: generating the second decision variable if the marginal benefit is greater than or equal to the current benefit threshold, otherwise generating the first decision variable.

[0009] In some embodiments, determining the decision variable based on the marginal revenue and the current revenue threshold includes: Obtain the total number of lexical units and the share budget of the second model, and obtain the total lexical unit quota corresponding to the second model based on the total number of lexical units and the share budget; Based on the share budget, determine the current quota quantity corresponding to the current time step, obtain the previous cumulative usage corresponding to the previous time step, and obtain the remaining budget based on the total word quota and the previous cumulative usage; If the marginal benefit is greater than or equal to the current benefit threshold, and the previous cumulative usage is less than the current quota, and the remaining budget is not less than the comparison parameter, the second decision variable is generated, where the comparison parameter is the maximum value between one and the preset number of steps minus one.

[0010] In some embodiments, obtaining the share budget of the second model includes: Obtain the power limit for service operation, the first average power of the first model, and the second average power of the second model; Calculate the first difference between the power upper limit and the first average power, and the second difference between the second average power and the first average power. Calculate the quotient of the first difference and the second difference to obtain the share upper limit. Set the share budget to be less than or equal to the share upper limit.

[0011] In some embodiments, obtaining the current revenue threshold corresponding to the current time step includes: Obtain the previous revenue threshold and the previous quota quantity at the previous time step, and obtain the quota error corresponding to the previous time step based on the previous cumulative usage and the previous quota quantity; The current revenue threshold is obtained by summing the previous revenue threshold and the quota error.

[0012] In some embodiments, obtaining the current revenue threshold corresponding to the current time step includes: Obtain the previous revenue threshold and the previous quota quantity at the previous time step; obtain the quota error corresponding to the previous time step based on the previous cumulative usage and the previous quota quantity; calculate the sign function value of the quota error; and accumulate the total error of the quota error before the current time step. The product of the sign function value and the multiplication ratio step size is added to one, and then multiplied by the previous revenue threshold to obtain the multiplication term. The product of the total error and the integral weight is calculated to obtain the integral term. The sum of the multiplication term and the integral term is calculated to obtain the reference threshold. The reference threshold is then set within a preset range using a pruning function to obtain the current revenue threshold.

[0013] To achieve the above objectives, a second aspect of this application provides a cross-model collaborative reasoning apparatus, comprising: Marginal revenue calculation module: used to obtain input prompts, send the input prompts into the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal revenue of the current time step based on the highest probability value; Decision variable generation module: used to obtain the current revenue threshold corresponding to the current time step, and determine decision variables based on the marginal revenue and the current revenue threshold. The decision variables include a first decision variable for indicating the reasoning of the current time step by the first model and a second decision variable for indicating the reasoning of the current time step by the second model. The model size of the first model is smaller than that of the second model. Decision reasoning module: used to obtain the target word at the current time step based on the decision variables, and to obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0016] The cross-model collaborative reasoning method, apparatus, device, and storage medium proposed in this application embodiment obtain input prompts, send the input prompts to a first model for reasoning generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal benefit of the current time step based on the highest probability value; obtain the current benefit threshold corresponding to the current time step, and determine decision variables based on the marginal benefit and the current benefit threshold. The decision variables include a first decision variable for indicating that the first model performs reasoning at the current time step and a second decision variable for indicating that the second model performs reasoning at the current time step. The model size of the first model is smaller than that of the second model; obtain the target word at the current time step based on the decision variables, and obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word. This application embodiment completes basic reasoning with a lightweight first model. The lightweight model has fewer parameters and lower computational complexity. The computing power and energy consumption during the reasoning process are much lower than those of the large model, which can significantly reduce the energy consumption expenditure of the basic reasoning link. Marginal revenue quantifies the inference confidence of the first model at the current time step, which is then used to call the second model, avoiding energy waste caused by blindly using a large model. When the marginal revenue is higher than the revenue threshold, it indicates that the inference confidence of the first model at the current time step is high enough to ensure the accuracy of word generation. At this time, the first decision variable is triggered, and the first model completes the inference at the current time step, giving full play to the low-energy consumption advantage of the lightweight model. When the marginal revenue is lower than the revenue threshold, it indicates that the inference confidence of the first model is insufficient, and continued use may lead to a decrease in accuracy. At this time, the second decision variable is triggered, switching to the second model, which is larger and has stronger inference capabilities, to perform inference. The high precision of the second model ensures the accuracy of the final output sequence. This dynamic switching strategy avoids the high energy consumption caused by using the second model throughout, and also avoids the accuracy problem caused by using the first model throughout. Overall, the embodiments of this application achieve on-demand switching between the two models by dynamically determining the marginal revenue and the revenue threshold. In time steps with high inference confidence, the first model is used to achieve low-energy inference, while in critical time steps with low inference confidence, the second model is used to ensure accuracy, thereby reducing inference energy consumption while ensuring inference accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart of the cross-model collaborative reasoning method provided in the embodiments of this application.

[0018] Figure 2This is a flowchart provided in an embodiment of the present application for determining decision variables based on marginal revenue and current revenue threshold.

[0019] Figure 3 This is a flowchart of obtaining the share budget of the second model provided in an embodiment of this application.

[0020] Figure 4 This is a flowchart of obtaining the current revenue threshold corresponding to the current time step, provided in an embodiment of this application.

[0021] Figure 5 This is a flowchart of obtaining the current revenue threshold corresponding to the current time step, provided in an embodiment of this application.

[0022] Figure 6 This is a flowchart of obtaining the target word at the current time step based on decision variables, provided in an embodiment of this application.

[0023] Figure 7 This is a structural block diagram of a cross-model collaborative reasoning device provided in another embodiment of this application.

[0024] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] First, let's analyze some of the terms used in this application: Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0029] The application of large language models in industry and scientific research continues to accelerate. Unlike the training phase, inference handles continuous online, user-facing interactive loads. Its peak demand and business surges are often unpredictable and long-term, requiring data centers to maintain service levels with higher power supply and cooling redundancy. Under these operational characteristics, inference energy consumption optimization becomes increasingly important. Even if the energy consumption reduction per request seems limited, the cumulative effect over a large-scale service window can result in significant differences in energy consumption and carbon emissions. Meanwhile, marginal improvements in energy efficiency on the computing hardware side are gradually slowing down. The benefits of operator-level kernel optimization and compiler integration are diminishing. Relying solely on operator-level and graph-level optimizations of a single model is no longer sufficient to further reduce energy consumption per unit answer without sacrificing model quality.

[0030] In related technologies, inference optimization schemes for large language models include optimization within a single model and collaboration across models or policies. Optimization within a single model refers to shortening inference time and reducing energy consumption through acceleration and approximation techniques, specifically including speculative sampling and beam search. The basic idea of ​​speculative sampling is to pre-generate several candidates using a cheap proxy process, which are then validated once by the target model, thereby reducing the number of iterations in the target model. This type of method essentially still uses the target model as the main thread and relies on the parallel characteristics of batch validation of candidates. Although it can achieve considerable speed improvements under certain hardware and batch processing conditions, it requires maintaining candidates and their validation logic, and once validation fails, it often degenerates into regular decoding. Energy efficiency gains are highly sensitive to load patterns and hit rates. Beam search achieves stronger global optimality by expanding multiple generation paths in parallel at each step and retaining several high-probability prefixes, but its computational overhead increases rapidly with the beam width, resulting in energy costs several times higher than single-track methods. Cross-model collaboration focuses on request-level or task-level model routing, selecting models of different sizes for the entire request or task based on metrics such as input features, length, and historical performance. This approach places selection at a higher granularity, making it easier to deploy in systems engineering and integrate with service orchestration systems. However, this method fails to consider the highly non-uniform lexical difficulty within the same request or task, thus sacrificing significant optimization opportunities.

[0031] Based on this, embodiments of this application provide a cross-model collaborative inference method, apparatus, device, and storage medium. Basic inference is performed using a lightweight first model, which has fewer parameters and lower computational complexity. The computational power and energy consumed during inference are significantly lower than those of a larger model, greatly reducing energy expenditure in the basic inference stage. Marginal revenue is used to quantify the inference confidence of the first model at the current time step, which is then used for subsequent calls to the second model, avoiding energy waste caused by blindly using a large model. When the marginal revenue is higher than a revenue threshold, it indicates that the inference confidence of the first model at the current time step is high enough to ensure the accuracy of word generation. At this point, a first decision variable is triggered, and the first model completes the inference at the current time step, fully leveraging the low-energy advantage of the lightweight model. When the marginal revenue is lower than the revenue threshold, it indicates that the inference confidence of the first model is insufficient, and continued use may lead to a decrease in accuracy. At this point, a second decision variable is triggered, switching to a larger, more powerful second model for inference, utilizing the high precision of the second model to ensure the accuracy of the final output sequence. This dynamic switching strategy avoids the high energy consumption caused by using the second model throughout the entire process, while also avoiding the accuracy problems caused by using the first model throughout the entire process. Overall, the embodiments of this application realize on-demand switching between the two models by dynamically determining the marginal benefit and the benefit threshold. At time steps with high inference confidence, the first model is used to achieve low-energy inference, while at critical time steps with low inference confidence, the second model is used to ensure accuracy, thereby reducing inference energy consumption while ensuring inference accuracy.

[0032] This application provides a cross-model collaborative reasoning method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the cross-model collaborative reasoning method in this application embodiment is described.

[0033] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] The cross-model collaborative reasoning method provided in this application relates to the field of intelligent computing device technology. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting cross-model collaborative reasoning, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. This cross-model collaborative reasoning method can be executed by the terminal or the server, or by the terminal and the server working together.

[0036] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] The cross-model collaborative reasoning method in the embodiments of this application is described below.

[0039] A fundamental fact of language generation is the highly non-uniform difficulty of tokens. Within the same response, conjunctions, punctuation, and common phrases tend to be highly predictable; however, named entities, long-distance references, rare technical terms, code parameters, and complex inference nodes place stronger demands on expressive power. Most inference optimization techniques in related technologies follow a single-model paradigm, where a single model is responsible for all tokens from beginning to end. This approach ignores the subtle variations within the response. If we can subdivide easy and difficult points within the response at the token level and dynamically adopt a smaller, more energy-efficient model or a larger, more reliable model accordingly, we can potentially achieve systematic benefits in terms of overall energy consumption and latency without modifying the user interface or introducing the engineering complexity of backoff branches. Therefore, this application proposes a cross-model collaborative inference method. Unlike the single-model optimization in related technologies, the method in this embodiment does not create candidate branches or backoff. It uses a small model to progressively output and determine tokens on a single generation track, only allowing the larger model to generate the token when necessary. Therefore, it does not incur the additional memory and synchronization costs of parallel branches, nor does it rely on the check hit rate. Furthermore, without altering the user interface, the switching granularity is reduced to the word unit level, making the switching cost sufficiently low. This allows for the efficient use of a large model at the word unit level. The specific method flow is described below.

[0040] Figure 1 This is an optional flowchart of the cross-model collaborative reasoning method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 110 to 130. It is also understood that this embodiment... Figure 1The order of steps 110 to 130 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0041] Step 110: Obtain input prompts, send the input prompts to the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal revenue of the current time step based on the highest probability value.

[0042] In one embodiment, assume the first model is S and the second model is L, where the first model is smaller than the second model; that is, the first model can be considered a small model and the second model a large model. Assume the input prompt is x, the target output word count (i.e., the total number of words) is T, and the final output sequence is... The collaborative reasoning generation adopts a single-track, single-step gating structure. The core is that at each time step, the first model S is used to obtain the decision variables for the current time step. Then, based on the decision variables, it is decided whether to call the second model L to generate the target word for this time step. The final target word is unconditionally written back to the context of the first model S to maintain consistency in the future.

[0043] Specifically, assuming the current time step is step t, the first model S, when used for inference, generates the input prompt and the output sequence of the previous time steps, represented as a text sequence. Understandably, at the first time step, the text sequence only includes the input prompt x. Then, the first model S provides the probability distribution corresponding to the probability value of each word at the current time step t. , among which, here The unnormalized raw output of the first model S is given, where V is the vocabulary size. Let V be a probability space of dimension V-1.

[0044] For example, if the input prompt x is "The weather is very...", assuming this is the first time step, t=1, at which point some data has already been generated. If the vocabulary is empty, assuming there are only 5 lexical units in the vocabulary, namely ["good", "bad", "sunny", "terrible", "warm"], then the original output of the first model S is: =[2.1,0.5,1.8,-0.3,1.2]. The probability distribution obtained by softmax is: =[0.410,0.083,0.304,0.037,0.166].

[0045] Next, the maximum value among the probability values ​​is selected as the highest probability value, denoted as: This can be called the top-1 probability; for example, the highest probability value in the above example is 0.410. Then, the marginal benefit at the current time step is determined based on the highest probability value, expressed as... Therefore, the marginal benefit is 0.59. Understandably, the marginal benefit reflects the potential quality improvement from switching to the second model. When the first model is confident, the highest probability value is close to 1, and the marginal benefit is close to 0. When the first model is not confident, the highest probability value is close to 0, indicating that the lower its confidence level, the closer the marginal benefit is to 1. Therefore, switching to the stronger second model is expected to bring higher quality benefits. For example, the first model is very confident that "weather" is followed by "sunny". =0.1, the marginal benefit of calling the second model is small, if the first model is hesitant about what to follow "weather". =0.7, at which point the marginal benefit of calling the second model is large.

[0046] Step 120: Obtain the current revenue threshold corresponding to the current time step, and determine the decision variables based on the marginal revenue and the current revenue threshold.

[0047] In one embodiment, different time steps correspond to different revenue thresholds. Therefore, the current revenue threshold corresponding to the current time step is obtained. The calculation process of the revenue threshold will be described in detail later. After obtaining the current revenue threshold, the decision variable can be determined based on the marginal revenue and the current revenue threshold. Here, the decision variable is a binary variable, represented as... It includes a first decision variable for indicating inference at the current time step by the first model and a second decision variable for indicating inference at the current time step by the second model. When When corresponds to the first decision variable, when The time corresponds to the second decision variable.

[0048] In one embodiment, if the marginal benefit is greater than or equal to the current benefit threshold, a second decision variable is generated; otherwise, a first decision variable is generated, represented as follows:

[0049] in, This indicates the current profit threshold. It represents marginal revenue.

[0050] Understandably, if The decision variable is the second decision variable, indicating the probability distribution output by the second model L. to generate Otherwise, the probability distribution is output using the first model S. to generate But regardless How did it originate? In the reasoning of the next time step, the selected... All of them were forcibly appended back to the context of the first model S, and the text sequence was... This ensures that the gating decision in step t+1 is based on a consistent prefix.

[0051] In one embodiment, since the power consumption of the second model L is higher than that of the first model S, in order to meet the upper limit of the average power or total energy consumption preset by the intelligent computing device, it is necessary to constrain the number of tokens generated by the second model L. This constraint is reflected in the process of determining the decision variables. (Refer to...) Figure 2 , Figure 2 This is a flowchart provided in an embodiment of the present application for determining decision variables based on marginal revenue and a current revenue threshold, specifically including the following steps: Step 210: Obtain the total number of lexical units and the share budget of the second model. Based on the total number of lexical units and the share budget, obtain the total lexical unit quota corresponding to the second model.

[0052] In one embodiment, assuming the total number of tokens in the entire output is T, the budgeted share of the entire response allowed to be output by the second model L is set to... Therefore, based on the total number of word units T and the share budget The total token quota corresponding to the second model can be calculated. .

[0053] Step 220: Determine the current quota amount corresponding to the current time step based on the share limit, and obtain the previous cumulative usage corresponding to the previous time step. Obtain the remaining budget based on the total word token quota and the previous cumulative usage.

[0054] In one embodiment, since the timing of the second model's invocation is uncertain, to avoid excessive consumption of the share budget in the early stages leading to inference out of control in the later stages, a quota quantity that increases linearly with time is introduced. For the current time step t, the corresponding current quota quantity is: , This indicates rounding down to the nearest integer.

[0055] Next, define the previous cumulative usage corresponding to the previous time step as... , represented as: Here, the previous cumulative usage represents the inference share already used by the second model before the current time step, and the remaining budget for the current time step is obtained based on the difference between the total token quota and the previous cumulative usage. , represented as: .

[0056] Step 230: If the marginal revenue is greater than or equal to the current revenue threshold, and the previous cumulative usage is less than the current quota, and the remaining budget is not less than the comparison parameter, generate a second decision variable. The comparison parameter is the maximum value between one and the preset number of steps minus one.

[0057] In one embodiment, if marginal revenue Greater than or equal to the current revenue threshold And the previous cumulative usage Less than the current quota And the remaining budget Not less than the comparison parameter At that time, a second decision variable is generated. The comparison parameter here is the maximum value between one and a preset number of steps minus one. The preset number of steps K is to avoid back-and-forth jitter. It is set that once the inference process of the second model L is triggered, the second model L must be called continuously for at least K steps. K can be understood as the cooldown length, corresponding to the preset number of steps. Internal Order All decision variables are secondary decision variables.

[0058] Therefore, the process of determining decision variables can be expressed as:

[0059] It can be seen that after executing the above rules, if the gating rules are satisfied at each step, then It is always true, therefore Ensure ultimate feasibility.

[0060] The following is a specific example to illustrate this.

[0061] Suppose we want to generate a text with a total length and a total number of words T of 10 words, and we set the share budget τ of the second model L to 0.5, meaning that at most 50% of the words will be generated by the second model L. Therefore, the total word quota Bmax corresponding to the second model is... 0.5×10 =5 tokens, assuming a preset step count K=2, meaning that once the second model L is called, at least 2 consecutive time steps must be used. The quota quantity At corresponding to time step t= 0.5×t Therefore, the quota quantity corresponding to each time step is shown in Table 1 below: Time step 1 2 3 4 5 6 7 8 9 10 Quota quantity 0 1 1 2 2 3 3 4 4 5 Table 1. Quota Quantity for Each Time Step The following is the step-by-step reasoning process. Assuming that the reward threshold for each step is 0.6, the reasoning process corresponding to each time step is shown below.

[0062] With time step t=1 and marginal revenue of 0.4, the reasoning and decision-making process is as follows: 0.4 < 0.6: not satisfied; The decision variable is 0 (choose S), and the cumulative usage is 0.

[0063] With time step t=2 and marginal revenue of 0.7, the reasoning and decision-making process is as follows: 0.7 0.6: Satisfied; :satisfy; :satisfy; The decision variable is 1 (select L), and the cumulative usage is 1.

[0064] With time step t=3 and marginal revenue of 0.3, the reasoning and decision-making process is as follows: K=2, and this step forces the second model to be called for cooling. The decision variable is 1 (select L), and the cumulative usage is 2.

[0065] With time step t=4 and marginal revenue of 0.8, the reasoning and decision-making process is as follows: 0.8 0.6: Satisfied. Not satisfied. The decision variable is 0 (choose S), and the cumulative usage is 2.

[0066] With time step t=5 and marginal revenue of 0.6, the reasoning and decision-making process is as follows: 0.6 0.6: Satisfied Not satisfied The decision variable is 0 (choose S), and the cumulative usage is 2.

[0067] With time step t=6 and marginal revenue of 0.9, the reasoning and decision-making process is as follows: 0.9 0.6: Satisfied; :satisfy; :satisfy; The decision variable is 1 (select L), and the cumulative usage is 3.

[0068] With time step t=7 and marginal revenue of 0.2, the reasoning and decision-making process is as follows: K=2, and this step forces the second model to be called for cooling. The decision variable is 1 (select L), and the cumulative usage is 4.

[0069] With time step t=8 and marginal revenue of 0.5, the reasoning and decision-making process is as follows: 0.5 < 0.6: not satisfied; The decision variable is 0 (choose S), and the cumulative usage is 4.

[0070] With time step t=9 and marginal revenue of 0.7, the reasoning and decision-making process is as follows: 0.7 0.6: Satisfied; Not satisfied; The decision variable is 0 (choose S), and the cumulative usage is 4.

[0071] With a time step t=10 and a marginal benefit of 0.8, the reasoning and decision-making process is as follows: 0.8 0.6: Satisfied; :satisfy; :satisfy; The decision variable is 1 (select L), and the cumulative usage is 5.

[0072] Based on the above reasoning process, it can be seen that the total number of calls to the second model L is 5, which is exactly equal to the total word quota Bmax=5.

[0073] Therefore, this embodiment improves inference quality by calling the second model and balances power consumption and computational budget by limiting the share budget of the second model. The current quota amount corresponding to each time step prevents the quota from being wasted in the early stages. The preset number of steps K is used to constrain and ensure that a segment is generated continuously when the second model L is called, avoiding the overhead of frequent switching. In this way, the system can call the second model L as much as possible when the first model S is uncertain, while meeting the total power consumption constraint, thereby improving the generation quality.

[0074] In one embodiment, the share budget of the second model is related to the power of the intelligent computing device, therefore referring to... Figure 3 , Figure 3 This is a flowchart of obtaining the share budget of the second model provided in an embodiment of this application, which specifically includes the following steps: Step 310: Obtain the power limit for service operation, the first average power of the first model, and the second average power of the second model.

[0075] In one embodiment, the maximum power required for the current AI service to run is obtained. Simultaneously, the first average power of the first model running alone is statistically analyzed. And the second average power when the second model is triggered. The second average power refers to the power consumption level when the second model is invoked.

[0076] Step 320: Calculate the first difference between the power limit and the first average power, and the second difference between the second average power and the first average power. Calculate the quotient of the first difference and the second difference to obtain the share limit. Set the share budget to be less than or equal to the share limit.

[0077] In one embodiment, the operating average power can be obtained from the first average power and the second average power, and is approximately expressed as follows: , represents the overall average power in the scenario where the first model and the second model operate together, where, This represents the token share, equivalent to the share budget, and its corresponding share cap is... .

[0078] Therefore, in order to satisfy the power upper limit constraint ≤ The share cap ρ must satisfy the following relationship:

[0079] Among them, the upper limit of power and first average power The first difference is Second average power and first average power The second difference is The upper limit of the share is obtained by quotient of the first difference and the second difference. .

[0080] Before actual operation, a share budget τ is set, ensuring that this share budget is less than or equal to the share upper limit calculated above. This transforms power constraints into lexical-level share restrictions, enabling a quantitative mapping from power budget to call share.

[0081] In one embodiment, reference is made to Figure 4 , Figure 4 This is a flowchart of obtaining the current revenue threshold corresponding to the current time step provided in an embodiment of this application, which specifically includes the following steps: Step 410: Obtain the previous revenue threshold and the previous quota quantity in the previous time step, and obtain the quota error corresponding to the previous time step based on the previous cumulative usage and the previous quota quantity.

[0082] In one embodiment, historical benchmark data and constraint feedback signals are provided for the iterative update of the current revenue threshold. From the perspective of constraint optimization, the cumulative revenue needs to be maximized while satisfying the word share constraint. The corresponding optimization problem can be defined as:

[0083] in, Let be the decision variable at time step t. Let T' be the marginal revenue at time step t, and T' be the total number of time steps.

[0084] Construct a Lagrangian function by incorporating the share constraint into the profit maximization objective and introducing non-negative Lagrange multipliers μ≥0:

[0085] Based on this optimization objective, the revenue threshold at time step t is obtained. and quota quantity This revenue threshold The reward threshold for triggering the second model at time step t is essentially a reparameterized form of the nonnegative Lagrange multiplier μ introduced in constrained optimization problems. Based on the dual ascent method, the global constraint violation quantity is... μ is updated as a gradient, the quota quantity is online to a single time step, and the quota error is used instead of the global violation quantity as a gradient feedback signal.

[0086] The quota error at time step t is expressed as: This directly reflects the degree of constraint violation of the word share within time step t. A positive error indicates that the actual usage exceeds the quota, while a negative error indicates that the quota is not fully utilized. It is understandable that, for the previous time step, its previous revenue threshold is... Quota error is .

[0087] Step 420: Accumulate the previous revenue threshold and quota error to obtain the current revenue threshold.

[0088] In one embodiment, a step size parameter α (α>0) is introduced to adjust the threshold update amplitude, avoiding drastic fluctuations during the update process. Simultaneously, a truncation operation is introduced to ensure that the revenue threshold remains non-negative, conforming to the practical physical meaning and the non-negativity constraint of the Lagrange multipliers in the optimization problem. The threshold iterative update accumulates the weighted value of the previous revenue threshold and the quota error, and the non-negativity constraint is handled through a truncation operation, thus obtaining the first-order update. ,in Step size, .

[0089] Furthermore, Converting this to a profit threshold yields the current profit threshold for the current time step. The calculation formula is as follows:

[0090] As can be seen, the above process dynamically adjusts the revenue threshold through real-time feedback of quota error. Essentially, it is an online iterative optimization of the Lagrange multiplier, ultimately achieving the dual goals of maximizing cumulative revenue and complying with word share constraints. This ensures that the current revenue threshold, as the threshold for triggering the second model, can accurately adapt to the quota usage status at different time steps.

[0091] In one embodiment, the revenue threshold can also be adjusted from a cybernetics perspective. (See also...) Figure 5 , Figure 5 This is a flowchart of obtaining the current revenue threshold corresponding to the current time step provided in an embodiment of this application, which specifically includes the following steps: Step 510: Obtain the previous revenue threshold and the previous quota quantity in the previous time step. Based on the previous cumulative usage and the previous quota quantity, obtain the quota error corresponding to the previous time step. Calculate the sign function value of the quota error and accumulate the total error of the quota error before the current time step.

[0092] In one embodiment, from a cybernetics perspective, the threshold update process is equivalent to updating the quota quantity. Proportional-integral (PI) control is implemented. Quota error serves as the tracking error; the proportional channel provides immediate response to deviation direction, while the integral channel eliminates steady-state error, ultimately achieving precise matching of the word share to the previous quota quantity Bmax without overshoot.

[0093] At this point, obtain the previous revenue threshold from the previous time step. and the previous quota quantity This serves as the target benchmark for quota tracking. The cumulative usage of the second model within the previous time step is simultaneously acquired. Based on the previous cumulative usage and the previous quota quantity, the quota error corresponding to the previous time step is calculated. This error directly reflects the degree of deviation between quota usage and the target benchmark in the previous time step. Then, the sign function value is calculated for the aforementioned quota error. The sign function value only reflects the deviation direction of the quota error and does not depend on the absolute scale of the error. This is beneficial to improving the robustness of threshold updates to changes in the scale of revenue and adapting to scene migration under different temperatures and domain distributions.

[0094] Next, the quota errors of all time steps prior to the current time step are accumulated to obtain the total quota error, which is used for the calculation of subsequent integration channels to suppress steady-state deviation, expressed as:

[0095] Step 520: Calculate the product of the sign function value and the multiplication ratio step size, add one, and then multiply it by the previous revenue threshold to obtain the multiplication term. Calculate the product of the total error and the integral weight to obtain the integral term. Calculate the sum of the multiplication term and the integral term to obtain the reference threshold. Use the clipping function to set the reference threshold within a preset range to obtain the current revenue threshold.

[0096] In one embodiment, a hybrid update method combining multiplicative proportional and additive integral approaches can be used, along with a pruning function to ensure stability, ultimately yielding the revenue threshold for the current time step. Here, the multiplicative proportional step size η > 0 is used to adjust the response intensity of the proportional channel; the second is the integral weight. ≥0 is used to control the degree to which the integral channel suppresses steady-state deviation. These two types of parameters can be flexibly configured according to the actual needs of the scenario.

[0097] First, calculate the product of the sign function value and the multiplication scale step size. Then, add 1 to the product result. Finally, multiply this product by the previous revenue threshold to obtain the multiplication term. This multiplication term depends only on the sign of the quota error and is insensitive to the absolute scale of marginal revenue, which can effectively enhance the robustness of threshold updates in different scenarios.

[0098] Next, the sum of quota errors is multiplied by the integration weight to obtain the integral term, which is expressed as:

[0099] Among them, the integral term can gradually eliminate steady-state error by accumulating historical deviations, ensuring that the final quota usage is close to the previous quota amount and avoiding overshoot.

[0100] Adding the multiplication term to the integration term yields the reference threshold, expressed as:

[0101] As can be seen, this reference threshold integrates the immediate response of the proportional channel and the steady-state correction capability of the integral channel. By pruning it, the current revenue threshold can be obtained. A pruning function `clip[,[λmin,λmax]]` is introduced, setting a preset threshold range `[λmin,λmax]`, where `λmin` is the minimum threshold and `λmax` is the maximum threshold. Limiting the reference threshold within this preset range ensures the stability of threshold updates under extreme noise and long-sequence scenarios, preventing outliers from affecting system decisions.

[0102] After pruning, the current revenue threshold corresponding to the current time step is finally obtained, and the transmission hi is:

[0103] When updating the revenue threshold, the above embodiments use a hybrid approach of proportional-integral control, combining gating methods and threshold update logic, to prioritize the second model being called at positions with higher marginal revenue without violating quota constraints. This achieves the dual goals of quota constraint compliance and maximizing cumulative revenue.

[0104] Step 130: Obtain the target word for the current time step based on the decision variables, and obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word.

[0105] In one embodiment, reference is made to Figure 6 , Figure 6 This is a flowchart of obtaining the target word at the current time step based on decision variables, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 610: If the decision variable is the first decision variable, determine the corresponding word as the target word based on the highest probability value.

[0106] In one embodiment, if the decision variable at the current time step is the first decision variable, the target word is directly determined based on the inference result of the first model. The probability values ​​of all word elements in the vocabulary generated by the first model inference at the current time step and the highest probability value of the current time step calculated previously are obtained. Among the probability values ​​of all word elements, the word element corresponding to the highest probability value is located, and the word element is determined as the target word at the current time step. The target word is added to the target word sequence as the basis for model input in subsequent time steps.

[0107] Step 620: If the decision variable is the second decision variable, input the target word sequence before the current time step into the second model for inference generation, obtain the decision probability value corresponding to each word, determine the corresponding word as the target word based on the maximum value of the decision probability value, and set the decision variable corresponding to the preset number of steps after the current time step as the second decision variable.

[0108] In one embodiment, if the decision variable at the current time step is the second decision variable, the second decision variable corresponds to the decision mode of "calling the second model for optimization and generation." This requires leveraging the stronger reasoning capabilities of the second model to generate more accurate target words. All target words determined before the current time step are extracted and arranged in the order of generation to form a complete target word sequence. This sequence, along with the input prompt, is then fed into the second model as input data. Based on the input target word sequence, the second model performs deep reasoning generation, outputting the decision probability value corresponding to each word element in the vocabulary. It iterates through all word element decision probability values, selecting the highest value and determining its corresponding word element as the target word for the current time step. This target word is then added to the target word sequence.

[0109] In one embodiment, as mentioned above, in order to ensure the consistency and stability of the generated results, the decision variables corresponding to the preset number of steps after the current time step are all set as the second decision variables. That is, the second model will be continuously called for inference and generation within the subsequent preset number of steps to avoid frequent model switching that leads to fluctuations in generation quality.

[0110] In one embodiment, during cross-model collaborative inference, the two models employ a KVCache synchronization mechanism to persistently store cached key (K) and value (V) data from the historical generation process. Subsequent time steps only perform a single inference calculation for newly added lexical units, while ensuring that the cached data of the first and second models are always precisely aligned with the currently generated target word sequence. This mechanism avoids repeated recalculation of historical lexical units and ensures that when the decision variable switches to the second decision variable, the second model can directly continue generation based on the correct context, improving collaborative efficiency and generation coherence. The specific process is as follows: The system first obtains complete input prompts and constructs a KVCache for the first model based on these prompts, enabling the first model to advance the inference process using incremental generation from the initial time step. The KVCache pre-filling strategy for the second model can be flexibly configured according to actual needs: if it is necessary to reduce the inference overhead when switching to the second decision variable for the first time, a small amount of pre-warming pre-filling can be performed on the second model; if system startup costs are prioritized, pre-filling can be temporarily suspended for the second model, and hardware resources can be concentrated on the first model to ensure the efficient execution of gated decisions (determining decision variables based on marginal revenue and the current revenue threshold).

[0111] Upon entering the incremental generation phase, since the first model bears the core responsibility for gating decisions at each time step, its KVCache immediately appends the K and V data corresponding to the target word to the cache after each time step produces the target word, ensuring real-time synchronization between the cache and the target word sequence. The second model only participates in decoding and generation when the decision variable is the second decision variable; before participating in generation, a cache balancing operation must be performed—that is, for newly added target word sequence fragments not included in the second model's KVCache, they are appended to its cache in batches at once, achieving context alignment with the first model's cache. To further optimize performance, the system can set a periodic balancing mechanism: even if the current decision variable is the first decision variable, if the first model continuously generates for a preset number of steps, it will also package and append the K and V data corresponding to the newly added target word to the second model's cache, pre-allocating the cache alignment cost when switching to the second decision variable in the future.

[0112] To ensure the accuracy of the attention mechanism, the system constructs a minimal Boolean causal mask for each KVCache data append operation and explicitly specifies the absolute position index. The Boolean causal mask strictly adheres to the principle that future positions are not visible, and combined with the query length and key length during incremental writes, ensures that the model's attention calculation focuses only on the generated historical target words. The explicit position index ensures that the first and second models reference the same time coordinates at any time step, avoiding contextual misalignment caused by broadcast mechanisms or differences in position offsets. This design effectively reduces memory usage and data transfer bandwidth pressure by constructing only the minimum mask and position data required for adding new words.

[0113] It is evident that the core advantage of this KVCache synchronization mechanism lies in compressing the additional computational overhead to key nodes: the first model performs only one incremental cache write per time step, and the second model only performs batch appends when the second decision variable is triggered or when periodically catching up. The target word can be generated directly using the aligned cache at the moment the decision variable switches, without having to recalculate the historical context. This avoids the engineering complexity caused by candidate branches and backtracking, while retaining the generation quality advantage of dual-model collaboration.

[0114] In one embodiment, the above mechanism was experimentally verified and quantitatively analyzed in an NPU hardware environment: Qwen3-0.6B was used as the first model and Qwen3-8B as the second model, and they were used in collaboration. In a representative task, compared with using Qwen3-8B alone, the task energy consumption was reduced from 2273J to 1876J, achieving an energy consumption reduction of about 17%, which fully verified the significant effect of the mechanism in balancing generation quality and system energy efficiency.

[0115] The cross-model collaborative reasoning method in this application adopts a word-level model collaborative reasoning paradigm on a single generation track. The system deploys two sets of model weights on the same device. The first model has a smaller model size and lower energy consumption, and is responsible for progressive decoding and generation by default. The second model has stronger performance but higher energy consumption and is only called on demand at specific word positions. At each time step, the method evaluates the confidence level of the first model and decides whether to call the second model to determine the current word. Regardless of which model generates the word, it is appended to the target word sequence in the context of the first model, ensuring that gating decisions in subsequent time steps are always based on the latest prefix.

[0116] This embodiment differs from the candidate construction and backoff mechanism of speculative sampling, from the maintenance of multiple parallel sequences in beam search, and from the coarse-grained switching of the entire route. Instead, it degrades the model switching granularity to a single word, which can accurately capture the fine-grained fluctuations in word generation difficulty without introducing parallel backoff costs, thus achieving a balance between generation quality and inference efficiency.

[0117] Since the energy consumption constraints of intelligent computing clusters stem from both long-term goals and the power budget and heat dissipation limits of the computing cluster, inference services often face the dual constraints of average power limits and peak power suppression strategies. To meet energy consumption constraints in cross-model collaborative inference, this embodiment proposes an adaptive switching framework under budget constraints: constructing quota data that grows over time, calculating the gradual upper bound of token usage based on the principle of avoiding premature exhaustion of second model call opportunities; and introducing a revenue threshold as a probability threshold for triggering the second model, ensuring that the actual token usage stably tracks the quota trajectory through online updates. The quota upper limit can be directly converted from the average power of the first and second models, combined with the actual operating power limit, thus translating the power / energy consumption budget into the second model share budget, implementing system-level energy consumption constraints into the decoding behavior before deployment.

[0118] The core of achieving word-level collaboration here lies in two points: First, the gating signal design needs to balance difficult site capture with low computational / communication overhead. It uses the probability information output by the first model to construct robust marginal benefits and combines the benefit threshold to achieve stepwise decision-making. Second, cross-model context consistency maintenance needs to enable the second model to quickly align prefixes with low compensation cost when it is triggered. Based on KVCache, a pre-filling and incremental writing mechanism and a batch catching-up strategy between the two models are designed to ensure the correctness and efficiency of the switching.

[0119] The technical solution provided in this application embodiment obtains input prompts, sends them to a first model for inference generation, obtains the probability value corresponding to each word at the current time step, determines the highest probability value from the probability values, and determines the marginal benefit of the current time step based on the highest probability value; obtains the current benefit threshold corresponding to the current time step, and determines decision variables based on the marginal benefit and the current benefit threshold. Decision variables include a first decision variable indicating that the first model performs inference at the current time step and a second decision variable indicating that the second model performs inference at the current time step. The model size of the first model is smaller than that of the second model; the target word at the current time step is obtained based on the decision variables, and the output sequence corresponding to the input prompt is obtained based on the target word sequence before the current time step and the target word. This application embodiment completes basic inference with a lightweight first model. The lightweight model has fewer parameters and lower computational complexity, and the computing power and energy consumption during inference are much lower than those of a large model, which can significantly reduce the energy consumption expenditure of the basic inference stage. The marginal benefit is used to quantify the inference confidence of the first model at the current time step for subsequent calls to the second model, avoiding energy waste caused by blindly using a large model. When the marginal benefit exceeds the benefit threshold, it indicates that the inference confidence of the first model at the current time step is high enough to guarantee the accuracy of word generation. In this case, the first decision variable is triggered, and the first model completes the inference for the current time step, fully leveraging the low-energy consumption advantage of the lightweight model. When the marginal benefit falls below the benefit threshold, it indicates that the inference confidence of the first model is insufficient, and continued use may lead to a decrease in accuracy. In this case, the second decision variable is triggered, switching to a larger, more powerful second model for inference. The high precision of the second model ensures the accuracy of the final output sequence. This dynamic switching strategy avoids the high energy consumption caused by using the second model throughout the entire process, while also avoiding the accuracy problem caused by using the first model throughout. Overall, this embodiment achieves on-demand switching between the two models through dynamic determination of marginal benefit and benefit threshold. At time steps with high inference confidence, the first model is used for low-energy inference; at critical time steps with low inference confidence, the second model is used to ensure accuracy, thereby reducing inference energy consumption while maintaining inference accuracy.

[0120] This application also provides a cross-model collaborative reasoning device that can implement the above-described cross-model collaborative reasoning method, referring to... Figure 7 The device includes: Marginal revenue calculation module 710: It is used to obtain input prompts, send the input prompts to the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal revenue at the current time step based on the highest probability value.

[0121] Decision variable generation module 720: used to obtain the current revenue threshold corresponding to the current time step, and determine decision variables based on marginal revenue and the current revenue threshold. The decision variables include a first decision variable used to indicate the reasoning of the current time step by the first model and a second decision variable used to indicate the reasoning of the current time step by the second model. The model size of the first model is smaller than that of the second model.

[0122] Decision reasoning module 730: It is used to obtain the target word at the current time step based on the decision variables, and to obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word.

[0123] The specific implementation of the cross-model collaborative reasoning device in this embodiment is basically the same as the specific implementation of the cross-model collaborative reasoning method described above, and will not be repeated here.

[0124] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the cross-model collaborative reasoning method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0125] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the cross-model collaborative inference method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0126] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described cross-model collaborative reasoning method.

[0127] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The cross-model collaborative reasoning method, apparatus, device, and storage medium proposed in this application embodiment obtain input prompts, send the input prompts to a first model for reasoning generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal benefit of the current time step based on the highest probability value; obtain the current benefit threshold corresponding to the current time step, and determine decision variables based on the marginal benefit and the current benefit threshold. The decision variables include a first decision variable for indicating that the first model performs reasoning at the current time step and a second decision variable for indicating that the second model performs reasoning at the current time step. The model size of the first model is smaller than that of the second model; obtain the target word at the current time step based on the decision variables, and obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word. This application embodiment completes basic reasoning with a lightweight first model. The lightweight model has fewer parameters and lower computational complexity. The computing power and energy consumption during the reasoning process are much lower than those of the large model, which can significantly reduce the energy consumption expenditure of the basic reasoning link. Marginal revenue quantifies the inference confidence of the first model at the current time step, which is then used to call the second model, avoiding energy waste caused by blindly using a large model. When the marginal revenue is higher than the revenue threshold, it indicates that the inference confidence of the first model at the current time step is high enough to ensure the accuracy of word generation. At this time, the first decision variable is triggered, and the first model completes the inference at the current time step, giving full play to the low-energy consumption advantage of the lightweight model. When the marginal revenue is lower than the revenue threshold, it indicates that the inference confidence of the first model is insufficient, and continued use may lead to a decrease in accuracy. At this time, the second decision variable is triggered, switching to the second model, which is larger and has stronger inference capabilities, to perform inference. The high precision of the second model ensures the accuracy of the final output sequence. This dynamic switching strategy avoids the high energy consumption caused by using the second model throughout, and also avoids the accuracy problem caused by using the first model throughout. Overall, the embodiments of this application achieve on-demand switching between the two models by dynamically determining the marginal revenue and the revenue threshold. In time steps with high inference confidence, the first model is used to achieve low-energy inference, while in critical time steps with low inference confidence, the second model is used to ensure accuracy, thereby reducing inference energy consumption while ensuring inference accuracy.

[0129] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0130] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0133] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0134] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A cross-model collaborative reasoning method, characterized in that, include: Get input prompts, send the input prompts into the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal benefit of the current time step based on the highest probability value; Obtain the current revenue threshold corresponding to the current time step, and determine decision variables based on the marginal revenue and the current revenue threshold. The decision variables include a first decision variable for indicating the reasoning of the current time step by the first model and a second decision variable for indicating the reasoning of the current time step by the second model. The model size of the first model is smaller than that of the second model. Based on the decision variables, the target word for the current time step is obtained, and the output sequence corresponding to the input prompt is obtained based on the target word sequence before the current time step and the target word.

2. The cross-model collaborative reasoning method according to claim 1, characterized in that, The step of obtaining the target word for the current time step based on the decision variables includes: If the decision variable is the first decision variable, the corresponding word element is determined as the target word based on the highest probability value; If the decision variable is the second decision variable, the target word sequence before the current time step is input into the second model for inference generation to obtain the decision probability value corresponding to each word element. Based on the maximum value of the decision probability value, the corresponding word element is determined as the target word, and the decision variable corresponding to the preset number of steps after the current time step is set as the second decision variable.

3. The cross-model collaborative reasoning method according to claim 1, characterized in that, The step of determining the decision variable based on the marginal revenue and the current revenue threshold includes: generating the second decision variable if the marginal revenue is greater than or equal to the current revenue threshold, and generating the first decision variable otherwise.

4. The cross-model collaborative reasoning method according to claim 1, characterized in that, The step of determining decision variables based on the marginal revenue and the current revenue threshold includes: Obtain the total number of lexical units and the share budget of the second model, and obtain the total lexical unit quota corresponding to the second model based on the total number of lexical units and the share budget; Based on the share budget, determine the current quota quantity corresponding to the current time step, obtain the previous cumulative usage corresponding to the previous time step, and obtain the remaining budget based on the total word quota and the previous cumulative usage; If the marginal benefit is greater than or equal to the current benefit threshold, and the previous cumulative usage is less than the current quota, and the remaining budget is not less than the comparison parameter, the second decision variable is generated, where the comparison parameter is the maximum value between one and the preset number of steps minus one.

5. The cross-model collaborative reasoning method according to claim 4, characterized in that, Obtaining the share budget for the second model includes: Obtain the power limit for service operation, the first average power of the first model, and the second average power of the second model; Calculate the first difference between the power upper limit and the first average power, and the second difference between the second average power and the first average power. Calculate the quotient of the first difference and the second difference to obtain the share upper limit. Set the share budget to be less than or equal to the share upper limit.

6. The cross-model collaborative reasoning method according to claim 4, characterized in that, The step of obtaining the current revenue threshold corresponding to the current time step includes: Obtain the previous revenue threshold and the previous quota quantity at the previous time step, and obtain the quota error corresponding to the previous time step based on the previous cumulative usage and the previous quota quantity; The current revenue threshold is obtained by summing the previous revenue threshold and the quota error.

7. The cross-model collaborative reasoning method according to claim 4, characterized in that, The step of obtaining the current revenue threshold corresponding to the current time step includes: Obtain the previous revenue threshold and the previous quota quantity at the previous time step; obtain the quota error corresponding to the previous time step based on the previous cumulative usage and the previous quota quantity; calculate the sign function value of the quota error; and accumulate the total error of the quota error before the current time step. The product of the sign function value and the multiplication ratio step size is added to one, and then multiplied by the previous revenue threshold to obtain the multiplication term. The product of the total error and the integral weight is calculated to obtain the integral term. The sum of the multiplication term and the integral term is calculated to obtain the reference threshold. The reference threshold is then set within a preset range using a pruning function to obtain the current revenue threshold.

8. A cross-model collaborative reasoning device, characterized in that, include: Marginal revenue calculation module: used to obtain input prompts, send the input prompts into the first model for inference generation, obtain the probability value corresponding to each word at the current time step, determine the highest probability value from the probability values, and determine the marginal revenue of the current time step based on the highest probability value; Decision variable generation module: used to obtain the current revenue threshold corresponding to the current time step, and determine decision variables based on the marginal revenue and the current revenue threshold. The decision variables include a first decision variable for indicating the reasoning of the current time step by the first model and a second decision variable for indicating the reasoning of the current time step by the second model. The model size of the first model is smaller than that of the second model. Decision reasoning module: used to obtain the target word at the current time step based on the decision variables, and to obtain the output sequence corresponding to the input prompt based on the target word sequence before the current time step and the target word.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the cross-model collaborative reasoning method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-model collaborative reasoning method as described in any one of claims 1 to 7.

Images