Model early-leaving reasoning acceleration method, system and device and storage medium

Abstract

The invention discloses a model early-leaving reasoning acceleration method, system and device and a storage medium. The method comprises: acquiring question and answer sample data of a large language model input by a user; based on an agent determination sub-model of a large language model, determining a first number of agents required for processing the question and answer sample data according to the question and answer sample data, and running a sample data reasoning sub-model of the large language model to reasone the question and answer sample data; an output result of each feature layer is extracted, and first accumulated time consumed when each feature layer outputs the result and a first memory occupied by the system when each feature layer outputs the result are determined; according to the first number, the first cumulative time and the first memory, determining a model early-leaving target layer number corresponding to each feature layer; and when the number of the feature layers continuously outputting the same result is the same as the number of the model early-leaving target layers, controlling the large language model to exit reasoning in advance and outputting the target result. The method can be widely applied to the technical field of natural language processing.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, system, device and storage medium for accelerating model early-retirement reasoning. Background Art

[0002] In the field of deep learning, especially for large-scale pre-trained models (such as Qwen, etc.), as the number of model parameters increases, the complexity of the model also increases. This not only increases the computational cost during the training process, but also leads to a significant increase in the time delay and computing resource consumption in the inference phase. This phenomenon is particularly evident on edge computing and mobile devices, as these devices often have limited computing power and power supply.

[0003] In the related technologies, in order to solve the above problems, researchers have proposed the model early exit technology. The core idea of ​​this technology is to set multiple exit points in the middle layer of the model. These exit points can evaluate the complexity of the input data and decide whether the model reasoning process should be terminated in advance. If the input sample is simple enough and a reliable result can be obtained at a shallower level, the result can be directly output at the corresponding exit point to avoid continuing to perform more complex calculation processes. However, although the model early exit technology can accelerate the reasoning process to a certain extent, most of the current implementations of the model early exit technology are based on fixed thresholds or preset strategies to determine when the model exits. Although such a strategy is simple and easy to implement, when faced with different data sets and a constantly changing model reasoning environment, the timeliness of the model early exit technology and the accuracy of model reasoning will show a certain decline. Therefore, the problems existing in the related technologies need to be solved urgently. Summary of the invention

[0004] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent.

[0005] To this end, an object of an embodiment of the present application is to provide a model early retirement reasoning acceleration method, system, device and storage medium, which can improve the timeliness and accuracy of large language model reasoning.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0007] On the one hand, an embodiment of the present application provides a model early exit reasoning acceleration method for accelerating the reasoning speed of a large language model, wherein the large language model includes an agent determination submodel and a sample data reasoning submodel; the sample data reasoning submodel includes several feature layers, and the method includes: obtaining the question and answer sample data of the user input into the large language model; based on the agent determination submodel, determining the first number of agents required to process the question and answer sample data according to the question and answer sample data, and running the sample data reasoning submodel to reason about the question and answer sample data; extracting the output result of each of the feature layers and determining the first cumulative time consumed when each of the feature layers outputs the result, and the first memory occupied by the system when each of the feature layers outputs the result; determining the model early exit target layer number corresponding to each feature layer according to the first number, the first cumulative time and the first memory; when the number of the feature layers that continuously output the same result is the same as the model early exit target layer number, controlling the large language model to exit reasoning early and output the target result.

[0008] The embodiments of the present application disclose a method, system, device and storage medium for accelerating model early exit inference, which obtains question and answer sample data input by a user into a large language model; determines a sub-model based on an intelligent agent of the large language model, determines a first number of intelligent agents required to process the question and answer sample data according to the question and answer sample data, and runs the sample data inference sub-model of the large language model to infer the question and answer sample data; extracts the output result of each feature layer and determines the first cumulative time consumed when each feature layer outputs the result, and the first memory occupied by the system when each feature layer outputs the result; determines the model early exit target layer number corresponding to each feature layer according to the first number, the first cumulative time and the first memory; when the number of feature layers that continuously output the same result is the same as the model early exit target layer number, controls the large language model to exit inference early and output the target result. The present application can determine a model early exit dynamic model early exit target layer number according to the first number, the first cumulative time and the first memory, determines the model early exit node through the dynamic model early exit target layer number, and timely controls the large language model to exit early and output the target result. This application can improve the timeliness and accuracy of large language model reasoning and increase the speed of model reasoning.

[0009] In addition, a model early-retirement reasoning acceleration method according to the above embodiment of the present application may also have the following additional technical features:

[0010] Further, in one embodiment of the present application, determining the target number of model early retirement layers corresponding to each feature layer according to the first number, the first accumulated time, and the first memory includes:

[0011] Determining an initial number of feature layers according to the first number;

[0012] determining a target time ratio according to the first accumulated time;

[0013] Determining a target memory utilization rate of the system according to the first memory;

[0014] The target number of layers for early retirement of the model is determined according to the initial number of layers, the target time ratio, and the target memory utilization.

[0015] Further, in one embodiment of the present application, determining the initial number of feature layers according to the first number includes:

[0016] When the first number is less than a first preset threshold, determining the initial number of layers to be a first preset number of layers;

[0017] When the first number is greater than or equal to the first preset threshold, the initial number of layers is determined to be a second preset number of layers; wherein the first preset number of layers is less than or equal to the second preset number of layers.

[0018] Further, in one embodiment of the present application, determining the target time ratio according to the first accumulated time includes:

[0019] Extracting the second time consumed when the sample data inference sub-model outputs the maximum length;

[0020] determining a target time difference according to the first accumulated time and the second time;

[0021] A target time ratio is determined according to the target time difference and the second time.

[0022] Further, in one embodiment of the present application, determining a target memory utilization of the system according to the first memory includes:

[0023] Extract the current system's maximum memory;

[0024] Determine the free memory of the system according to the first memory and the maximum memory;

[0025] The target memory utilization is determined according to the free memory and the maximum memory.

[0026] Further, in one embodiment of the present application, determining the target memory utilization rate according to the free memory and the maximum memory includes:

[0027] Taking the free memory and the maximum memory as a quotient to obtain a first quotient;

[0028] The first quotient is used as the target memory utilization.

[0029] Further, in one embodiment of the present application, determining the target number of layers for early retirement of the model according to the initial number of layers, the target time ratio, and the target memory utilization rate includes:

[0030] The initial number of floors is multiplied by a preset emergency coefficient to obtain a first product;

[0031] Multiplying the first product by the target memory utilization to obtain a second product;

[0032] Taking the second product as a quotient and dividing it by the target time ratio to obtain a second quotient;

[0033] The second quotient is used as the target number of layers.

[0034] On the other hand, an embodiment of the present application provides a model early-dead reasoning acceleration system for accelerating the reasoning speed of a large language model, wherein the large language model includes an agent determination sub-model and a sample data reasoning sub-model; the sample data reasoning sub-model includes several feature layers, and the system includes:

[0035] An acquisition module is used to obtain question and answer sample data input by users into a large language model;

[0036] A first processing module is used to determine a sub-model based on the agent, determine a first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model to reason about the question and answer sample data;

[0037] A second processing module is used to extract the output results of each of the feature layers and determine a first cumulative time consumed when each of the feature layers outputs the results, and a first memory occupied by the system when each of the feature layers outputs the results;

[0038] A third processing module, configured to determine the target number of model early retirement layers corresponding to each feature layer according to the first number, the first accumulated time, and the first memory;

[0039] The fourth processing module is used to control the large language model to exit reasoning early and output the target result when the number of the feature layers that continuously output the same result is the same as the target number of layers for early exit of the model.

[0040] On the other hand, an embodiment of the present application provides a model early-deadline inference acceleration device, comprising:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned model early-retirement reasoning acceleration method.

[0044] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned model early-retirement inference acceleration method.

[0045] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0046] The embodiments of the present application disclose a method, system, device and storage medium for accelerating model early exit inference, which obtains question and answer sample data input by a user into a large language model; determines a sub-model based on an intelligent agent of the large language model, determines a first number of intelligent agents required to process the question and answer sample data according to the question and answer sample data, and runs the sample data inference sub-model of the large language model to infer the question and answer sample data; extracts the output result of each feature layer and determines the first cumulative time consumed when each feature layer outputs the result, and the first memory occupied by the system when each feature layer outputs the result; determines the model early exit target layer number corresponding to each feature layer according to the first number, the first cumulative time and the first memory; when the number of feature layers that continuously output the same result is the same as the model early exit target layer number, controls the large language model to exit inference early and output the target result. The present application can determine a model early exit dynamic model early exit target layer number according to the first number, the first cumulative time and the first memory, determines the model early exit node through the dynamic model early exit target layer number, and timely controls the large language model to exit early and output the target result. This application can improve the timeliness and accuracy of large language model reasoning and increase the speed of model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A schematic diagram of the steps of a method for accelerating model early-retirement reasoning provided in an embodiment of the present application;

[0049] Figure 2A schematic diagram of the steps of determining the number of model early retirement target layers corresponding to each feature layer according to a first quantity, a first accumulated time, and a first memory in a model early retirement inference acceleration method provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the steps of determining a target time ratio according to a first accumulated time in a model early-retirement reasoning acceleration method provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the steps of determining a target memory utilization rate of a system according to a first memory in a model early retirement reasoning acceleration method provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the steps of determining a target memory utilization rate according to free memory and maximum memory in a model early retirement reasoning acceleration method provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the steps of determining the target number of layers for early retirement of a model according to the initial number of layers, the target time ratio, and the target memory utilization rate in a method for accelerating early retirement of a model provided in an embodiment of the present application;

[0054] Figure 7 A flowchart of another model early-retirement reasoning acceleration method provided in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of the structure of a model early-retirement reasoning acceleration system provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of a model early-retirement reasoning acceleration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0058] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, the nouns appearing in this application are explained.

[0061] Agent: Agent is a core concept in the field of artificial intelligence, which refers to an agent that can autonomously perceive the environment, make decisions, and perform actions to achieve specific goals. The following is a detailed explanation of the agent:

[0062] An agent is an entity that can perceive the environment and take actions to achieve a specific goal. It can be software, hardware or a system with autonomy, adaptability and interaction capabilities. Agents can be divided into physical agents and virtual agents. Physical agents are robots, while virtual agents include various software agents. Agents are reactive, autonomous, proactive, social and evolvable. Autonomy: Agents can automatically adjust their behavior and state according to environmental changes and have the ability to self-manage. This enables agents to flexibly respond to various changes and achieve predetermined goals in dynamic and complex environments. Reactivity: Agents can respond to external stimuli, which is the basis for interaction with the environment. Initiative: Agents can not only respond to stimuli, but also take the initiative to take actions to achieve goals. Sociality: Agents can cooperate with other agents or people to achieve common goals through communication. Evolution: Agents can accumulate knowledge and experience and adapt to new environments. Agents perceive changes in the environment (such as through sensors or data input), make judgments and decisions based on their own learned knowledge and algorithms, and then perform actions to affect the environment or achieve predetermined goals. Its core lies in its ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments.

[0063] Prompt splicing: Prompt splicing is a technique in the field of natural language processing (NLP). Especially when using large language models (LLMs), Prompt splicing is widely used to guide the model to generate more accurate and flexible results. The following is a detailed explanation of Prompt splicing:

[0064] U-Prompt concatenation refers to combining multiple independent prompt words or sentences together as input to guide large language models to generate outputs related to multiple prompts. This method can provide richer and more comprehensive information, thereby accurately guiding the model to generate content expected by users. Prompt concatenation is divided into simple concatenation, templated concatenation, and chained concatenation. Simple concatenation: Multiple prompts are directly concatenated together to form a longer prompt. This method is simple and direct, but additional processing of the generated output may be required to remove unnecessary parts or integrate information. Templated concatenation: Use preset templates to combine different prompts. Templates can contain placeholders for inserting specific prompt content. This method ensures that the generated prompts have a certain structure and consistency. Chained concatenation: Link multiple prompts together to form a chain structure. The output of each prompt is used as the input of the next prompt, forming a continuous generation process. This method can handle more complex tasks, but requires careful design of the associations and dependencies between prompts. Application scenarios of prompt concatenation may include chatbots, content generation, and personalized recommendations. Chatbots: In chat scenarios, Prompt splicing can be used to guide intelligent chatbots to generate responses for specific topics or situations. By providing multiple prompts, chatbots can better grasp user intentions and give corresponding responses, further improving user experience. Content generation: For content generation applications, Prompt splicing can help the model more accurately generate articles, summaries, or other text content on specific topics. Guided by multiple prompts, the model can better understand user needs and generate content that is more in line with expectations. Personalized recommendations: In the field of personalized recommendations, Prompt splicing can be used to guide the model to have a more accurate understanding of user interests and preferences, thereby providing recommendation results that are more in line with user expectations. This method can effectively narrow the information gap between the recommendation system and users and improve the accuracy of recommendations.

[0065] Secondly, the technical problems existing in the relevant technology are explained.

[0066] In the field of deep learning, especially for large-scale pre-trained models (such as Qwen, etc.), as the number of model parameters increases, the complexity of the model also increases. This not only increases the computational cost during the training process, but also leads to a significant increase in the time delay and computing resource consumption in the inference phase. This phenomenon is particularly evident on edge computing and mobile devices, as these devices often have limited computing power and power supply.

[0067] In related technologies, in order to solve the above problems, researchers have proposed model early exit technology. The core idea of ​​this technology is to set multiple exit points in the middle layer of the model. These exit points can evaluate the complexity of the input data and decide whether the model reasoning process should be terminated early. If the input sample is simple enough and a reliable result can be obtained at a shallower level, the result can be directly output at the corresponding exit point to avoid continuing to perform more complex calculations. However, although the model early exit technology can speed up the reasoning process to a certain extent, most of the current implementations of the model early exit technology are based on fixed thresholds or preset strategies to determine when the model should exit. Specifically, traditional model early exit methods usually rely on fixed thresholds to determine whether the sample should exit early. The softmax probability y of each category is calculated at the exit of each intermediate layer. c , in the entropy sum entropy(y) = ∑ c∈C y c log(y c ), if entropy meets the threshold, the inference is terminated, otherwise it continues. The specific threshold depends on the trade-off between classification effect and inference speed, and needs to be determined based on actual task experiments. Although such a strategy is simple and easy to implement, when faced with different data sets and a constantly changing model inference environment, due to its fixed threshold, the timeliness of the model early retirement technology and the accuracy of model inference will show a certain decline. Therefore, there are still technical problems that need to be solved in the relevant technology.

[0068] Therefore, refer to Figure 1 , Figure 1 The figure is a schematic diagram of the steps of the model early retirement reasoning acceleration method of the present application. The model early retirement reasoning acceleration method of the present application can be used to speed up the reasoning speed of a large language model, wherein the large language model can include an agent determination sub-model and a sample data reasoning sub-model. The sample data reasoning sub-model can include several feature layers. Figure 1 In the method for accelerating model early retirement inference, the method may include steps S101 to S105.

[0069] S101. Obtain question and answer sample data input by the user into a large language model.

[0070] It is understandable that the question and answer sample data input by the user into the large language model may be voice sample data, text sample data or image sample data.

[0071] In some feasible embodiments of the present application, the acquisition device can establish a wired or wireless connection with the processor. After the connection is established, the processor can obtain the question and answer sample data of the user input into the large language model from the acquisition device.

[0072] It should be noted that the acquisition device can be a host computer, a mobile phone or other mobile devices. It should be noted that the wired connection method of the above embodiment may include the connection between the mobile device and the processing module, and may also include the connection between the processing module and the hardware device, and the wired connection between other devices currently known or to be developed in the future and the processing module; and the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (Ultra Wide Band) connection and other wireless connection methods currently known or to be developed in the future.

[0073] For example, the user can input "the difference between a three-phase motor and a common motor" through the text input box of the host computer, and then the processor connected to the host computer can obtain question and answer sample data with the content "the difference between a three-phase motor and a common motor".

[0074] For example, the user can also input "What is the difference between a three-phase motor and an ordinary motor" through a voice capture device, and then the processor connected to the voice capture device can first parse the voice content and then obtain question and answer sample data of "What is the difference between a three-phase motor and an ordinary motor".

[0075] Exemplarily, the user can also input captured or stored pictures through the host computer picture input box or camera, and then the processor connected to the host computer or camera can first analyze the picture content and then obtain question and answer sample data such as "what is the picture" or "what is in the picture".

[0076] S102. Determine a sub-model based on the agent, determine a first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model to reason about the question and answer sample data.

[0077] It is understandable that during the reasoning process of the large language model, the processing of some question and answer sample data requires the participation of the agent and the sample data reasoning sub-model. Therefore, the large language model needs to have at least one agent determination sub-model and at least one sample data reasoning sub-model. The agent determination sub-model can determine the required number of question and answer sample data to be processed during the entire large language model reasoning process based on the question and answer sample data. The sample data reasoning sub-model may include several transformer layers, that is, feature layers. The sample data reasoning sub-model outputs the results of each transformer layer during the process of reasoning the question and answer sample data. When the processing of some question and answer sample data does not require the participation of the agent, its number is 0.

[0078] In some feasible embodiments of the present application, the processor may input the question and answer sample data into the agent determination sub-model to obtain the first number of agents required to process the question and answer sample data. At the same time, the processor may run the sample data reasoning sub-model to reason about the question and answer sample data.

[0079] It should be noted that the agent determination sub-model is a trained model. Its training process can be to splice all existing agents, their descriptions, and various current input questions through prompts, and send them to the big model for training as a whole, and then continuously adjust the model parameters until the loss value is minimized to obtain the corresponding agent determination sub-model.

[0080] S103, extracting the output result of each feature layer and determining the first accumulated time consumed when each feature layer outputs the result, and the first memory occupied by the system when each feature layer outputs the result.

[0081] It is understandable that each feature layer can correspond to an output result, that is, each transformer layer corresponds to an output result. The first cumulative time is the cumulative time starting from the moment when the user inputs the question and answer sample data and ending with the output result of each feature layer. Since the large language model is a streaming output, each transformer layer has a corresponding output, and the time end corresponding to the output result of each feature layer is also different. The first memory can be the first memory occupied by the system when each feature layer outputs a result during the large language model inference process.

[0082] In some feasible embodiments of the present application, the processor can extract the output results of each feature layer, and at the same time determine the first cumulative time consumed when each feature layer outputs the results, and determine the first memory occupied by the system when each feature layer outputs the results.

[0083] It should be noted that the large model can be installed in a data processing system. The data system can correspond to a real-time memory when each feature layer outputs the result during the model reasoning process.

[0084] S104. Determine the target number of model early retirement layers corresponding to each feature layer according to the first quantity, the first accumulated time, and the first memory.

[0085] It is understandable that the model early withdrawal target number of layers corresponding to each feature layer may be different. Specifically, the model early withdrawal target number of layers corresponding to two adjacent feature layers may be different. For example, the model early withdrawal target number of layers corresponding to the first feature layer, that is, the transformer layer, is a, and the model early withdrawal target number of layers corresponding to the second feature layer, that is, the transformer layer, may be b. The model early withdrawal target number of layers corresponding to the second feature layer, that is, the transformer layer, may be a or c. The model early withdrawal target number of layers corresponding to each feature layer may be dynamically adjusted according to the first quantity, the first accumulated time, and the first memory.

[0086] In some feasible embodiments of the present application, the processor can calculate the model early retirement target layer number corresponding to each feature layer based on the first quantity, the first accumulated time and the first memory.

[0087] S105. When the number of feature layers that continuously output the same result is the same as the target number of model early exit layers, control the large language model to exit reasoning early and output the target result.

[0088] It is understandable that the process of determining the target number of layers for early model regression can be performed simultaneously with the process of detecting whether the output results of each extracted feature layer are the same.

[0089] In some feasible embodiments of the present application, when the number of target layers for model early exit is the same as the number of feature layers with the same continuous output results, the large language model can exit the reasoning process early and output the target result corresponding to the question and answer sample data.

[0090] In summary, the model early exit reasoning acceleration method, system, device and storage medium disclosed in the embodiments of the present application obtain the question and answer sample data of the user input into the large language model; determine the sub-model based on the agent of the large language model, determine the first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model of the large language model to reason the question and answer sample data; extract the output result of each feature layer and determine the first cumulative time consumed when each feature layer outputs the result, and the first memory occupied by the system when each feature layer outputs the result; determine the model early exit target layer number corresponding to each feature layer according to the first number, the first cumulative time and the first memory; when the number of feature layers that continuously output the same result is the same as the model early exit target layer number, control the large language model to exit the reasoning early and output the target result. The present application can determine a model early exit dynamic model early exit target layer number according to the first number, the first cumulative time and the first memory, determine the model early exit node through the dynamic model early exit target layer number, and timely control the large language model to exit early and output the target result. This application can improve the timeliness and accuracy of large language model reasoning and increase the speed of model reasoning.

[0091] Further, refer to Figure 2 , Figure 2 This is a schematic diagram of the steps of determining the target number of model early retirement layers corresponding to each feature layer according to the first quantity, the first accumulated time and the first memory in an embodiment of the present application. Figure 2 In the step, the step may include steps S201 to S204.

[0092] S201. Determine an initial number of feature layers according to a first number.

[0093] S202: Determine a target time ratio according to the first accumulated time.

[0094] S203: Determine a target memory utilization rate of the system according to the first memory.

[0095] S204: Determine the target number of layers for early retirement of the model according to the initial number of layers, the target time ratio, and the target memory utilization.

[0096] It is understandable that the initial number of layers may vary with the number of agents required to process the question and answer sample data. Specifically, the first number may have a predetermined mapping relationship with the initial number of layers. The mapping relationship may be adjusted according to user needs.

[0097] In some feasible embodiments of the present application, the processor may determine the initial number of feature layers according to the first quantity based on a mapping table stored in a database. At the same time, the processor may calculate and determine the target time ratio based on the first accumulated time. At the same time, the processor may determine the target memory utilization of the system based on the first memory. After obtaining the initial number of layers, the target time ratio, and the target memory utilization, the processor may calculate the target number of layers for early model retirement based on the initial number of layers, the target time ratio, and the target memory utilization.

[0098] In some feasible embodiments of the present application, the processor may first determine the initial number of feature layers based on the first number based on the mapping table stored in the database. Then, the processor may calculate and determine the target time ratio based on the first accumulated time. Then, the processor may determine the target memory utilization of the system based on the first memory. After successively obtaining the initial number of layers, the target time ratio, and the target memory utilization, the processor may calculate the target number of layers for early model retirement based on the initial number of layers, the target time ratio, and the target memory utilization.

[0099] Further, in some feasible embodiments of the present application, the step of determining the initial number of characteristic layers according to the first number may include step S301 or step S302.

[0100] S301: When the first number is less than a first preset threshold, determine that the initial number of layers is the first preset number of layers.

[0101] S302: When the first number is greater than or equal to a first preset threshold, determine that the initial number of layers is a second preset number of layers; wherein the first preset number of layers is less than or equal to the second preset number of layers.

[0102] It is understandable that the first preset threshold value may be any positive integer, such as 3, 5, 7 or 9.

[0103] In some feasible embodiments of the present application, when the first number is less than a first preset threshold, the processor may determine that the initial number of layers is the first preset number of layers. When the first number is greater than or equal to the first preset threshold, the processor may determine that the initial number of layers is the second preset number of layers. The first preset number of layers may be less than the second preset number of layers.

[0104] Specifically, when the number of agents required to process the question and answer sample data is less than 3, the question and answer sample data can be determined to be a simple question. After being determined to be a simple question, the first preset number of layers is a. When the number of agents required to process the question and answer sample data is greater than or equal to 3, the question and answer sample data can be determined to be a complex question. After being determined to be a complex question, the processor can determine the first preset number of layers to be b, where a can be equal to b.

[0105] Specifically, when the number of agents required to process the question and answer sample data is less than 5, the question and answer sample data can be determined to be a simple question. After being determined to be a simple question, the first preset number of layers is c. When the number of agents required to process the question and answer sample data is greater than or equal to 5, the question and answer sample data can be determined to be a complex question. After being determined to be a complex question, the processor can determine the first preset number of layers to be d, where c can be equal to d.

[0106] Specifically, when the number of agents required to process the question and answer sample data is less than 7, the question and answer sample data can be determined to be a simple question. After being determined to be a simple question, the first preset number of layers is e. When the number of agents required to process the question and answer sample data is greater than or equal to 7, the question and answer sample data can be determined to be a complex question. After being determined to be a complex question, the processor can determine the first preset number of layers to be f, where e can be less than f.

[0107] Specifically, when the number of agents required to process the question and answer sample data is less than 9, the question and answer sample data can be determined to be a simple question. After being determined to be a simple question, the first preset number of layers is g. When the number of agents required to process the question and answer sample data is greater than or equal to 9, the question and answer sample data can be determined to be a complex question. After being determined to be a complex question, the processor can determine the first preset number of layers to be h, where g can be equal to h.

[0108] Further, refer to Figure 3 , Figure 3 Schematic diagram of the steps of determining the target time ratio according to the first accumulated time in the embodiment of the present application. Figure 3 In the step, the step may include step S401-step S403.

[0109] S401, extracting the second time consumed when the sample data inference sub-model outputs the maximum length.

[0110] S402: Determine a target time difference according to the first accumulated time and the second time.

[0111] S403: Determine a target time ratio according to the target time difference and the second time.

[0112] It is understandable that the maximum length of the sample data reasoning sub-model output may be the total number of feature layers in the corresponding sample data reasoning sub-model. The feature layers of the sample data reasoning sub-model may be fixed, and the second time consumed by all its outputs may also be fixed.

[0113] In some feasible embodiments of the present application, the processor may extract the second time consumed when the sample data reasoning sub-model outputs the maximum length. After obtaining the second time, the processor may determine the target time difference according to the first accumulated time and the second time. After obtaining the target time difference, the processor may determine the target time ratio according to the target time difference and the second time.

[0114] In some feasible embodiments of the present application, the processor may extract the second time consumed when the sample data reasoning sub-model outputs the maximum length, and after obtaining the second time, input the second time and the first accumulated time into the following formula to obtain the target time ratio. The formula is:

[0115] t r =(time(max_output_len)-time(use)) / time(max_output_len)

[0116] Among them, t r is the target time ratio, time(max_output_ln) is the second time, and time(use) is the first accumulated time.

[0117] Further, refer to Figure 4 , Figure 4 1 is a schematic diagram of the steps of determining the target memory utilization of the system according to the first memory in an embodiment of the present application. Figure 4 This step may include steps S501 to S503.

[0118] S501. Extract the maximum memory of the current system.

[0119] S502: Determine the free memory of the system according to the first memory and the maximum memory.

[0120] S503: Determine the target memory utilization rate according to the free memory and the maximum memory.

[0121] In some feasible embodiments of the present application, the processor may extract the maximum memory of the current system. After obtaining the maximum memory, the processor may determine the free memory of the system according to the memory occupied by the system when each of the feature layers outputs the result and the maximum memory. After obtaining the free memory, the processor may determine the target memory utilization rate according to the free memory and the maximum memory.

[0122] In some feasible embodiments of the present application, the processor can extract the maximum memory of the current system, and after obtaining the maximum memory, the maximum memory and the first memory occupied by the system when each feature layer outputs the result are input into the following formula to obtain the target memory utilization. The formula is:

[0123] memory=(max_spare_gpu_memory-used_memory) / max_spare_gpu_memory

[0124] Among them, memory is the target memory utilization, max_spare_gpu_memory is the maximum memory of the current system, and used_memory is the first memory occupied by the system when each feature layer outputs results.

[0125] Further, refer to Figure 5 , Figure 5 This is a schematic diagram of the steps of determining the target memory utilization rate according to the free memory and the maximum memory in the embodiment of the present application. Figure 5 In the step, the step may include step S601-step S602.

[0126] S601. Take the free memory and the maximum memory as the quotient to obtain a first quotient.

[0127] S602: Use the first quotient as the target memory utilization.

[0128] In some feasible embodiments of the present application, the processor may calculate the quotient of the free memory and the maximum memory to obtain a first quotient. After obtaining the first quotient, the processor may use the first quotient as the target memory utilization.

[0129] In some feasible embodiments of the present application, after obtaining the free memory and the maximum memory, they can be input into the formula:

[0130] memory=f_memory / max_spare_gpu_memory

[0131] Among them, f_memory is the free memory and max_spare_gpu_memory is the maximum memory.

[0132] Further, refer to Figure 6 , Figure 6 This is a schematic diagram of the steps for determining the target number of layers for early retirement of the model according to the initial number of layers, the target time ratio, and the target memory utilization in the embodiment of the present application. Figure 6 In the step, the step may include steps S701 to S704.

[0133] S701. Multiply the initial number of floors by a preset emergency coefficient to obtain a first product.

[0134] S702: Multiply the first product by the target memory utilization to obtain a second product.

[0135] S703. Divide the second product by the target time ratio to obtain a second quotient.

[0136] S704: Use the second quotient as the target number of layers.

[0137] It can be understood that the preset emergency coefficient can be any number greater than 0 and less than or equal to 1.

[0138] In some feasible embodiments of the present application, the processor may multiply the initial number of layers by a preset emergency coefficient to obtain a first product. After obtaining the first product, the processor may multiply the first product by a target memory utilization rate to obtain a second product. After obtaining the second product, the processor may divide the second product by a target time ratio to obtain a second quotient. After obtaining the second quotient, the processor may use the second quotient as the target number of layers.

[0139] In some other embodiments of the present application, the processor may input the initial number of layers, the preset emergency coefficient, the target memory utilization rate, and the target time ratio into any one of the following two formulas to obtain the target number of layers. The two formulas are:

[0140] N t1 =N 0* e*memory / t r

[0141] N t2 =N 1* e*memory / t r

[0142] Among them, N t1 is the target number of layers when it is less than 3, N t2is the target number of layers when the number of agents is more than 3, N0 is the target number of layers when the number of agents is less than 3, N1 is the target number of layers when the number of agents is more than 3, memory is the target memory utilization, t r is the target time ratio.

[0143] The following is combined with Figure 7 The implementation principle of this application is described.

[0144] This embodiment proposes a more flexible and efficient model early exit reasoning acceleration method to address the problems existing in the current model early exit technology. The core of this method is to introduce a learnable decision module that can calculate the decision boundary of each question and answer sample at each exit point in real time and dynamically adjust these decision boundaries according to the characteristics of the question and answer sample and the state of the current model.

[0145] Traditional model early exit methods usually rely on a fixed threshold to determine whether a sample should be exited early. Calculate the softmax probability y for each category at the exit of the intermediate layer of each model c , in the entropy sum entropy(y) = ∑ c∈C y c log(y c ), if the entropy meets the threshold, the reasoning is launched, otherwise it continues. The specific threshold depends on the trade-off between classification effect and reasoning speed, and needs to be determined based on actual task experiments.

[0146] Reference Figure 7 The system of this embodiment includes a model module and an early exit decision module. The model module generates common large model results. According to the question and answer content input by the user, the results are generated layer by layer and the deeper structure of the model is further entered to complete the reasoning. The early exit decision module is used to determine whether the current model has output a result with a high confidence level and exit the reasoning stage in advance.

[0147] The core decision-making layer of the system mainly consists of two parts: a deep learning-based discrimination module and a decision boundary adjustment layer.

[0148] The deep learning discriminant module determines the credibility of the character with the highest current confidence by training a linear layer. This discriminant module is trained or fine-tuned together with the model.

[0149] The specific adjustment method of the decision boundary depends on the characteristics of the sample and the state of the model. For example, for simple question-answering samples, the decision boundary can be set more leniently to allow them to exit earlier; while for complex question-answering samples, the decision boundary should be set more strictly to ensure that accuracy is not affected.

[0150] Specifically, when a question-and-answer sample is input into the model, the large model must first be used to determine whether the question in the current question-and-answer sample requires the invocation of an agent and which agents to use.

[0151] Specifically, this part refers to the common form in the current langchain large model reasoning framework. All existing intelligent agents, their descriptions and current problems are spliced ​​through prompts and sent as a whole to the trained large model to determine which agents are needed to participate in the output of the large language model.

[0152] When the large language model determines that the current problem needs to call three or more agents, it can be determined that the problem is more complex and it is necessary to determine whether each of the five consecutive layers of the transformer structure has obtained the same classification result. If so, it can exit early during reasoning and output the target result corresponding to the question and answer sample data.

[0153] When it is judged to be a simple question, it is necessary to determine whether the transformer structure of each of the three consecutive layers has obtained the same classification result. If so, the model can be directly retired early and the target result corresponding to the question and answer sample data can be output.

[0154] In addition, the decision module also considers the processing time of the model's question and answer samples, the system's resource consumption, and the urgency of the task. The time consumption corresponding to the question and answer samples can be evaluated using the target time ratio, which can evaluate the computing time required for the current sample processing, so as to shorten the inference time as much as possible while ensuring accuracy. Resource consumption can consider the computing resources consumed during the model operation (such as CPU, GPU utilization, etc.) to ensure the effective use of resources, which can be evaluated using the target memory utilization. The urgency of the task can be based on the importance and urgency of the task, giving priority to those tasks with high timeliness requirements. The target time ratio is:

[0155] t r =(time(max_output_len)-time(use)) / time(max_output_len)

[0156] Among them, t r is the target time ratio, time(max_output_len) is the time consumed when the sample data inference sub-model outputs the maximum length, and time(use) is the cumulative time consumed when each feature layer outputs the result.

[0157] Resource consumption can be determined by dynamically obtaining the target memory utilization of the current GPU or CPU. The specific formula is:

[0158] memory=(max_spare_gpu_memory-used_memory) / max_spare_gpu_memory

[0159] Among them, memory is the target memory utilization, max_spare_gpu_memory is the maximum memory of the current system, and used_memory is the first memory occupied by the system when each feature layer outputs results.

[0160] After obtaining the target memory utilization and target time ratio, the target number of model early retirement layers corresponding to each feature layer can be calculated. Specifically, the formula is:

[0161] N t1 =N0*e*memory / t r

[0162] N t2 =N1*e*memory / t r

[0163] Among them, N t1 The target number of model early retirement layers for each feature layer corresponding to the simple problem, N t2 is the target number of model early retirement layers for each feature layer corresponding to the complex problem, N0 is the number of feature layers initially set for the simple problem, N1 is the number of feature layers initially set for the complex problem, and memory is the target memory utilization. r is the target time ratio.

[0164] In summary, the present application can improve the reasoning efficiency of the model when processing data of different complexities. By introducing a dynamic decision boundary, simple question-and-answer samples can be identified and exited early in the early stages of the model, thereby avoiding subsequent unnecessary calculations. In this way, not only the reasoning speed of simple samples is accelerated, but also more computing resources are reserved for processing complex samples. This mechanism effectively balances the accuracy and computational efficiency of the model, especially when processing data sets containing a large number of simple samples. The decision boundaries in traditional model early exit techniques are usually static, which means that they cannot adapt well to changing data sets and reasoning environments. However, the present application introduces a learnable decision module so that the decision boundary can be dynamically adjusted according to the characteristics of the data and the current state of the model. This dynamic adjustment capability enables the model to make more accurate decisions in different application scenarios, which can improve the overall adaptability and accuracy of the model.

[0165] This application can provide a new reasoning acceleration method for large models by introducing dynamic decision boundaries and multi-dimensional evaluation mechanisms. This method can effectively improve the reasoning efficiency and accuracy of the model under diverse data and different complexity requirements, and has broad application prospects and commercial value.

[0166] Reference Figure 8 , the embodiment of the present application also provides a model early retirement reasoning acceleration system. The system may include an acquisition module 1001, a first processing module 1002, a second processing module 1003, a third processing module 1004 and a fourth processing module 1005. Among them, the acquisition module 1001 is used to obtain the question and answer sample data of the user input large language model. The first processing module 1002 is used to determine the sub-model based on the agent, determine the first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model to reason about the question and answer sample data. The second processing module 1003 is used to extract the output results of each of the feature layers and determine the first cumulative time consumed when each of the feature layers outputs the results, and the first memory occupied by the system when each of the feature layers outputs the results. The third processing module 1004 is used to determine the number of model early retirement target layers corresponding to each feature layer according to the first number, the first cumulative time and the first memory. The fourth processing module 1005 is used to control the large language model to exit reasoning early and output the target result when the number of the feature layers that continuously output the same result is the same as the target number of layers for early exit of the model.

[0167] It should be noted that the acquisition module 1001 can be any integrated circuit module or microprocessor module obtained by integrating a chip with a processing function and its peripheral circuits through existing integration technology. The first processing module 1002 can also be any integrated circuit module or microprocessor module obtained by integrating a chip with a processing function and its peripheral circuits through existing integration technology. The processing module can also include one or more memories. The second processing module 1003 can also be any integrated circuit module or microprocessor module obtained by integrating a chip with a processing function and its peripheral circuits through existing integration technology. The processing module can also include one or more memories. One or more memories can be used to store data such as mapping table information or retrieval results. The third processing module 1004 can also be any integrated circuit module or microprocessor module obtained by integrating a chip with a processing function and its peripheral circuits through existing integration technology. The processing module can also include one or more memories.

[0168] In some embodiments of the present application, the specific connection method and device settings of the acquisition module 1001 and the first processing module 1002, the first processing module 1002 and the second processing module 1003, and the third processing module 1004 and the second processing module 1003 are not limited.

[0169] Understandably, Figure 1 The contents of the model early-departure reasoning acceleration method embodiment shown in the figure are applicable to the model early-departure reasoning acceleration device embodiment. The functions specifically implemented by the model early-departure reasoning acceleration device embodiment are similar to those of Figure 1 The model early retirement reasoning acceleration method embodiment shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the model early retirement reasoning acceleration embodiment shown are also the same.

[0170] Reference Fig. 9 The present application also discloses a model early-retirement reasoning acceleration device, including:

[0171] at least one processor 1011;

[0172] At least one memory 1012, used to store at least one program;

[0173] When at least one program is executed by at least one processor 1011, at least one processor 1011 implements the following Figure 1 An embodiment of a method for accelerating model early retirement reasoning is shown.

[0174] It is understandable that if Figure 1 The model early-retirement reasoning acceleration method embodiment shown is applicable to the computer device embodiment. The functions specifically implemented by the computer device embodiment are similar to those in the example Figure 1 The model early retirement reasoning acceleration method embodiment shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the model early retirement reasoning acceleration method embodiment shown are also the same.

[0175] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 1 An embodiment of a method for accelerating model early retirement reasoning is shown.

[0176] It is understandable that if Figure 1 The model early-retirement reasoning acceleration method embodiment shown is applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are similar to those in the embodiment shown in FIG. Figure 3 The model early retirement reasoning acceleration method embodiment shown is the same as that shown in FIG. Figure 1The beneficial effects achieved by the model early retirement reasoning acceleration method embodiment shown are also the same.

[0177] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0178] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0179] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0182] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0183] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0184] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0185] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

[0186] In the description of this specification, the description with reference to the terms "one embodiment", "another embodiment" or "certain embodiments" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0187] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for accelerating model early-retirement reasoning, characterized in that: Used to speed up the reasoning of a large language model, wherein the large language model includes an agent determination sub-model and a sample data reasoning sub-model; The sample data reasoning sub-model includes several feature layers, and the method includes: Obtaining question and answer sample data input by the user into the large language model; Determine a sub-model based on the agent, determine a first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model to reason about the question and answer sample data; Extracting the output results of each of the feature layers and determining a first cumulative time consumed when each of the feature layers outputs the results, and a first memory occupied by the system when each of the feature layers outputs the results; Determine, according to the first number, the first accumulated time, and the first memory, a target number of model early retirement layers corresponding to each feature layer; When the number of the feature layers that continuously output the same result is the same as the target number of layers for early exit of the model, the large language model is controlled to exit reasoning early and output the target result.

2. A method for accelerating model early retirement reasoning according to claim 1, characterized in that: The determining, according to the first quantity, the first accumulated time, and the first memory, the number of target layers of model early retirement corresponding to each feature layer includes: Determining an initial number of feature layers according to the first number; determining a target time ratio according to the first accumulated time; Determining a target memory utilization rate of the system according to the first memory; The target number of layers for early retirement of the model is determined according to the initial number of layers, the target time ratio, and the target memory utilization.

3. A method for accelerating model early-retirement reasoning according to claim 2, characterized in that: The step of determining the initial number of feature layers according to the first number includes: When the first number is less than a first preset threshold, determining the initial number of layers to be a first preset number of layers; When the first number is greater than or equal to the first preset threshold, the initial number of layers is determined to be a second preset number of layers; wherein the first preset number of layers is less than or equal to the second preset number of layers.

4. A method for accelerating model early-retirement reasoning according to claim 2, characterized in that: The step of determining a target time ratio according to the first accumulated time includes: Extracting the second time consumed when the sample data inference sub-model outputs the maximum length; determining a target time difference according to the first accumulated time and the second time; A target time ratio is determined according to the target time difference and the second time.

5. A method for accelerating model early retirement reasoning according to claim 2, characterized in that: The determining, according to the first memory, a target memory utilization rate of the system includes: Extract the current system's maximum memory; Determine the free memory of the system according to the first memory and the maximum memory; The target memory utilization is determined according to the free memory and the maximum memory.

6. A method for accelerating model early-retirement reasoning according to claim 5, characterized in that: The determining the target memory utilization rate according to the free memory and the maximum memory includes: Taking the free memory and the maximum memory as a quotient to obtain a first quotient; The first quotient is used as the target memory utilization.

7. A method for accelerating model early-retirement reasoning according to claim 2, characterized in that: Determining the target number of layers for early retirement of the model according to the initial number of layers, the target time ratio, and the target memory utilization rate includes: The initial number of floors is multiplied by a preset emergency coefficient to obtain a first product; Multiplying the first product by the target memory utilization to obtain a second product; Taking the second product as a quotient and dividing it by the target time ratio to obtain a second quotient; The second quotient is used as the target number of layers.

8. A model early-retirement reasoning acceleration system, characterized in that: Used to speed up the reasoning speed of a large language model, the large language model includes an agent determination sub-model and a sample data reasoning sub-model; the sample data reasoning sub-model includes several feature layers, and the system includes: An acquisition module is used to obtain question and answer sample data input by users into a large language model; A first processing module is used to determine a sub-model based on the agent, determine a first number of agents required to process the question and answer sample data according to the question and answer sample data, and run the sample data reasoning sub-model to reason about the question and answer sample data; A second processing module is used to extract the output results of each of the feature layers and determine a first cumulative time consumed when each of the feature layers outputs the results, and a first memory occupied by the system when each of the feature layers outputs the results; A third processing module, configured to determine the target number of model early retirement layers corresponding to each feature layer according to the first number, the first accumulated time, and the first memory; The fourth processing module is used to control the large language model to exit reasoning early and output the target result when the number of the feature layers that continuously output the same result is the same as the target number of layers for early exit of the model.

9. A model early-retirement reasoning acceleration device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the model early retirement inference acceleration method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor executable program is used to implement the model early-retirement reasoning acceleration method as described in any one of claims 1 to 7 when executed by the processor.

Images