Model inference control method, device, medium, and product

Abstract

The application discloses a model inference control method and device, medium and product, relates to the technical field of artificial intelligence, and comprises the following steps: obtaining a probability distribution sequence of a word element in a pre-training language model inference process, wherein the probability distribution sequence comprises a current probability distribution of the word element in a current inference step and a historical probability distribution of the word element in a historical inference step; extracting the current probability distribution and the historical probability distribution, identifying a current ranking of a pre-set target word element in the current step and a historical ranking of the target word element in the historical step, and generating a ranking sequence according to the current ranking and the historical ranking; locating an inference termination step of the pre-training language model according to the ranking sequence and a pre-set inference termination threshold, and controlling the inference process of the pre-training language model based on the inference termination step, so that the technical problem that a convergence critical point cannot be accurately identified in the prior art, resulting in overthinking and calculation waste, is solved, and the technical effects of reducing redundant thinking and reducing calculation overhead are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model inference control method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the wide application of large language models in mathematical solving and logical reasoning tasks, the problem of "overthinking" has also become prominent. Models often generate lengthy, repetitive or irrelevant reasoning content, not only occupying computing resources and prolonging response time, but also making the reasoning process deviate from the correct path, and even falling into a cycle.

[0003] To suppress overthinking, related technologies have proposed the idea of early termination of inference, but there are still obvious limitations: fixed budget methods are difficult to adapt to the complexity of different tasks; the scene dependence of the trigger word-based strategy is strong, making it difficult to cover diverse reasoning behaviors; while using external models to judge the inference stage in real time is more intelligent, but the computing cost is too high, making it difficult to apply. SUMMARY

[0004] The present application provides a model inference control method, an electronic device, a storage medium and a program product to at least solve the technical problem that the convergence critical point cannot be accurately identified in related technologies, resulting in overthinking and waste of computing resources.

[0005] The present application provides a model inference control method, comprising: obtaining a probability distribution sequence of a word element in a pre-training language model inference process, wherein the probability distribution sequence comprises a current probability distribution of the word element in a current inference step of the pre-training language model and a historical probability distribution of the word element in a historical inference step of the pre-training language model; extracting the current probability distribution and the historical probability distribution from the probability distribution sequence, identifying a current ranking of a target word element pre-set from the current probability distribution, identifying a historical ranking of the target word element from the historical probability distribution, and generating a ranking sequence according to the current ranking and the historical ranking; locating an inference termination step of the pre-training language model according to the ranking sequence and a pre-set inference termination threshold, generating an inference termination instruction if the current inference step is the inference termination step, controlling the inference termination of the pre-training language model based on the inference termination instruction, and generating a continuous inference instruction if the current inference step is not the inference termination step, controlling the continuous inference of the pre-training language model based on the continuous inference instruction.

[0006] The application further provides a model inference control device, comprising: an acquisition module, configured to acquire a probability distribution sequence of a token in a pre-training language model inference process, wherein the probability distribution sequence comprises a current probability distribution of the token in a current inference step of the pre-training language model and a historical probability distribution of the token in a historical inference step of the pre-training language model; a sorting module, configured to extract the current probability distribution and the historical probability distribution from the probability distribution sequence, identify a current ranking of a preset target token from the current probability distribution, identify a historical ranking of the target token from the historical probability distribution, and generate a ranking sequence according to the current ranking and the historical ranking; and a control module, configured to locate an inference termination step of the pre-training language model according to the ranking sequence and a preset inference termination threshold, generate an inference termination instruction if the current inference step is the inference termination step, control the pre-training language model to terminate inference based on the inference termination instruction, generate a continuous inference instruction if the current inference step is not the inference termination step, and control the pre-training language model to continuously infer based on the continuous inference instruction.

[0007] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement the steps of the model inference control method when executing the computer program.

[0008] The application further provides a nonvolatile computer readable storage medium, wherein the nonvolatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the model inference control method.

[0009] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the model inference control method.

[0010] By the present application, firstly, a sequence of token probability distributions generated by the pre-trained language model in the inference process is acquired, which includes a token current probability distribution of the current inference step and token historical probability distributions of the historical inference steps, so as to completely grasp the real-time state and historical trajectory of model inference; secondly, the current probability distribution and the historical probability distributions are extracted from the sequence of probability distributions, the rankings of the target token in the current and each historical step are identified respectively, and a ranking sequence reflecting the dynamic change rule of probability is generated based on the rankings, so that the probability distribution changes of the model at different stages in the inference process are clearly distinguished, thereby the convergence trend of the inference gradually tending to be stable can be detected; then, the stopping inference step of the pre-trained language model is located according to the ranking sequence and the pre-set inference stopping threshold, if the current inference step is the stopping inference step, a stopping inference instruction is generated and the model is controlled to stop inference based on the instruction; if the current inference step is not the stopping step, a continuing inference instruction is generated and the model is controlled to continue inference based on the instruction, so that the present application can flexibly stop or continue inference according to the real-time inference state. Therefore, the technical problem of being unable to accurately identify the convergence critical point in the related art, resulting in excessive thinking and calculation waste, can be solved, and the technical effects of reducing redundant thinking and reducing calculation overhead are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A flowchart of a model inference control method provided for the embodiments of the present application;

[0013] Figure 2 A decision flowchart of the model inference control method provided for the embodiments of the present application;

[0014] Figure 3 A schematic diagram of a model inference control device provided for the embodiments of the present application;

[0015] Figure 4 A structural schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0017] It should be noted that in the description of the present application, the term "comprising", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or inherent to such a process, method, article or apparatus. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0018] In order for those skilled in the art to better understand the present application, the following will further describe the present application in conjunction with the accompanying drawings. Figure 1 and specific embodiments.

[0019] Figure 1 A flow chart of a model inference control method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps: Figure 1

[0020] In step S101, a probability distribution sequence of a word piece in a pre-training language model inference process is obtained, wherein the probability distribution sequence comprises a current probability distribution of the word piece in the pre-training language model in a current inference step, and a historical probability distribution of the word piece in the pre-training language model in a historical inference step.

[0021] It should be noted that the pre-training language model generally represents a model that is trained based on a large-scale corpus (including sentence, paragraph and other language training materials) by designing a language model training task, training a large-scale neural network algorithm structure, learning language feature representation, and finally obtaining a neural network model containing model structure and parameters. Other tasks can be implemented on the basis of this model through feature extraction or task fine-tuning to achieve specific task purposes. The idea of pre-training is to first train a task to obtain a set of model parameters, then initialize the network model parameters using the set of model parameters, and then train other tasks using the initialized network model to obtain a model adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and extract rich syntactic and semantic information from text. The pre-training language model can provide word piece and sentence-level features containing rich semantic information for downstream tasks, or directly fine-tune the pre-training model for downstream tasks, which is convenient and fast to obtain a downstream exclusive model.

[0022] ​The neural network algorithm structure trained by the pre-trained language model can be a convolutional neural network, a recurrent neural network, a long short-term memory network, or a model constructed by an attention network such as a transformer, a bert, a GPT, a Clip, and the like, which are not limited in the present application. The attention network represents a network model trained by using an attention mechanism, which extracts more important feature information in the input sequence by giving different weights to each part of the input sequence, so that the model finally obtains more accurate output; the word element is the smallest text unit in language model processing; the probability distribution sequence is a data sequence composed of word element probability distributions in time sequence in multiple reasoning steps, which represents a combined sequence containing the current probability distribution of the current reasoning step and the historical probability distribution of the historical reasoning step in the present application; the reasoning step is a reasoning calculation process experienced by the pre-trained language model when generating a word element, which represents each calculation step from the start of the input sequence to the current reasoning time in the present application, each step corresponding to a set of probability distributions.

[0023] It can be understood that the embodiments of the present application can accurately record the generation probability of each word element at the current step and the historical step, thereby providing a data basis for calculating the current ranking and the historical ranking of the target word element; based on the ranking information, a complete historical ranking sequence can be constructed to determine whether the model reasoning process reaches the termination condition, thereby realizing real-time monitoring and control of the pre-trained language model reasoning process.

[0024] In an embodiment of the present application, before obtaining the probability distribution sequence of the word element in the pre-trained language model reasoning process, the following steps are included: obtaining the word element text output by the current reasoning step; extracting the total number of word elements and the word element list in the word element text; obtaining the number of the same word element from the word element list, calculating the current probability value of the corresponding word element according to the number of the same word element and the total number of word elements; generating the current probability distribution according to the current probability value of the word element.

[0025] The word element text is a complete text segment actually output by the pre-trained language model in the current reasoning step, which represents the original output content for counting the occurrence of word elements in the present application; the word element list is a word element sequence formed by cutting the word element text according to a preset segmentation rule, which represents a data basis for counting the occurrence times of each word element in the present application; the current probability value is the ratio of the occurrence times of a certain word element to the total number of word elements, which represents the basic calculation result of generating the current probability distribution in the present application.

[0026] It can be understood that the embodiments of the present application can quantify the uncertainty of the model to each word element at the current step by counting the word element output by the current reasoning step, calculating the occurrence probability of each word element and generating the current probability distribution, thereby providing accurate data basis for subsequent ranking calculation and probability analysis.

[0027] The inference monitoring module of the present application receives the results generated by the pre-trained language model in the inference process in real time, including the current inference step output word token text and the current probability value of each word token, and records the total number of word tokens generated cumulatively to calculate the thinking length. Based on the above information, the present application further extracts the total number of word tokens and the word token list from the word token text output by the current inference step, counts the number of the same word tokens, and calculates the current probability value of the corresponding word token according to the number of the same word tokens and the total number of word tokens to generate the current probability distribution.

[0028] For example, when the three word tokens "today", "weather" and "very good" are generated, the present application identifies the current probability value of each word token, and accumulates the thinking length as 3, while adding the current probability distribution of the three word tokens to the historical probability distribution sequence. Therefore, the present application embodiment can completely obtain the word token probability distribution sequence of the current inference and the cumulative length, providing a data basis for calculating the current ranking and historical ranking of the target word token.

[0029] In an embodiment of the present application, the probability distribution sequence of the word token in the inference process of the pre-trained language model is obtained, comprising: reading the probability distribution sequence of the word token from the cache queue, wherein the cache queue stores the current probability distribution and the historical probability distribution.

[0030] The cache queue is a data structure for storing multi-round inference data in chronological order, which represents a temporary storage unit for saving the current probability distribution and the historical probability distribution in the present application.

[0031] It can be understood that the present application embodiment realizes fast access and unified management of word token probability information by reading the current probability distribution and the historical probability distribution from the cache queue, thereby reducing the repeated calculation overhead.

[0032] Specifically, the inference monitoring module of the present application adopts a streaming data processing method, receives each generated word token in the inference process of the pre-trained language model in real time, and updates the thinking length counter immediately after receiving, which is used to record the number of word tokens generated in the current inference step, i.e. the number of word tokens generated cumulatively. For each received word token, the present application extracts its probability distribution information, which contains the generation probability of all word tokens in the large model vocabulary in the current step, which is used to determine the current probability distribution ranking of the target word token subsequently. Subsequently, the present application stores the probability distribution of the word token in the cache queue, which stores the probability distributions of the last N steps in the generation order, wherein N can be set according to actual application, which includes the probability distribution of the current step and the probability distributions of the previous N-1 historical steps, thereby forming a complete probability distribution sequence.

[0033] In a subsequent step, the current probability distribution and the historical probability distribution can be extracted from the cache queue by the application for calculating the current ranking and the historical ranking of the target token, and judging whether the termination reasoning condition is reached based on the ranking sequence, so as to realize the dynamic control of the reasoning process of the pre-trained language model. In this way, the application not only can record the generation probability of each token in the current step and the historical step, but also can provide a reliable data basis for the subsequent termination reasoning judgment.

[0034] wherein the thinking length counter is used to record the number of tokens generated in the current reasoning step; the large model vocabulary is a set of all tokens that the model can generate, and each token corresponds to a probability value.

[0035] In step S102, the current probability distribution and the historical probability distribution are extracted from the probability distribution sequence, the current ranking of the pre-set target token is identified from the current probability distribution, the historical ranking of the target token is identified from the token historical probability distribution, and the ranking sequence is generated according to the current ranking and the historical ranking.

[0036] wherein the current probability distribution is the probability distribution result given by the pre-trained language model to all tokens in the current reasoning step; the historical probability distribution is the probability distribution result generated by the pre-trained language model in the historical reasoning step; the target token is a specific token pre-set for termination judgment or state monitoring, which represents a token whose probability ranking change needs to be monitored in the application; the current ranking is the probability sorting position of the target token in the current probability distribution; the historical ranking is the probability sorting position of the target token in the historical probability distribution; and the ranking sequence is a data sequence combined by the current ranking and the historical ranking in a predetermined order.

[0037] It can be understood that by extracting the current probability distribution and the historical probability distribution and ranking the target token, the application embodiment can intuitively reflect the generation priority of each token in the current reasoning step and the historical reasoning step, which helps to capture the dynamic change trend of the token in the reasoning process.

[0038] In an embodiment of the application, before identifying the current ranking of the pre-set target token from the current probability distribution, it further includes: obtaining historical reasoning data of the pre-trained language model; extracting token data of the termination reasoning step from the historical reasoning data; and determining the target token according to the token data of the termination reasoning step.

[0039] wherein the historical reasoning data is the reasoning record data generated by the pre-trained language model in the past reasoning process; and the termination reasoning step is a step in the historical reasoning process of the pre-trained language model, which is used to judge whether the reasoning should be terminated, and represents a key step for determining the position of the termination signal in the application.

[0040] It can be understood that by using the historical inference data and the word information of the aborted inference step, the target word can be accurately determined, so as to ensure that the selected word is representative and key in subsequent ranking calculation and aborted inference judgment, improve the reliability and accuracy of the aborted inference judgment, and avoid misjudgment or inference efficiency reduction caused by random selection or non-key word.

[0041] The feature extraction module of the present application first receives the historical probability distribution sequence and the number of generated word in the current inference step. The word information of the aborted inference step is extracted from the historical inference data, and the target word of the current step is determined (for example), for the target word, the probability value of the word in the current probability distribution is first obtained, and the ranking of the word in the current probability distribution is calculated . Then, according to the inference execution sequence, the ranking of the current (step N) is combined with the N 1 step history ranking , …, to form a historical probability ranking sequence , which is used to analyze the ranking change trend of the target word. By observing the change of the sequence, such as the case of continuous decline or fluctuation, the present application can judge whether the pre-trained language model enters the inference convergence stage, so as to assist in accurately controlling the aborted inference step and avoid excessive calculation.

[0042] In an embodiment of the present application, the current ranking of the pre-set target word in the current probability distribution comprises: identifying the current probability value of the word in the current probability distribution; sorting the word based on the current probability value to obtain a first sorting result; determining the current probability distribution ranking of the word according to the first sorting result; identifying the current probability distribution ranking of the target word from the current probability distribution ranking.

[0043] Wherein, the current probability value is the generation probability of the word in the current inference step; the first sorting result is the sorting list obtained by sorting all words in the current inference step according to their current probability value from large to small, which represents the sorting basis for determining the ranking position of the word in the present application; the current probability distribution ranking is the specific sorting position of the word in the first sorting result.

[0044] It can be understood that by sorting all words in the current probability distribution and determining the ranking of the target word, the present embodiment can accurately quantify the generation possibility of the word in the current inference step, which is convenient for subsequent comparison with the historical ranking, so as to provide accurate data basis for judging whether the inference needs to be aborted.

[0045] Since the pre-trained language model actually generates a word piece that is not necessarily the word piece with the highest probability at each step in the process of generating each word piece, the present application first obtains the current probability value of each word piece from the current probability distribution for the word piece generated in the current inference step; then, based on the current probability values, all word pieces are sorted to obtain a first sorting result, which is a sequence of word pieces arranged from high to low in probability. According to the first sorting result, the present application determines the probability distribution ranking of each word piece in the current step, and further identifies the current probability distribution ranking of the pre-set target word piece (for example) from it, denoted as , which provides a reliable data basis for subsequent construction of the historical probability ranking sequence of the target word piece and judgment of the termination of the inference step.

[0046] For example, in an embodiment of the present application, the number of word pieces generated in the current inference step is ≥ 1), for ease of description, the state will be simplified as step In the present application, the current probability distribution ranking of the target word piece generated in step is defined as . = 1 indicates that the target word piece has the highest generation probability in the current probability distribution in step , and = 30 indicates that the target word piece has the 30th highest generation probability in the current probability distribution.

[0047] In an embodiment of the present application, identifying the historical ranking of the target word piece from the historical probability distribution comprises: identifying the historical probability value of the word piece from the historical probability distribution; sorting the word pieces based on the historical probability value to obtain a second sorting result; determining the historical probability distribution ranking of the word piece according to the second sorting result; and identifying the historical probability distribution ranking of the target word piece from the historical probability distribution ranking.

[0048] Wherein, the historical probability value is the generation probability of the word piece in the historical inference step; the second sorting result is a sorting list obtained by sorting all word pieces in the historical inference step according to their historical probability values from large to small, which represents the sorting basis for determining the historical ranking of the word piece in the present application; and the historical probability distribution ranking is the specific sorting position of the word piece in the second sorting result.

[0049] It can be understood that the target word piece's performance in the historical inference step can be accurately reflected by sorting the historical probability distribution and obtaining the historical ranking, which can capture the stability and trend of the word piece in the entire inference process, providing a reliable historical basis for subsequent judgment of whether the current inference step needs to be terminated.

[0050] ​For example, in the current inference step, the historical probability ranking sequence of the target token is denoted as , which is composed of the probability rankings of the tokens generated in the current step and the previous N-1 steps, and is used to describe the relative generation possibility of the target token in each step.

[0051] For example, in an embodiment of the present application, in the current inference step, the target token is assumed to be. First, the application obtains the probability distribution of the tokens generated in the current step and the previous historical steps (i.e., the historical probability distribution). Then, for each token generated in a historical step, the application extracts the probability value of the token from its probability distribution, sorts the probability values from high to low, and determines the historical ranking of the target token in the step accordingly. Repeat the above process to combine the historical rankings of the target token in the current step N and the previous N 1 steps in sequence to form the historical probability ranking sequence . The ranking sequence is used to describe the relative generation possibility of the target token in each step and its trend, thereby providing a basis for subsequent judgment of whether the model enters the inference convergence phase.

[0052] In an embodiment of the present application, the ranking sequence is generated according to the current ranking and the historical ranking, including: obtaining the execution order of the historical inference steps and the current inference step; determining the respective filling positions of the current ranking and the historical ranking according to the execution order, identifying whether at least one of the current ranking and the historical ranking is a null value; if the current ranking and the historical ranking are non-null values, filling the current ranking and the historical ranking to the corresponding filling positions of the ranking sequence; if at least one of the current ranking and the historical ranking is a null value, setting the ranking of the inference step corresponding to the null value to a preset value, and filling the preset value to the corresponding filling position of the ranking sequence.

[0053] Wherein, the filling position is a specific position reserved in the ranking sequence for the ranking of the corresponding inference step, which represents the fixed mapping position of the current ranking and the historical ranking in the sequence in the present application; the null value is a state in which there is no valid ranking data in a certain inference step, which represents that the step does not generate target token related probability or data is missing in the present application; the preset value is a default value used to replace the real ranking in the case of null value, which represents a placeholder value for uniformly filling the missing ranking to keep the length and structure of the ranking sequence consistent in the present application.

[0054] It can be understood that the embodiments of the present application can integrate the ranking of the current inference step and the ranking of the historical inference step into a complete ranking sequence according to the actual execution order, ensure that the ranking sequence has valid values at each inference step by identifying the null values in the ranking and filling in the preset values, thereby avoiding inference termination judgment errors caused by missing historical data or the current ranking not being generated. At the same time, this step enables the ranking sequence to accurately reflect the performance trend of the target word element in the entire inference process, providing a reliable data basis for subsequent termination of inference according to the ranking sequence and the termination threshold.

[0055] In an embodiment of the present application, if the current step number t is less than the historical trace depth N, special processing is performed on the missing previous steps when constructing the historical probability ranking sequence of the target word element. Specifically, the present application first obtains the execution order of the current step and the historical step, and determines the filling position of each step in the historical probability ranking sequence; then, it identifies whether there are missing previous steps in the sequence, i.e., the current ranking or the historical ranking is null; for non-null steps, the corresponding probability ranking is filled into the corresponding position of the sequence; for the missing previous steps, the present application embodiment sets the corresponding probability ranking to infinity to represent that these steps have not generated any probability information related to the target word element. Through the above operations, the completed historical probability ranking sequence can fully reflect the probability changes of the target word element in the current step and the historical step.

[0056] In step S103, the termination inference step of the pre-trained language model is located according to the ranking sequence and the pre-set inference termination threshold. If the current inference step is the termination inference step, a termination inference instruction is generated, and the termination inference of the pre-trained language model is controlled based on the termination inference instruction. If the current inference step is not the termination inference step, a continue inference instruction is generated, and the continue inference of the pre-trained language model is controlled based on the continue inference instruction.

[0057] Among them, the inference termination threshold is a preset value or interval used to determine whether the inference termination condition is reached, which represents the judgment standard for determining the termination inference step based on the ranking sequence in the present application; the termination inference step is the specific inference step determined to meet the inference termination condition, which represents the position of the termination inference; the continue inference instruction represents the control signal that allows the pre-trained language model to continue generating subsequent word elements in the present application.

[0058] It can be understood that the embodiments of the present application can accurately judge each inference step using the ranking sequence and the preset termination threshold, thereby identifying the key steps that meet the termination condition in the model inference process. By locating the termination inference step, the present application can generate control instructions for termination or continue inference according to the actual inference state, and control the inference process of the pre-trained language model according to the control instructions.

[0059] In an embodiment of the present application, before locating the step of stopping reasoning of the pre-trained language model according to the ranking sequence and the pre-set reasoning stopping threshold, further comprising: obtaining a training data set, the training data set comprising labeled samples and real stopping reasoning steps; training the pre-trained language model using the training data set, obtaining actual stopping reasoning steps of the pre-trained language model in the training process; and adjusting the reasoning stopping threshold according to the real stopping reasoning steps and the actual stopping reasoning steps.

[0060] Wherein, the training data set is a sample set for optimizing the stopping threshold; the labeled sample is a reasoning example confirmed by human in the training data set; the real stopping reasoning step is the stopping point confirmed by human in the labeled sample; and the actual stopping reasoning step is the stopping point predicted by the model according to the existing threshold in the training process, which represents the step of actually triggering the stopping reasoning of the model in the present application.

[0061] It can be understood that by training the pre-trained language model using the training data set with labeled samples and real stopping reasoning steps, the behavior data of the model in actual reasoning, i.e. the actual stopping reasoning step, can be obtained, thereby providing a basis for adjusting the pre-set reasoning stopping threshold.

[0062] The present application extracts the key features for stopping reasoning detection, i.e. the probability ranking sequence of historical word elements, from the data stream generated by the reasoning monitoring module, which is used to reflect the relative generation possibility of the target word element in each step and its trend. On this basis, combined with a small amount of labeled samples, including the word element sequence of the reasoning process of each sample, the manually labeled stopping reasoning step and the task type label, and combined with the initial rule threshold, learning and optimization are carried out.

[0063] In the optimization process, the present application compares the deviation between the actual stopping reasoning step predicted by the pre-trained language model and the labeled real stopping step, adjusts the initial rule threshold according to the deviation size, and thereby generates an optimized threshold for the current pre-trained language model and a specific task type.

[0064] In an embodiment of the present application, locating the step of stopping reasoning of the pre-trained language model according to the ranking sequence and the pre-set reasoning stopping threshold comprises: extracting the current ranking and at least one historical ranking of the target word element from the ranking sequence; obtaining a plurality of stopping thresholds from the pre-set reasoning stopping threshold; and locating the step of stopping reasoning of the pre-trained language model according to the current ranking, the at least one historical ranking and the plurality of stopping thresholds.

[0065] It can be understood that by combining the current ranking and historical ranking of the target word item and a plurality of pre-set abort thresholds, it can be accurately determined whether the model has reached the abort condition in the reasoning process, and the positioning of the abort reasoning step is realized. Considering the importance of the current word item, the historical reasoning information is taken into account, which makes the abort judgment more stable and reliable; at the same time, through the hierarchical strategy of multiple thresholds, different situations can be flexibly handled, and false positives caused by a single indicator can be avoided.

[0066] The threshold fine-tuning module of the present application first prepares labeled samples, selects samples, each sample contains an artificially confirmed abort point position, and records the historical probability ranking sequence of the sample at the abort point. For example, the abort point position of sample 1 is =350, and the historical probability ranking sequence is = [6, 12, 18, 25, 35, 45], which describes the change trend of the target word item at each step. Then the abort threshold is initialized, and the default multi-level initial threshold is used to detect the abort point of the samples, and the predicted abort point is determined according to the relationship between the current ranking and each threshold in combination with the historical ranking sequence. Taking sample 1 as an example, the predicted abort step is =360, while the artificial labeling is =350, so the deviation is 10. Calculate the average deviation of all samples, and adjust the abort threshold to minimize the average deviation, thereby optimizing the rule parameters. Through this process, the present application can generate optimized thresholds for the current pre-trained language model and specific task types, improve the accuracy and adaptability of abort point detection, and realize reasoning control in cross-task and cross-model scenarios.

[0067] In an embodiment of the present application, according to the current ranking, at least one historical ranking and a plurality of abort thresholds, the abort inference step of the pre-trained language model is located, comprising: obtaining a plurality of first abort thresholds, second abort thresholds, third abort thresholds and fourth abort thresholds from the plurality of abort thresholds, the second abort threshold being greater than the first abort threshold, the third abort threshold being greater than the second abort threshold, and the fourth abort threshold being greater than the third abort threshold; if the current ranking is less than or equal to the first abort threshold, it is determined that the current inference step is an abort inference step, and if the current ranking is greater than the first abort threshold, it is determined whether the current ranking is less than or equal to the second abort threshold; if the current ranking is less than or equal to the second abort threshold, the abort inference step of the current inference step is located according to the plurality of historical rankings, if the current ranking is greater than the second abort threshold, it is determined whether the current ranking is less than or equal to the third abort threshold; if the current ranking is less than or equal to the third abort threshold, the abort inference step of the current inference step is located according to the plurality of historical rankings, if the current ranking is greater than the third abort threshold, it is determined whether the current ranking is less than or equal to the fourth abort threshold; if the current ranking is less than or equal to the fourth abort threshold, the abort inference step of the current inference step is located according to the plurality of historical rankings, and if the current ranking is greater than the fourth abort threshold, it is determined that the current inference step is not an abort inference step.

[0068] Among them, the first abort threshold represents the highest sensitivity level in the application, which is used to determine the abort inference; the second abort threshold represents the abort determination threshold suitable for considering the historical ranking; the third abort threshold represents the threshold suitable for the more relaxed historical ranking determination abort; the fourth abort threshold represents the threshold that allows the longest time to observe the historical ranking to still determine the abort.

[0069] It can be understood that, by setting a plurality of hierarchical abort thresholds and combining the current ranking and the historical ranking for layered judgment, the embodiments of the present application can realize fine control of the abort step in the inference process of the pre-trained language model. The first to fourth abort thresholds form a judgment interval from strict to relaxed, so that the model can flexibly determine whether to abort the inference according to the importance of the current token and the historical trend; if the current ranking is below the strict threshold, the inference can be aborted immediately to improve efficiency; if the current ranking is in the middle threshold interval, the historical ranking is combined for judgment to avoid misjudgment; if the current ranking exceeds the highest threshold, the inference is continued to ensure the integrity of the inference.

[0070] In the embodiments of the present application, the abort point decision module extracts the probability ranking sequence of the historical tokens from the data stream output by the inference monitoring module is implemented as a feature. The decision is based on pre-set rule strategies, which are verified in order of core rules from high to low, and the threshold is the default value, which can be adjusted by the user according to the characteristics of the pre-trained language model and the data characteristics.

[0071] Specific strategies include: strategy 1 is a current ranking priority strategy, if the token ranking of the current step ≤5 (i.e. [0]≤5) is determined to be the reasoning step; strategy 2 is a recent ranking ladder strategy, if the current step ranking ≤10, and the previous 1 step ranking ≤50, the previous 2 step ranking ≤100, the previous 3 step ranking ≤1000 (i.e. [0]≤10 and [1]≤50 and [2]≤100 and [3]≤1000) is determined to be the reasoning step; strategy 3 is a short-term ranking stability strategy, if the current step and the previous 2 steps are less than or equal to 20 (i.e. [0]≤20 and [1]≤20 and [2]≤20) is determined to be the reasoning step; strategy 4 is a mid-term ranking stability strategy, if the current step and the previous 5 steps are less than or equal to 50 (i.e. [0]≤50 and [1]≤50 and [2]≤50 and [3]≤50 and [4]≤50 and [5]≤50) is determined to be the reasoning step. In turn, verify strategy 1 to strategy 4, if any one of the strategies is met, immediately output the decision result of "stop reasoning step"; if all strategies are not met, output the decision result of "continue reasoning".

[0072] In addition, the strategy design of the present application is not randomly established, but is based on the analysis of the reasoning dynamic characteristics of the pre-trained language model, and the general principles of decision logic design. The essence of stopping the reasoning step is that the model enters the critical node from the "reflection reasoning phase" to the "reasoning convergence phase", at this time the model has formed a complete reasoning framework and generated a clear conclusion, and continuing reasoning may cause overthinking.

[0073] In the insufficient exploration phase, the model has not formed a complete logic, and believes that it needs to continue to think, the probability of the target token is very low, and the ranking is usually at the back. In the compensation reasoning phase, the model gradually corrects the logical loopholes, the judgment of "whether the thinking is complete" fluctuates greatly, and the ranking of the target token shows an upward trend but is unstable. In the reasoning convergence phase, the model confirms that the reasoning is complete, the probability of the target token significantly increases and the ranking is stable in the front position, at this time triggering the stop reasoning step can avoid redundancy.

[0074] In the embodiments of the present application, the decision to suspend the reasoning step is achieved by analyzing the data stream output by the reasoning monitoring module, and the core feature is the historical probability ranking sequence of the target token (for example) in each reasoning step . This sequence reflects the relative generation possibility and change trend of the target token in the current step N and the previous N 1 steps, which is used to determine whether the model has entered the reasoning convergence stage. The decision to suspend reasoning is based on preset rule strategies, which are verified in turn according to the core rule from high to low priority. The threshold used in the strategy is the default initial value, which can be adjusted according to the characteristics of the pre-trained language model and the task data characteristics.

[0075] Assuming that the first suspension threshold is 5, the second suspension threshold is 10, the third suspension threshold is 20, and the fourth suspension threshold is 50, the specific strategies include:

[0076] Strategy 1 (current ranking priority strategy): if the ranking of the target token in the current step ≤5 (i.e. [0]≤5), it is determined that the current reasoning step is a suspended reasoning step;

[0077] Strategy 2 (recent ranking ladder strategy): if ≤10, and the historical rankings ≤50, ≤100, ≤1000 (i.e. [0]≤10 and [1]≤50 and [2]≤100 and [3]≤1000), it is determined that the current reasoning step is a suspended reasoning step;

[0078] Strategy 3 (short-term ranking stability strategy): if the rankings of the current step and the previous two steps are all less than or equal to 20 (i.e. [0]≤20 and [1]≤20 and [2]≤20), it is determined that the current reasoning step is a suspended reasoning step;

[0079] Strategy 4 (mid-term ranking stability strategy): if the rankings of the current step and the previous five steps are all less than or equal to 50 (i.e. [0]≤50 and [1]≤50 and [2]≤50 and [3]≤50 and [4]≤50 and [5]≤50), it is determined that the current reasoning step is a suspended reasoning step.

[0080] The policy verification can be executed in priority order, if any policy is met, the inference step is immediately outputted to be aborted; if all policies are not met, the inference step is outputted to be continued.

[0081] The above policy design is not randomly set, but is determined based on the analysis of the inference dynamic characteristics of the pre-trained language model and the general principles of decision logic. The essence of the abort point is the critical node of the model from the "reflection reasoning phase" to the "reasoning convergence phase". At this time, the model has formed a complete reasoning framework and generated a clear conclusion, and continuing reasoning may cause redundant or excessive calculation. The probability and ranking changes of the target token in each step directly reflect the dynamic changes of the model reasoning state. In the insufficient exploration phase, the probability of the target token is very low, and the ranking is at the back; in the compensation reasoning phase, the model gradually corrects the logic, and the ranking of the target token rises but is unstable; in the reasoning convergence phase, the probability of the target token significantly increases and the ranking is stable and at the front. At this time, triggering the abort reasoning can avoid redundant calculation.

[0082] In an embodiment of the present application, if the current ranking is less than or equal to the second abort threshold, the abort reasoning step of the current inference step is located according to the plurality of historical rankings, comprising: obtaining a target gradient condition for meeting the abort reasoning step, wherein the target gradient condition comprises that the current ranking is less than the previous historical ranking, and the previous historical ranking is greater than the next historical ranking in the plurality of historical rankings; if it is identified that the current ranking and the plurality of historical rankings meet the target gradient condition, it is determined that the current inference step is the abort reasoning step, and if it is identified that at least one of the current ranking and the plurality of historical rankings does not meet the target gradient condition, it is determined whether the current ranking is less than or equal to the third abort threshold.

[0083] The target gradient condition is a condition for judging the ranking change trend of the current step and the historical step.

[0084] It can be understood that by introducing the target gradient condition, the change trend of the current ranking and the historical ranking is considered in the determination basis of the abort reasoning, so that the determination depends not only on a single threshold, but also considers the dynamic evolution of the model reasoning state. In this way, the critical point of the transition of the model from the reflection reasoning phase to the reasoning convergence phase can be more accurately identified, and premature or late abortion can be effectively prevented, thereby reducing redundant reasoning steps and computational resource consumption while ensuring reasoning accuracy.

[0085] In an embodiment of the present application, the design of strategy 2 (recent ranking ladder strategy) is based on the fact that when the pre-trained language model enters the reasoning convergence phase, the probability ranking of the target token (for example) does not immediately rise to the top, but presents a step-by-step descending trend. To this end, if the current ranking does not meet the first suspension threshold, but is less than or equal to the second suspension threshold, the present application does not directly determine the current reasoning step as a suspended reasoning step, but further combines multiple historical rankings to structurally verify the trend.

[0086] Specifically, under the premise of meeting the condition that the current ranking is less than or equal to the second suspension threshold, the present application obtains a target gradient condition for determining the suspended reasoning step, which is composed of the relative size relationship between multiple consecutive historical rankings, to reflect whether the target token ranking presents a stable and monotonic rising trend. The target gradient condition includes: the current ranking is less than the previous historical ranking, and the previous historical ranking is greater than the next historical ranking among the multiple historical rankings, that is , and When it is identified that the current ranking and the multiple historical rankings all meet the target gradient condition, it is determined that the current reasoning step is a suspended reasoning step, thereby avoiding the suspension delay caused by the strict threshold of strategy 1 and improving the reasoning efficiency.

[0087] If it is identified that at least one of the current ranking and the multiple historical rankings does not meet the target gradient condition, it indicates that the present application has not formed a stable convergent probability structure, and the current does not meet the triggering condition of strategy 2. The present application continues to determine whether the current ranking is less than or equal to the third suspension threshold, thereby entering the next level of suspension judgment process.

[0088] In an embodiment of the present application, if the current ranking is less than or equal to the third suspension threshold, the suspended reasoning step of the current reasoning step is located according to the multiple historical rankings, including: obtaining a first number of historical rankings from the multiple historical rankings; if the first number of historical rankings are all less than or equal to the third suspension threshold, it is determined that the current reasoning step is a suspended reasoning step, and if at least one of the first number of historical rankings is greater than the third suspension threshold, it is determined whether the current ranking is less than or equal to the fourth suspension threshold.

[0089] It can be understood that by counting the number of historical rankings that meet the threshold condition, the overall consideration of historical reasoning trend is introduced, rather than relying on a single ranking. In this way, the suspended reasoning step can be more stably located, the risk of misjudgment caused by single fluctuation can be reduced, and premature or late suspension can be effectively avoided, thereby reducing redundant calculation while maintaining model reasoning accuracy, improving reasoning efficiency and resource utilization.

[0090] ​​In the present application, the design logic of strategy 3 is as follows: for part of the reasoning task, even if the pre-trained language model enters the reasoning convergence stage, the probability ranking of the target word element may still fluctuate slightly within a certain interval. This fluctuation is mainly due to the self-checking or self-doubt mechanism existing in the model during the generation process, for example, the model may evaluate "whether there is a more concise or better derivation path". Although there is such fluctuation, the core reasoning logic of the model has been completely established, and the generated conclusion is basically clear.

[0091] If only strategy 1 (current ranking priority strategy) or strategy 2 (recent ranking ladder strategy) is relied on, it may not be able to capture the suspension signal within the fluctuation range in time, resulting in delay in judging the suspension reasoning step and generating irrelevant word elements, increasing the calculation cost. Therefore, strategy 3 combines the target word element ranking of the current step with the historical rankings of the previous two steps through the short-term ranking stability strategy , to identify the reasoning convergence state within the allowed fluctuation interval, thereby achieving accurate judgment of the suspension reasoning step and effectively avoiding unnecessary over-reasoning.

[0092] Therefore, in an embodiment of the present application, a third suspension threshold judgment mechanism based on short-term ranking stability is introduced: if the target word element ranking of the current step is less than or equal to the third suspension threshold, short-term stability detection is triggered. The detection obtains a first number of historical rankings, such as the rankings of the previous two steps , from multiple historical rankings. If the first number of historical rankings are all less than or equal to the third suspension threshold, it indicates that the ranking has remained stable within the allowed fluctuation range, and thus it can be determined that the current reasoning step is the suspension reasoning step.

[0093] On the contrary, if at least one of the above historical rankings is greater than the third suspension threshold, it means that the ranking has not stabilized within the expected interval. At this time, it is further judged whether the current ranking is less than or equal to the fourth suspension threshold; if so, it is still determined that the current step is the suspension reasoning step, otherwise it is considered that it has not converged.

[0094] In an embodiment of the present application, if the current ranking is less than or equal to the fourth suspension threshold, the suspension reasoning step of the current reasoning step is located according to multiple historical rankings, including: obtaining a second number of historical rankings from multiple historical rankings, the second number being greater than the first number; if the second number of historical rankings are all less than or equal to the fourth suspension threshold, it is determined that the current reasoning step is the suspension reasoning step, and if at least one of the second number of historical rankings is greater than the fourth suspension threshold, it is determined that the current reasoning step is not the suspension reasoning step.

[0095] It can be understood that the ability to judge the reasoning trend is further enhanced by introducing more historical ranking data. Compared with relying only on a small amount of historical rankings, the present application can more accurately identify whether the model has truly entered the reasoning convergence stage, reduce the risk of misjudgment caused by single or local fluctuations, and thus effectively avoid premature or late termination while ensuring reasoning accuracy.

[0096] In the present application, for a small part of complex or special tasks, the pre-trained language model may still have a probability ranking of the target word element Continuous repeated fluctuations, that is, the model continuously evaluates whether the thinking has been completed during the generation process, resulting in difficulty in natural termination of reasoning, and even the risk of infinite reasoning or generation of redundant word elements. To solve this problem, the present application further proposes strategy 4, that is, the mid-term ranking stability strategy, which identifies the reasoning convergence state by introducing ranking stability judgment of a longer time span.

[0097] Specifically, in an embodiment of the present application, if the ranking of the current step is less than or equal to the fourth termination threshold, the mid-term stability detection process is triggered. The process obtains a second number of historical rankings from a plurality of historical rankings, for example, the rankings of the previous five steps 、 、 、 、 , wherein the second number is greater than the first number used in strategy 3.

[0098] If the second number of historical rankings are all less than or equal to the fourth termination threshold, it means that the ranking of the target word element has been continuously maintained within the allowed fluctuation range in a longer window, the reasoning logic of the model has been stable and the conclusion has been basically formed, and therefore the current reasoning step can be determined as the termination reasoning step. On the contrary, if there is at least one ranking greater than the fourth termination threshold in the above historical rankings, it means that the model still has significant fluctuations in the mid-term time range, and the reasoning has not completely converged, and therefore the current reasoning step is determined to be not the termination reasoning step.

[0099] Through the above mid-term ranking stability strategy, strategy 4 can effectively make up for the shortcomings of strategies 1 to 3 in long-term stability judgment, ensure that the model can be terminated in time when facing complex reasoning tasks, and avoid generating redundant word elements, thereby avoiding the problems of excessive reasoning and infinite loop.

[0100] In an embodiment of the present application, before locating the stop reasoning step of the pre-trained language model according to the ranking sequence and the pre-set reasoning stop threshold, it further comprises: obtaining a training data set, the training data set comprising labeled samples and real stop reasoning steps; training the pre-trained language model using the training data set to obtain the actual stop reasoning step of the pre-trained language model in the training process; and adjusting the reasoning stop threshold according to the real stop reasoning step and the actual stop reasoning step.

[0101] Wherein, the stop reasoning instruction represents a control signal triggering the pre-trained language model to immediately stop the current reasoning process in the present application.

[0102] It can be understood that the present application realizes accurate control of the reasoning process of the pre-trained language model by generating the stop reasoning instruction or the continue reasoning instruction. If the current step is the stop reasoning step, the model ends the thinking process to avoid redundant reasoning; otherwise, the pre-trained language model continues to generate the next word element to ensure the continuity of reasoning.

[0103] The present application receives the result of the stop point decision module and controls the reasoning process of the pre-trained language model based on the result: if the decision determines that the current reasoning step is the stop reasoning step, a stop reasoning instruction is generated, and a thinking process stop symbol is additionally added at the end of the current thinking process to control the stop reasoning of the pre-trained language model based on the stop reasoning instruction to make it generate the final conclusion; if the decision determines that the current reasoning step is not the stop reasoning step, a continue reasoning instruction is generated, and the pre-trained language model is controlled to continue generating the next word element based on the continue reasoning instruction to maintain the continuation of the thinking process, thereby outputting the final reasoning content.

[0104] According to the model inference control method provided in the embodiments of the present application, firstly, a token probability distribution sequence generated by the pre-trained language model in the inference process is obtained, which includes a token current probability distribution of the current inference step and token historical probability distributions of the historical inference steps, so that the real-time state and historical trajectory of the model inference are completely mastered; secondly, the current probability distribution and the historical probability distributions are extracted from the probability distribution sequence, the rankings of the target token in the current step and each historical step are identified respectively, and a ranking sequence reflecting the dynamic change rule of the probability is generated based on the rankings, so that the probability distribution changes of the model at different stages in the inference process are clearly distinguished, and the convergence trend of the inference gradually tending to be stable can be detected; then, the inference termination step of the pre-trained language model is located according to the ranking sequence and the pre-set inference termination threshold, if the current inference step is the inference termination step, an inference termination instruction is generated and the model is controlled to terminate the inference based on the instruction; if the current inference step is not the termination step, a continue inference instruction is generated and the model is controlled to continue the inference based on the instruction, so that the present application can flexibly stop or continue the inference according to the real-time inference state. Therefore, the technical problem that the convergence critical point cannot be accurately identified in the related art, resulting in excessive thinking and calculation waste, can be solved, and the technical effects of reducing redundant thinking and reducing calculation overhead are achieved.

[0105] In summary, the overall working mechanism of the present application is as shown in Figure 2 Firstly, the inference monitoring module monitors the state changes or output data in the inference process in real time, and inputs the monitoring information into the feature extraction module; the feature extraction module extracts key features such as statistical indicators, trend information or pattern features from the monitoring data, providing reliable basis for subsequent decision-making. Subsequently, the termination point decision module determines whether the inference process needs to be terminated in advance according to the extracted features and the current threshold. In order to improve the accuracy and adaptability of the decision, the present application also sets a threshold fine-tuning module to dynamically adjust the threshold through error feedback, learning mechanism or external input, and real-time feedback the adjusted threshold to the termination point decision module, thereby forming a closed-loop optimization structure. Finally, when the decision module determines that the termination condition is met, the inference termination and output module will immediately terminate the current inference process and output the result; if the termination condition has not been triggered, the inference will continue or return to the monitoring stage, forming a robust and self-adaptive inference control process.

[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0107] The embodiments of the present application also provide a model inference control device, Figure 3A structural schematic diagram of a model inference control device provided in an embodiment of the present application is shown in Figure 3 The model inference control device 10 includes an acquisition module 100, a sorting module 200, and a control module 300.

[0108] The acquisition module 100 is configured to acquire a probability distribution sequence of a token in a pre-training language model inference process, where the probability distribution sequence includes a current probability distribution of the token in a current inference step of the pre-training language model and a historical probability distribution of the token in a historical inference step of the pre-training language model. The sorting module 200 is configured to extract the current probability distribution and the historical probability distribution from the probability distribution sequence, identify a current ranking of a target token from the current probability distribution, identify a historical ranking of the target token from the historical probability distribution, and generate a ranking sequence according to the current ranking and the historical ranking. The control module 300 is configured to locate an inference termination step of the pre-training language model according to the ranking sequence and a pre-set inference termination threshold, generate an inference termination instruction if the current inference step is the inference termination step, control the pre-training language model to terminate the inference based on the inference termination instruction, generate a continuous inference instruction if the current inference step is not the inference termination step, and control the pre-training language model to continuously infer based on the continuous inference instruction.

[0109] In an embodiment of the present application, the model inference control device further includes a calculation module configured to acquire a token text output by the current inference step before acquiring the probability distribution sequence of the token in the pre-training language model inference process, extract a total number of tokens and a token list from the token text, acquire a number of identical tokens from the token list, calculate a current probability value of the corresponding token according to the number of identical tokens and the total number of tokens, and generate the current probability distribution according to the current probability value of the token.

[0110] In an embodiment of the present application, the sorting module 200 is further configured to identify a current probability value of the token from the current probability distribution, sort the token based on the current probability value to obtain a first sorting result, determine a current probability distribution ranking of the token according to the first sorting result, and identify the current probability distribution ranking of the target token from the current probability distribution ranking.

[0111] In an embodiment of the present application, the sorting module 200 is further configured to identify a historical probability value of the token from the historical probability distribution, sort the token based on the historical probability value to obtain a second sorting result, determine a historical probability distribution ranking of the token according to the second sorting result, and identify the historical probability distribution ranking of the target token from the historical probability distribution ranking.

[0112] In an embodiment of the present application, the ranking module 200 is further configured to obtain an execution order of the historical reasoning step and the current reasoning step; determine respective filling positions of the current ranking and the historical ranking according to the execution order, identify whether at least one of the current ranking and the historical ranking is a null value; if the current ranking and the historical ranking are not null values, fill the current ranking and the historical ranking into the corresponding filling positions of the ranking sequence; and if at least one of the current ranking and the historical ranking is a null value, set the ranking of the reasoning step corresponding to the null value to a preset value, and fill the preset value into the corresponding filling position of the ranking sequence.

[0113] In an embodiment of the present application, the determining module is further configured to obtain historical reasoning data of the pre-trained language model before identifying the current ranking of the pre-set target token from the current probability distribution; extract token data of the aborted reasoning step from the historical reasoning data; and determine the target token according to the token data of the aborted reasoning step.

[0114] In an embodiment of the present application, the control module 300 is further configured to extract the current ranking and at least one historical ranking of the target token from the ranking sequence; obtain a plurality of aborted threshold values from the pre-set reasoning aborted threshold value, and locate the aborted reasoning step of the pre-trained language model according to the current ranking, the at least one historical ranking, and the plurality of aborted threshold values.

[0115] In an embodiment of the present application, the control module 300 is further configured to obtain a plurality of first aborted threshold values, second aborted threshold values, third aborted threshold values, and fourth aborted threshold values from the plurality of aborted threshold values, the second aborted threshold values being greater than the first aborted threshold values, the third aborted threshold values being greater than the second aborted threshold values, and the fourth aborted threshold values being greater than the third aborted threshold values; if the current ranking is less than or equal to the first aborted threshold value, determine that the current reasoning step is the aborted reasoning step; if the current ranking is greater than the first aborted threshold value, determine whether the current ranking is less than or equal to the second aborted threshold value; if the current ranking is less than or equal to the second aborted threshold value, locate the aborted reasoning step of the current reasoning step according to the plurality of historical rankings; if the current ranking is greater than the second aborted threshold value, determine whether the current ranking is less than or equal to the third aborted threshold value; if the current ranking is less than or equal to the third aborted threshold value, locate the aborted reasoning step of the current reasoning step according to the plurality of historical rankings; if the current ranking is greater than the third aborted threshold value, determine whether the current ranking is less than or equal to the fourth aborted threshold value; if the current ranking is less than or equal to the fourth aborted threshold value, locate the aborted reasoning step of the current reasoning step according to the plurality of historical rankings; and if the current ranking is greater than the fourth aborted threshold value, determine that the current reasoning step is not the aborted reasoning step.

[0116] In an embodiment of the present application, the control module 300 is further configured to obtain a target gradient condition for determining the suspension reasoning step, wherein the target gradient condition comprises that the current ranking is less than the previous historical ranking, and the previous historical ranking is greater than the next historical ranking in the plurality of historical rankings; if it is identified that the current ranking and the plurality of historical rankings satisfy the target gradient condition, it is determined that the current reasoning step is the suspension reasoning step; if it is identified that at least one of the current ranking and the plurality of historical rankings does not satisfy the target gradient condition, it is determined whether the current ranking is less than or equal to a third suspension threshold.

[0117] In an embodiment of the present application, the control module 300 is further configured to obtain a first number of historical rankings from the plurality of historical rankings; if the first number of historical rankings are all less than or equal to the third suspension threshold, it is determined that the current reasoning step is the suspension reasoning step; if at least one of the first number of historical rankings is greater than the third suspension threshold, it is determined whether the current ranking is less than or equal to a fourth suspension threshold.

[0118] In an embodiment of the present application, the control module 300 is further configured to obtain a second number of historical rankings from the plurality of historical rankings, the second number being greater than the first number; if the second number of historical rankings are all less than or equal to the fourth suspension threshold, it is determined that the current reasoning step is the suspension reasoning step; if at least one of the second number of historical rankings is greater than the fourth suspension threshold, it is determined that the current reasoning step is not the suspension reasoning step.

[0119] In an embodiment of the present application, further comprising: an adjustment module configured to obtain a training data set before locating the suspension reasoning step of the pre-trained language model according to the ranking sequence and the pre-set reasoning suspension threshold, the training data set comprising a labeled sample and a real suspension reasoning step; training the pre-trained language model using the training data set to obtain an actual suspension reasoning step of the pre-trained language model in the training process; and adjusting the reasoning suspension threshold according to the real suspension reasoning step and the actual suspension reasoning step.

[0120] The model inference control device provided in the embodiments of the present application firstly acquires a token probability distribution sequence generated by the pre-trained language model in the inference process, which includes a token current probability distribution of the current inference step and token historical probability distributions of the historical inference steps, so as to completely grasp the real-time state and historical trajectory of the model inference; secondly, the current probability distribution and the historical probability distributions are extracted from the probability distribution sequence, the rankings of the target token in the current step and each historical step are respectively identified, and a ranking sequence reflecting the dynamic change rule of the probability is generated based on the rankings, so that the probability distribution changes of different stages of the model in the inference process are clearly distinguished, and the convergence trend of the inference gradually tending to be stable can be detected; then, the inference termination step of the pre-trained language model is located according to the ranking sequence and the pre-set inference termination threshold, if the current inference step is the inference termination step, an inference termination instruction is generated and the model is controlled to terminate the inference based on the instruction; if the current inference step is not the termination step, a continue inference instruction is generated and the model is controlled to continue the inference based on the instruction, so that the present application can flexibly stop or continue the inference according to the real-time inference state. Therefore, the technical problem of being unable to accurately identify the convergence critical point in the related art, resulting in excessive thinking and calculation waste, can be solved, and the technical effects of reducing redundant thinking and reducing calculation overhead are achieved.

[0121] The features of the embodiments corresponding to the model inference control device can be referred to the related descriptions of the embodiments of the model inference control method, which will not be repeated here.

[0122] The embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, Figure 4 The electronic device includes a memory 401 and a processor 402, the memory 401 stores a computer program 403, and the processor 402 is configured to run the computer program 403 to execute the steps in the above-mentioned model inference control method embodiments.

[0123] The embodiments of the present application also provide a non-volatile computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above-mentioned model inference control method embodiments when running.

[0124] In an exemplary embodiment, the above-mentioned computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0125] Embodiments of the present application also provide a computer program product, the computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in the model inference control method embodiments.

[0126] Those skilled in the art will further appreciate that the functions of the examples described herein-based units and algorithm steps can be implemented using electronic hardware, computer software, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein generally in terms of their functionality, without necessarily describing the corresponding detailed hardware or software mechanisms that implement that functionality. Those skilled in the art will appreciate that the mechanisms that implement the examples described herein are not limited to the examples described herein, but can be implemented using any number of techniques, including those described herein.

[0127] The above has carried out the detailed introduction to the model inference control method provided by the present application. The principle and implementation mode of the present application are described herein by applying specific examples, and the above example description is only applicable to help understand the method and core idea of the present application. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principle of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model inference control method, characterized in that, include: The method involves obtaining a sequence of probability distributions of lexical units during the inference process of a pre-trained language model. This sequence includes the current probability distribution of lexical units in the current inference step and the historical probability distributions of lexical units in previous inference steps. Before obtaining the sequence of probability distributions of lexical units during the inference process, the method further includes: obtaining the lexical text output by the current inference step; extracting the total number of lexical units and a lexical list from the lexical text; obtaining the number of identical lexical units from the lexical list; calculating the current probability value of the corresponding lexical unit based on the number of identical lexical units and the total number of lexical units; and generating the current probability distribution based on the current probability value of the lexical unit. Extract the current probability distribution and historical probability distribution from the probability distribution sequence, identify the current ranking of the target word from the current probability distribution, identify the historical ranking of the target word from the historical probability distribution, and generate a ranking sequence based on the current ranking and the historical ranking; The inference termination step of the pre-trained language model is located based on the ranking sequence and the pre-set inference termination threshold. If the current inference step is the inference termination step, an inference termination instruction is generated, and the inference termination of the pre-trained language model is controlled based on the inference termination instruction. If the current inference step is not the inference termination step, an inference continuation instruction is generated, and the inference continuation of the pre-trained language model is controlled based on the inference continuation instruction.

2. The model inference control method according to claim 1, characterized in that, The step of identifying the current ranking of a pre-set target word from the current probability distribution includes: Identify the current probability value of a word from the current probability distribution; The lexical units are sorted based on the current probability values ​​to obtain a first sorting result; The current probability distribution ranking of the word element is determined based on the first ranking result; Identify the current probability distribution ranking of the target word from the current probability distribution ranking.

3. The model inference control method according to claim 1, characterized in that, The step of identifying the historical ranking of the target word from the historical probability distribution includes: Identify the historical probability values ​​of lexical units from the historical probability distribution; The lexical units are sorted based on the historical probability values ​​to obtain a second sorting result; The historical probability distribution ranking of word elements is determined based on the second ranking result; Identify the historical probability distribution ranking of the target word from the historical probability distribution ranking.

4. The model inference control method according to claim 1, 2 or 3, characterized in that, The step of generating a ranking sequence based on the current ranking and the historical ranking includes: Obtain the execution order of the historical reasoning steps and the current reasoning step; The filling positions of the current ranking and the historical ranking are determined according to the execution order, and it is identified whether at least one of the current ranking and the historical ranking is null. If the current ranking and the historical ranking are not empty, then the current ranking and the historical ranking are filled into the corresponding filling positions of the ranking sequence; If at least one of the current ranking and the historical ranking is null, the ranking of the reasoning step corresponding to the null value is set to a preset value, and the preset value is filled into the corresponding filling position of the ranking sequence.

5. The model inference control method according to claim 1, 2 or 3, characterized in that, Before identifying the current ranking of a pre-set target word from the current probability distribution, the method further includes: Obtain the historical inference data of the pre-trained language model; Extract the metadata of the aborted reasoning step from the historical reasoning data; The target lexical element is determined based on the lexical data of the aborted reasoning step.

6. The model inference control method according to claim 1, characterized in that, The step of locating the termination of inference of the pre-trained language model based on the ranking sequence and a pre-set inference termination threshold includes: Extract the current ranking and at least one historical ranking of the target word from the ranking sequence; Multiple termination thresholds are obtained from the pre-set inference termination thresholds, and the termination inference step of the pre-trained language model is located based on the current ranking, at least one of the historical rankings, and the multiple termination thresholds.

7. The model inference control method according to claim 6, characterized in that, The step of locating the termination inference step of the pre-trained language model based on the current ranking, at least one of the historical rankings, and multiple termination thresholds includes: Multiple termination thresholds are obtained from a plurality of termination thresholds, namely a first termination threshold, a second termination threshold, a third termination threshold, and a fourth termination threshold, wherein the second termination threshold is greater than the first termination threshold, the third termination threshold is greater than the second termination threshold, and the fourth termination threshold is greater than the third termination threshold; If the current ranking is less than or equal to the first termination threshold, then the current reasoning step is determined to be the terminated reasoning step; if the current ranking is greater than the first termination threshold, then it is determined whether the current ranking is less than or equal to the second termination threshold. If the current ranking is less than or equal to the second termination threshold, then the termination reasoning step of the current reasoning step is located based on multiple historical rankings; if the current ranking is greater than the second termination threshold, then it is determined whether the current ranking is less than or equal to the third termination threshold. If the current ranking is less than or equal to the third termination threshold, then the termination reasoning step of the current reasoning step is located based on multiple historical rankings; if the current ranking is greater than the third termination threshold, then it is determined whether the current ranking is less than or equal to the fourth termination threshold. If the current ranking is less than or equal to the fourth termination threshold, the termination reasoning step of the current reasoning step is located based on multiple historical rankings. If the current ranking is greater than the fourth termination threshold, the current reasoning step is determined not to be the termination reasoning step.

8. The model inference control method according to claim 7, characterized in that, If the current ranking is less than or equal to the second abort threshold, then the abort inference step of the current inference step is located based on multiple historical rankings, including: Obtain the target gradient condition that satisfies the termination reasoning step, wherein the target gradient condition includes the current ranking being less than the previous historical ranking, and the previous historical ranking being greater than the next historical ranking among the multiple historical rankings; If the current ranking and multiple historical rankings are found to satisfy the target gradient condition, then the current inference step is determined to be the terminated inference step. If at least one of the current ranking and multiple historical rankings is found not to satisfy the target gradient condition, then it is determined whether the current ranking is less than or equal to the third termination threshold.

9. The model inference control method according to claim 7, characterized in that, If the current ranking is less than or equal to the third abort threshold, then the abort inference step of the current inference step is located based on multiple historical rankings, including: Obtain the first number of historical rankings from the multiple historical rankings mentioned above; If all historical rankings of the first quantity are less than or equal to the third termination threshold, then the current reasoning step is determined to be the terminated reasoning step. If at least one historical ranking of the first quantity is greater than the third termination threshold, then it is determined whether the current ranking is less than or equal to the fourth termination threshold.

10. The model inference control method according to claim 9, characterized in that, If the current ranking is less than or equal to the fourth abort threshold, then the abort inference step of the current inference step is located based on multiple historical rankings, including: Obtain a second number of historical rankings from the plurality of historical rankings, wherein the second number is greater than the first number; If all historical rankings of the second quantity are less than or equal to the fourth termination threshold, then the current reasoning step is determined to be the terminated reasoning step; if at least one historical ranking of the second quantity is greater than the fourth termination threshold, then the current reasoning step is determined not to be the terminated reasoning step.

11. The model inference control method according to claim 1, characterized in that, Before locating the abort inference step of the pre-trained language model based on the ranking sequence and a pre-set inference abort threshold, the method further includes: Obtain a training dataset, which includes labeled samples and real aborted inference steps; The pre-trained language model is trained using the training dataset, and the actual inference steps that the pre-trained language model stops during the training process are obtained. The reasoning termination threshold is adjusted based on the actual reasoning termination steps and the actual reasoning termination steps.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model inference control method as described in any one of claims 1 to 11.

13. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model inference control method as described in any one of claims 1 to 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the model inference control method as described in any one of claims 1 to 11.

Images