Model reasoning method, system and equipment and computer readable storage medium

Abstract

The invention discloses a model reasoning method, system and device and a computer readable storage medium, and relates to the technical field of artificial intelligence. Screening out an intermediate layer meeting a skipping condition as a candidate layer; generating reasoning information of the big language model skipping the candidate layer for reasoning; screening out a target layer from the candidate layers according to the reasoning information; obtaining a to-be-processed text; inputting the to-be-processed text into the large language model skipping the target layer to obtain a candidate sequence; and inputting the to-be-processed text and the candidate sequence into the large language model to obtain a target reasoning result. According to the method, the target layer can be matched with the reasoning information for reasoning of the large language model, so that the reasoning accuracy of the large language model is improved; moreover, speculation sampling is carried out by means of the large language model, other models do not need to be introduced, implementation is easy, and therefore the reasoning speed of the large language model is increased.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model reasoning method, system, device and computer-readable storage medium. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. However, due to factors such as structure, large language models have slow reasoning speed and inaccuracy, which affects the user experience of large language models.

[0003] In summary, how to improve the reasoning speed and accuracy of large language models is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] The purpose of this application is to provide a model reasoning method, which can solve the technical problem of how to improve the reasoning speed and accuracy of a large language model to a certain extent. This application also provides a model reasoning system, an electronic device and a computer-readable storage medium.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A model reasoning method, comprising:

[0007] Identify the various intermediate layers in the large language model;

[0008] Filter out the intermediate layers that meet the skipping conditions as candidate layers;

[0009] Generate inference information for large language models to skip candidate layers for inference;

[0010] According to the reasoning information, the target layer is selected from the candidate layers;

[0011] Get the text to be processed;

[0012] Inputting the text to be processed into the large language model that skips the target layer to obtain a candidate sequence;

[0013] The text to be processed and the candidate sequence are input into a large language model to obtain a target inference result.

[0014] Preferably, the step of selecting the intermediate layers satisfying the skipping condition as candidate layers includes:

[0015] Acquire an evaluation data set, wherein the evaluation data set includes prompt words and prompt inference results corresponding to the prompt words;

[0016] Input the prompt word into the large language model that skips the middle layer, and generate the maximum likelihood estimate corresponding to the middle layer based on the probability of the prompt inference result generated by the large language model that skips the middle layer;

[0017] Generate the maximum likelihood estimate corresponding to the intermediate layer relative to the gradient of the intermediate layer;

[0018] Generate the Fisher information matrix of the middle layer according to the gradient of the middle layer;

[0019] Generate a screening threshold based on the trace of the Fisher information matrix of all intermediate layers;

[0020] The intermediate layers whose trace of the Fisher information matrix is ​​smaller than the screening threshold are taken as candidate layers.

[0021] Preferably, the generating of the reasoning information of the large language model skipping the candidate layer for reasoning includes:

[0022] generating a first output distribution of a large language model;

[0023] Sampling the candidate layers under time constraints to obtain candidate groups;

[0024] generating a second output distribution of the large language model that skips the candidate group, and determining an inference time of the large language model that skips the candidate group;

[0025] Inference information is generated according to the first output distribution, the second output distribution, and the inference time.

[0026] Preferably, generating the reasoning information according to the first output distribution, the second output distribution and the reasoning time includes:

[0027] Performing a KL divergence operation on the first output distribution and the second output distribution to generate prediction accuracy information;

[0028] Inference information is generated according to the prediction accuracy information and the weight calculation result of the inference time.

[0029] Preferably, the step of selecting a target layer from candidate layers according to the inference information includes:

[0030] Generates cumulative value of inference information;

[0031] The target layer is determined according to the candidate group corresponding to the maximum cumulative value.

[0032] Preferably, determining the target layer according to the candidate group corresponding to the maximum cumulative value includes:

[0033] The candidate layer in the candidate group corresponding to the maximum cumulative value is used as the candidate layer;

[0034] Generate a target loss value based on the total inference time of the large language model that skips the candidate layer;

[0035] The candidate layer corresponding to the minimum target loss value is determined as the target layer.

[0036] Preferably, before inputting the text to be processed and the candidate sequence into the large language model, the method further includes:

[0037] Inputting the candidate sequence into a large language model to obtain a probability distribution of the candidate sequence;

[0038] Determine the candidate text unit corresponding to the maximum probability distribution;

[0039] In response to the candidate text unit being consistent with the candidate sequence, the step of inputting the to-be-processed text and the candidate sequence into a large language model is performed.

[0040] A model reasoning system, comprising:

[0041] A first determination module, used to determine each intermediate layer in the large language model;

[0042] A first screening module is used to screen out intermediate layers that meet the skipping condition as candidate layers;

[0043] The first generation module is used to generate reasoning information for the large language model to skip the candidate layer for reasoning;

[0044] The second screening module is used to screen the target layer from the candidate layers according to the reasoning information;

[0045] A first acquisition module, used to acquire the text to be processed;

[0046] A first input module, configured to input the text to be processed into a large language model that skips the target layer to obtain a candidate sequence;

[0047] The second input module is used to input the text to be processed and the candidate sequence into a large language model to obtain a target reasoning result.

[0048] An electronic device, comprising:

[0049] Memory for storing computer programs;

[0050] A processor is used to implement the steps of any of the above-mentioned model reasoning methods when executing the computer program.

[0051] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned model reasoning methods are implemented.

[0052] The present application provides a model reasoning method, which determines each intermediate layer in a large language model; selects the intermediate layers that meet the skipping conditions as candidate layers; generates reasoning information for the large language model to skip the candidate layers for reasoning; based on the reasoning information, selects the target layer from the candidate layers; obtains the text to be processed; inputs the text to be processed into the large language model that skips the target layer to obtain a candidate sequence; inputs the text to be processed and the candidate sequence into the large language model to obtain the target reasoning result. The present application uses the intermediate layers that meet the skipping conditions in the large language model as candidate layers, and selects the target layer from the candidate layers based on the reasoning information for the large language model to skip the candidate layers for reasoning, so that the target layer can be adapted to the reasoning information for reasoning of the large language model, ensuring the reasoning performance of the large language model after skipping the target layer, thereby improving the reasoning accuracy of the large language model; and then first inputs the text to be processed into the large language model that skips the target layer to obtain the candidate sequence, and inputs the text to be processed and the candidate sequence into the large language model to obtain the target reasoning result, realizing speculative sampling with the help of the large language model itself, without the need to introduce other models, and is easy to implement, thereby improving the reasoning speed of the large language model. The model reasoning system, electronic device and computer-readable storage medium provided in this application also solve corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 A flowchart of a model reasoning method provided in an embodiment of the present application;

[0055] Figure 2 It is a schematic diagram of the structure of the middle layer;

[0056] Figure 3 A schematic diagram of the structure of a model reasoning system provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0058] Figure 5 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] See also Figure 1 , Figure 1 A flowchart of a model reasoning method provided in an embodiment of the present application.

[0061] A model reasoning method provided in an embodiment of the present application may include the following steps:

[0062] Step S101: determine each intermediate layer in the large language model.

[0063] Step S102: Filter out intermediate layers that meet the skipping condition as candidate layers.

[0064] In practical applications, a large language model is composed of multiple layers, such as convolutional layers, pooling layers, and normalization layers. Some layers in the large language model can be skipped so that the skipped large language model can be used to perform speculative sampling on the large language model itself, thereby speeding up the reasoning efficiency of the large language model. In this process, it is necessary to determine the various intermediate layers in the large language model, and select the intermediate layers that meet the skipping conditions as candidate layers. This intermediate layer refers to the layer that can be skipped. The type of intermediate layer can be flexibly determined according to the application scenario. For example, the intermediate layer can include the attention layer (Attention Layers) and the multi-layer perceptron layer (Multi-Layer Perceptron, MLP) between the input layer and the output layer, so that the subsequent application of the candidate layer determines the way to skip the layer of the large language model.

[0065] In a specific application scenario, in the process of screening out an intermediate layer that meets the skipping condition as a candidate layer, an evaluation data set can be obtained, the evaluation data set including a prompt word and a prompt inference result corresponding to the prompt word; the prompt word is input into a large language model that skips the intermediate layer, and a maximum likelihood estimate corresponding to the intermediate layer is generated according to the probability of the prompt inference result generated by the large language model that skips the intermediate layer; a gradient of the maximum likelihood estimate corresponding to the intermediate layer relative to the intermediate layer is generated; a Fisher information matrix of the intermediate layer is generated according to the gradient of the intermediate layer; a screening threshold is generated according to the trace of the Fisher information matrix of all the intermediate layers; and an intermediate layer whose trace of the Fisher information matrix is ​​less than the screening threshold is taken as a candidate layer.

[0066] For ease of understanding, assume that the evaluation dataset is , x i Indicates the prompt word, yi represents the prompt inference result, and n represents the total number of prompt words; then the generation formula of the maximum likelihood estimation can be: , represents the value of the maximum likelihood estimate, Z * represents the middle layer, X represents the evaluation dataset, Indicates that the large language model skipping the intermediate layer generates the observed output y i The conditional probability of . Correspondingly, the gradient generation formula can be ,in, Denotes the derivative. The formula for generating the Fisher information matrix can be: , or ,in, represents the value of the Fisher information matrix, and E represents the expectation function. The formula for generating the screening threshold can be ,in, represents the screening threshold, and Tr represents the trace of the Fisher information matrix. At this time, The middle layer is the candidate layer. , M represents a large language model.

[0067] Step S103: Generate inference information for the large language model to skip the candidate layer for inference.

[0068] Step S104: Filter out the target layer from the candidate layers according to the inference information.

[0069] In practical applications, although candidate layers have been screened out according to the skipping conditions, whether a candidate layer can actually be skipped needs to be further verified. In this process, reasoning information of the large language model skipping the candidate layer for reasoning can be generated to reflect the reasoning performance of the large language model after skipping the candidate layer. Then, a suitable target layer can be screened out from the candidate layers according to the reasoning information.

[0070] In a specific application scenario, in the process of generating reasoning information for skipping a candidate layer for reasoning in a large language model, a first output distribution of the large language model can be generated; the candidate layer is sampled under a time constraint to obtain a candidate group; a second output distribution of the large language model that skips the candidate group is generated, and the reasoning time of the large language model that skips the candidate group is determined; reasoning information is generated based on the first output distribution, the second output distribution, and the reasoning time. Further, in the process of generating reasoning information based on the first output distribution, the second output distribution, and the reasoning time, a KL (Kullback-Leibler, relative entropy) divergence operation can be performed on the first output distribution and the second output distribution to generate prediction accuracy information; reasoning information is generated based on the weighted operation result of the prediction accuracy information and the reasoning time. Correspondingly, in the process of screening out a target layer from the candidate layer based on the reasoning information, a cumulative value of the reasoning information can be generated; the target layer is determined based on the candidate group corresponding to the maximum cumulative value.

[0071] For ease of understanding, assume represents the first output distribution, Represents the second output distribution, then the generation formula of the prediction accuracy information can be: , Represents the prediction accuracy information, KL represents the divergence function, represents the candidate group. The formula for generating the candidate group can be: , represents the k-step sampling function under the time constraint t. The formula for generating inference information can be , R represents the reasoning information, T represents the reasoning time; Reward scaling factor for prediction accuracy, used to balance the importance of distribution similarity; represents the time penalty coefficient, which is used to balance output quality and inference speed. At this time, the target layer is , S represents all skippable intermediate layers. In this way, the present application selects the target layer based on the distribution similarity and the evaluation of the inference time, and double-considers and weighs the similarity of the predicted distribution and the reduction of the inference time under each adjustment of the model structure, which can ensure that the skipping of the selected target layer can effectively improve the computational efficiency, so that the model after skipping the target layer is close to the distribution of the original complete model.

[0072] In specific application scenarios, the candidate group corresponding to the maximum cumulative value can be directly determined as the target layer, or the candidate group can be further selected to obtain the target layer. That is, in the process of determining the target layer according to the candidate group corresponding to the maximum cumulative value, the candidate layer in the candidate group corresponding to the maximum cumulative value can be used as the candidate layer; the target loss value is generated according to the total inference time of the large language model that skips the candidate layer; the candidate layer corresponding to the minimum target loss value is determined as the target layer. In this process, it is assumed that the generation process of the target loss value is , then the target layer is ,in, represents the total inference time required for the large language model in non-skipped layers, represents the target layer, and arg represents the complex argument.

[0073] Step S105: Obtain the text to be processed.

[0074] Step S106: Input the text to be processed into the large language model that skips the target layer to obtain a candidate sequence.

[0075] Step S107: input the text to be processed and the candidate sequence into the large language model to obtain the target inference result.

[0076] In practical applications, after determining the target layer, the text to be processed can be obtained. The text to be processed can be a question text input by the user, a text that needs feature extraction, a text that needs classification, a text that needs question and answer, or a dialogue text, etc.; then the text to be processed can be input into the large language model that skips the target layer to obtain a candidate sequence, so as to estimate the candidate sequence that can be used for model reasoning by skipping the target layer. The candidate sequence is a sequence used to obtain text processing results, which can be a word sequence, etc.; finally, the text to be processed and the candidate sequence are input into the large language model, so as to obtain the target reasoning result of the large language model for the text to be processed, which can be the feature extraction result, classification result or question and answer result of the text to be processed, etc.

[0077] In specific application scenarios, before the text to be processed and the candidate sequence are input into the large language model, the candidate sequence can also be input into the large language model to obtain the probability distribution of the candidate sequence. , , K represents the length of the candidate sequence, ; Determine the candidate text unit corresponding to the maximum probability distribution In response to the candidate text unit being consistent with the candidate sequence, the step of inputting the text to be processed and the candidate sequence into the large language model is executed to ensure that the candidate sequence meets the quality standards of the large language model, can correctly guide the large language model to perform model reasoning, and output verified sentences, etc. as the target reasoning result.

[0078] For ease of understanding, assume that the middle layer structure is as follows Figure 2 As shown, in the process of generating the answer result of "Where is Zurich?" according to the solution of this application, the verification stage (Verification) uses the complete model to verify the generated token sequence, and the generation stage (Drafting) skips some intermediate layers to quickly generate tokens, and the underlined dot indicates the accepted proposed token, the underlined horizontal dot indicates the unaccepted proposed token, the underlined horizontal dot indicates the token predicted by the complete model in the verification stage, and the unmarked one indicates the next token. The question-answering process is as follows:

[0079] <s>Where ls Zur lch? Ans: Zur ;

[0080] <s>Where is Zur ich? Ans : Zur ich is the largest ;

[0081] <s>Where is Zur lch? Ans: Zur lch is the largest c ity in Switzerland . It ;

[0082] <s>Where is Zur Ich? Ans : Zur Ich is the largest city in Switzerland.It is located in ;

[0083] <s>Where is Zur ich? Ans : Zur ich is the largest cty in Swltzerlend. It is located in the eanton north ;

[0084] <s>Where is Zur ich? Ans : Zur ich is the largest sity inSehtzerlang, It is located in the north- central ;

[0085] <s>Where ls Zur lch? Ans: Zur toh is the largest city in Switzerland.It is located in the north-central Switzerland part ;

[0086] <s>Wherg is Zur ich?Ans: Zar loh is the largest city in Switzerland.It is located in the nom-central part of Switzerland the ;

[0087] <s>Where ls zur ich? Ans i Zur ich is the largest cify in Switzerland. It is located in the north - central part of the country ....

[0088] The present application provides a model reasoning method, which determines each intermediate layer in a large language model; selects the intermediate layers that meet the skipping conditions as candidate layers; generates reasoning information for the large language model to skip the candidate layers for reasoning; based on the reasoning information, selects the target layer from the candidate layers; obtains the text to be processed; inputs the text to be processed into the large language model that skips the target layer to obtain a candidate sequence; inputs the text to be processed and the candidate sequence into the large language model to obtain the target reasoning result. The present application uses the intermediate layers that meet the skipping conditions in the large language model as candidate layers, and selects the target layer from the candidate layers based on the reasoning information for the large language model to skip the candidate layers for reasoning, so that the target layer can be adapted to the reasoning information for reasoning of the large language model, thereby ensuring the reasoning performance of the large language model after skipping the target layer, thereby improving the reasoning accuracy of the large language model; and then first inputs the text to be processed into the large language model that skips the target layer to obtain the candidate sequence, and inputs the text to be processed and the candidate sequence into the large language model to obtain the target reasoning result, realizing speculative sampling with the help of the large language model itself, without introducing other models, easy to implement, and improving the reasoning decoding speed without affecting the answer quality and occupying additional hardware resources.

[0089] In order to facilitate understanding of a model reasoning method provided by the present application, the process of answering user image retrieval using a question-answering large language model is now described. This process may include the following operations:

[0090] Identify the various intermediate layers in the large question-answering language model;

[0091] Filter out the intermediate layers that meet the skipping conditions as candidate layers;

[0092] Generate reasoning information for large question-answering language models that skip candidate layers for reasoning;

[0093] According to the reasoning information, the target layer is selected from the candidate layers;

[0094] Obtaining image retrieval information input by the user, the image retrieval information may include content information, color information, size information, etc. of the image to be retrieved;

[0095] The image retrieval information is input into the large question-answering language model that skips the target layer to obtain the image candidate sequence;

[0096] The image retrieval information and the image candidate sequence are input into the large language model to obtain the target image adapted to the image retrieval information inferred by the question-answering large language model.

[0097] See also Figure 3 , Figure 3 A schematic diagram of the structure of a model reasoning system provided in an embodiment of the present application.

[0098] A model reasoning system provided in an embodiment of the present application may include:

[0099] A first determination module 101 is used to determine each intermediate layer in the large language model;

[0100] A first screening module 102, used to screen out intermediate layers that meet the skipping condition as candidate layers;

[0101] A first generating module 103 is used to generate reasoning information for skipping a candidate layer for reasoning in a large language model;

[0102] A second screening module 104 is used to screen out a target layer from the candidate layers according to the reasoning information;

[0103] A first acquisition module 105, used to acquire the text to be processed;

[0104] A first input module 106, configured to input the text to be processed into a large language model that skips the target layer to obtain a candidate sequence;

[0105] The second input module 107 is used to input the text to be processed and the candidate sequence into the large language model to obtain the target inference result.

[0106] In a model reasoning system provided in an embodiment of the present application, the first screening module may include:

[0107] A first acquisition unit is used to acquire an evaluation data set, where the evaluation data set includes prompt words and prompt inference results corresponding to the prompt words;

[0108] A first generating unit is used to input the prompt word into the large language model that skips the intermediate layer, and generate a maximum likelihood estimate corresponding to the intermediate layer according to the probability of the prompt inference result generated by the large language model that skips the intermediate layer;

[0109] A second generating unit, used to generate a maximum likelihood estimate corresponding to the middle layer relative to the gradient of the middle layer;

[0110] A third generating unit, used for generating a Fisher information matrix of the middle layer according to the gradient of the middle layer;

[0111] a fourth generating unit, for generating a screening threshold value according to the traces of the Fisher information matrices of all the intermediate layers;

[0112] The first setting unit is used to select the intermediate layer whose trace of Fisher information matrix is ​​smaller than the screening threshold as the candidate layer.

[0113] In a model reasoning system provided in an embodiment of the present application, the first generation module may include:

[0114] a fifth generating unit, configured to generate a first output distribution of the large language model;

[0115] A first sampling unit, used for sampling the candidate layer under time constraints to obtain a candidate group;

[0116] a sixth generating unit, configured to generate a second output distribution of the large language model for skipping the candidate group, and determine an inference time of the large language model for skipping the candidate group;

[0117] The seventh generating unit is used to generate reasoning information according to the first output distribution, the second output distribution and the reasoning time.

[0118] In a model reasoning system provided by an embodiment of the present application, the seventh generation unit can be specifically used to: perform KL divergence operation on the first output distribution and the second output distribution to generate prediction accuracy information; and generate reasoning information based on the weighted operation result of the prediction accuracy information and the reasoning time.

[0119] In a model reasoning system provided in an embodiment of the present application, the second screening module may include:

[0120] an eighth generating unit, configured to generate a cumulative value of the reasoning information;

[0121] The first determination unit is used to determine the target layer according to the candidate group corresponding to the maximum cumulative value.

[0122] In a model inference system provided by an embodiment of the present application, the first determination unit can be used to: select a candidate layer in a candidate group corresponding to a maximum cumulative value as a candidate layer; generate a target loss value according to the total inference time of a large language model that skips the candidate layer; and determine the candidate layer corresponding to the minimum target loss value as the target layer.

[0123] A model reasoning system provided in an embodiment of the present application may also include:

[0124] The first detection module is used to input the candidate sequence into the large language model before the second input module inputs the text to be processed and the candidate sequence into the large language model to obtain the probability distribution of the candidate sequence; determine the candidate text unit corresponding to the maximum probability distribution; in response to the candidate text unit being consistent with the candidate sequence, execute the step of inputting the text to be processed and the candidate sequence into the large language model.

[0125] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the model reasoning method provided in the embodiment of the present application. Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0126] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the model reasoning method described in any of the above embodiments are implemented.

[0127] See also Figure 5 , another electronic device provided in the embodiment of the present application may also include: an input port 203 connected to the processor 202, for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, for displaying the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, for realizing communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-DefinitionMultimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.

[0128] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the model reasoning method described in any of the above embodiments are implemented.

[0129] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0130] For the description of the relevant parts of a model reasoning system, an electronic device, and a computer-readable storage medium provided in the embodiment of the present application, please refer to the detailed description of the corresponding parts in a model reasoning method provided in the embodiment of the present application, which will not be repeated here. In addition, the parts of the above-mentioned technical solutions provided in the embodiment of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0131] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0132] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A model reasoning method, characterized in that: include: Identify the various intermediate layers in the large language model; Filter out the intermediate layers that meet the skipping conditions as candidate layers; Generate inference information for large language models to skip candidate layers for inference; According to the reasoning information, the target layer is selected from the candidate layers; Get the text to be processed; Inputting the text to be processed into the large language model that skips the target layer to obtain a candidate sequence; The text to be processed and the candidate sequence are input into a large language model to obtain a target inference result.

2. The method according to claim 1, characterized in that: The step of selecting the intermediate layers satisfying the skipping condition as candidate layers includes: Acquire an evaluation data set, wherein the evaluation data set includes prompt words and prompt inference results corresponding to the prompt words; Input the prompt word into the large language model that skips the middle layer, and generate the maximum likelihood estimate corresponding to the middle layer based on the probability of the prompt inference result generated by the large language model that skips the middle layer; Generate the maximum likelihood estimate corresponding to the intermediate layer relative to the gradient of the intermediate layer; Generate the Fisher information matrix of the middle layer according to the gradient of the middle layer; Generate a screening threshold based on the trace of the Fisher information matrix of all intermediate layers; The intermediate layers whose trace of the Fisher information matrix is ​​smaller than the screening threshold are taken as candidate layers.

3. The method according to claim 2, characterized in that The generating of the reasoning information of the large language model skipping the candidate layer for reasoning includes: generating a first output distribution of a large language model; Sampling the candidate layers under time constraints to obtain candidate groups; generating a second output distribution of the large language model that skips the candidate group, and determining an inference time of the large language model that skips the candidate group; Inference information is generated according to the first output distribution, the second output distribution, and the inference time.

4. The method according to claim 3, characterized in that The generating the reasoning information according to the first output distribution, the second output distribution and the reasoning time includes: Performing a KL divergence operation on the first output distribution and the second output distribution to generate prediction accuracy information; Inference information is generated according to the prediction accuracy information and the weight calculation result of the inference time.

5. The method according to claim 4, characterized in that The step of selecting a target layer from candidate layers according to the inference information includes: Generates cumulative value of inference information; The target layer is determined according to the candidate group corresponding to the maximum cumulative value.

6. The method according to claim 5, characterized in that The step of determining the target layer according to the candidate group corresponding to the maximum cumulative value includes: The candidate layer in the candidate group corresponding to the maximum cumulative value is used as the candidate layer; Generate a target loss value based on the total inference time of the large language model that skips the candidate layer; The candidate layer corresponding to the minimum target loss value is determined as the target layer.

7. The method according to claim 1, characterized in that Before inputting the text to be processed and the candidate sequence into the large language model, the method further includes: Inputting the candidate sequence into a large language model to obtain a probability distribution of the candidate sequence; Determine the candidate text unit corresponding to the maximum probability distribution; In response to the candidate text unit being consistent with the candidate sequence, the step of inputting the to-be-processed text and the candidate sequence into a large language model is performed.

8. A model reasoning system, characterized in that: include: A first determination module, used to determine each intermediate layer in the large language model; A first screening module is used to screen out intermediate layers that meet the skipping condition as candidate layers; The first generation module is used to generate reasoning information for the large language model to skip the candidate layer for reasoning; The second screening module is used to screen the target layer from the candidate layers according to the reasoning information; A first acquisition module, used to acquire the text to be processed; A first input module, configured to input the text to be processed into a large language model that skips the target layer to obtain a candidate sequence; The second input module is used to input the text to be processed and the candidate sequence into a large language model to obtain a target reasoning result.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the model reasoning method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the model reasoning method according to any one of claims 1 to 7 are implemented.

Images