A method and apparatus for generating an inference result for text data

Abstract

Embodiments of the present application provide a method and device for generating an inference result of text data, generating an initial draft for the text data through an approximate model; generating a target draft based on the initial draft through an original model, and determining the inference result of the text data based on the target draft, which can accelerate the inference and greatly shorten the inference time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of generating inference results for text data, and in particular, to a method for generating inference results for text data, an apparatus for generating inference results for text data, an electronic device, and a computer-readable storage medium. BACKGROUND

[0002] A large language model (LLM) is a neural network-based machine learning model that can generate text, translate languages, write different creative content, and answer your questions, and even execute the instructions you give, by learning a large amount of text data.

[0003] Autoregressive sampling is a commonly used sampling method in generative models, especially for sequence generation models like LLMs. The core idea is that the model takes the previously generated elements as input when generating the next element of the sequence to predict the most likely next element.

[0004] Inference of large language models usually requires autoregressive sampling. Their inference process is quite slow and needs to be performed serially token by token. Therefore, the inference process of large models is often limited by memory access speed, and each token generated requires all parameters to be transferred from the storage unit to the computing unit, so the memory access bandwidth becomes a serious bottleneck, resulting in low inference speed. SUMMARY

[0005] The embodiments of the present application provide a method and apparatus for generating inference results for text data, an electronic device, and a computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0006] The embodiments of the present application disclose a method for generating inference results for text data, comprising:

[0007] When the user input text data is obtained, an initial draft for the text data is generated by an approximate model;

[0008] A target draft is generated based on the initial draft by an original model; the approximate model is a distilled version of the original model;

[0009] The inference result for the text data is determined based on the target draft.

[0010] Optionally, the original model is deployed on a cloud server, and the approximate model is deployed on a client device.

[0011] Optionally, before the step of generating an initial draft for the text data through the approximation model, further comprising:

[0012] constructing training data for a large language model;

[0013] determining an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculating a first loss function between the intermediate layer output and a preset target value;

[0014] determining a core layer parameter and an additional layer parameter of the large language model based on the training data, and calculating a second loss function for the core layer parameter and the additional layer parameter;

[0015] determining an activation mode of a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode;

[0016] generating an original model based on the first loss function, the second loss function, and the third loss function; the original model is a complete model, and the original model includes a task target layer having an association with a task type;

[0017] generating an approximation model based on the task target layer.

[0018] Optionally, the step of determining an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculating a first loss function between the intermediate layer output and a preset target value, comprises:

[0019] determining the intermediate layer based on a network structure of the large language model and characteristics of a task type;

[0020] inputting the training data to the intermediate layer to obtain the intermediate layer output;

[0021] selecting a first initial loss function according to the task type and the preset target value;

[0022] weighting the first initial loss function for different intermediate layers to calculate the first loss function between the intermediate layer output and the preset target value.

[0023] Optionally, the large language model includes a core layer and an additional layer, and the step of determining a core layer parameter and an additional layer parameter of the large language model based on the training data, and calculating a second loss function for the core layer parameter and the additional layer parameter, comprises:

[0024] determining an original parameter matrix and a decomposition method of the large language model, and decomposing the original parameter matrix into a plurality of sub-matrices based on the decomposition method;

[0025] determine core layer parameters and additional layer parameters based on the sub-matrix and the original parameter matrix;

[0026] determine a second initial loss function for the core layer and the additional layer;

[0027] input the training data into the large language model to obtain core layer and additional layer outputs of the core layer and the additional layer;

[0028] generate core layer and additional layer loss values based on the core layer and additional layer outputs through the second initial loss function;

[0029] perform a weighted summation operation on the core layer and additional layer loss values to determine a second loss function for the core layer parameters and the additional layer parameters.

[0030] Optionally, the step of determining an activation mode of a target layer and a target branch of the large language model based on the activation mode includes:

[0031] determining the target layer and the target branch of the large language model based on a task type;

[0032] determining an attention mechanism, a gating mechanism, and a routing algorithm for the target layer and the target branch;

[0033] determining a third loss function based on the activation mode through the attention mechanism, the gating mechanism, and the routing algorithm.

[0034] Optionally, the step of generating an original model based on the first loss function, the second loss function, and the third loss function includes:

[0035] determining initial weights for the first loss function, the second loss function, and the third loss function;

[0036] adjusting the initial weights through hyperparameter search and determining target weights for the initial weights;

[0037] calculating a total loss function through the target weights, the first loss function, the second loss function, and the third loss function;

[0038] generating an original model based on the total loss function.

[0039] The embodiment of the application also discloses a device for generating an inference result of text data, which comprises:

[0040] an initial draft generation module configured to generate an initial draft for the text data through an approximate model when the text data input by a user is obtained;

[0041] a target draft generation module configured to generate a target draft based on the initial draft by an original model; the approximate model is a distilled version of the original model;

[0042] a reasoning result determination module configured to determine a reasoning result for the text data based on the target draft.

[0043] The embodiment of the present application further discloses an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0044] The memory is used for storing a computer program.

[0045] The processor is used for executing the program stored on the memory, and realizes the method as described in the embodiment of the present application.

[0046] The embodiment of the present application further discloses a computer readable storage medium, which stores instructions, and when the instructions are executed by one or more processors, the processors execute the method as described in the embodiment of the present application.

[0047] The embodiment of the present application has the following advantages:

[0048] According to the embodiment of the present application, the initial draft for the text data is generated by the approximate model, the target draft is generated based on the initial draft by the original model, and the reasoning result for the text data is determined based on the target draft, so that the following beneficial effects can be achieved.

[0049] Accelerated reasoning: greatly shorten the reasoning time.

[0050] Improved model adaptability: can adapt to different tasks and inputs.

[0051] Balancing speed and accuracy: providing flexible solutions. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a step flow chart of a generation method of a reasoning result for text data provided in the embodiment of the present application;

[0053] Figure 2 is a step flow chart of a generation method of speculative sampling reasoning provided in the embodiment of the present application;

[0054] Figure 3 is a structure block diagram of a generation device of a reasoning result for text data provided in the embodiment of the present application;

[0055] Figure 4is a hardware structure block diagram of an electronic device provided in an embodiment of the present application.

[0056] Figure 5 is a schematic diagram of a computer readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0058] Large Language Model (LLM) is a neural network-based machine learning model that can generate text, translate languages, write different creative content, and answer your questions, and even execute the instructions you give, by learning a large amount of text data.

[0059] Autoregressive sampling is a commonly used sampling method in generative models, especially suitable for sequence generation models like LLM. Its core idea is that when the model generates the next element of the sequence, it takes the previously generated elements as input to predict the next most likely element.

[0060] One bottleneck problem encountered by Large Language Model (LLM) in the inference process is slow inference speed. This bottleneck is mainly due to the autoregressive generation method of Large Language Model (LLM) and the huge number of parameters. In order to solve this problem, the technical personnel in the field proposed the method of speculative sampling. However, the current speculative sampling method still has some problems, which limit its practical application.

[0061] There are three main problems in the current speculative sampling inference process:

[0062] 1. Approximate model accuracy problem:

[0063] Problem description: The performance gap between the current approximate model used for speculative sampling and the original LLM is large. This leads to many generated tokens not being accurate during the decoding process, and ultimately still needs to rely on the original LLM for correction.

[0064] Impact: Although the approximate model can speed up part of the decoding process, due to its accuracy problem, it cannot significantly reduce the call to the original LLM, thereby increasing the communication cost and offsetting part of the acceleration effect.

[0065] 2. Fixed size of approximate model problem:

[0066] Problem description: The current approximate model is usually of fixed size. When deployed on the edge side, only a smaller model that matches the computing capacity of the device can be selected, and the full computing resources of the device cannot be fully utilized.

[0067] Impact: This limits the performance of speculative sampling methods on different devices, and cannot achieve better acceleration on devices with sufficient computing resources.

[0068] 3. Low network bandwidth utilization problem:

[0069] Problem description: The current speculative sampling method does not fully utilize the network bandwidth, and the effect of speculative sampling will be greatly affected in poor network conditions.

[0070] Impact: This leads to unstable generation speed and unsatisfactory generation effect in practical application of the speculative sampling method.

[0071] The embodiment of the present application solves the problems of low precision of approximate model, fixed size of model, and low network bandwidth utilization in the traditional speculative sampling method by dynamic speculative sampling, model modularization, and module transmission caching technology.

[0072] First, in order to solve the problem of too large difference between the approximate model and the original model, the embodiment of the present application adopts the strategy of dynamically increasing the approximate model. By gradually increasing the size of the approximate model, it gradually approaches the original model, thereby significantly improving the generation ability of the approximate model and reducing the burden of the original model in the evaluation and correction process.

[0073] Second, the embodiment of the present application modularizes the original model in fine granularity. Through mathematical transformation and predefined rules, the model is split into multiple functionally independent modules. This modular design not only ensures the consistency of the model, but also enables the approximate model to flexibly adjust its size and ability by combining different modules. Therefore, the embodiment of the present application can select the most suitable module combination according to the computing ability of the end-side device, and maximize the use of hardware resources.

[0074] Finally, considering the instability of the network environment, the embodiment of the present application introduces a module transmission and caching mechanism in the process of speculative sampling. In good network conditions, the module most relevant to the current task in the original model can be transmitted to the approximate model end for caching. In this way, even in the case of network limitation, the approximate model can still generate high-quality text using the cached module, thereby ensuring the stability and efficiency of the generation process.

[0075] In summary, the embodiment of the present application optimizes the speculative sampling process through dynamic adjustment, modularization, and caching technology, effectively improves the generation efficiency, reduces the dependence on the original model, and enhances the robustness of the system in complex network environments.

[0076] Reference Figure 1, shows a step flow chart of a method for generating inference results for text data according to an embodiment of the present application, which can specifically include the following steps:

[0077] Step 101, when the user input text data is obtained, an initial draft for the text data is generated by an approximate model;

[0078] Step 102, a target draft is generated based on the initial draft by an original model; the approximate model is a distilled version of the original model;

[0079] Step 103, determining the inference result for the text data based on the target draft.

[0080] Reference Figure 2 , Figure 2 is a step flow chart of a speculative sampling inference generation method according to an embodiment of the present application;

[0081] Speculative sampling is a technique to optimize the inference efficiency of large models. It introduces an approximate model and a dynamic caching mechanism during the inference process to speed up the inference process. The whole process mainly consists of two stages:

[0082] Model deployment: deploy an approximate model with weak computing power but fast response on the client device side, and deploy an original model with large number of parameters and strong performance on the cloud server.

[0083] Dynamic interactive inference: after the user inputs the question, the approximate model and the original model interact in real time to complete the inference task together.

[0084] The detailed steps can be as follows:

[0085] 1. Model deployment:

[0086] Approximate model: usually a small parameter model, used to quickly generate initial results or candidate answers. It can be a distilled version of the original model, or a small model specially trained for a specific task.

[0087] Original model: large number of parameters, strong computing power, can generate high-quality output. It is usually deployed in the cloud to provide powerful computing resources.

[0088] 2. Dynamic interactive inference:

[0089] 2-1. Establish connection and judgment:

[0090] After the user inputs the text data, the approximate model and the original model are connected.

[0091] The system determines whether the current task is suitable for dynamic caching. If the task is relatively simple or does not require high real-time performance, the approximate model can be used to generate results directly.

[0092] To determine whether a task is complex, the following factors can be considered:

[0093] Whether the number of factors involved in the task / question in the text data is greater than a preset threshold;

[0094] Whether the field in which the task / question in the text data is located is the target field;

[0095] Uncertainty contained in the task / question in the text data:

[0096] Simple task / question: The question conditions are clear, the answer is relatively certain, and there are no too many ambiguous or unknown factors.

[0097] Complex task / question: The question contains many uncertainties, may have multiple possible answers, or requires different judgments based on different situations.

[0098] Knowledge background required for the task / question in the text data:

[0099] Simple question: The knowledge required to solve the problem is relatively basic and does not require in-depth professional knowledge.

[0100] Complex question: Requires relatively specialized knowledge or requires the integration of knowledge from multiple fields to answer.

[0101] Logical relationship contained in the task / question in the text data:

[0102] Simple task / question: The logical relationship of the question is relatively simple and can be answered through simple reasoning.

[0103] Complex task / question: The logical relationship of the question is complex and may involve multiple layers of nested conditions, causal relationships, etc., requiring complex logical reasoning.

[0104] 2-2. Generating initial draft tokens and caching deployment:

[0105] The approximate model generates an initial draft token based on user input.

[0106] At the same time, the original model starts sending additional modules to the approximate model. These modules can be new neural network layers, attention mechanisms, etc., to enhance the expressive power of the approximate model.

[0107] After receiving the additional modules, the approximate model caches them for subsequent reasoning. It is important to note that the newly cached modules will not participate in the current segment generation.

[0108] 2-3. Original model scoring and output:

[0109] The original model scores the initial draft token generated by the approximate model and modifies it based on the score to generate a new, higher-quality target draft token.

[0110] 2-4. Return results and update:

[0111] The approximate model returns the target draft token returned by the original model to the user.

[0112] At the same time, the approximate model detects whether there are available deployed caches. If so, these cache modules are integrated into its own model to improve the accuracy of subsequent inference.

[0113] 2-5. Loop iteration:

[0114] Repeat steps 2-2 to 2-4 until the complete output result is generated.

[0115] The role of dynamic caching:

[0116] Speed up inference: By caching part of the original model, the approximate model can use these modules to improve the generation quality and reduce the dependence on the original model, thereby speeding up the inference process.

[0117] Improve model adaptability: As the task progresses, the approximate model will continuously update its knowledge base, thereby better adapting to different inputs and outputs.

[0118] Balance speed and accuracy: Speculative sampling provides a flexible solution for users by balancing speed and accuracy.

[0119] Speculative sampling distributes the computational tasks of large models to multiple models and uses dynamic caching mechanisms to improve model adaptability, thereby achieving efficient and high-quality inference. This method has broad application prospects in natural language processing, machine translation, and other fields.

[0120] The embodiment of the invention generates an initial draft for the text data through the approximate model; generates a target draft based on the initial draft through the original model, and determines the inference result for the text data based on the target draft, which can achieve the following beneficial effects.

[0121] Speed up inference: significantly shorten the inference time.

[0122] Improve model adaptability: can adapt to different tasks and inputs.

[0123] Balancing speed and accuracy: providing flexible solutions.

[0124] On the basis of the above-mentioned embodiments, variant embodiments of the above-mentioned embodiments are proposed, and it should be noted that, in order to make the description brief, only the differences from the above-mentioned embodiments are described in the variant embodiments.

[0125] In an optional embodiment of the present application, before the step of generating an initial draft for text data through an approximate model, the following steps are further included:

[0126] Constructing training data of a large language model;

[0127] Determining an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculating a first loss function between the intermediate layer output and a preset target value;

[0128] Determining a core layer parameter and an additional layer parameter of the large language model based on the training data, and calculating a second loss function for the core layer parameter and the additional layer parameter;

[0129] Determining an activation mode of a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode;

[0130] Generating an original model based on the first loss function, the second loss function and the third loss function; the original model is a complete model, and the original model includes a task target layer having an associated relationship with a task type;

[0131] Generating an approximate model based on the task target layer.

[0132] The embodiment of the present application can construct training data of a large language model, and the purpose is to provide sufficient and high-quality data for training a large language model.

[0133] Exemplarily, a large amount of text data such as books, articles, codes and the like can be collected and arranged, and cleaning and preprocessing can be performed to make it meet the requirements of model training, so that the quality of training data directly affects the performance of the model. High-quality and diversified data can enable the model to learn more rich language knowledge and patterns.

[0134] The embodiment of the present application can determine the intermediate layer output and calculate the first loss function, and the purpose is to ensure that the intermediate layer of the model also contains useful information, and to provide a basis for subsequent modularization and construction of the approximate model.

[0135] Exemplarily, the meaning of the intermediate layer output loss calculation is:

[0136] Intermediate layer output: when a neural network processes information, each layer will produce some intermediate results, which are called intermediate layer outputs.

[0137] logits: In neural networks, logits usually refer to the outputs that have been linearly transformed but not yet passed through an activation function. Computing the loss on logits can more directly reflect the features learned by the model.

[0138] Computing loss: By calculating the difference between the output of an intermediate layer and the target value, the learning effect of the model at each level is measured.

[0139] The specific purpose is:

[0140] Ensure the effectiveness of each part of the model: Ensure that each part of the model is learning useful features, rather than relying solely on the last layer to complete the task.

[0141] Promote modularity: Provide the basis for subsequent modular decomposition. If the output of a certain intermediate layer can well predict the target, it can be considered as an independent module.

[0142] The implementation is as follows:

[0143] Select intermediate layers: According to the network structure and task characteristics, select appropriate intermediate layers for loss calculation.

[0144] Define loss function: Common loss functions such as cross-entropy loss, mean square error, etc. can be used.

[0145] Weight assignment: The loss of different intermediate layers can be weighted to balance the importance of different layers.

[0146] For example, suppose a language model has 10 layers, and the logits of the 5th and 8th layers can be calculated for loss, and the cross-entropy loss with the target label can be calculated.

[0147] By calculating the difference between the logits of the output of the intermediate layer of the model and the preset target value, the first loss function is obtained, which can ensure that the intermediate layer of the model can effectively extract features and provide support for subsequent tasks.

[0148] The embodiment of the application can determine the core layer parameters and the additional layer parameters, and calculate the second loss function, which aims to split the model into core generation ability and enhancement ability, and facilitate flexible combination to build different sizes of approximate models.

[0149] Exemplarily, the meaning of matrix decomposition and low-rank transformation to split the model:

[0150] Matrix decomposition: Decompose the parameter matrix of the model into the product of two or more smaller matrices.

[0151] Low-rank transformation: Approximate a high-rank matrix as a low-rank matrix to reduce the number of parameters.

[0152] Core layer parameters and additional layer parameters: Divide the parameters of the model into two parts, core layer parameters responsible for the basic generation ability of the model, and additional layer parameters responsible for enhancing the performance of the model on specific tasks.

[0153] The specific purpose is:

[0154] Reducing model complexity: Through matrix decomposition and low-rank transformation, the number of model parameters is reduced, thereby reducing the computational load and storage space of the model.

[0155] Improving model interpretability: Splitting the model into multiple sub-modules helps understand the internal working mechanism of the model.

[0156] Promote modularization: Provide basis for subsequent modularization.

[0157] Implementation:

[0158] Select decomposition method: You can use singular value decomposition (SVD), non-negative matrix factorization (NMF), etc.

[0159] Determine core layer and additional layer: According to the task requirements and model structure, determine which parameters belong to the core layer and which parameters belong to the additional layer.

[0160] Calculate loss: Calculate the loss of core layer parameters and additional layer parameters respectively.

[0161] For example: Decompose the parameter matrix of a fully connected layer into two smaller matrices, one of which represents the core layer parameters and the other represents the additional layer parameters.

[0162] By matrix decomposition and other methods, the model parameters are split into core layer parameters and additional layer parameters, and their loss functions are calculated respectively, which can improve the interpretability of the model, facilitate subsequent modularization and pruning.

[0163] The embodiment of the application can determine the activation mode of the target layer and the target branch, and calculate the third loss function, which aims to improve the adaptability of the model so that it can adapt to different inputs and tasks.

[0164] Exemplarily, the meaning of the routing distribution module:

[0165] Routing distribution: According to different inputs, dynamically select different model components to process information.

[0166] Different layers, different branches: There can be multiple parallel branches or multiple layers in the model, and the routing distribution module can activate different branches or layers as needed.

[0167] Objective:

[0168] Improve the adaptability of the model: make the model adapt to different inputs and tasks.

[0169] Enhance the expressive ability of the model: by combining different model components, improve the expressive ability of the model.

[0170] Implementation:

[0171] Attention mechanism: attention mechanism can be used to dynamically select different model components.

[0172] Gating mechanism: gating mechanism can be used to control the flow of information.

[0173] Routing algorithm: a special routing algorithm can be designed to select the optimal path.

[0174] The model selects the activation of different layers and different branches according to the input by increasing the routing distribution module, to realize the generation of text generation, and calculates different losses for different activation methods.

[0175] For example, in the machine translation task, different translation model components can be dynamically selected according to the length and complexity of the input sentence.

[0176] According to different tasks, different target layers and target branches are selected, and the corresponding loss function is calculated, which can enhance the expressive ability of the model and make it better complete various natural language processing tasks.

[0177] The embodiment of the application can generate an original model, which aims to build a complete and high-performance large language model.

[0178] Based on the three loss functions calculated above, the model is optimized as a whole to obtain an original model with excellent performance, which has strong language generation ability and can be used as the basis for subsequent tasks.

[0179] The embodiment of the application can generate an approximate model based on the task target layer, since the task target layer has a correlation with the task type, which is equivalent to constructing a smaller, faster, and more specific model for a specific task type, which can be deployed on resource-constrained devices.

[0180] According to the task requirements, part of the layers or modules in the original model are selected as the approximate model, so as to reduce the calculation amount and storage space of the model, and improve the inference speed.

[0181] The embodiment of the application can generate an inference result for text data, which aims to:

[0182] Leverage the strengths of both the original model and the approximate model: the original model provides high-quality output, while the approximate model provides fast response. By combining the capabilities of both, efficient and high-quality text generation is achieved.

[0183] Dynamic model selection: dynamically choose to use the original model or the approximate model, or a combination of both, based on the complexity and real-time requirements of the task.

[0184] Improve user experience: for simple tasks, the approximate model can quickly generate results to meet the user's real-time needs; for complex tasks, the original model can provide more accurate and comprehensive answers.

[0185] By feeding the input text into the original model and the approximate model, the final inference result is obtained, so that various natural language processing tasks such as text generation, machine translation, and question and answer systems can be achieved according to the training target of the model.

[0186] In the embodiment of the present application, the training data of the large language model is constructed; the intermediate layer output of the intermediate layer of the large language model is determined based on the training data, and the first loss function between the intermediate layer output and the preset target value is calculated; the core layer parameters and the additional layer parameters of the large language model are determined based on the training data, and the second loss function for the core layer parameters and the additional layer parameters is calculated; the activation mode of the target layer and the target branch of the large language model is determined, and the third loss function is determined based on the activation mode; the first loss function, the second loss function and the third loss function are used to generate an original model; the original model is a complete model, and the original model includes a task target layer having an associated relationship with a task type; an approximate model is generated based on the task target layer; when the user input text data is obtained, the inference result for the text data is generated through the original model and the approximate model, thereby improving the model inference speed.

[0187] Further, the following beneficial effects can be achieved.

[0188] Improve inference efficiency: the fast response of the approximate model can significantly shorten the inference time.

[0189] Improve generation quality: the high-quality output of the original model can ensure the accuracy and fluency of the generated results.

[0190] Enhance the adaptability of the model: the dynamic caching mechanism enables the model to adjust its behavior according to the specific circumstances of the task, improving the generalization ability of the model.

[0191] Reduce computing cost: by reasonably allocating computing tasks, the consumption of computing resources can be reduced.

[0192] In an optional embodiment of the present application, the step of determining the intermediate layer output of the intermediate layer of the large language model based on the training data and calculating the first loss function between the intermediate layer output and the preset target value comprises:

[0193] Determining the intermediate layer based on the network structure of the large language model and the characteristics of the task type;

[0194] Inputting the training data into the intermediate layer to obtain the intermediate layer output;

[0195] According to the task type and the preset target value, selecting a loss function;

[0196] Weighting the loss functions of different intermediate layers to calculate the first loss function between the intermediate layer output and the preset target value.

[0197] For example, the first loss function can be calculated as follows.

[0198] Determining the intermediate layer:

[0199] Analyzing the model structure: carefully study the network structure of the large language model, including the type, number of neurons and connection method of each layer.

[0200] Consider the task characteristics: according to the specific task (such as text generation, question answering, translation, etc.), select the intermediate layer that has important influence on the task. For example, for text generation tasks, more attention may be paid to the intermediate layer related to semantics.

[0201] Experience judgment: combine past experience and research results to select some intermediate layers that usually have strong feature extraction ability, such as the last few layers of the encoder or the output layer of the attention mechanism.

[0202] Calculate the intermediate layer output:

[0203] Forward propagation: input the training data into the model and perform forward propagation calculation to obtain the output of each intermediate layer.

[0204] Extract logits: extract logits from the selected intermediate layer. Logits are usually the output after linear transformation but before activation function.

[0205] Determine the preset target value:

[0206] According to the task definition: for different tasks, the preset target value will be different.

[0207] Text generation task: the target value can be the probability distribution of the next word.

[0208] Classification task: the target value can be the probability of each category.

[0209] Regression task: The target value can be a continuous numerical value.

[0210] Compute the loss function:

[0211] Select the loss function: Choose an appropriate first initial loss function based on the task type and the nature of the target value.

[0212] Cross-entropy loss: Commonly used for classification tasks and text generation tasks.

[0213] Mean squared error: Commonly used for regression tasks.

[0214] Compute the loss: Plug the logits of the intermediate layer output and the preset target value into the first initial loss function to calculate the loss function.

[0215] Weighted processing:

[0216] Assign weights: If multiple intermediate layers are selected to calculate the loss, assign different weights to the first initial loss of each intermediate layer.

[0217] Weight adjustment: Determine appropriate weights through experiments or experience, calculate the first loss function between the intermediate layer output and the preset target value, and balance the influence of different intermediate layers on the final loss.

[0218] For example, assume there is a Transformer model for text generation, and the output of the last encoder layer and the first decoder layer are selected as the intermediate layers.

[0219] Input the training data into the model to get the output of these two intermediate layers.

[0220] Compare the logits of these two outputs with the one-hot encoding of the next word (as the target value).

[0221] Calculate the loss of these two intermediate layers using the cross-entropy loss function.

[0222] Assign weights to the two losses, for example, the loss weight of the encoder output is 0.6 and the loss weight of the decoder output is 0.4.

[0223] Sum the two weighted losses to get the final first loss function.

[0224] By minimizing this loss function, the model's intermediate layers can better predict the next word, thereby improving the quality of the model's generation.

[0225] In an optional embodiment of the present application, the large language model comprises a core layer and an additional layer, and the step of determining the core layer parameters and the additional layer parameters of the large language model based on the training data comprises:

[0226] determining an original parameter matrix and a decomposition method of the large language model, and decomposing the original parameter matrix into a plurality of sub-matrices based on the decomposition method;

[0227] determining the core layer parameters and the additional layer parameters based on the sub-matrices and the original parameter matrix;

[0228] determining a second initial loss function for the core layer and the additional layer;

[0229] inputting the training data into the large language model to obtain core layer and additional layer outputs of the core layer and the additional layer;

[0230] generating core layer and additional layer loss values through the second initial loss function based on the core layer and additional layer outputs;

[0231] performing a weighted sum operation on the core layer and additional layer loss values to determine a second loss function for the core layer parameters and the additional layer parameters.

[0232] Core layer:

[0233] Definition: In a neural network model, the core layer usually refers to those layers that have the greatest impact on the model output and contain the most critical features of the model.

[0234] Role: The core layer is responsible for extracting the main features of the input data, which are the basis for the model to make predictions or generate results.

[0235] Example: In an image classification model, the deep part of the convolutional layer is usually considered as the core layer, because it extracts high-level semantic features of the image.

[0236] Additional layer:

[0237] Definition: In contrast to the core layer, the additional layer refers to those layers that have a smaller impact on the model output and are mainly used to assist the work of the core layer.

[0238] Role: The additional layer can play the role of regularization, feature enhancement, information fusion, etc., helping the model to better learn and generalize.

[0239] Example: In a sequence-to-sequence model, the attention mechanism module can be regarded as an additional layer, which helps the model to focus on the relevant parts of the input sequence when generating sequences.

[0240] Original parameter matrix:

[0241] Definition: The original parameter matrix refers to the matrix composed of all the parameters that a neural network model initializes at the beginning of training. These parameters are what the model needs to learn and adjust through continuous training to better fit the data.

[0242] Function: The original parameter matrix contains all the information of the model and is the basis for the model to make predictions and generate results.

[0243] Example: In a fully connected layer, the weight matrix and bias vector form the original parameter matrix of that layer.

[0244] Relationship between the three

[0245] The core layer is learned from the original parameter matrix and contains the most critical features of the model.

[0246] The additional layer is also learned from the original parameter matrix, but its role is to assist the core layer.

[0247] The original parameter matrix is the starting point of the model, which is continuously updated through the training process and eventually forms a model that can complete a specific task.

[0248] Exemplarily, the second loss function can be calculated as follows.

[0249] 1. Parameter matrix decomposition:

[0250] Choose decomposition method:

[0251] Singular Value Decomposition (SVD): Decompose the original parameter matrix into the product of three matrices, which can effectively extract the main components of the matrix.

[0252] Non-negative Matrix Factorization (NMF): Only applicable to non-negative matrices, the sub-matrix obtained by decomposition is also non-negative. This is useful when dealing with some matrices with physical meaning, such as term frequency matrices.

[0253] Other methods: Depending on the specific model and task, other matrix decomposition methods such as Tucker decomposition, CP decomposition, etc. can also be considered.

[0254] Determine the decomposition dimension: Determine the dimension of the decomposed matrix based on experience or through cross-validation, which will affect the effect of decomposition and the complexity of the model.

[0255] 2. Core layer and additional layer parameter division:

[0256] Based on matrix decomposition results: Correspond the sub-matrix obtained by decomposition with the original parameter matrix to determine which part corresponds to the core layer parameter and which part corresponds to the additional layer parameter.

[0257] Task-based requirements: Analyze which parameters are more important for the completion of specific tasks. For example, for text generation tasks, parameters related to semantics may be more important.

[0258] Empirical judgment: Based on the structure of the model and understanding of the model, artificially divide the core layer and the additional layer.

[0259] 3. Define the loss function:

[0260] For the core layer:

[0261] Reconstruction loss: Measure the difference between the decomposed matrix product and the original matrix to ensure that the core layer can retain the main information of the original model.

[0262] Task-related loss: Define additional loss functions to constrain the behavior of the core layer according to specific tasks. For example, for text generation tasks, you can add language model loss.

[0263] For the additional layer:

[0264] Regularization loss: Prevent overfitting of the additional layer and improve the generalization ability of the model.

[0265] Task-specific loss: Define the corresponding loss function according to the function of the additional layer. For example, if the additional layer is used to enhance the model's ability in a certain field, you can add data in that field for training.

[0266] 4. Calculate the loss:

[0267] Forward propagation: Input the training data into the model to calculate the output of the core layer and the additional layer.

[0268] Calculate the loss: Substitute the calculated output and the corresponding target value into the defined loss function to get the loss value.

[0269] Weighted sum: Weighted sum of each loss value according to certain weights to get the final second loss function.

[0270] Through the above steps, the parameter matrix of the large language model can be decomposed, and the loss function for the core layer and the additional layer can be defined. The purpose of this is:

[0271] Improve the interpretability of the model: Split the model into multiple sub-modules to help understand the internal working mechanism of the model.

[0272] Enhance the flexibility of the model: Adjust the parameters of the core layer and the additional layer to control the performance of the model on different tasks.

[0273] Reducing the complexity of the model: Through matrix decomposition, the number of parameters of the model can be reduced, thereby reducing the computational load of the model.

[0274] For example:

[0275] Suppose there is a Transformer model for text generation, and you want to decompose the model's parameter matrix using SVD.

[0276] SVD decompose the weight matrix of the model to get three matrices U, Σ, V.

[0277] The first few columns of the U matrix are used as the core layer parameters, representing the semantic information of the model.

[0278] The last few rows of the V matrix are used as the additional layer parameters, representing the style information of the model.

[0279] Define the reconstruction loss to measure the difference between the product of the decomposed matrices and the original matrix.

[0280] Define the language model loss to encourage the core layer to generate text that conforms to language rules.

[0281] Define the style loss to encourage the additional layer to generate text with a specific style.

[0282] Weighted sum of reconstruction loss, language model loss and style loss to get the final second loss function.

[0283] By minimizing this loss function, the core layer of the model can capture the semantic information of the text well, and the additional layer can add different styles to the generated text.

[0284] In an optional embodiment of the present application, the step of determining the activation mode of the target layer and the target branch of the large language model based on the activation mode to determine the third loss function comprises:

[0285] Determine the target layer and target branch of the large language model based on the task type;

[0286] Determine the attention mechanism, gating mechanism and routing algorithm for the target layer and the target branch;

[0287] Determine the third loss function based on the activation mode through the attention mechanism, the gating mechanism and the routing algorithm.

[0288] For example, the third loss function can be calculated as follows.

[0289] 1. Define the target layer and the target branch:

[0290] Target layer:

[0291] Determine which layers have the most impact on the current task's output based on task requirements. For example, for a text generation task, focus on the output layer of the decoder.

[0292] This can be the last layer of the model or an intermediate layer.

[0293] Target branch:

[0294] If the model has multiple output branches, determine which branch to activate. For example, in multi-task learning, the model may predict multiple attributes simultaneously, and the branch related to the current task needs to be selected.

[0295] 2. Introduce attention mechanism:

[0296] Calculate attention weights:

[0297] Take the output of the target layer and other layers as input, and calculate the attention weight of each position through the attention mechanism.

[0298] Attention weights represent the degree of dependence of the current position on information from different positions.

[0299] Weighted sum:

[0300] According to the attention weight, perform weighted sum on the features of different positions to obtain the weighted feature representation.

[0301] Dynamic selection:

[0302] Through the attention mechanism, the model can dynamically select information from different positions, thus adapting to different inputs and tasks.

[0303] 3. Introduce gating mechanism:

[0304] Design a gating unit:

[0305] Design a gating unit to control the flow of information.

[0306] The output of the gating unit is a value between 0 and 1, representing the amount of information allowed to pass through.

[0307] Control information flow:

[0308] Multiply the output of the gating unit with the output of the target layer to control the flow of information.

[0309] Adaptive adjustment:

[0310] The parameters of the gating unit can be learned, allowing the model to adaptively adjust the flow of information.

[0311] 4. Design routing algorithm:

[0312] Define nodes and edges:

[0313] Consider different layers and branches of the model as nodes in the graph, and the connections between nodes as edges.

[0314] Calculate path weights:

[0315] Calculate the weight of each path based on task requirements and current input.

[0316] Select the optimal path:

[0317] Use routing algorithms (such as Dijkstra's algorithm, A* algorithm) to find the optimal path from input to output.

[0318] Activate target branches:

[0319] Activate target branches along the selected path.

[0320] 5. Define the third loss function:

[0321] Based on the activation method:

[0322] Define the corresponding loss function according to the selected activation method.

[0323] Attention mechanism: Use cross-entropy loss to measure whether the distribution of attention weights is reasonable.

[0324] Gating mechanism: Use binary cross-entropy loss to measure whether the output of the gating unit is correct.

[0325] Routing algorithm: Use path length or loss on the path as the loss function.

[0326] Combine other losses:

[0327] Weighted sum the third loss function and other loss functions (such as reconstruction loss, classification loss) to get the final loss function.

[0328] Through the above steps, flexible activation methods can be designed for the target layer and target branch of the large language model, and the corresponding loss function can be defined. This method can improve the adaptability of the model, making it better handle different tasks and inputs.

[0329] Example:

[0330] Suppose there is a Transformer model for machine translation. You can:

[0331] Define the target layer: the last layer of the decoder.

[0332] Introduce attention mechanism: Calculate the attention weights of the decoder self-attention layer to emphasize the context information related to the current translation word.

[0333] Introduce gating mechanism: Design a gating unit to control the influence of different encoder layers on the decoder.

[0334] Define loss function: Use cross-entropy loss to measure the difference between the model's predicted translation and the true translation, and use KL divergence to measure the smoothness of the attention distribution.

[0335] In this way, the model can dynamically adjust the attention and gating mechanisms according to the input source language sentence, so as to generate more accurate and fluent translation results.

[0336] In an optional embodiment of the present application, the step of generating the original model based on the first loss function, the second loss function and the third loss function comprises:

[0337] Determine the initial weights for the first loss function, the second loss function and the third loss function;

[0338] Adjust the initial weights through hyperparameter search and determine the target weights for the initial weights;

[0339] Calculate the total loss function through the target weights, the first loss function, the second loss function and the third loss function;

[0340] Generate the original model based on the total loss function.

[0341] Exemplarily, the original model can be generated in the following way.

[0342] 1. Weighted combination of loss functions:

[0343] Determine the weights:

[0344] Empirical setting: Assign an initial weight to each loss function based on experience or prior knowledge.

[0345] Hyperparameter search: Through methods such as grid search, random search or Bayesian optimization, dynamically adjust the weights during training to find the optimal combination.

[0346] Weighted sum:

[0347] Multiply each loss function by the corresponding weight and add them up to get the final total loss function.

[0348] 2. Selection of optimization algorithm:

[0349] Gradient descent algorithm:

[0350] Stochastic Gradient Descent (SGD): Randomly select one sample for update in each iteration.

[0351] Mini-batch Gradient Descent: Randomly select a small batch of samples for update in each iteration.

[0352] Adaptive learning rate algorithms such as Adam, RMSprop, etc.: Automatically adjust the learning rate to speed up convergence.

[0353] 3. Model training:

[0354] Initialize model parameters:

[0355] Randomly initialize the parameters of the model.

[0356] Iterative training:

[0357] Forward propagation: Input the input data into the model and calculate the output of the model.

[0358] Calculate loss: Calculate the loss value according to the total loss function.

[0359] Backward propagation: Calculate the gradient of the loss function with respect to the model parameters.

[0360] Parameter update: Update the model parameters using the selected optimization algorithm.

[0361] Early stopping mechanism:

[0362] Set a validation set, when the performance on the validation set no longer improves, stop training to prevent overfitting.

[0363] 4. Model saving:

[0364] Save the best model:

[0365] Save the model with the best performance on the validation set during training.

[0366] Save multiple models:

[0367] Save multiple models at different training stages for model integration or selection.

[0368] Example:

[0369] Calculate each loss function: According to the loss function defined above, calculate the first loss function (intermediate layer loss), the second loss function (core layer and additional layer loss), and the third loss function (target layer and target branch loss).

[0370] Weighted combination of loss functions: Add the three loss functions according to certain weights to get the total loss function.

[0371] Select optimizer: Choose a suitable optimizer, such as Adam, SGD, etc.

[0372] Initialize model parameters: Randomly initialize all parameters of the model.

[0373] Iterative training:

[0374] Forward propagation: Input training data into the model and calculate the output.

[0375] Calculate loss: Calculate the total loss.

[0376] Backward propagation: Calculate the gradient.

[0377] Parameter update: Update the model parameters using the optimizer.

[0378] Verification: Evaluate the model performance on the validation set.

[0379] Early stopping: Stop training when the performance on the validation set no longer improves.

[0380] Save model: Save the trained model.

[0381] Notes

[0382] Hyperparameter tuning: The weights of the loss function, learning rate, batch size, and other hyperparameters need to be carefully adjusted to achieve the best training results.

[0383] Regularization: L1 regularization, L2 regularization, and other methods can be used to prevent overfitting.

[0384] Data augmentation: Data augmentation can increase the diversity of training data and improve the generalization ability of the model.

[0385] Model integration: Multiple trained models can be integrated to improve the robustness of the model.

[0386] Through the above steps, multiple loss functions can be integrated to train an original model with excellent performance. This original model can serve as the basis for subsequent speculative sampling and provide services for different downstream tasks.

[0387] In order to make those skilled in the art better understand the embodiments of the present application, the following is an example of the embodiments of the present application.

[0388] In practical applications, the inference of large language models usually requires the use of autoregressive sampling. Their inference process is quite slow and needs to be decoded serially token by token. Therefore, the inference process of large models is often subject to memory access speed, and the generation of each token requires the transfer of all parameters from the storage unit to the computing unit, so that the memory access bandwidth becomes a serious bottleneck. Speculative sampling is a method that can fundamentally decode the computation-memory ratio, ensuring and using the exact same sampling distribution as the original model.

[0389] However, there are mainly three problems in the current speculative sampling inference process:

[0390] 1. The difference between the current approximate model and the original model is often large, resulting in inaccurate generation of many tokens in the decoding process, and ultimately still relying on the generation of large models, thereby increasing the communication cost.

[0391] 2. The current approximate model is fixed in size, and when deployed on the edge side, only a smaller approximate model that meets the requirements can be selected, and the full computing power of the edge side cannot be fully utilized.

[0392] 3. The current speculative sampling inference does not fully utilize network bandwidth, and in the case of limited network, the effect of speculative sampling will be affected, the generation speed is unstable, and the generation effect is poor.

[0393] The embodiments of the present application adopt a dynamic speculative sampling method for inference, which solves the above three problems in the following ways:

[0394] First, for the problem of too large difference between the generation effect of the approximate model and the original model, the present solution dynamically increases the size of the approximate model to make the approximate model as close as possible to the size of the original model, which can improve the generation ability of the approximate model as much as possible, and make the generation of the approximate model and the original model more similar, so as to alleviate the generation pressure of the original large model during evaluation, and reduce the evaluation and correction cost.

[0395] Second, the present solution modularizes the original language model, and realizes high modularization through equivalent mathematical transformation or adding predefined rules during training of the original model. Due to the consistency of the model ensured by modularization, the model can have generation ability only with core modules, and the effect of the approximate model can be gradually improved by adding optional modules; due to the fine granularity of modularization, the approximate model can be approximately considered to be continuously increased, so the approximate model can select the parameter size under the limit condition of the edge chip to utilize the computing power of all edge chips.

[0396] Third, the present scheme considers the network limited scenario, introduces the module transmission and cache of the original model in the process of using speculative sampling, as much as possible, when the network condition is good, the relevant modules of the original model can be transmitted to the approximate model as cache, so that the network bandwidth can be used as much as possible to make the effect of the approximate model as good as possible, and the network unlimited time is also used as much as possible, when the network transmission capacity is limited or there is no network transmission capacity, the better generation effect and stable generation speed can still be obtained based on the more relevant modules cached at present

[0397] In summary, the present scheme increases the utilization rate of network bandwidth in the speculative sampling process by performing fine-grained modularization on the original model, dynamically transmitting different modules during token speculative sampling, and caching modules to the approximate model, improves the generation effect and stability of the approximate model, and speeds up the generation speed, and maximizes the relevant ability when the network is limited.

[0398] The main technical solutions of the embodiments of the present application mainly include two parts: a speculative sampling model construction part and a speculative sampling inference part.

[0399] The following steps are taken for the speculative sampling model construction:

[0400] Step S1: Collect a large amount of text data, mainly from web pages, books, etc., and perform data matching on the data to construct training data for the large language model.

[0401] Step S2: Train the large language model, and introduce a pre-defined modularization loss function restriction during the training process, which can select to add restrictions such as:

[0402] (S2-1) To ensure that the model takes the output results of the intermediate layers, it can still produce effective generation effect, and the loss of the output logits of each layer is calculated.

[0403] (S2-2) The model is split into core layer parameters and additional layer parameters through matrix decomposition, low-rank transformation and other mathematical methods, the core layer can have core generation capability, and the additional layer can enhance certain aspects of capability to produce additional gain, and the loss of the single core layer and the additional layer is calculated.

[0404] (S2-3) The model increases the routing distribution module, selects to activate different layers and different branches of parameters according to the input, and realizes the generation of text generation, and different losses are calculated for different activation methods.

[0405] (S2-4) The granularity of the selected method in (S2-1) to (S2-3) is refined, and the loss of all layers and all branches is weighted and combined for optimization.

[0406] After training, the intermediate layer, core layer or shared expert layer can be used as an approximation model, and the complete model can be used as an original model for speculative sampling.

[0407] The following steps are used for speculative sampling model inference:

[0408] Step A1: deploy the approximation model on the edge side and the original model on the cloud data center.

[0409] Step A2: the user calls the inference service, and the approximation model and the original model interact dynamically for speculative sampling, and finally returns the inference result. The speculative sampling steps of dynamic caching are as follows:

[0410] (A2-1) The user inputs a question, links are established on both sides of the approximation model and the original model, and it is determined whether dynamic caching can be performed.

[0411] (A2-2) The approximation model generates a draft, and at the same time the original model starts sending additional modules to the approximation model, the approximation model accepts and deploys caching, and the newly cached modules do not participate in this generation.

[0412] (A2-3) The original model scores the generated content, modifies to obtain an output token, and returns it to the approximation model.

[0413] (A2-4) The approximation model returns the token to the user, and detects whether there is a deployed cache available, and if so, updates the approximation model.

[0414] (A2-5) Repeat (A2-2) to (A2-4) until the user can generate completely.

[0415] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the action sequence described, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0416] It should be noted that for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the action sequence described, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0417] With reference to Figure 3 , a structural block diagram of a generation apparatus for an inference result of text data provided in an embodiment of the present application is shown, and specifically can include the following modules:

[0418] An initial draft generation module 301 is configured to generate an initial draft for the text data by using an approximate model when the text data input by a user is acquired;

[0419] A target draft generation module 302 is configured to generate a target draft based on the initial draft by using an original model; the approximate model is a distilled version of the original model;

[0420] An inference result determination module 303 is configured to determine the inference result for the text data based on the target draft.

[0421] For the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0422] In addition, an embodiment of the present application further provides an electronic device, as shown in the figure, Figure 4 which comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404,

[0423] The memory 403 is configured to store a computer program;

[0424] The processor 401 is configured to execute the program stored in the memory 403, and realize the generation method of the inference result of the text data in any of the above embodiments;

[0425] The communication bus mentioned above can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0426] The communication interface is configured to realize the communication between the above terminal and other devices.

[0427] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0428] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0429] like Figure 5 As shown, in another embodiment of the present invention, a computer-readable storage medium 501 is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the method for generating inference results for text data as described in the above embodiments.

[0430] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0431] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0432] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0433] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The units as divided can or can not be physically reallocated, and can or can not be components independent of each other. In some embodiments, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electric, mechanical or other forms.

[0434] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0435] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0436] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts of the technical solutions that make contributions to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes various media that can store program codes, such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc.

[0437] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating reasoning results from text data, characterized in that, include: When the user inputs text data, an initial draft is generated based on the text data using an approximation model; Generate the target draft from the initial draft using the original model; The approximate model is a distilled version of the original model; Determine the reasoning result for the text data based on the target draft; Prior to the step of generating an initial draft for the text data using an approximation model, the method further includes: Training data for building large language models; Based on the training data, determine the intermediate layer output of the intermediate layer of the large language model, and calculate the first loss function between the intermediate layer output and the preset target value; Based on the training data, determine the core layer parameters and additional layer parameters of the large language model, and calculate the second loss function for the core layer parameters and additional layer parameters; Determine the activation methods for the target layer and target branch of the large language model, and determine the third loss function based on the activation methods; An original model is generated based on the first loss function, the second loss function, and the third loss function; the original model is a complete model, and the original model includes a task target layer that is related to the task type. An approximate model is generated based on the task objective layer.

2. The method according to claim 1, characterized in that, The original model is deployed on a cloud server, and the approximate model is deployed on a client device.

3. The method according to claim 1, characterized in that, The step of determining the intermediate layer output of the intermediate layer of the large language model based on the training data, and calculating the first loss function between the intermediate layer output and the preset target value includes: The intermediate layer is determined based on the characteristics of the network structure and task types of the large language model. The training data is input into the intermediate layer to obtain the output of the intermediate layer; Based on the task type and the preset target value, a first initial loss function is selected; The first initial loss function of the intermediate layer is weighted for different intermediate layers to calculate the first loss function between the output of the intermediate layer and the preset target value.

4. The method according to claim 1, characterized in that, The large language model includes a core layer and an additional layer. The step of determining the core layer parameters and additional layer parameters of the large language model based on the training data, and calculating the second loss function for the core layer parameters and additional layer parameters, includes: The original parameter matrix and decomposition method of the large language model are determined, and the original parameter matrix is ​​decomposed into multiple sub-matrices based on the decomposition method; The core layer parameters and the additional layer parameters are determined based on the submatrix and the original parameter matrix; Determine a second initial loss function for the core layer and the additional layer; The training data is input into the large language model to obtain the core layer and the additional layer outputs; Based on the outputs of the core layer and the additional layer, loss values ​​for the core layer and the additional layer are generated using the second initial loss function. A weighted summation operation is performed on the loss values ​​of the core layer and the additional layer to determine a second loss function for the parameters of the core layer and the parameters of the additional layer.

5. The method according to claim 1, characterized in that, The step of determining the activation methods for the target layer and target branch of the large language model, and determining the third loss function based on the activation methods, includes: The target layer and target branch of the large language model are determined based on the task type. Determine the attention mechanism, gating mechanism, and routing algorithm for the target layer and the target branch; The third loss function is determined based on the activation method using the attention mechanism, the gating mechanism, and the routing algorithm.

6. The method according to claim 1, characterized in that, The step of generating the original model based on the first loss function, the second loss function, and the third loss function includes: Determine the initial weights for the first loss function, the second loss function, and the third loss function; The initial weights are adjusted by hyperparameter search, and the target weights for the initial weights are determined. The total loss function is generated by calculating the target weight, the first loss function, the second loss function, and the third loss function. The original model is generated based on the total loss function.

7. An apparatus for generating reasoning results from text data, characterized in that, include: The initial draft generation module is used to generate an initial draft of the text data when the user inputs it, using an approximation model. The target draft generation module is used to generate a target draft based on an initial draft from an original model; the approximate model is a distilled version of the original model; The reasoning result determination module is used to determine the reasoning result for the text data based on the target draft. The initial draft generation module is also used to construct training data for a large language model; Based on the training data, determine the intermediate layer output of the intermediate layer of the large language model, and calculate the first loss function between the intermediate layer output and the preset target value; Based on the training data, the core layer parameters and additional layer parameters of the large language model are determined, and a second loss function is calculated for the core layer parameters and the additional layer parameters. The activation methods for the target layer and target branch of the large language model are determined, and a third loss function is determined based on the activation methods. An original model is generated based on the first loss function, the second loss function, and the third loss function. The original model is a complete model, and the original model includes a task target layer that is related to the task type. An approximate model is generated based on the task target layer.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-6.

Images