Method and device for generating reasoning result for text data

Abstract

The embodiment of the invention provides a method and a device for generating a reasoning result aiming at text data. The method comprises the following steps: generating an initial draft aiming at the text data through an approximation model; the target draft is generated based on the initial draft through the original model, and the reasoning result for the text data is determined based on the target draft, so that reasoning can be accelerated, and the reasoning time is greatly shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of generating inference results for text data, and in particular to a method for generating inference results for text data, a device for generating inference results for text data, an electronic device and a computer-readable storage medium. Background Art

[0002] The Large Language Model (LLM) is a machine learning model based on a neural network. By learning from a large amount of text data, it can generate text, translate languages, write different creative content, answer your questions, and even execute the instructions you give.

[0003] Autoregressive sampling is a commonly used sampling method in generative models, especially for sequence generation models like LLM. Its core idea is that when the model generates the next element of the sequence, it takes the previously generated elements as input to predict the next most likely element.

[0004] Inference of large language models usually requires the use of autoregressive sampling. Their inference process is quite slow and requires serial decoding token by token. Therefore, the inference process of large models is often constrained by memory access speed. Generating each token requires transferring all parameters from the storage unit to the computing unit, so the memory access bandwidth becomes a serious bottleneck, resulting in low inference speed. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for generating inference results for text data, so as to overcome the above problems or at least partially solve the above problems.

[0006] The embodiment of the present invention discloses a method for generating an inference result for text data, comprising:

[0007] When the text data input by the user is obtained, an initial draft for the text data is generated through the approximate model;

[0008] Generate a target draft based on the initial draft through the original model; the approximate model is a distilled version of the original model;

[0009] An inference result for the text data is determined based on the target draft.

[0010] Optionally, the original model is deployed on a cloud server, and the approximate model is deployed on a client device.

[0011] Optionally, before the step of generating an initial draft for the text data by using the approximate model, the method further includes:

[0012] Building training data for large language models;

[0013] Determine an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculate a first loss function between the intermediate layer output and a preset target value;

[0014] Determine core layer parameters and additional layer parameters of the large language model based on the training data, and calculate a second loss function for the core layer parameters and the additional layer parameters;

[0015] Determining an activation mode for a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode;

[0016] Generate an original model based on the first loss function, the second loss function and the third loss function; the original model is a complete model, and the original model includes a task target layer associated with a task type;

[0017] An approximate model is generated based on the task target layer.

[0018] Optionally, the step of determining an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculating a first loss function between the intermediate layer output and a preset target value includes:

[0019] Determining the intermediate layer based on the network structure of the large language model and the characteristics of the task type;

[0020] Input the training data into the intermediate layer to obtain the intermediate layer output;

[0021] Selecting a first initial loss function according to the task type and the preset target value;

[0022] The first initial loss functions of different intermediate layers are weighted to calculate the first loss function between the intermediate layer output and the preset target value.

[0023] Optionally, the large language model includes a core layer and an additional layer, and the steps of determining the core layer parameters and the additional layer parameters of the large language model based on the training data, and calculating the second loss function for the core layer parameters and the additional layer parameters include:

[0024] Determining an original parameter matrix and a decomposition method of the large language model, and decomposing the original parameter matrix into a plurality of sub-matrices based on the decomposition method;

[0025] Determining core layer parameters and additional layer parameters based on the sub-matrix and the original parameter matrix;

[0026] determining a second initial loss function for the core layer and the additional layer;

[0027] Inputting the training data into the large language model, obtaining core layer and additional layer outputs of the core layer and the additional layer;

[0028] Generate core layer and additional layer loss values ​​based on the core layer and additional layer outputs by using the second initial loss function;

[0029] A weighted sum operation is performed on the core layer and additional layer loss values ​​to determine a second loss function for the core layer parameters and the additional layer parameters.

[0030] Optionally, the step of determining an activation mode for a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode comprises:

[0031] Determining a target layer and a target branch of the large language model based on a task type;

[0032] Determining an attention mechanism, a gating mechanism, and a routing algorithm for the target layer and the target branch;

[0033] A third loss function is determined based on the activation method through the attention mechanism, the gating mechanism and the routing algorithm.

[0034] Optionally, the step of generating an original model based on the first loss function, the second loss function and the third loss function includes:

[0035] Determining initial weights for the first loss function, the second loss function, and the third loss function;

[0036] Adjusting the initial weights by hyperparameter search and determining target weights for the initial weights;

[0037] Generate a total loss function by calculating the target weight, the first loss function, the second loss function and the third loss function;

[0038] An original model is generated based on the total loss function.

[0039] The embodiment of the present invention also discloses a device for generating an inference result for text data, comprising:

[0040] An initial draft generation module, used to generate an initial draft for the text data through an approximate model when obtaining text data input by a user;

[0041] A target draft generation module, configured to generate a target draft based on an initial draft through an original model; the approximate model being a distilled version of the original model;

[0042] An inference result determination module is used to determine an inference result for the text data based on the target draft.

[0043] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] The memory is used to store computer programs;

[0045] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0046] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0047] The embodiments of the present invention include the following advantages:

[0048] In an embodiment of the present invention, an initial draft for the text data is generated through the approximate model; a target draft is generated based on the initial draft through the original model, and an inference result for the text data is determined based on the target draft, so as to achieve the following beneficial effects.

[0049] Accelerated reasoning: Significantly shorten the reasoning time.

[0050] Improve model adaptability: Be able to adapt to different tasks and inputs.

[0051] Balancing speed and precision: Providing flexible solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of a method for generating an inference result for text data provided in an embodiment of the present invention;

[0053] Figure 2 is a flowchart of the steps of a method for generating speculative sampling inference provided in an embodiment of the present invention;

[0054] Figure 3 It is a structural block diagram of a device for generating inference results for text data provided in an embodiment of the present invention;

[0055] Figure 4is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] The Large Language Model (LLM) is a machine learning model based on a neural network. By learning from a large amount of text data, it can generate text, translate languages, write different creative content, answer your questions, and even execute the instructions you give.

[0059] Autoregressive sampling is a commonly used sampling method in generative models, especially for sequence generation models like LLM. Its core idea is that when the model generates the next element of the sequence, it takes the previously generated elements as input to predict the next most likely element.

[0060] A bottleneck problem encountered by large language models (LLMs) during reasoning is slow reasoning speed. This bottleneck mainly stems from the autoregressive generation method and huge number of parameters of large language models (LLMs). To solve this problem, those skilled in the art have proposed a method called speculative sampling. However, there are still some problems with the current speculative sampling method, which limits its practical application.

[0061] There are three main problems in the current speculative sampling reasoning process:

[0062] 1. Approximate model accuracy problem:

[0063] Problem description: The performance gap between the current approximate model used for speculative sampling and the original LLM is large. As a result, many generated tokens are not accurate during the decoding process, and ultimately still need to rely on the original LLM for correction.

[0064] Impact: Although the approximate model can accelerate part of the decoding process, it cannot significantly reduce the calls to the original LLM due to its accuracy, thereby increasing the communication cost and offsetting part of the acceleration effect.

[0065] 2. Problem of approximate model size fixation:

[0066] Problem description: Current approximate models are usually of fixed size. When deployed on the client side, only a smaller model that matches the computing power of the device can be selected, which cannot fully utilize all the computing resources of the device.

[0067] Impact: This limits the performance of speculative sampling methods on different devices and cannot achieve better acceleration effects on devices with sufficient computing resources.

[0068] 3. Low network bandwidth utilization:

[0069] Problem description: The current speculative sampling method does not fully utilize the network bandwidth, and the effect of speculative sampling will be greatly affected when the network conditions are poor.

[0070] Impact: This results in the speculative sampling method having unstable generation speed and unsatisfactory generation effect in practical applications.

[0071] The embodiments of the present invention effectively solve the problems of low approximate model accuracy, fixed model size, low network bandwidth utilization, etc. existing in traditional speculative sampling methods through dynamic speculative sampling combined with technologies such as model modularization and module transmission cache.

[0072] First, in order to solve the problem of too large a difference between the effects of the approximate model and the original model, the embodiment of the present invention adopts a strategy of dynamically increasing the approximate model. By gradually increasing the size of the approximate model, it gradually approaches the original model, thereby significantly improving the generation ability of the approximate model and reducing the burden of the original model during the evaluation and correction process.

[0073] Secondly, the embodiment of the present invention performs fine-grained modularization on the original model. Through mathematical transformation and predefined rules, the model is split into multiple modules with relatively independent functions. This modular design not only ensures the consistency of the model, but also enables the approximate model to flexibly adjust its scale and capabilities by combining different modules. Therefore, the embodiment of the present invention can select the most appropriate module combination according to the computing power of the terminal device to maximize the use of hardware resources.

[0074] Finally, considering the instability of the network environment, the embodiment of the present invention introduces a module transmission and caching mechanism in the process of speculative sampling. When the network conditions are good, the module in the original model that is most relevant to the current task can be transferred to the approximate model end for caching. In this way, even when the network is limited, the approximate model can still use the cached modules to generate high-quality text, thereby ensuring the stability and efficiency of the generation process.

[0075] In summary, the embodiments of the present invention optimize the speculative sampling process through dynamic adjustment, modularization, caching and other technologies, effectively improve the generation efficiency, reduce the dependence on the original model, and enhance the robustness of the system in a complex network environment.

[0076] Reference Figure 1, shows a flowchart of a method for generating an inference result for text data provided in an embodiment of the present invention, which may specifically include the following steps:

[0077] Step 101, when the text data input by the user is obtained, an initial draft for the text data is generated by using an approximate model;

[0078] Step 102, generating a target draft based on the initial draft through the original model; the approximate model is a distilled version of the original model;

[0079] Step 103: Determine an inference result for the text data based on the target draft.

[0080] refer to Figure 2 , Figure 2 is a flowchart of the steps of a method for generating speculative sampling inference provided in an embodiment of the present invention;

[0081] Speculative sampling is a technique for optimizing the efficiency of large model reasoning. It accelerates the reasoning process by introducing approximate models and dynamic caching mechanisms during the reasoning process. The whole process is mainly divided into two stages:

[0082] Model deployment: Deploy an approximate model with weak computing power but fast response on the client device side, and deploy an original model with large parameters and powerful performance on the cloud server.

[0083] Dynamic interactive reasoning: After the user enters a question, the approximate model and the original model interact in real time to jointly complete the reasoning task.

[0084] The detailed steps can be shown as follows:

[0085] 1. Model deployment:

[0086] Approximate model: Usually a model with a small number of parameters, used to quickly generate initial results or candidate answers. It can be a distilled version of the original model or a small model specially trained for a specific task.

[0087] Original model: It has a large number of parameters, strong computing power, and can generate high-quality output. It is usually deployed in the cloud to provide powerful computing resources.

[0088] 2. Dynamic interactive reasoning:

[0089] 2-1. Establish connection and judgment:

[0090] After the user inputs text data, a connection is established between the approximate model and the original model.

[0091] The system determines whether the current task is suitable for dynamic caching. If the task is simple or does not require high real-time performance, the approximate model can be used directly to generate results.

[0092] To determine whether a task is complex, you can consider the following aspects:

[0093] Whether the number of factors involved in the task / question in the text data is greater than a preset threshold;

[0094] Whether the domain of the task / problem in the text data is the target domain;

[0095] The tasks / questions in the text data contain uncertainties:

[0096] Simple tasks / problems: The conditions of the problem are clear, the answers are relatively certain, and there are not too many vague or unknown factors.

[0097] Complex tasks / problems: Problems contain a lot of uncertainty, may have multiple possible answers, or require different judgments based on different situations.

[0098] The knowledge background required for the tasks / problems in the text data:

[0099] Simple problems: The knowledge required to solve the problems is relatively basic and does not require in-depth professional knowledge.

[0100] Complex questions: They require relatively professional knowledge or require the integration of knowledge from multiple fields to answer.

[0101] The tasks / questions in the text data contain logical relationships:

[0102] Simple tasks / questions: The logical relationship of the questions is relatively simple and the answers can be obtained through simple reasoning.

[0103] Complex tasks / problems: The logical relationships of the problems are complex and may involve multiple layers of nested conditions, cause-and-effect relationships, etc., requiring complex logical reasoning.

[0104] 2-2. Generate initial draft token and cache deployment:

[0105] The approximate model generates an initial draft token based on user input.

[0106] At the same time, the original model begins to send additional modules to the approximate model. These modules can be new neural network layers, attention mechanisms, etc., which are used to enhance the expressive power of the approximate model.

[0107] After the approximate model receives the additional module, it caches it for subsequent reasoning. It should be noted that the newly cached module will not participate in the generation of the current segment.

[0108] 2-3. Original model scoring and output:

[0109] The original model scores the initial draft token generated by the approximate model and modifies the initial draft token based on the scoring result to generate a new, higher quality target draft token.

[0110] 2-4. Return results and updates:

[0111] The approximate model returns the target draft token returned by the original model to the user.

[0112] At the same time, the approximate model detects whether there are any deployed caches available. If so, these cache modules are integrated into its own model to improve the accuracy of subsequent reasoning.

[0113] 2-5. Loop iteration:

[0114] Repeat steps 2-2 to 2-4 until the complete output result is generated.

[0115] The role of dynamic cache:

[0116] Accelerate reasoning: By caching some modules of the original model, the approximate model can use these modules to improve the generation quality and reduce the dependence on the original model, thereby accelerating the reasoning process.

[0117] Improve model adaptability: As the task progresses, the approximate model will continuously update its knowledge base to better adapt to different inputs and outputs.

[0118] Balancing speed and accuracy: Speculative sampling provides users with a flexible solution by making a trade-off between speed and accuracy.

[0119] Speculative sampling distributes the computational tasks of a large model to multiple models and uses a dynamic cache mechanism to improve the adaptability of the model, thereby achieving efficient and high-quality reasoning. This method has broad application prospects in natural language processing, machine translation and other fields.

[0120] In an embodiment of the present invention, an initial draft for the text data is generated through the approximate model; a target draft is generated based on the initial draft through the original model, and an inference result for the text data is determined based on the target draft, so as to achieve the following beneficial effects.

[0121] Accelerated reasoning: Significantly shorten the reasoning time.

[0122] Improve model adaptability: Be able to adapt to different tasks and inputs.

[0123] Balancing speed and precision: Providing flexible solutions.

[0124] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0125] In an optional embodiment of the present invention, before the step of generating an initial draft for text data by using an approximate model, the method further includes:

[0126] Building training data for large language models;

[0127] Determine an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculate a first loss function between the intermediate layer output and a preset target value;

[0128] Determine core layer parameters and additional layer parameters of the large language model based on the training data, and calculate a second loss function for the core layer parameters and the additional layer parameters;

[0129] Determining an activation mode for a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode;

[0130] Generate an original model based on the first loss function, the second loss function and the third loss function; the original model is a complete model, and the original model includes a task target layer associated with a task type;

[0131] An approximate model is generated based on the task target layer.

[0132] The embodiments of the present invention can construct training data for a large language model, with the purpose of providing sufficient and high-quality data for training the large language model.

[0133] For example, a large amount of text data, such as books, articles, codes, etc., can be collected and sorted, and cleaned and preprocessed to meet the model training requirements, so that the quality of training data directly affects the performance of the model. High-quality and diverse data can enable the model to learn richer language knowledge and patterns.

[0134] The embodiment of the present invention can determine the intermediate layer output and calculate the first loss function, the purpose of which is to ensure that the intermediate layer of the model also contains useful information and provide a basis for the construction of subsequent modularization and approximate models.

[0135] For example, the meaning of the intermediate layer output calculation loss is:

[0136] Intermediate layer output: When the neural network processes information, each layer will produce some intermediate results, which are called intermediate layer outputs.

[0137] Logits: In neural networks, logits usually refer to the output that has been linearly transformed but has not yet been activated. Calculating the loss on logits can more directly reflect the features learned by the model.

[0138] Calculate loss: Measure the learning effect of the model at each level by calculating the difference between the output of the intermediate layer and the target value.

[0139] The specific purposes are:

[0140] Ensure the effectiveness of each part of the model: Make sure that each part of the model is learning useful features, rather than relying solely on the last layer to complete the task.

[0141] Promote modularity: Provide a basis for subsequent modular decomposition. If the output of an intermediate layer can predict the target well, then it can be considered as an independent module.

[0142] The implementation is as follows:

[0143] Select the middle layer: Select the appropriate middle layer for loss calculation based on the network structure and task characteristics.

[0144] Define loss function: You can use common loss functions, such as cross entropy loss, mean square error, etc.

[0145] Weight distribution: The losses of different intermediate layers can be weighted to balance the importance of different layers.

[0146] For example, assuming a language model has 10 layers, the loss can be calculated for the output logits of the 5th and 8th layers, and the cross entropy loss can be calculated with the target label respectively.

[0147] By calculating the difference between the logits output by the middle layer of the model and the preset target value, the first loss function is obtained, which can ensure that the middle layer of the model can effectively extract features and provide support for subsequent tasks.

[0148] The embodiment of the present invention can determine the core layer parameters and the additional layer parameters and calculate the second loss function, with the purpose of splitting the model into two parts: core generation capability and enhancement capability, so as to facilitate flexible combination and build approximate models of different scales.

[0149] Exemplarily, the meaning of matrix decomposition, low-rank transformation split model:

[0150] Matrix factorization: Factorize the model's parameter matrix into the product of two or more smaller matrices.

[0151] Low-rank transformation: Approximate a high-rank matrix to a low-rank matrix to reduce the number of parameters.

[0152] Core layer parameters and additional layer parameters: The model parameters are divided into two parts. The core layer parameters are responsible for the basic generation ability of the model, and the additional layer parameters are responsible for enhancing the performance of the model on specific tasks.

[0153] The specific purposes are:

[0154] Reduce model complexity: Through matrix decomposition and low-rank transformation, the number of model parameters can be reduced, thereby reducing the computational complexity and storage space of the model.

[0155] Improve model interpretability: Splitting the model into multiple submodules helps to understand the internal working mechanism of the model.

[0156] Promote modularization: provide a basis for subsequent modular decomposition.

[0157] Implementation:

[0158] Select the decomposition method: You can use methods such as singular value decomposition (SVD) and non-negative matrix factorization (NMF).

[0159] Determine the core layer and additional layer: According to the task requirements and model structure, determine which parameters belong to the core layer and which parameters belong to the additional layer.

[0160] Calculate loss: Calculate loss for core layer parameters and additional layer parameters separately.

[0161] For example: decomposing the parameter matrix of a fully connected layer into two smaller matrices, one matrix represents the core layer parameters and the other matrix represents the additional layer parameters.

[0162] By splitting the model parameters into core layer parameters and additional layer parameters through methods such as matrix decomposition, and calculating their loss functions separately, the interpretability of the model can be improved and subsequent modularization and pruning can be facilitated.

[0163] The embodiment of the present invention can determine the activation mode of the target layer and the target branch and calculate the third loss function, the purpose of which is to improve the adaptability of the model so that it can adapt to different inputs and tasks.

[0164] For example, the meaning of the routing distribution module is:

[0165] Routing distribution: Dynamically select different model components to process information based on different inputs.

[0166] Different layers and different branches: There can be multiple parallel branches or multiple layers in the model, and the routing distribution module can choose to activate different branches or layers as needed.

[0167] Purpose:

[0168] Improve the adaptability of the model: Make the model adaptable to different inputs and tasks.

[0169] Enhance the expressiveness of the model: Improve the expressiveness of the model by combining different model components.

[0170] Implementation:

[0171] Attention Mechanism: Attention mechanism can be used to dynamically select different model components.

[0172] Gating Mechanism: Gating mechanism can be used to control the flow of information.

[0173] Routing algorithm: A special routing algorithm can be designed to select the optimal path.

[0174] The model adds a routing distribution module and selects to activate the parameters of different layers and branches according to the input to generate text, and calculates different losses for different activation methods.

[0175] For example, in a machine translation task, different translation model components can be dynamically selected based on the length and complexity of the input sentence.

[0176] According to different tasks, selecting different target layers and target branches and calculating the corresponding loss functions can enhance the expressiveness of the model and enable it to better complete various natural language processing tasks.

[0177] The embodiment of the present invention can generate an original model, the purpose of which is to build a complete and high-performance large language model.

[0178] Based on the three loss functions calculated above, the model is optimized as a whole to obtain an original model with excellent performance. This original model has strong language generation capabilities and can serve as the basis for subsequent tasks.

[0179] The embodiment of the present invention can generate an approximate model based on the task target layer. Since the task target layer is associated with the task type, it is equivalent to building a smaller, faster model that can be targeted at specific task types for deployment on resource-constrained devices.

[0180] According to task requirements, some layers or modules in the original model are selected as approximate models, thereby reducing the computational complexity and storage space of the model and improving the reasoning speed.

[0181] The embodiment of the present invention can generate reasoning results for text data, the purpose of which is to:

[0182] Leverage the advantages of the original model and the approximate model: the original model provides high-quality output, while the approximate model provides fast response. By combining the capabilities of the two, efficient and high-quality text generation is achieved.

[0183] Dynamically adjust model selection: Dynamically choose to use the original model or the approximate model, or a combination of the two, based on the complexity and real-time requirements of the task.

[0184] Improve user experience: For simple tasks, the approximate model can quickly generate results to meet the real-time needs of users; for complex tasks, the original model can provide more accurate and comprehensive answers.

[0185] By feeding the input text into the original model and the approximate model, the final inference result is obtained, thereby achieving various natural language processing tasks such as text generation, machine translation, question-answering systems, etc. according to the training objectives of the model.

[0186] In an embodiment of the present invention, training data of a large language model is constructed; based on the training data, an intermediate layer output of an intermediate layer of the large language model is determined, and a first loss function between the intermediate layer output and a preset target value is calculated; based on the training data, core layer parameters and additional layer parameters of the large language model are determined, and a second loss function for the core layer parameters and the additional layer parameters is calculated; an activation method for a target layer and a target branch of the large language model is determined, and a third loss function is determined based on the activation method; an original model is generated based on the first loss function, the second loss function, and the third loss function; the original model is a complete model, and the original model includes a task target layer associated with a task type; an approximate model is generated based on the task target layer; when text data input by a user is obtained, an inference result for the text data is generated through the original model and the approximate model, thereby improving the model inference speed.

[0187] Furthermore, the following beneficial effects can be achieved.

[0188] Improve inference efficiency: The fast response of the approximate model can significantly shorten the inference time.

[0189] Improve generation quality: High-quality output of the original model can ensure the accuracy and fluency of the generated results.

[0190] Enhance the adaptability of the model: The dynamic caching mechanism enables the model to adjust its behavior according to the specific circumstances of the task, improving the generalization ability of the model.

[0191] Reduce computing costs: By reasonably allocating computing tasks, the consumption of computing resources can be reduced.

[0192] In an optional embodiment of the present invention, the step of determining the intermediate layer output of the intermediate layer of the large language model based on the training data and calculating the first loss function between the intermediate layer output and a preset target value includes:

[0193] Determining the intermediate layer based on the network structure of the large language model and the characteristics of the task type;

[0194] Input the training data into the intermediate layer to obtain the intermediate layer output;

[0195] Selecting a loss function according to the task type and the preset target value;

[0196] The loss functions of different intermediate layers are weighted to calculate a first loss function between the intermediate layer output and a preset target value.

[0197] Exemplarily, the first loss function may be calculated as follows.

[0198] Determine the middle layer:

[0199] Analyze model structure: Carefully study the network structure of the large language model, including the type of each layer, the number of neurons, and the connection method.

[0200] Consider the characteristics of the task: According to the specific task (such as text generation, question answering, translation, etc.), select the middle layer that has an important impact on the task. For example, for the text generation task, you may pay more attention to the middle layer related to semantics.

[0201] Empirical judgment: Based on previous experience and research results, select some intermediate layers that usually have strong feature extraction capabilities, such as the last few layers of the encoder layer or the output layer of the attention mechanism.

[0202] Calculate the intermediate layer output:

[0203] Forward propagation: Input the training data into the model, perform forward propagation calculations, and obtain the output of each intermediate layer.

[0204] Extract logits: Extract logits from the selected intermediate layer. Logits are usually the output after linear transformation but not after activation function.

[0205] Determine the preset target value:

[0206] According to the task definition: For different tasks, the preset target value will be different.

[0207] Text generation tasks: The target value can be the probability distribution of the next word.

[0208] Classification task: The target value can be the probability of each category.

[0209] Regression task: The target value can be a continuous value.

[0210] Calculate the loss function:

[0211] Choose a loss function: Choose an appropriate first initial loss function based on the task type and the nature of the target value.

[0212] Cross entropy loss: commonly used in classification tasks and text generation tasks.

[0213] Mean squared error: commonly used in regression tasks.

[0214] Calculate loss: Substitute the logits output by the intermediate layer and the preset target value into the first initial loss function to calculate the loss function.

[0215] Weighted processing:

[0216] Assign weights: If you choose multiple intermediate layers to calculate the loss, you can assign different weights to the first initial loss of each intermediate layer.

[0217] Weight adjustment: Determine the appropriate weights through experiments or experience, calculate the first loss function between the intermediate layer output and the preset target value, and balance the impact of different intermediate layers on the final loss.

[0218] For example, suppose there is a Transformer model for text generation, and the encoder output of the last layer and the output of the first decoder layer are selected as the intermediate layers.

[0219] The training data is fed into the model to obtain the output of these two intermediate layers.

[0220] These two output logits are compared with the one-hot encoding of the next word (as the target value).

[0221] The losses of these two intermediate layers are calculated using the cross entropy loss function.

[0222] Assign weights to the two losses separately, for example, the loss weight of the encoder output is 0.6, and the loss weight of the decoder output is 0.4.

[0223] The two losses are weighted and summed to obtain the final first loss function.

[0224] By minimizing this loss function, the intermediate layer of the model can better predict the next word, thereby improving the generation quality of the model.

[0225] In an optional embodiment of the present invention, the large language model includes a core layer and an additional layer, and the steps of determining the core layer parameters and the additional layer parameters of the large language model based on the training data, and calculating the second loss function for the core layer parameters and the additional layer parameters include:

[0226] Determining an original parameter matrix and a decomposition method of the large language model, and decomposing the original parameter matrix into a plurality of sub-matrices based on the decomposition method;

[0227] Determining core layer parameters and additional layer parameters based on the sub-matrix and the original parameter matrix;

[0228] determining a second initial loss function for the core layer and the additional layer;

[0229] Inputting the training data into the large language model, obtaining core layer and additional layer outputs of the core layer and the additional layer;

[0230] Generate core layer and additional layer loss values ​​based on the core layer and additional layer outputs by using the second initial loss function;

[0231] A weighted sum operation is performed on the core layer and additional layer loss values ​​to determine a second loss function for the core layer parameters and the additional layer parameters.

[0232] Core layer:

[0233] Definition: In a neural network model, core layers usually refer to those layers that have the greatest impact on the model output and contain the most critical features of the model.

[0234] Function: The core layer is responsible for extracting the main features in the input data, which are the basis for the model to make predictions or generate results.

[0235] For example: In an image classification model, the deep part of the convolutional layers are often considered to be the core layers because they extract high-level semantic features of the image.

[0236] Additional Layers:

[0237] Definition: In contrast to the core layer, additional layers are those that have little impact on the model output and are mainly used to assist the core layer in its work.

[0238] Function: Additional layers can play a role in regularization, feature enhancement, information fusion, etc., helping the model to learn and generalize better.

[0239] For example: In a sequence-to-sequence model, the attention mechanism module can be seen as an additional layer that helps the model focus on relevant parts of the input sequence when generating sequences.

[0240] Original parameter matrix:

[0241] Definition: The original parameter matrix refers to the matrix composed of all the parameters initialized at the beginning of the neural network model training. These parameters are what the model needs to learn and are adjusted through continuous training so that the model can better fit the data.

[0242] Function: The original parameter matrix contains all the information of the model and is the basis for the model to predict and generate.

[0243] For example: In a fully connected layer, the weight matrix and the bias vector constitute the original parameter matrix of the layer.

[0244] The relationship between the three

[0245] The core layer is learned from the original parameter matrix and contains the most critical features of the model.

[0246] The additional layers are also learned from the original parameter matrix, but their role is to assist the core layer.

[0247] The original parameter matrix is ​​the starting point of the model, which is continuously updated through the training process to eventually form a model that can complete a specific task.

[0248] Exemplarily, the second loss function may be calculated as follows.

[0249] 1. Parameter matrix decomposition:

[0250] Select the decomposition method:

[0251] Singular Value Decomposition (SVD): Decomposing the original parameter matrix into the product of three matrices can effectively extract the main components of the matrix.

[0252] Non-negative matrix factorization (NMF): It is only applicable to non-negative matrices, and the decomposed sub-matrices are also non-negative. This is useful when dealing with matrices with physical meanings, such as word frequency matrices.

[0253] Other methods: Depending on the specific model and task, you can also consider using other matrix decomposition methods, such as Tucker decomposition, CP decomposition, etc.

[0254] Determine the decomposition dimension: Determine the dimension of the decomposed matrix based on experience or through cross-validation, which will affect the decomposition effect and the complexity of the model.

[0255] 2. Division of core layer and additional layer parameters:

[0256] Based on the matrix decomposition results: match the decomposed sub-matrices with the original parameter matrix to determine which parts correspond to the core layer parameters and which parts correspond to the additional layer parameters.

[0257] Based on task requirements: Combined with specific tasks, analyze which parameters are more important for completing the task. For example, for text generation tasks, you may pay more attention to parameters related to semantics.

[0258] Empirical judgment: According to the structure of the model and the understanding of the model, the core layer and additional layer are artificially divided.

[0259] 3. Define the loss function:

[0260] For the core layer:

[0261] Reconstruction loss: measures the difference between the decomposed matrix product and the original matrix, ensuring that the core layer can retain the main information of the original model.

[0262] Task-related loss: Depending on the specific task, additional loss functions are defined to constrain the behavior of the core layer. For example, for text generation tasks, the loss of the language model can be added.

[0263] For additional layers:

[0264] Regularization loss: prevents overfitting of additional layers and improves the generalization ability of the model.

[0265] Task-specific loss: Define the corresponding loss function according to the function of the additional layer. For example, if the additional layer is used to enhance the model's ability in a specific field, data from that field can be added for training.

[0266] 4. Calculate the loss:

[0267] Forward propagation: Input the training data into the model and calculate the output of the core layer and additional layers.

[0268] Calculate the loss: Substitute the calculated output and the corresponding target value into the defined loss function to obtain the various loss values.

[0269] Weighted summation: Each loss value is weighted and summed according to a certain weight to obtain the final second loss function.

[0270] Through the above steps, the parameter matrix of the large language model can be decomposed and the loss function can be defined for the core layer and the additional layer respectively. The purpose of this is to:

[0271] Improve model interpretability: Splitting the model into multiple submodules helps understand the internal working mechanism of the model.

[0272] Enhance the flexibility of the model: The performance of the model on different tasks can be controlled by adjusting the parameters of the core layer and additional layers.

[0273] Reduce the complexity of the model: Through matrix decomposition, the number of model parameters can be reduced, thereby reducing the amount of model calculations.

[0274] For example:

[0275] Suppose there is a Transformer model for text generation, and you want to perform SVD decomposition on the parameter matrix of the model.

[0276] Perform SVD decomposition on the model's weight matrix to obtain three matrices U, Σ, and V.

[0277] The first few columns of the U matrix are used as core layer parameters to represent the semantic information of the model.

[0278] The last few rows of the V matrix are used as additional layer parameters to represent the style information of the model.

[0279] Define the reconstruction loss to measure the difference between the factorized matrix product and the original matrix.

[0280] Define the language model loss to encourage the core layer to generate text that conforms to language rules.

[0281] Define a style loss to encourage additional layers to generate text with a specific style.

[0282] The reconstruction loss, language model loss and style loss are weighted and summed to obtain the final second loss function.

[0283] By minimizing this loss function, the core layer of the model can capture the semantic information of the text well, while the additional layers can add different styles to the generated text.

[0284] In an optional embodiment of the present invention, the step of determining an activation mode for a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode comprises:

[0285] Determining a target layer and a target branch of the large language model based on a task type;

[0286] Determining an attention mechanism, a gating mechanism, and a routing algorithm for the target layer and the target branch;

[0287] A third loss function is determined based on the activation method through the attention mechanism, the gating mechanism and the routing algorithm.

[0288] Exemplarily, the third loss function may be calculated as follows.

[0289] 1. Define the target layer and target branch:

[0290] Target layer:

[0291] According to the task requirements, determine which layers have the greatest impact on the output of the current task. For example, for a text generation task, you may pay more attention to the output layer of the decoder.

[0292] It can be the last layer of the model or a layer in the middle.

[0293] Target branch:

[0294] If the model has multiple output branches, you need to determine which branch to activate. For example, in multi-task learning, the model may predict multiple attributes at the same time, and you need to select the branch that is relevant to the current task.

[0295] 2. Introducing the attention mechanism:

[0296] Calculate the attention weights:

[0297] The outputs of the target layer and other layers are taken as input, and the attention weight of each position is calculated through the attention mechanism.

[0298] The attention weight indicates the degree of dependence of the current position on information at different positions.

[0299] Weighted sum:

[0300] The features at different positions are weighted and summed according to the attention weights to obtain the weighted feature representation.

[0301] Dynamic Selection:

[0302] Through the attention mechanism, the model can dynamically select information at different positions to adapt to different inputs and tasks.

[0303] 3. Introduce gating mechanism:

[0304] Design of the gate control unit:

[0305] Design a gating unit to control the flow of information.

[0306] The output of the gating unit is a value between 0 and 1, indicating the amount of information allowed to pass.

[0307] Controlling information flow:

[0308] The output of the gating unit is multiplied by the output of the target layer to control the flow of information.

[0309] Adaptive Adjustment:

[0310] The parameters of the gating unit can be learned, allowing the model to adaptively adjust the information flow.

[0311] 4. Design routing algorithm:

[0312] Define nodes and edges:

[0313] The different layers and branches of the model are considered as nodes in the graph, and the connections between nodes are considered as edges.

[0314] Calculate path weight:

[0315] Calculate the weight of each path based on task requirements and current input.

[0316] Select the best path:

[0317] Use routing algorithms (such as Dijkstra algorithm, A* algorithm) to find the optimal path from input to output.

[0318] Activate the target branch:

[0319] Activates the target branch along the selected path.

[0320] 5. Define the third loss function:

[0321] Based on activation method:

[0322] According to the selected activation method, define the corresponding loss function.

[0323] Attention mechanism: Cross entropy loss can be used to measure whether the distribution of attention weights is reasonable.

[0324] Gating mechanism: Binary cross entropy loss can be used to measure whether the output of the gating unit is correct.

[0325] Routing algorithm: You can use path length or loss on the path as the loss function.

[0326] Combined with other losses:

[0327] The third loss function is weightedly summed with other loss functions (such as reconstruction loss and classification loss) to obtain the final loss function.

[0328] Through the above steps, we can design flexible activation methods for the target layer and target branch of the large language model and define the corresponding loss function. This approach can improve the adaptability of the model and enable it to better handle different tasks and inputs.

[0329] For example:

[0330] Suppose there is a Transformer model for machine translation. You can:

[0331] Define the target layer: the last layer of the decoder.

[0332] Introducing the attention mechanism: Calculate the attention weights of the decoder’s self-attention layer to emphasize the contextual information related to the current translated word.

[0333] Introduce gating mechanism: Design a gating unit to control the impact of different encoder layers on the decoder.

[0334] Define the loss function: Use cross entropy loss to measure the difference between the translation results predicted by the model and the true translation, and use KL divergence to measure the smoothness of the attention distribution.

[0335] In this way, the model can dynamically adjust the attention and gating mechanisms according to the input source language sentence, thereby generating more accurate and fluent translation results.

[0336] In an optional embodiment of the present invention, the step of generating an original model based on the first loss function, the second loss function and the third loss function includes:

[0337] Determining initial weights for the first loss function, the second loss function, and the third loss function;

[0338] Adjusting the initial weights by hyperparameter search and determining target weights for the initial weights;

[0339] Generate a total loss function by calculating the target weight, the first loss function, the second loss function and the third loss function;

[0340] An original model is generated based on the total loss function.

[0341] Exemplarily, the original model may be generated in the following manner.

[0342] 1. Weighted combination of loss functions:

[0343] Determine the weights:

[0344] Empirical setting: Assign an initial weight to each loss function based on experience or prior knowledge.

[0345] Hyperparameter search: Dynamically adjust weights during training to find the optimal combination through methods such as grid search, random search, or Bayesian optimization.

[0346] Weighted sum:

[0347] Multiply each loss function by the corresponding weight and then add them together to get the final total loss function.

[0348] 2. Selection of optimization algorithm:

[0349] Gradient Descent Algorithm:

[0350] Stochastic Gradient Descent (SGD): A sample is randomly selected for update at each iteration.

[0351] Mini-batch gradient descent (mini-batch SGD): A mini-batch of samples is randomly selected for update at each iteration.

[0352] Adaptive learning rate algorithms such as Adam and RMSprop: can automatically adjust the learning rate to speed up convergence.

[0353] 3. Model training:

[0354] Initialize model parameters:

[0355] Randomly initialize the parameters of the model.

[0356] Iterative training:

[0357] Forward propagation: Input data is fed into the model to calculate the output of the model.

[0358] Calculate loss: Calculate the loss value according to the total loss function.

[0359] Backpropagation: Calculate the gradient of the loss function with respect to the model parameters.

[0360] Parameter Update: Update the model parameters using the selected optimization algorithm.

[0361] Early stopping mechanism:

[0362] Set a validation set and stop training when the performance on the validation set no longer improves to prevent overfitting.

[0363] 4. Model saving:

[0364] Save the best model:

[0365] During the training process, save the model with the best performance on the validation set.

[0366] To save multiple models:

[0367] Models at multiple different training stages can be saved for model integration or selection.

[0368] For example:

[0369] Calculate each loss function: According to the loss function defined above, calculate the first loss function (intermediate layer loss), the second loss function (core layer and additional layer loss), and the third loss function (target layer and target branch loss).

[0370] Weighted combined loss function: Add the three loss functions according to certain weights to obtain the total loss function.

[0371] Select optimizer: Select a suitable optimizer, such as Adam, SGD, etc.

[0372] Initialize model parameters: Randomly initialize all parameters of the model.

[0373] Iterative training:

[0374] Forward propagation: input training data into the model and calculate the output.

[0375] Calculate loss: Calculate the total loss.

[0376] Backpropagation: Calculate gradients.

[0377] Parameter Update: Update the model parameters using the optimizer.

[0378] Validation: Evaluate model performance on the validation set.

[0379] Early stopping: Stop training when the performance on the validation set no longer improves.

[0380] Save Model: Save the trained model.

[0381] Precautions

[0382] Hyperparameter tuning: Hyperparameters such as the weight of the loss function, learning rate, batch size, etc. need to be carefully tuned to achieve the best training results.

[0383] Regularization: You can use methods such as L1 regularization and L2 regularization to prevent overfitting.

[0384] Data enhancement: Data enhancement can increase the diversity of training data and improve the generalization ability of the model.

[0385] Model integration: Multiple trained models can be integrated to improve the robustness of the model.

[0386] Through the above steps, multiple loss functions can be integrated to train an original model with excellent performance. This original model can serve as the basis for subsequent speculative sampling and provide services for different downstream tasks.

[0387] In order to enable those skilled in the art to better understand the embodiment of the present invention, the embodiment of the present invention is described below using an example.

[0388] In practical applications, the reasoning of large language models usually requires the use of autoregressive sampling. Their reasoning process is quite slow and requires serial decoding token by token. Therefore, the reasoning process of large models is often constrained by memory access speed. Generating each token requires transferring all parameters from the storage unit to the computing unit, so the memory access bandwidth becomes a serious bottleneck. Speculative sampling is a method that can fundamentally decode the memory access ratio and ensure that the sampling distribution is exactly the same as that of the original model.

[0389] However, there are three main problems in the current speculative sampling reasoning process:

[0390] 1. The effect of the approximate model currently used is often quite different from that of the original model, resulting in inaccurate generation of many tokens during the decoding process. In the end, it still mainly relies on the generation of large models, which in turn increases communication costs.

[0391] 2. The current approximate model is of fixed size. When deployed on the end side, only a smaller approximate model that meets the requirements can be selected, which cannot bring into play the full computing power of the end side.

[0392] 3. The current speculative sampling reasoning does not fully utilize the network bandwidth, and when the network is limited, the effect of speculative sampling will be affected, the generation speed will be unstable, and the generation effect will be poor.

[0393] The embodiment of the present invention adopts dynamic speculative sampling to perform reasoning, and solves the above three problems in the following ways:

[0394] First, to address the problem of a large gap between the generation effect of the approximate model and the effect of the original model, this solution dynamically increases the size of the approximate model to make the approximate model as close to the size of the original model as possible. This can improve the generation ability of the approximate model as much as possible, making the generation of the approximate model closer to the original model, thereby alleviating the generation pressure of the original large model during evaluation and reducing the cost of evaluation and correction.

[0395] Second, this solution modularizes the original language model, and achieves high modularity through equivalent mathematical transformations or adding predefined rules when training the original model. Since modularization ensures the consistency of the model, only the core module is required to have the ability to generate the model, and adding optional modules will gradually improve the effect of the approximate model; since the modularization granularity is fine enough, it can be roughly considered that the approximate model can be continuously increased, so the approximate model can choose the parameter size under the extreme conditions of the end-side chip and give full play to the computing power of all end-side chips.

[0396] Third, this solution takes into account the scenario of network constraints. In the process of using speculative sampling, the module transmission and caching of the original model are introduced. When the network conditions are good, the relevant modules of the original model can be transferred to the approximate model for use as cache. In this way, the network bandwidth can be used as much as possible to make the effect of the approximate model as good as possible, and the network is not restricted. When the network transmission capacity is limited or there is no network transmission capacity, it is still possible to obtain better generation effects and stable generation speed based on the currently cached more relevant modules.

[0397] In summary, this solution modularizes the original model into fine-grained blocks, dynamically transmits different modules while speculatively sampling tokens, and caches modules to approximate models. This increases the utilization of network bandwidth during the speculative sampling process, improves the generation effect and stability of the approximate model, speeds up the generation, and maximizes the retention of relevant capabilities when the network is limited.

[0398] The main technical solution of the embodiment of the present invention is mainly divided into two parts, a speculative sampling model construction part and a speculative sampling reasoning part.

[0399] There are the following steps for speculative sampling model construction:

[0400] Step S1: Collect a large amount of text data, mainly from web pages, books, etc., match these data, and build training data for a large language model.

[0401] Step S2: Train the large language model and introduce predefined modular loss function restrictions during the training process. You can choose to add restrictions such as:

[0402] (S2-1) To ensure that the model can still produce effective generation results by taking the output results of some intermediate layers, the loss is calculated for the output logits of each intermediate layer.

[0403] (S2-2) The model is split into core layer parameters and additional layer parameters through mathematical methods such as matrix decomposition and low-rank transformation. The core layer can have the core generation capability, and the additional layer can enhance certain capabilities and generate additional gains. The loss is calculated separately for the core layer and the additional layer.

[0404] (S2-3) The model adds a routing distribution module and selects parameters of different layers and branches to generate text according to the input, and calculates different losses for different activation methods.

[0405] (S2-4) Refine the granularity of the method selected from (S2-1) to (S2-3), perform weighted combination of the losses of all layers and all branches, and perform optimization.

[0406] Once trained, you can use the intermediate layers, core layers, or shared expert layers as approximate models, and the full model as the original model for speculative sampling.

[0407] There are the following steps for speculative sampling model inference:

[0408] Step A1: Deploy the approximate model on the client side and the original model in the cloud data center.

[0409] Step A2: The user calls the inference service, and the approximate model and the original model interact with each other through dynamic speculative sampling, and finally returns the inference result. The steps of dynamic cache speculative sampling are as follows:

[0410] (A2-1) The user inputs a problem, the approximate model is linked to the original model, and it is determined whether dynamic caching can be performed.

[0411] (A2-2) The approximate model generates a draft, and the original model starts to send additional modules to the approximate model. The approximate model accepts and caches the modules, and the newly cached modules do not participate in the generation of this segment.

[0412] (A2-3) The original model scores the generated content, modifies it to obtain an output token, and returns it to the approximate model.

[0413] (A2-4) The approximate model returns the segment token to the user and checks whether there is an available deployed cache. If so, the approximate model is updated.

[0414] (A2-5) Repeat (A2-2) to (A2-4) until the user can be generated.

[0415] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0416] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0417] Reference Figure 3 , shows a structural block diagram of a device for generating reasoning results for text data provided in an embodiment of the present invention, which may specifically include the following modules:

[0418] An initial draft generation module 301 is used to generate an initial draft for the text data by using an approximate model when obtaining text data input by a user;

[0419] A target draft generation module 302 is used to generate a target draft based on the initial draft through the original model; the approximate model is a distilled version of the original model;

[0420] The inference result determination module 303 is used to determine the inference result for the text data based on the target draft.

[0421] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0422] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0423] Memory 403, used for storing computer programs;

[0424] The processor 401 is used to implement the method for generating an inference result for text data described in any of the above embodiments when executing the program stored in the memory 403:

[0425] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0426] The communication interface is used for communication between the above terminal and other devices.

[0427] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0428] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0429] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is also provided, in which instructions are stored. When the computer-readable storage medium 501 is run on a computer, the computer executes the method for generating reasoning results for text data described in the above embodiment.

[0430] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0431] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0432] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0433] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0434] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0435] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0436] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0437] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating inference results for text data, characterized in that: include: When the text data input by the user is obtained, an initial draft for the text data is generated through the approximate model; Generate a target draft based on the initial draft through the original model; The approximate model is a distilled version of the original model; An inference result for the text data is determined based on the target draft.

2. The method according to claim 1, characterized in that The original model is deployed on a cloud server, and the approximate model is deployed on a client device.

3. The method according to claim 1 or 2, characterized in that: Before the step of generating an initial draft for text data by using an approximate model, the method further includes: Building training data for large language models; Determine an intermediate layer output of an intermediate layer of the large language model based on the training data, and calculate a first loss function between the intermediate layer output and a preset target value; Determine core layer parameters and additional layer parameters of the large language model based on the training data, and calculate a second loss function for the core layer parameters and the additional layer parameters; Determining an activation mode for a target layer and a target branch of the large language model, and determining a third loss function based on the activation mode; Generate an original model based on the first loss function, the second loss function and the third loss function; the original model is a complete model, and the original model includes a task target layer associated with a task type; An approximate model is generated based on the task target layer.

4. The method according to claim 3, characterized in that The step of determining the intermediate layer output of the intermediate layer of the large language model based on the training data and calculating a first loss function between the intermediate layer output and a preset target value comprises: Determining the intermediate layer based on the network structure of the large language model and the characteristics of the task type; Input the training data into the intermediate layer to obtain the intermediate layer output; Selecting a first initial loss function according to the task type and the preset target value; The first initial loss functions of different intermediate layers are weighted to calculate the first loss function between the intermediate layer output and the preset target value.

5. The method according to claim 3, characterized in that: The large language model includes a core layer and an additional layer, and the steps of determining the core layer parameters and the additional layer parameters of the large language model based on the training data, and calculating the second loss function for the core layer parameters and the additional layer parameters include: Determining an original parameter matrix and a decomposition method of the large language model, and decomposing the original parameter matrix into a plurality of sub-matrices based on the decomposition method; Determining core layer parameters and additional layer parameters based on the sub-matrix and the original parameter matrix; determining a second initial loss function for the core layer and the additional layer; Inputting the training data into the large language model, obtaining core layer and additional layer outputs of the core layer and the additional layer; Generate core layer and additional layer loss values ​​based on the core layer and additional layer outputs by using the second initial loss function; A weighted sum operation is performed on the core layer and additional layer loss values ​​to determine a second loss function for the core layer parameters and the additional layer parameters.

6. The method according to claim 3, characterized in that The step of determining an activation mode for a target layer and a target branch of the large language model and determining a third loss function based on the activation mode comprises: Determining a target layer and a target branch of the large language model based on a task type; Determining an attention mechanism, a gating mechanism, and a routing algorithm for the target layer and the target branch; A third loss function is determined based on the activation method through the attention mechanism, the gating mechanism and the routing algorithm.

7. The method according to claim 3, characterized in that The step of generating an original model based on the first loss function, the second loss function and the third loss function comprises: Determining initial weights for the first loss function, the second loss function, and the third loss function; Adjusting the initial weights by hyperparameter search and determining target weights for the initial weights; Generate a total loss function by calculating the target weight, the first loss function, the second loss function and the third loss function; An original model is generated based on the total loss function.

8. A device for generating inference results for text data, characterized in that: include: An initial draft generation module, used for generating an initial draft for the text data by using an approximate model when obtaining text data input by a user; A target draft generation module, configured to generate a target draft based on an initial draft through an original model; the approximate model being a distilled version of the original model; An inference result determination module is used to determine an inference result for the text data based on the target draft.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images