Text generation method and electronic device based on MoE large model

Abstract

The present disclosure relates to a text generation method and electronic device. The method includes: inputting prompt information into a MoE large model, determining a vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training; using a text evaluation model, based on the vector representation, determining a score value for the prompt information, wherein the score value for the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information; based on the score value for the prompt information, determining the number of experts to be activated in the MoE large model, wherein the number of experts to be activated is not greater than the maximum number of experts; and generating an output corresponding to the prompt information based on the activated experts of the MoE large model. In this way, the present disclosure can enable the MoE large model to dynamically adjust the number of activated experts according to the complexity of the prompt word, thereby improving the accuracy of the output and reducing hardware costs.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computers, and more particularly to a text generation method and an electronic device. Background Art

[0002] Generative large language models are deep learning models capable of generating natural language text. These models are based on complex neural network architectures, such as transformers, and are pre-trained on large amounts of text data to learn the statistical properties and patterns of language. The core capability of generative large language models is to generate coherent, logical, and grammatically correct text. This means that the models can output entirely new textual content, such as articles, conversations, and poems, rather than simply reproducing or restating content from the training data.

[0003] In large language models, a mixture of experts (MoE) can be integrated into the transformation layer to reduce computational complexity by activating a smaller number of experts. However, current solutions only activate a fixed number of experts, making the model inflexible. Summary of the Invention

[0004] According to exemplary embodiments of the present disclosure, a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating text are provided. The method can determine the number of experts to activate in a large MoE model based on the complexity of the prompt information, thereby enabling dynamic control and use of the model.

[0005] In a first aspect of the present disclosure, an information processing method is provided, comprising: obtaining prompt information; inputting the prompt information into a MoE large model, and determining a vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training; further comprising using a text evaluation model to determine a score value of the prompt information based on the vector representation, wherein the score value of the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information; determining the number of experts to be activated in the MoE large model based on the score value of the prompt information, wherein the number of experts to be activated is not greater than the maximum number of experts; and generating an output corresponding to the prompt information based on the activated experts of the MoE large model.

[0006] In a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which instructions, when executed by the at least one processing unit, enable the electronic device to perform the method described in accordance with the first aspect of the present disclosure.

[0007] In a third aspect of the present disclosure, a text generation device is provided, including: an acquisition unit configured to acquire prompt information; a first determination unit configured to input the prompt information into the MoE large model, and determine the vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training; a second determination unit configured to use a text evaluation model to determine a score value of the prompt information based on the vector representation, wherein the score value of the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information; a third determination unit configured to determine the number of experts to be activated in the MoE large model based on the score value of the prompt information, wherein the number of experts to be activated is not greater than the maximum number of experts; and a generation unit configured to generate an output corresponding to the prompt information based on the activated experts of the MoE large model.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which has machine-executable instructions stored thereon, and when the machine-executable instructions are executed by a device, the device can perform the method described according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, wherein the computer-executable instructions implement the method described according to the first aspect of the present disclosure when executed by a processor.

[0010] In a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processing circuit configured to execute the method described according to the first aspect of the present disclosure.

[0011] The purpose of providing the summary of the invention section is to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary of the invention section is not intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram showing a system according to some embodiments of the present disclosure is shown;

[0014] Figure 2 A schematic flow chart of a method for text generation according to some embodiments of the present disclosure is shown;

[0015] Figure 3A schematic diagram illustrating a text generation process according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A block diagram illustrating an example apparatus according to some embodiments of the present disclosure; and

[0017] Figure 5 A block diagram is shown of an example device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] In the embodiments of the present disclosure, the MoE large model may be referred to as a MoE large language model (LLM), a large model based on the MoE architecture, a large language model based on the MoE, a generative large language model, a large language model based on the transformer architecture, a large language model, a large model, etc., without limitation in the present disclosure. The large model in the embodiments of the present disclosure can be applied to a variety of application scenarios, such as writing articles, generating dialogues, creating poetry, generating content for social media / news, generating code / annotations / documents, generating teaching materials, and assisting language learning.

[0020] Large language models can deeply understand and process natural language, using neural networks to accomplish a variety of complex tasks, such as text generation and knowledge question answering. Large language models can be based on a transformer architecture, which effectively handles long-range dependencies through a self-attention mechanism and excels in parallel computing. Large language models can include multiple transformer layers, such as an encoder or decoder layer. Transformer layers can include self-attention layers, feedforward neural network (FNN) layers, and layer normalization (LN) layers.

[0021] The self-attention layer allows the model to establish connections between words or phrases at different positions, capturing long-range dependencies. The FNN layer further applies nonlinear transformations to the output of the self-attention layer. The LN layer normalizes all activations for each sample. For each sample, the LN layer calculates the mean and standard deviation of all activations for that sample and then uses these statistics to normalize each activation to have unit mean and unit variance.

[0022] In practical applications, the MoE architecture is introduced to reduce the cost of large language models during training and inference.

[0023] However, traditional large language models based on MoEs (MoEs) don't fully address the cost issue. MoEs, through gating functions, can only activate a fixed number of expert networks and cannot dynamically adjust the number of activated experts based on the difficulty of the prompt. Their processing power is limited for complex prompts, making it difficult to generate effective responses. Meanwhile, simple prompts can lead to excessive resource consumption.

[0024] To address the aforementioned issues and potential other problems, embodiments of the present disclosure provide a method for text generation. In embodiments of the present disclosure, a text evaluation model can be used to assess the complexity of prompt information input to the MoE model, thereby dynamically adjusting the number of experts activated by the MoE model. The text evaluation model can score the complexity of the vector representation corresponding to the prompt information, and based on the score, the number of experts to be activated in the MoE model can be determined.

[0025] In this way, the complexity of the prompt is scored using a text evaluation model, dynamically adjusting the number of experts activated by the MoE model. For complex prompts, the number of experts can be increased to generate high-quality responses; for simple prompts, the MoE model can reduce the number of activated experts. This approach avoids the resource mismatch problem associated with a fixed number of activated experts, effectively improving resource utilization, reducing hardware and energy costs, and providing users with higher-quality and more efficient text generation services.

[0026] Figure 1 Schematic diagram of system 100 according to some embodiments of the present disclosure is shown. System 100 includes a MoE large model 102 and an evaluation system 104, wherein evaluation system 104 includes a context manager 106, a text evaluator 108 (or simply evaluator), and a system dynamic scheduler 110 (or simply scheduler).

[0027] like Figure 1As shown, the MoE model 102 and the evaluation system 104 can receive and transmit information to each other, collaborating to complete related tasks. The MoE model 102 is a deep learning model architecture for natural language processing. It is composed of multiple sub-models, or experts. Each expert is essentially a feed-forward neural network (FFN) with specific parameter settings and computational structure. After specific training, it can perform specialized learning for different data features or tasks. For example, some experts excel at semantic understanding, while others are more proficient in grammatical analysis. This specialized division of labor enables the MoE model 102 to have stronger generalization capabilities, adapt to a wider range of input information, and reduce the limitations that a single model may encounter when handling complex tasks. Furthermore, the MoE model 102 also includes a gating function. The gating function is a special mechanism whose input is the current input data and whose output is a weight vector for each expert, reflecting the importance of each expert in processing the current input data, forming a probability distribution. Based on this probability distribution, the model can determine which experts to activate to process the input data.

[0028] For example, assume that there are N experts, and they can be expressed as: . Exemplarily, the gating function can be expressed as:

[0029] (1)

[0030] In formula (1), represents the scoring function, Indicates the expert 's rating.

[0031] For example, the output of MoE can be expressed as:

[0032] (2)

[0033] Sparse MoE is an implementation of MoE that, for an MoE with N expert groups, only activates the most important expert models. For example, when N = 16, K = 4 experts can be activated based on the output of the gating function, thereby achieving sparsity and reducing computational complexity. This approach is called Top-K MoE, for example, Top-4 MoE. For example, the output of Top-K MoE can be expressed as:

[0034] (3)

[0035] For example, in a large MoE model, the encoding layer or decoding layer may include a self-attention layer, a Top-K-MoE layer, and a LN layer. Alternatively, it can be understood as replacing the FFN layer in the large language model with a MoE layer.

[0036] The configuration of the MoE large model, including the number of layers, vector dimensions, and the MoE parameter K, is fixed before the model training process and cannot be changed after training. These configurations directly determine the computational cost consumed by the large model, including hardware and energy costs.

[0037] In the embodiment of the present disclosure, an evaluation system 104 is introduced. By working in conjunction with the evaluation system 104, the number of activated experts can be dynamically adjusted according to the complexity of the prompt information input into the MoE large model 102.

[0038] In some embodiments of the present disclosure, after receiving the prompt information, the MoE big model 102 can process the prompt information and convert it into a vector representation. It can be understood that the prompt information input by the user is mostly presented in the form of natural language. The prompt information input by the user to the big model can be a piece of text used to guide the model to generate a specific output. For example, the prompt information can be a question, a sentence fragment, an instruction, or any other form of text. The big model can predict the next content based on the prompt information. For example, in a task of generating text, if the prompt information is "Today's weather", the big model may generate words such as "sunny" or "cloudy" to complete the output sentence.

[0039] Exemplarily, the prompt information may include a word sequence, such as a plurality of word tokens.

[0040] The MoE large model 102 can encode semantic, grammatical, and other features of the prompt information and convert them into vector representations. Exemplarily, the vector representations output by the intermediate layers of the decoder of the MoE large model 102 can be stored in the context manager 106. For example, the context manager 106 can store the intermediate hierarchical features in a vector buffer. For example, the decoder outputs of the 16th and 32nd layers of the 32-layer decoder of the MoE large model 102 can be stored in the buffer, where the outputs of the 16th and 32nd layers are both 4096-dimensional vectors.

[0041] Furthermore, the text evaluator 108 can calculate the difficulty and complexity of the prompt information for the MoE model 102 based on the vector representation of the prompt information, and represent this complexity as a score. Specifically, the text evaluator 108 can process the vector representation using a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0042] Specifically, the text evaluator 108 may use the decoder output of the vector buffer to predict the difficulty and complexity of the prompt word to the MoE large model 102.

[0043] The system dynamic scheduler 110 can be implemented as a configurable scheduling function. In some embodiments of the present disclosure, the developer of the MoE large model 102 can freely define multiple thresholds for the complexity of prompt information, with the aim of adapting to different application scenarios and task requirements. Different scenarios have different requirements for the accuracy and efficiency of prompt information processing. As a quantitative standard, the threshold can accurately define the complexity range and provide a basis for resource allocation. Exemplarily, the system dynamic scheduler 110 can determine the number of experts to be activated based on multiple thresholds. Specifically, based on the score value output by the text evaluator 108, the minimum first-level threshold higher than the score value can be screened out from the preset threshold to clarify the threshold level of the prompt information and achieve an accurate match between complexity and resource allocation.

[0044] Furthermore, based on the results of the system dynamic scheduler 110, the MoE large model 102 activates a corresponding number of experts and generates output. Furthermore, in embodiments of the present disclosure, the threshold can also be adjusted based on the server's operating status. When the server is highly loaded, the threshold is raised to reduce the number of activated experts and ensure system stability; when the server is under low load, the threshold is lowered to fully utilize resources and improve processing efficiency.

[0045] Figure 2 A schematic flow chart of a method 200 for text generation according to some embodiments of the present disclosure is shown. In box 202, prompt information is obtained. In box 204, the prompt information is input into the MoE large model, and a vector representation corresponding to the prompt information is determined, wherein the MoE large model has a maximum number of experts determined through training. In box 206, a text evaluation model is used to determine a score value for the prompt information based on the vector representation, wherein the score value for the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information. In box 208, based on the score value for the prompt information, the number of experts to be activated in the MoE large model is determined, wherein the number of experts to be activated is not greater than the maximum number of experts. In box 210, an output corresponding to the prompt information is generated based on the activated experts of the MoE large model.

[0046] Exemplarily, the vector representation corresponding to the prompt information includes: hierarchical features corresponding to the prompt information output by the intermediate layer of the MoE large model. As mentioned above, the hierarchical features can be output by the intermediate layer of the decoder of the MoE large model, and these hierarchical features can be saved in the vector buffer by the context manager.

[0047] For example, the text evaluation model can be understood as follows Figure 1 The evaluation system 104 or text evaluator 108 shown in FIG.

[0048] In some embodiments of the present disclosure, a mode in which a text evaluation model and a MoE large model work together is adopted to process prompt information more efficiently and energy-efficiently. The text evaluation model can evaluate the complexity of the prompt information for the MoE large model, and present the evaluation results in the form of a score value. It can be understood that the model cannot directly understand the prompt information in natural language, so the MoE large model will pre-process the input prompt information, output the corresponding hierarchical features from the middle layer, and convert the word sequence in the prompt information into a vector representation. Specifically, when the prompt information is input, it will undergo a tokenization process to divide the prompt information into a series of word units, which are then mapped with the vocabulary and converted into corresponding vector representations, also known as embeddings.

[0049] For example, the text evaluation model may use quantitative performance indicators to represent the complexity of the prompt information. For example, the obtained score value may be any quantitative performance indicator, such as the accuracy, recall rate, cross entropy, etc. of sequence prediction.

[0050] In some embodiments, the text evaluation model can be based on CNN, which extracts features through convolutional layers and pooling layers and is suitable for discovering local patterns in sequences. For example, the convolution operation of the convolutional layer can be expressed as:

[0051] (4)

[0052] In formula (4) is the input sequence, is the convolution kernel (or filter), is the position subscript in the sequence.

[0053] In CNN, each convolution operation can be followed by a nonlinear activation function, for example, represented as:

[0054] (5)

[0055] The pooling layer in CNN is used to reduce the dimension of the data. For example, the maximum pooling operation can be expressed as:

[0056] (6)

[0057] In Equation (6), k represents the size of the pooling window. The output layer in a CNN may include only one neuron and does not use an activation function.

[0058] In some embodiments, the text evaluation model can be based on RNN, which processes sequence data through recursive connections and can be suitable for processing context dependencies in sequences. Exemplarily, the hidden state update in RNN can be expressed as:

[0059] (7)

[0060] In formula (7), Is the first The hierarchical features of word units, and is the weight matrix, is the bias term, is the time step input.

[0061] In RNN, the output of the model ( ) can be determined based on the hidden state, for example, as:

[0062] (8)

[0063] In formula (8), represents the output weight matrix, Represents the output bias term.

[0064] In an embodiment of the present disclosure, a neural network training system may be used to train a text evaluation model. Exemplarily, the training system may include back propagation, stochastic gradient descent, parallel training, and the like.

[0065] For example, during forward propagation, the loss function can be calculated using the model. The loss function is then differentiated with respect to the model parameters to obtain the gradient. The gradient is then backpropagated back to the model to update the model parameters. For example, during stochastic gradient descent, an appropriate learning rate can be selected based on the task and model, and the model parameters are then iteratively updated based on the gradient during an iterative process. For example, during parallel training, the training data can be divided into multiple batches, with each batch trained on a single GPU. If the model is too large to be completed on a single GPU, the model can be split into multiple parts and trained separately on multiple GPUs.

[0066] Optionally, during model training, a batch normalization layer can be added to the neural network model to speed up training. Optionally, during model training, a regularization term can be added to the loss function to prevent overfitting.

[0067] Optionally, during the model training process, forward propagation, backpropagation, and stochastic gradient descent can be repeated until a preset stopping condition is reached. In this way, a trained text evaluation model can be obtained and better adapted to specific tasks.

[0068] In some embodiments, multiple thresholds may be determined, for example, multiple threshold intervals formed by the multiple thresholds corresponding to multiple numbers of candidate experts, where the difference between the number of the multiple threshold intervals and the number of the multiple thresholds is 1. In some embodiments, in response to a score falling within a first threshold interval among the multiple threshold intervals, the number of candidate experts corresponding to the first threshold interval is determined as the number of experts to be activated.

[0069] In some embodiments of the present disclosure, in order to process prompt information more accurately and determine the complexity score of prompt information, multiple thresholds for the complexity of prompt information for large models can be defined based on usage scenarios, customer needs and other factors. The threshold intervals formed by these thresholds correspond one-to-one to the number of multiple candidate experts, where the difference between the number of multiple threshold intervals and the number of multiple thresholds is one. The score value is compared with multiple thresholds, and the minimum first-level threshold value higher than the score value is selected. According to the threshold interval where the score value is located, the corresponding number of candidate experts, that is, the number of experts to be activated, is determined. For example, assuming that the multiple thresholds include threshold 1 and threshold 2, and threshold 1 is less than threshold 2, then the multiple threshold intervals are: less than threshold 1, threshold 1 to threshold 2, and greater than threshold 2.

[0070] It's understandable that each of the multiple candidate expert numbers is less than the maximum number of experts, and the maximum number of experts is determined by the MoE model during training. Furthermore, different usage scenarios and customer needs may require different models to process prompt information. For example, in real-time interactive scenarios, customers may prioritize response speed. In this case, a relatively low threshold can be set to allow the model to activate fewer experts when prompt information is less complex, speeding up processing. In scenarios where the accuracy of the results is extremely high, a higher threshold can be set to ensure that a sufficient number of experts are involved in processing complex prompt information.

[0071] Taking a large MoE model with N=16 experts and a maximum of K=4 experts activated simultaneously as an example, 4-1=3 thresholds can be configured in advance, from small to large. For example, 0.5, 0.8, and 0.9 correspond to four threshold intervals: (0-0.5), (0.5-0.8), (0.8-0.9), and (>0.9). Each threshold interval corresponds to a different number of candidate experts (e.g., 1, 2, 3, and 4, respectively). After the text evaluator predicts the complexity and difficulty of the prompt information and assigns a score, the system dynamic scheduler 110 can perform a threshold determination, selecting the lowest first-level threshold above the score, thereby determining the threshold interval in which the score falls. For example, for a prompt word with a score of 0.47, 0.5>0.47, and the first-level threshold of 0.5 is selected. The score of this prompt word falls within the threshold interval of (0-0.5), and the number of experts to be activated can be 1 accordingly. For a prompt word with a degree value of 0.87, since 0.8 < 0.87 < 0.9, the third-level threshold of 0.9 is selected. This places the prompt word's score within the threshold range of (0.8-0.9), and accordingly, the number of experts to be activated is 3. For prompt words exceeding all three thresholds, the scheduler selects the fourth-level threshold range, and accordingly, the number of experts to be activated is 4. In this way, the MoE model can accurately allocate expert resources based on the actual complexity of the prompt information, ensuring processing effectiveness while avoiding resource waste.

[0072] In some embodiments of the present disclosure, when determining multiple thresholds of the MoE large model, the working status of the server running the model can be comprehensively considered. The working status of the server changes dynamically, and the threshold can be adjusted dynamically according to the working status of the server to ensure the efficiency and quality of the output generated by the MoE large model based on the prompt information. The working status of the server can include one or more key indicators. The number of tasks processed simultaneously by the server (such as the number of prompt words) can reflect its current load pressure. If too many tasks are processed simultaneously, the processing capacity of the server will be challenged, which may lead to slower response speed or even freezes. The utilization rate of the graphics processing unit (GPU), the memory occupancy rate of the server and the running speed of the server are also important indicators. Among them, the GPU plays a key role in the calculation of the large model. High utilization may mean that the server is in a high-intensity computing state.

[0073] Based on the working status information of the server, multiple thresholds can be determined more scientifically. When the server is in a busy state, for example, when more than a certain number (such as 20) of prompt messages are processed at the same time, or the GPU utilization rate exceeds a certain proportion (such as 50%), or the memory occupancy rate is too high or the running speed is too slow, the system dynamic scheduler 110 can determine a higher threshold, such as increasing the previously set thresholds by a predetermined value, such as increasing the original thresholds of 0.5, 0.8, and 0.9 by 0.05 to 0.55, 0.85, and 0.95. By raising the threshold, each prompt message is responded to by fewer activated experts, thereby reducing the burden on the server, ensuring the stable operation of the system, and maintaining processing efficiency to a certain extent. In this way, the threshold is dynamically adjusted according to the real-time working status of the server, which can achieve the optimal configuration of computing resources and improve the overall performance of the MoE large model.

[0074] In some embodiments of the present disclosure, the system has two operating modes, illustratively referred to as the first operating mode (also referred to as Mode A) and the second operating mode (also referred to as Mode B), providing more flexible solutions for different application requirements. In the first operating mode, after prompt information is input into the MoE model, a score is output for the prompt information as a whole, which determines the number of experts to be activated. Based on the activated experts in the MoE model, a complete output is generated in one go based on the prompt information. Specifically, the evaluator and scheduler select a corresponding level k threshold based on the complexity of the prompt information. The MoE model then activates k experts to provide a comprehensive response, which in turn generates the model input. When generating the output, these activated experts combine their respective expertise to conduct in-depth analysis and processing of the prompt information, ultimately generating a complete output. This operating mode provides complete and coherent responses, meeting users' needs for comprehensive information. By activating an appropriate number of experts, the accuracy and professionalism of the responses can be guaranteed, improving content quality. This is suitable for scenarios requiring complete and comprehensive responses, such as writing articles, reports, and answering detailed questions. After the user enters a prompt word, the system can provide a complete answer covering all key points in one go. It can be understood that in the first working mode, the number of activated experts is determined once and for all based on the complexity of the prompt information.

[0075] In the second operating mode, the text evaluation model can be used to determine a current score based on the vector representation of the prompt information and the vector representations of multiple word-grams in the generated output. Furthermore, based on the current score, the number of experts to be activated in the MoE large model to generate the next word-gram of the multiple word-grams is determined. Specifically, the system determines the corresponding vector representations based on the prompt information and the last preset number of word-grams generated in the output generated by the MoE large model in the current state (assuming the first t word-grams in the generated output). The text evaluator can determine a score based on the current vector representation and, based on a comparison of this score with a preset threshold, determine the number of experts to be activated to generate the next word-gram (i.e., the t+1th word-gram). The MoE large model can then generate the next word-gram based on these activated experts. This process repeats until the complete text output is generated. For example, a fixed-size window can be used to select the last t words generated, and the number of experts to be activated to generate the next word-gram can be determined based on the vector representations of the words within the window.

[0076] It is understandable that the preset number of word units that have been generated provide important contextual information for predicting the next word unit. This working mode can make dynamic predictions based on the previous content generated in real time, adjust the reply strategy in a timely manner, has strong real-time performance, and can dynamically adjust the reply according to the context, providing a more natural and smooth interactive experience. By gradually generating words, it can better adapt to different contexts and user needs, and improve the flexibility and pertinence of the reply. It is suitable for scenarios that require real-time interaction, automatic completion or text generation, such as chatbot conversations, intelligent writing assistance, etc. Subsequent content can be dynamically generated based on existing replies to make conversation or text generation smoother. It is understandable that in the second working mode, the number of experts to be activated is re-determined for the next word to be output. That is to say, in the process of determining the complete output, the number of experts to be activated is dynamically updated.

[0077] Figure 3 FIG. 3 is a schematic flow chart showing a process 300 of text generation according to some embodiments of the present disclosure. Figure 3 As shown, at the initial stage of the entire system operation, the MoE large model can be loaded. For example, the trained MoE large model has determined the maximum number of experts, denoted as K, such as K=4 or other values.

[0078] All parameters and functions of the MoE model are loaded into the server's GPU or CPU. These parameters, such as weights and biases learned during training, determine the model's processing and ability to process input data. Functions, for example, contain the program code for the model's calculations, transformations, and other operations, such as the self-attention mechanism and feedforward neural network functions. These parameters and functions are fundamental to the proper operation of the MoE model. After the MoE model is successfully loaded, the system receives requests through the network portal. These requests contain user-entered prompts, and users expect the MoE model to generate responses based on these prompts to answer their questions. The difficulty of the prompts varies, requiring different numbers of experts to be activated. By integrating a context manager, a text evaluator, and a system dynamic scheduler, working in conjunction with the MoE model, the number of activated experts can be dynamically adjusted based on the difficulty of the prompt. Complex prompts require more experts to process, while simple prompts require fewer, thus avoiding wasted resources.

[0079] For example, after receiving a request, the MoE model processes the prompt information and outputs hierarchical features from the intermediate layer. The context manager stores these intermediate features in a vector buffer. The text evaluator calculates the complexity of the prompt information based on the vector representation and expresses it as a score.

[0080] Furthermore, the dynamic scheduler can read the configured multi-level thresholds and adjust them based on the current operating status of the server in the MoE large model. Specifically, the dynamic scheduler adjusts these thresholds based on the current server hardware conditions, such as GPU utilization and memory usage, as well as the number of requests. It then selects the lowest threshold that is greater than the score given by the text evaluator.

[0081] like Figure 3As shown, the system has two operating modes. In operating mode A, after determining a score based on the prompt, the dynamic scheduler determines the number of experts to activate based on the score. The activated experts then conduct an in-depth analysis of the prompt, integrating their expertise to generate a complete response to the prompt in one go. This mode is suitable for scenarios where completeness and accuracy of responses are critical, such as document generation and professional question answering. In operating mode B, the MoE model re-invokes the context manager when determining the next word to respond. The context manager determines the vector representation of multiple words and the prompt in the currently generated output. The text evaluator recalculates the score, and the dynamic scheduler recalculates the number of experts to activate to predict the next word. This cycle repeats until a complete response is generated. This mode is suitable for real-time interactive scenarios, such as chatbot conversations, where subsequent content can be dynamically generated based on the previous content, making responses more timely and natural, and improving the user experience.

[0082] It is understood that the embodiments of the present disclosure do not limit the training method of the MoE large model. For example, the MoE large model can be obtained through training through processes such as backpropagation and stochastic gradient descent. Exemplarily, the trained MoE large model is a Top-K-MoE large model with a determined K value, where the K value is the aforementioned maximum number of experts.

[0083] It should be understood that in the embodiments of the present disclosure, "first", "second", "third", etc. are only used to indicate that multiple objects may be different, but at the same time do not exclude that two objects are the same, and should not be interpreted as any limitation on the embodiments of the present disclosure.

[0084] It should also be understood that the division of the modes, situations, categories and embodiments in the embodiments of the present disclosure is only for the convenience of description and should not constitute a special limitation. The features in various modes, categories, situations and embodiments can be combined with each other when it is logical.

[0085] It should also be understood that the above content is only intended to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Those skilled in the art may make various modifications, variations, or combinations based on the above content. Such modifications, variations, or combinations are also within the scope of the embodiments of the present disclosure.

[0086] It should also be understood that the description of the above content focuses on emphasizing the differences between the various embodiments, and the same or similar points can be referenced or borrowed from each other. For the sake of brevity, they will not be repeated here.

[0087] Figure 41 shows a schematic block diagram of an example apparatus 400 according to some embodiments of the present disclosure. The apparatus 400 may be implemented in software, hardware, or a combination of both. Figure 4 As shown, the apparatus 400 includes an acquiring unit 402 , a first determining unit 404 , a second determining unit 406 , a third determining unit 408 and a generating unit 410 .

[0088] The acquisition unit 402 is configured to acquire prompt information. The first determination unit 404 is configured to input the prompt information into the MoE large model and determine the vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training. The second determination unit 406 is configured to use a text evaluation model to determine a score value of the prompt information based on the vector representation, wherein the score value of the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information. The third determination unit 408 is configured to determine the number of experts to be activated in the MoE large model based on the score value of the prompt information, wherein the number of experts to be activated is not greater than the maximum number of experts. The generation unit 410 is configured to generate an output corresponding to the prompt information based on the activated experts of the MoE large model.

[0089] In some embodiments, the third determination unit 408 is configured to: determine multiple thresholds, multiple threshold intervals consisting of the multiple thresholds correspond one-to-one to the multiple numbers of candidate experts, wherein the difference between the number of the multiple threshold intervals and the number of the multiple thresholds is one; and in response to the first threshold interval of the score value in the multiple threshold intervals, determine the number of candidate experts corresponding to the first threshold interval as the number of experts to be activated.

[0090] In some embodiments, the third determining unit 408 is configured to: obtain a working status of a server running the MoE large model; and determine a plurality of thresholds based on the working status of the server.

[0091] In some embodiments, the working status of the server includes at least one of the following: the number of tasks processed simultaneously by the server, the utilization rate of a GPU of the server, the memory occupancy rate of the server, or the running speed of the server.

[0092] In some embodiments, each of the plurality of candidate numbers of experts is less than the maximum number of experts.

[0093] In some embodiments, the vector representation corresponding to the prompt information includes: hierarchical features corresponding to the prompt information output by the middle layer of the MoE large model.

[0094] In some embodiments, the generation unit 410 may be configured to generate a complete output based on the activated experts of the MoE large model.

[0095] In some embodiments, the generating unit 410 is configured to generate a next word-gram of the plurality of word-grams based on the activated experts of the MoE large model and the plurality of word-grams in the generated output.

[0096] In some embodiments, the second determining unit 406 is further configured to determine a score value using a text evaluation model based on the vector representation of the prompt information and the vector representations of the plurality of word tokens in the generated output. Optionally, the plurality of word tokens are a preset number of word tokens generated last in the generated output.

[0097] In some embodiments, the text evaluation model adopts a CNN or RNN structure.

[0098] Figure 4 The device 400 can be used to achieve the above combination Figures 1 to 3 For the sake of brevity, the above process will not be described in detail here.

[0099] The division of modules or units in the embodiments of the present disclosure is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the disclosed embodiments may be integrated into a single unit, exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0100] Figure 5 1 shows a block diagram of an example device 500 that can be used to implement embodiments of the present disclosure. It should be understood that Figure 5 The device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figures 1 to 3 The process described.

[0101] like Figure 5 As shown, device 500 is in the form of a general-purpose computing device. Components of computing device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 500.

[0102] Computing device 500 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media such as a flash drive, a disk, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 500.

[0103] The computing device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0104] Communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of computing device 500 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, computing device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0105] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, speaker, or printer. Computing device 500 may also communicate with one or more external devices (not shown) via communication unit 540, as needed. External devices such as storage devices, display devices, and the like may also be used to communicate with one or more devices that allow a user to interact with computing device 500, or with any device that allows computing device 500 to communicate with one or more other computing devices (e.g., a network card or modem). Such communication may be performed via an input / output (I / O) interface (not shown).

[0106] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0107] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0109] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0110] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0111] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A text generation method, comprising: Get prompt information; Inputting the prompt information into a mixture of experts (MoE) large model to determine a vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training; Determine a score for the prompt information based on the vector representation using a text evaluation model independent of the MoE model, wherein the score for the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information, wherein the text evaluation model adopts a convolutional neural network (CNN) or a recurrent neural network (RNN) structure; Determining the number of experts to be activated in the MoE large model based on a threshold range within which the score value of the prompt information lies, wherein the number of experts to be activated is not greater than the maximum number of experts; as well as Based on the activated expert of the MoE large model, an output corresponding to the prompt information is generated, The step of determining the number of experts to be activated in the MoE large model based on the threshold interval of the score value includes: Determining a plurality of thresholds, wherein a plurality of threshold intervals formed by the plurality of thresholds correspond one-to-one to a plurality of numbers of candidate experts, wherein a difference between the number of the plurality of threshold intervals and the number of the plurality of thresholds is one; and In response to the score value being within a first threshold interval among the plurality of threshold intervals, the number of candidate experts corresponding to the first threshold interval is determined as the number of experts to be activated.

2. The method of claim 1 , wherein determining the plurality of thresholds comprises: Obtaining the working status of the server running the MoE large model; as well as The plurality of thresholds are determined based on the working status of the server.

3. The method according to claim 2, wherein the working status of the server comprises at least one of the following: The number of tasks processed simultaneously by the server, the utilization rate of the graphics processing unit (GPU) of the server, the memory occupancy rate of the server, or the operating speed of the server. The method according to claim 1 , wherein each of the plurality of candidate numbers of experts is smaller than the maximum number of experts.

5. The method according to claim 1, wherein the vector representation corresponding to the prompt information comprises: The hierarchical features corresponding to the prompt information output by the middle layer of the MoE large model.

6. The method of claim 1 , wherein generating the output comprises: Based on the activated experts of the MoE large model, a complete output is generated.

7. The method of claim 1 , wherein generating the output comprises: A next word-gram of the plurality of word-grams is generated based on the activated expert of the MoE large model and a plurality of word-grams in the generated output.

8. The method of claim 7, wherein determining the score value comprises: The rating value is determined using the text evaluation model based on the vector representation of the prompt information and the vector representation of a plurality of word units in the generated output.

9. The method according to claim 7, wherein the plurality of word units are a preset number of word units generated last in the generated output.

10. An electronic device comprising: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 9.

11. A text generation device, comprising: an acquiring unit, configured to acquire prompt information; a first determining unit configured to input the prompt information into a mixture of experts (MoE) large model and determine a vector representation corresponding to the prompt information, wherein the MoE large model has a maximum number of experts determined through training; A second determining unit is configured to determine a score value of the prompt information based on the vector representation using a text evaluation model independent of the MoE large model, wherein the score value of the prompt information is a quantitative performance indicator for characterizing the complexity of the prompt information, wherein the text evaluation model adopts a convolutional neural network (CNN) or a recurrent neural network (RNN) structure; a third determining unit configured to determine the number of experts to be activated in the MoE large model based on a threshold range within which the score value of the prompt information lies, wherein the number of experts to be activated is not greater than the maximum number of experts; as well as a generating unit configured to generate an output corresponding to the prompt information based on the activated expert of the MoE large model, wherein the third determining unit is configured to: determine a plurality of threshold values, wherein a plurality of threshold intervals formed by the plurality of threshold values ​​correspond one-to-one to a plurality of numbers of candidate experts, wherein a difference between the number of the plurality of threshold intervals and the number of the plurality of threshold values ​​is one; And in response to the score value being in a first threshold interval among the multiple threshold intervals, the number of candidate experts corresponding to the first threshold interval is determined as the number of experts to be activated.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images