Generative model building method, processing method, computing device and storage medium

Abstract

The invention discloses a building method and processing method of a generative model, computing equipment and a computer readable storage medium, and the building method of the generative model comprises the following steps: carrying out knowledge distillation processing on a first generative language model to obtain a second generative language model, the parameter scale of the second generative language model is smaller than that of the first generative language model; fine tuning training is conducted on the second generative language model through a target training set to update the second generative language model, and the target training set comprises samples representing the mapping relation between the natural language and the standard terms of the indexes. According to the technical scheme, the generative language model which is quick in response and high in query accuracy can be quickly established.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a generative model construction method, a processing method, a computing device, and a computer-readable storage medium. Background Art

[0002] In some implementations, the generative language model can interact with data through natural language to reduce the complexity of data analysis, such as querying data and obtaining analysis results in a chat-like manner.

[0003] At present, generative language models with high query accuracy usually have billions to hundreds of billions of parameters, which requires a large amount of computing resources when processing complex natural language queries, resulting in long response times. Therefore, how to quickly build a generative language model with fast response and high query accuracy is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The purpose of this application is to provide a generative model construction method, processing method, computing device and computer-readable storage medium, which can quickly build a generative language model with fast response and high query accuracy.

[0005] To achieve the above objectives: In a first aspect, an embodiment of the present application provides a method for building a generative model, comprising: performing knowledge distillation processing on a first generative language model to obtain a second generative language model, wherein a parameter scale of the second generative language model is smaller than a parameter scale of the first generative language model; fine-tuning the second generative language model through a target training set to update the second generative language model, wherein the target training set includes samples representing a mapping relationship between standard terms of natural language and indicators.

[0006] In one embodiment, the step of performing knowledge distillation processing on the first generative language model to obtain a second generative language model includes: extracting input samples from a pre-trained data set to the first generative language model to generate probability distribution information; adjusting the smoothness of the probability distribution according to the temperature parameter and the probability distribution information to generate and output soft target information; and training the initial generative language model according to the soft target information to obtain a second generative language model that fits the first generative language model.

[0007] In one embodiment, the step of training the initial generative language model according to the soft target information to obtain a second generative language model that fits the first generative language model includes: in the process of training the initial generative language model according to the soft target information, minimizing the loss function by a back propagation algorithm and a gradient descent method to obtain a second generative language model that fits the first generative language model.

[0008] In one embodiment, before the step of fine-tuning the second generative language model through the target training set to update the second generative language model, it includes: constructing a standard set of standard terms including indicators; annotating the collected natural language corresponding to the standard set to obtain the target training set; and / or, using data augmentation technology to perform sample processing on the target training set to update the target training set.

[0009] In one embodiment, after fine-tuning the second generative language model through the target training set to update the second generative language model, the method further includes: setting decoding constraints on the updated second generative language model according to a standard set corresponding to the standard terms of the indicator to obtain a final second generative language model.

[0010] In one embodiment, according to the standard set corresponding to the standard terms of the indicator, the step of setting decoding constraints on the updated second generative language model to obtain the final second generative language model includes: defining an output format; constructing a constraint set according to the output format and the standard set; using the vocabulary corresponding to the constraint set as the target vocabulary of the updated second generative language model to obtain the final second generative language model, wherein the final second generative language model is used to select the standard terms of the target indicator from the target vocabulary for output.

[0011] In one embodiment, the method further includes: collecting the accuracy evaluation corresponding to the output result of the second generative language model; when the accuracy evaluation representation is to be optimized, correcting the output result, and generating a new sample according to the corrected output result; and fine-tuning the second generative language model through the new sample to update the second generative language model.

[0012] In a second aspect, an embodiment of the present application provides a processing method, including: obtaining a query statement; inputting the query statement into a generative language model, and outputting indicators of standard terms, wherein the generative language model is constructed by the steps of the generative model construction method described in any of the above items.

[0013] In a third aspect, an embodiment of the present application provides a computing device, comprising: a processor and a memory storing a computer program, and when the processor runs the computer program, the steps of the method for building a generative model described in any one of the above items are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for building a generative model described in any one of the above items are implemented.

[0015] The generative model construction method, processing method, computing device and computer-readable storage medium provided in the embodiment of the present application, the generative model construction method includes: performing knowledge distillation processing on the first generative language model to obtain a second generative language model, the parameter scale of the second generative language model is smaller than the parameter scale of the first generative language model; fine-tuning the second generative language model through the target training set to update the second generative language model, the target training set includes samples of the mapping relationship between the standard terminology representing the natural language and the indicator. In this way, the technical solution of the present application can select a second generative language model with a smaller parameter scale, perform knowledge distillation processing on the first generative language model with high query accuracy and a larger parameter scale, so that the second generative language model with a faster response speed can quickly fit the query function of the first generative language model, and fine-tune the second generative language model through samples of the mapping relationship between the standard terminology representing the natural language and the indicator, so that the second generative language model can better convert the natural language into the corresponding standard terminology indicator, so that the query result corresponding to the natural language can be accurately output according to the standard terminology indicator later. Therefore, the technical solution of the present application can realize the rapid construction of a generative language model with fast response and high query accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0017] Figure 1 It is a flow chart of the method for building a generative model provided in an embodiment of the present application.

[0018] Figure 2 It is a schematic diagram of an application scenario of a model constructed in an example of an embodiment of the present application.

[0019] Figure 3 It is a flowchart of a processing method of an example of an embodiment of the present application.

[0020] Figure 4It is a schematic diagram of the structure of the computing device provided in an embodiment of the present application.

[0021] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings have shown clear embodiments of this application, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0024] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0025] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0026] It should be noted that, in this article, step codes such as S11, S12, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing it specifically, those skilled in the art may execute S12 first and then S11, etc., but these should all be within the scope of protection of this application.

[0027] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0029] See also Figure 1 , is a method for building a generative model provided in an embodiment of the present application. The method can be executed by a computing device provided in an embodiment of the present application, and the computing device can be implemented in software and / or hardware.

[0030] The present embodiment provides a method for building a generative model, which may include the following steps (for example, step S11 to step S12): Step S11: performing knowledge distillation processing on the first generative language model to obtain a second generative language model, wherein the parameter scale of the second generative language model is smaller than the parameter scale of the first generative language model.

[0031] In one embodiment, the first generative language model has a large parameter scale, which can represent a generative language model with a high accuracy of natural language-based queries. Optionally, the first generative language model can be referred to as a "teacher model". Optionally, the second generative language model with a small parameter scale and a fast response speed can be referred to as a "student model".

[0032] In one embodiment, the first generative language model may adopt a more advanced Transformer architecture with multiple layers of self-attention mechanisms and a large number of parameters, which enables it to capture complex patterns and long-distance dependencies in text.

[0033] In one embodiment, the first generative language model is, for example, deepseek236B or llama3-405B.

[0034] In one embodiment, the first generative language model may have tens to hundreds of billions of parameters and may be able to demonstrate excellent performance in a wide range of natural language processing tasks.

[0035] In one embodiment, the first generative language model may be selected based on the performance capabilities of general and professional business domains. Optionally, the evaluation factors of the performance capabilities of general and professional business domains include fluency in processing natural language, ability to understand complex contexts, and accuracy and generalization capabilities on specific data sets.

[0036] In one embodiment, the parameter size of the second generative language model may be significantly smaller than the parameter size of the first generative language model.

[0037] In one embodiment, the second generative language model may be a generative language model with a parameter size less than 10B, in other words, a generative language model with less than 1 billion parameters. Optionally, the second generative language model may be a small Transformer model.

[0038] In one embodiment, step S11: performing knowledge distillation processing on the first generative language model to obtain a second generative language model may include: extracting input samples from a pre-trained data set to the first generative language model to generate and output probability distribution information; adjusting the smoothness of the probability distribution according to the temperature parameter and the probability distribution information to generate soft target information; training the initial generative language model according to the soft target information to obtain a second generative language model that fits the first generative language model.

[0039] In one embodiment, the pre-training dataset may include a large amount of text data samples (eg, natural language samples).

[0040] In one embodiment, the probability distribution information may include a probability vector of each indicator corresponding to the input sample.

[0041] In one embodiment, the probability distribution information may represent the confidence level of the first generative language model for all indicators corresponding to the input sample.

[0042] In one embodiment, the soft target information may characterize the probability distribution that the first generative language model may generate. Optionally, the soft target information provides uncertainty information about the output of the first generative language model, which helps the initial generative language model better understand the task and be more robust during training.

[0043] In one embodiment, the smoothness of the probability distribution is adjusted according to the temperature parameter and the probability distribution information. For example, by increasing the temperature parameter, the probability distribution corresponding to the probability distribution information is made smoother, so that the soft target information finally generated helps the second generative language model to better learn the dark knowledge of the first generative language model, thereby facilitating the second generative language model to better fit the data processing function of the first generative language model. Optionally, the dark knowledge can represent the predicted probability of the first generative language model for the incorrect indicator of the input sample.

[0044] In one implementation, the initial generative language model may represent a model infrastructure of the first generative language model.

[0045] In one implementation, the initial generative language model is trained according to the soft target information, which can represent the processing and output process of deriving the soft target information through the initial generative language model.

[0046] In one embodiment, the step of training the initial generative language model according to the soft target information to obtain a second generative language model that fits the first generative language model may include: in the process of training the initial generative language model according to the soft target information, minimizing the loss function by a back propagation algorithm and a gradient descent method to obtain a second generative language model that fits the first generative language model.

[0047] In one embodiment, the setting of the initial generative language model includes defining an objective function, configuring a loss function, and configuring a training strategy.

[0048] In one embodiment, the objective function may represent a function that can fit the first generative language model to generate soft target information.

[0049] In one embodiment, when defining the objective function, it can be achieved by minimizing the gap between the output probability distribution of the initial generative language model and the soft target information of the first generative language model.

[0050] In one embodiment, the loss function may be a Kullback-Leibler divergence (KL divergence for short) loss function. Optionally, KL divergence is an asymmetric metric used to measure the difference between two probability distributions. Optionally, the probability distribution corresponding to the first generative language model in the preceding step (i.e., soft target information) is recorded as 𝑃, which indicates the confidence level of the first generative language model for each possible indicator; the probability distribution output by the initial generative language model is recorded as 𝑄, which is the output result of the initial generative language model under a given input. The KL divergence loss function is defined as:

[0051] in, is the probability distribution output and processed by the first generative language model, is the probability distribution output by the initial generative language model.

[0052] In one embodiment, the loss function may also be a cross entropy loss.

[0053] In one embodiment, the training strategy can represent various processing methods that enable the initial generative language model to fit the data processing function of the first generative language model to obtain the second generative language model, for example, in the process of training the initial generative language model according to the soft target information, the loss function is minimized by the back propagation algorithm and the gradient descent method.

[0054] In one embodiment, in the process of training the initial generative language model according to the soft target information, minimizing the loss function by the back propagation algorithm and the gradient descent method may include the following steps: (1) Forward propagation: The input sample corresponding to the soft target information is forward propagated through the initial generative language model, and the probability distribution of the output is obtained; (2) Calculate the loss: Input the probability distribution output by the initial generative language model and the soft target information corresponding to the first generative language model into the loss function to calculate the loss value; (3) Back propagation: The back propagation algorithm is used to calculate the gradient of the loss function with respect to the initial generative language model. The back propagation algorithm automatically calculates the gradient of each parameter through the chain rule; (4) Gradient descent: Use gradient descent or other optimization algorithms (such as Adam, RMSprop, etc.) to update the parameters of the initial generative language model according to the calculated gradient. The basic idea of ​​gradient descent is to adjust the parameters in the opposite direction of the gradient (i.e. the direction of gradient descent) to minimize the loss function; (5) Iterative training: Repeat the above process until the performance of the initial generative language model on the validation set no longer improves or the predetermined number of iterations is reached; (6) Regularization and techniques: Regularization techniques (such as L1 and L2 regularization) can be used during training to prevent overfitting, and other techniques (such as dropout and batch normalization) can be used to improve the generalization ability of the model; (7) Evaluation and adjustment: During the training process, the performance of the model is regularly evaluated on the validation set, and hyperparameters such as learning rate and batch size are adjusted as needed; (8) Early stopping: To prevent overfitting, training can be stopped when the performance on the validation set no longer improves. This is called early stopping. (9) Model saving: Once the initial generative language model is trained and the performance reaches a satisfactory level, the model can be saved to obtain a second generative language model.

[0055] Through these steps, the initial generative language model can gradually learn and fit the data processing function of the first generative language model, and finally obtain a second generative language model that can generate output similar to the first generative language model. This method is particularly suitable for text generation tasks in natural language processing.

[0056] Step S12: fine-tune the second generative language model through the target training set to update the second generative language model, where the target training set includes samples representing the mapping relationship between natural language and standard terms of the indicator.

[0057] In one embodiment, the indicator may represent the query intent, that is, the specific information or data point that the user wants to know.

[0058] In one embodiment, natural language refers to the language used by humans to communicate and express thoughts, feelings and information in daily life. It is a manifestation of human wisdom and has rich structures and complex rules. The characteristics of natural language include: (1) Natural formation: Natural language is not designed by humans, but is formed naturally in the process of human social development; (2) Diversity: There are thousands of different natural languages ​​in the world, each with its own unique vocabulary, grammar, and pronunciation rules; (3) Flexibility: Natural language allows people to express the same meaning in a flexible way. Different people may use different words and sentence structures to express the same concept. (4) Ambiguity: There is a lot of ambiguity in natural language. The same word may have different meanings in different contexts. (5) Context dependence: The meaning of language often depends on the context. The same sentence may have different interpretations in different situations. (6) Recursion: Natural languages ​​are recursive, allowing a limited number of words and syntactic structures to be infinitely combined to express an infinite number of ideas. (7) Cultural relevance: Language is closely related to culture, and different cultural backgrounds will affect the use and understanding of language; (8) Emotion and intention: Natural language is not only a carrier of information, but also contains the speaker’s emotions and intentions; (9) Unstructured: Compared with computer language, natural language is unstructured and has no fixed format and strict grammatical rules.

[0059] In one embodiment, standard terminology is a set of standardized words and expressions developed for accurate communication in a specific field or scenario, with the following specific features: (1) Be rigorously structured and follow specific rules and formats to ensure accuracy and consistency; (2) The meaning is clear, aiming to eliminate ambiguity and ensure that each term has a precise definition; (3) Usually excludes emotional coloring and focuses on objective and factual descriptions.

[0060] In one embodiment, the second generative language model can predict corresponding indicators based on the input natural language, and output the standard terms corresponding to the predicted indicators, so as to subsequently accurately output the query results corresponding to the natural language based on the indicators of the standard terms.

[0061] In one embodiment, step S12: fine-tuning the second generative language model through the target training set to update the second generative language model may include: constructing a standard set of standard terms including indicators; annotating the collected natural language corresponding to the standard set to obtain the target training set; and / or, using data augmentation technology to perform sample processing on the target training set to update the target training set.

[0062] In one implementation, the collected natural language is annotated according to the standard set, which may include: Experts directly annotate natural language to correspond to the standard set to map natural language to standard terms in the standard set; Construct Q&A pairs for natural language and standard terms in a standard set.

[0063] In one embodiment, a set of question-answer pairs (Q, A), where Q is natural language input by the user, and A is the standard terminology of the corresponding indicator, for example, (𝑄: "how many new customers are added in Central China this month", 𝐴: "customer growth").

[0064] In one embodiment, data enhancement technology is used to perform sample processing on the target training set, which may include: synonym replacement, text translation and then back translation, random insertion, deletion or replacement of words, etc.

[0065] In one embodiment, the data augmentation technology can generate more diverse training samples to update the target training set, so as to use the target training set to fine-tune the second generative language model, thereby enhancing the robustness of the second generative language model.

[0066] In one embodiment, after fine-tuning the second generative language model through the target training set to update the second generative language model, the method further includes: setting decoding constraints on the updated second generative language model according to the standard set corresponding to the standard terminology of the indicator to obtain the final second generative language model. In this way, the technical solution of this embodiment can constrain the final second generative language model to accurately output the predicted indicator according to the standard terminology of the indicator through the decoding constraint setting.

[0067] In one embodiment, according to the standard set corresponding to the standard terms of the indicator, the step of setting decoding constraints on the updated second generative language model to obtain the final second generative language model includes: defining an output format; constructing a constraint set according to the output format and the standard set; using the vocabulary corresponding to the constraint set as the target vocabulary of the updated second generative language model to obtain the final second generative language model, wherein the final second generative language model is used to select the standard terms of the target indicator from the target vocabulary for output.

[0068] In one implementation, defining the output format may represent defining the format, structure, etc. of the final output indicator of the second generative language model.

[0069] The technical solution of this embodiment enables the final second generative language model to set the probabilities of all words that do not belong to the constraint set to zero or a minimum value in the decoding step, ensuring that these words will not be selected in the output. Optionally, at each decoding step, the final second generative language model can generate a probability distribution of candidate words based on the context and decoding strategy, and use a filter to delete all candidate words that are not in the constraint set, and then renormalize the probability distribution of the remaining candidate words, and select a standard term of a vocabulary generation index from the constraint set based on the probability distribution of the remaining candidate words, and continue decoding until the standard term of the output index is output. Therefore, the second generative language model can better convert natural language into the index of the corresponding standard term, so that the query result corresponding to the natural language can be accurately output according to the index of the standard term later.

[0070] In one embodiment, it also includes: collecting the accuracy evaluation corresponding to the output result of the second generative language model; when the accuracy evaluation indicates that it needs to be optimized, correcting the output result, and generating new samples based on the corrected output result; fine-tuning the second generative language model through the new samples to update the second generative language model. Therefore, after the evaluation system of the output result is built, the technical solution of this embodiment can indicate that the output result needs to be optimized when the accuracy evaluation corresponding to the output result of the second generative language model is poor, so that the natural language corresponding to the output result can be accurately annotated and processed, and the generated new samples are fed back to the second generative language model for learning. In this way, the second generative language model in the technical solution of this embodiment can have a self-learning function to ensure long-term use and higher query accuracy.

[0071] The method for building a generative model provided in this embodiment includes: step S11: performing knowledge distillation processing on the first generative language model to obtain a second generative language model, the parameter scale of the second generative language model is smaller than the parameter scale of the first generative language model; step S12: fine-tuning the second generative language model through the target training set to update the second generative language model, the target training set includes samples of the mapping relationship between the standard terminology representing the natural language and the indicator. In this way, the technical solution of this embodiment can select a second generative language model with a smaller parameter scale, perform knowledge distillation processing on the first generative language model with high query accuracy and large parameter scale, so that the second generative language model with a faster response speed can quickly fit the query function of the first generative language model, and fine-tune the second generative language model through samples of the mapping relationship between the standard terminology representing the natural language and the indicator, so that the second generative language model can better convert the natural language into the corresponding standard terminology indicator, so that the query result corresponding to the natural language can be accurately output according to the standard terminology indicator. Therefore, the technical solution of this embodiment can realize the rapid construction of a generative language model with fast response and high query accuracy.

[0072] Based on the same inventive concept as the above embodiments, see Figure 2 The above embodiment is exemplarily described below through a specific application scenario: The steps of building an indicator parsing model (i.e., the second generative language model) may include: 1. The model distillation process (or knowledge distillation process) from the teacher model (i.e. the first generative language model) to the student model (i.e. the initial generative language model): 1.1 Select the teacher model T; Optionally, a generative large language model with excellent performance is selected as the teacher model T, such as deepseek236B or llama3-405B. The deepseek236B or llama3-405B selected by the teacher model T usually has tens of billions to hundreds of billions of parameters and can show excellent performance in a wide range of natural language processing tasks.

[0073] Optionally, the teacher model T is selected based on general and specialized business domain performance capabilities, including fluency in generating natural language, ability to understand complex contexts, and accuracy and generalization on specific datasets.

[0074] Optionally, the teacher model T typically adopts an advanced Transformer architecture with multiple layers of self-attention mechanisms and a large number of parameters, which enables it to capture complex patterns and long-range dependencies in text.

[0075] 1.2 Generate Soft Targets: Use the pre-training dataset D (including a large amount of text data) as input and input it into the teacher model T. The teacher model T predicts the dataset D and generates an output probability vector (equivalent to a soft target). For example, for each input sample, the teacher model T outputs the probability distribution of each possible indicator. It shows the confidence level of the teacher model T for all possible indicators.

[0076] Optionally, when the teacher model T predicts the data set D and generates an output probability vector, a temperature parameter can be added to the soft target to adjust the smoothness of the output probability distribution. By increasing the temperature parameter, the output probability distribution can be made smoother, which enables the student model S to better learn the "dark knowledge" of the teacher model T (i.e., the prediction probability of the teacher model T for multiple incorrect indicators).

[0077] 1.3 Train the student model S: Optionally, the student model S is typically a smaller generative language model with significantly fewer parameters than the teacher model T (eg, the model is selected to be less than 10B, i.e., less than 1 billion parameters), such as a small Transformer model.

[0078] Optionally, the training process of the student model S may include the following steps: objective function definition, loss function design, and training strategy design.

[0079] 1.3.1 Objective function definition: The training goal of the student model S is to try to fit the soft target generated by the teacher model T. This is usually achieved by minimizing the gap between the probability distribution output by the student model S and the soft target output by the teacher model T.

[0080] 1.3.2 Loss function design: Use the Kullback-Leibler divergence (KL Divergence) loss function. KL divergence is an asymmetric metric used to measure the difference between two probability distributions. The probability distribution (i.e., soft target) output by the teacher model T in the previous step is denoted as 𝑃, which represents the confidence level of the teacher model T in each possible indicator. The probability distribution output by the student model S is denoted as 𝑄, which is the prediction result of the student model S under a given input. The KL divergence loss function is defined as:

[0081] in, is the output probability distribution of the teacher model T, is the output probability distribution of the student model S.

[0082] 1.3.3 Training strategy: Use a step-by-step optimization method to train the student model S, and use the backpropagation algorithm (Backpropagation) and gradient descent method (such as Adam Optimizer) to minimize the loss function, so that the student model S gradually approaches the output of the teacher model T.

[0083] 2. The fine-tuning process of using the student model S as the indicator parsing model can include the following steps: 2.1 Formulate standard terminology for indicators and form a standard set M Optionally, comprehensively sort out and analyze the indicators of all data that may appear in the business scenario. Commonly used indicator expressions are unified into a standardized term to form a standard term sample, such as "how many new customers", "increased number of customers", and "how much customer growth" are unified into a standardized term "customer growth". A well-defined standard set M is obtained through the standard term sample, so that the student model S can better understand the semantics of the natural language of the business and identify and match the indicators of the data in the user query.

[0084] Optionally, the standard terms of all indicators form a set, denoted as the standard set M. The standard set M contains possible standard terms and indicator definitions, for example: 𝑀=[“Customer growth”, “Month-on-month growth”, “Sales amount”, “Product profit margin”, “Inventory turnover rate”, …] 2.2 Collect and create indicator parsing training set X (i.e. target training set); Alternatively, this example can collect natural language stationery used in actual business applications, annotate it through a data indicator expert team, and map it to standard terms in the standard set M. For example, "how many new customers are added in the Central China region this month" is annotated as "customer growth". On this basis, the indicator parsing training set X is constructed.

[0085] Optionally, this example can construct Q&A pairs to construct the indicator parsing training set X. A set of question-answer pairs (Q, A), where Q is the user's natural language query input, and A is the standard terminology of the corresponding indicator, for example: 𝑄: "How many new customers are there in Central China this month", 𝐴: "Customer growth".

[0086] Optionally, this example can also use data augmentation techniques (such as synonym replacement, text translation and then translation back, random insertion, deletion or replacement of words) to generate more diverse training samples to update the indicator parsing training set X, so as to enhance the robustness of the indicator parsing model by fine-tuning the student model S through the indicator parsing training set X.

[0087] 2.3 Fine-tune the student model S by using the indicator parsing training set X; Optionally, this example can adopt a supervised learning method and use the indicator parsing training set X to supervise the student model S. With natural language queries as input and the standard terms of indicators as output labels, the student model S learns by minimizing the loss between the predicted output and the labels of the true standardized terms, and finally obtains the indicator parsing model. The loss function and training strategy are the same as the previous steps.

[0088] Optionally, this example can also use an independent validation set to evaluate the performance of the metric parsing model, measuring its accuracy, recall, F1 score, and other indicators to ensure the generalization ability of the metric parsing model.

[0089] 3. Set constraint decoding for the indicator parsing model; Optionally, this example introduces the Constrained Decoding technique to enhance the accuracy and consistency of the indicator parsing model when generating indicators with standard terms. Constrained Decoding helps ensure that the generated results meet business requirements and indicator standards by limiting the generated vocabulary of the indicator parsing model so that it only generates outputs that meet predefined rules.

[0090] Optionally, the step of setting constraint decoding for the indicator parsing model may include: 3.1 Construct constraint set C: The construction of constraint set C can be expressed as: C = [ "[", "]", ","] ∪ M (all word collections) Among them, [ "[", "]", ","] is the basic character set used to define the format and structure of the generated output.

[0091] M is the standard set constructed in the previous step, which contains the standard terms of all indicators (such as "customer growth", "year-on-year growth", etc.).

[0092] 3.2 Indicator parsing model decoding is constrained by constraint sets Optionally, this example implements constraint decoding by directly switching the vocabulary of the indicator parsing model to the constraint set C. During decoding, the original vocabulary of the indicator parsing model is replaced with the vocabulary in the constraint set 𝐶.

[0093] Optionally, during the decoding step of the indicator parsing model, the probabilities of all words that are not in the constraint set 𝐶 are set to zero or a very small value to ensure that these words are not selected in the output. At each decoding step, the indicator parsing model generates a probability distribution of candidate words based on the context and the decoding strategy. Using a filter, remove all candidate words that are not in the constraint set 𝐶, and then renormalize the probability distribution of the remaining candidate words. Based on the remaining probability distribution, a word is selected from the constraint set 𝐶 to generate, and decoding continues until the entire output is completed.

[0094] The indicator parsing model of this example can be applied to the current ChatBI (business intelligence chat system), using natural language processing (NLP) and the indicator parsing model to parse the indicators that users want to query, execute queries for the corresponding indicator algorithms, and output the query results.

[0095] (1) In some implementations, ChatBI is completely understood by large language models, but large models take a long time to execute: Large language models (such as GPT-4 or LLaMA) usually have billions to hundreds of billions of parameters and require a lot of computing resources when processing complex queries. This high consumption of computing resources leads to long system response times, especially in application scenarios with high real-time requirements, which makes it difficult to meet the fast response expectations of users. The reasoning process of large models has high requirements for computing resources and energy consumption, resulting in high operating costs and is difficult to popularize in large-scale commercial applications.

[0096] Therefore, the indicator parsing model obtained in this example has a smaller parameter scale and can handle complex queries. Moreover, due to the lower demand for computing resources and energy consumption, it can improve the query response speed and save operating costs, making it easier to popularize in large-scale commercial applications.

[0097] (2) In some implementations, ChatBI's indicators are recognized through named entity recognition in the sequence labeling task, but the recognized indicators cannot be converted into standard terms, which makes the model prone to misunderstanding when parsing ambiguous or unclear queries. For example, for indicator names with unclear meanings or multiple interpretations, the model may not be able to accurately map them to indicators or database fields in standard terms. Sequence labeling methods can identify data indicators in text, but cannot understand the specific meaning or semantics of these indicators, nor can they map them to standard terms or database entries. For example, "sales amount" may appear in different documents in multiple ways such as "revenue", "sales income", and "sales volume". NER can only identify entities related to "sales amount" but cannot uniformly map them to indicator fields in standard terms.

[0098] Therefore, compared with the named entity recognition method of the sequence labeling task, the indicator parsing model obtained in this example can improve the recognition accuracy of the indicator, thereby making the output query results more accurate and without the need to use a large amount of data sets.

[0099] (3) In some implementations, ChatBI can identify indicators through traditional methods such as vocabulary and similarity. However, the vocabulary needs to be constantly updated to include standard terms or expressions of new indicators, and traditional vocabulary methods are difficult to respond to changes in standard terms or business terms of new indicators in a timely manner. In addition, traditional similarity methods rely on static dictionaries or simple text matching, lack in-depth understanding of the context, and have difficulty handling complex contexts or polysemous words. That is, when identifying indicators through traditional methods such as vocabulary and similarity, the accuracy of parsing based solely on rules and regular expressions is difficult to guarantee, and there is no generalization capability.

[0100] Therefore, compared with traditional methods of identifying indicators through vocabulary, similarity, etc., the indicator parsing model obtained in this example not only has a more accurate indicator recognition rate, but also has good generalization ability.

[0101] In this way, the indicator parsing model obtained in this example can ensure the balance between performance and accuracy when applied in the chatBI scenario, and optimize the indicator parsing in the chatBI scenario.

[0102] See also Figure 3 , is a processing method provided in an embodiment of the present application. The method can be executed by a computing device provided in an embodiment of the present application. The computing device can be implemented in software and / or hardware.

[0103] Based on the same inventive concept as the above-mentioned embodiment, the embodiment of the present application further provides a processing method, including (for example, step S21 to step S22): Step S21: Obtain a query statement.

[0104] Step S22: Input the query statement into the generative language model and output the index of the standard terminology.

[0105] In one implementation, the query statement may be a natural language or a standard term.

[0106] In one implementation, the generative language model is constructed by following the steps of any of the above described methods for constructing a generative model.

[0107] In one implementation, the specific implementation and beneficial effects of the generative language model may refer to the second generative language model in the above embodiment, which will not be described in detail here.

[0108] In this way, the processing method provided in this example can input the query statement into the generative language model, and quickly and accurately output the standard terms of the corresponding indicators, so as to subsequently accurately output the query results corresponding to the natural language query statement according to the indicators of the standard terms.

[0109] Based on the same inventive concept as the above embodiments, the present application embodiment provides a computing device, such as Figure 4 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 4 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 4 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the construction method or processing method of the generative model applied to the above device is implemented.

[0110] The device may also include: at least one network interface 312. The various components in the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 4 Various buses are labeled as bus system 313.

[0111] The memory 311 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0112] The memory 311 in the embodiment of the present application is used to store various types of data to support the operation of the device. Examples of these data include: any computer program for operating on the device, such as an operating system and an application program. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program may include various application programs for implementing various application services. Here, a program for implementing a method for building or processing a generative model of an embodiment of the present application may be included in an application program.

[0113] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, in which a computer program is stored. The computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it may also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by the processor, the above-mentioned generative model construction method or processing method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 or Figure 3 The description of the illustrated embodiment will not be repeated here.

[0114] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0116] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for building a generative model, characterized in that: include: Performing knowledge distillation processing on the first generative language model to obtain a second generative language model, wherein a parameter scale of the second generative language model is smaller than a parameter scale of the first generative language model; Fine-tuning the second generative language model through a target training set to update the second generative language model, wherein the target training set includes samples representing a mapping relationship between natural language and standard terms of indicators.

2. The method according to claim 1, characterized in that The step of performing knowledge distillation processing on the first generative language model to obtain a second generative language model includes: Extracting input samples from the pre-trained data set to the first generative language model to generate probability distribution information; The smoothness of the probability distribution is adjusted according to the temperature parameter and the probability distribution information to generate and output the soft target information; The initial generative language model is trained according to the soft target information to obtain the second generative language model that fits the first generative language model.

3. The method according to claim 2, characterized in that The step of training the initial generative language model according to the soft target information to obtain the second generative language model that fits the first generative language model includes: In the process of training the initial generative language model according to the soft target information, the loss function is minimized by a back propagation algorithm and a gradient descent method to obtain the second generative language model that fits the first generative language model.

4. The method according to claim 1, characterized in that: Before the step of fine-tuning the second generative language model using the target training set to update the second generative language model, the method further comprises: Constructing a standard set including standard terminology for indicators; Performing annotation processing corresponding to the standard set on the collected natural language to obtain the target training set; and / or, The target training set is processed by using data enhancement technology to update the target training set.

5. The method according to any one of claims 1 to 4, characterized in that: After the step of fine-tuning the second generative language model using the target training set to update the second generative language model, the method further comprises: According to the standard set corresponding to the standard terminology of the indicator, decoding constraints are set for the updated second generative language model to obtain a final second generative language model.

6. The method according to claim 5, characterized in that The step of setting decoding constraints on the updated second generative language model according to the standard set corresponding to the standard term of the indicator to obtain the final second generative language model includes: Define the output format; constructing a set of constraints based on the output format and the set of criteria; The vocabulary corresponding to the constraint set is used as the target vocabulary of the updated second generative language model to obtain a final second generative language model, wherein the final second generative language model is used to select standard terms of target indicators from the target vocabulary for output.

7. The method according to claim 5, characterized in that Also includes: Collecting accuracy evaluations corresponding to output results of the second generative language model; When the accuracy evaluation representation is to be optimized, correcting the output result, and generating a new sample according to the corrected output result; Fine-tune the second generative language model using the new sample to update the second generative language model.

8. A processing method, characterized in that: include: Get the query statement; The query statement is input into a generative language model, and an index of a standard term is output. The generative language model is constructed by the steps of the method for constructing a generative model according to any one of claims 1 to 7.

9. A computing device, characterized in that include: A processor and a memory storing a computer program, which implements the steps of the method for building a generative model according to any one of claims 1 to 7 when the processor runs the computer program.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the method for building a generative model according to any one of claims 1 to 7 are implemented.

Images