Language model training method and device and nonvolatile storage medium

Abstract

The invention discloses a language model training method and device and a nonvolatile storage medium. The method comprises the steps that multiple pieces of initial sequence data are input into a preset target sparse predictor, prediction parameter subsets corresponding to the multiple pieces of initial sequence data are output, and the prediction parameter subsets comprise prediction parameter subsets corresponding to multiple levels in a preset original language model; the original language model is used for generating texts; determining a target parameter subset based on the prediction parameter subsets corresponding to the initial sequence data; based on a preset training sample, parameters included in a corresponding target parameter subset in a preset original language model are adjusted, a target language model is obtained, and the training sample comprises a sample text and a generated text corresponding to the sample text. According to the method and the device, the technical problems of long time consumption and low efficiency in a model training process due to a large number of existing parameters during language model training at present are solved.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a language model training method, device and non-volatile storage medium. Background Art

[0002] Increasing the parameters of large language models can greatly improve the contextual learning ability of the model, and is often widely used in fields such as text generation and classification. However, due to the large number of parameters, they are very expensive and inefficient to train and infer.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a language model training method, device and non-volatile storage medium to at least solve the technical problem that the model training process is time-consuming and inefficient due to the large number of parameters in the current language model training.

[0005] According to one aspect of an embodiment of the present invention, a method for training a language model is provided, comprising: inputting a plurality of initial sequence data into a preset target sparse predictor respectively, and outputting prediction parameter subsets corresponding to each of the plurality of initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of a plurality of levels in a preset original language model, and the original language model is used to generate text; determining a target parameter subset based on the prediction parameter subsets corresponding to each of the plurality of initial sequence data; and adjusting parameters included in the corresponding target parameter subset in the preset original language model based on a preset training sample to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0006] Optionally, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to each of the multiple initial sequence data are output, including: obtaining multiple initial texts; inputting the multiple initial texts into multiple levels in the original language model, and outputting output data of multiple levels corresponding to each of the multiple initial texts, wherein the multiple levels include a perceptron layer and an attention mechanism layer; and determining multiple initial sequence data based on the output data of multiple levels corresponding to each of the multiple initial texts.

[0007] Optionally, before inputting multiple initial sequence data into a preset target sparse predictor respectively and outputting prediction parameter subsets corresponding to each of the multiple initial sequence data, it includes: obtaining training text; extracting text features from the training text; and training a preset original sparse trainer based on the text features and preset multiple sample parameter subsets to obtain a target sparse predictor.

[0008] Optionally, based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, a target parameter subset is determined, including: determining the occurrence probability of each parameter in the prediction parameter subsets corresponding to each of the multiple initial sequence texts; and determining the target parameter subset based on the parameters whose occurrence probability exceeds a preset threshold.

[0009] Optionally, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to the multiple initial sequence data are output. It also includes: inputting the initial sequence data into a target sparse predictor and a preset asynchronous predictor, and outputting prediction parameter subsets corresponding to the multiple initial sequence data, wherein the asynchronous predictor is used to predict the prediction parameter subset of the next level corresponding to the level predicted by the target sparse predictor.

[0010] Optionally, the target sparse predictor is based on a Hidden Markov Model and a classical Viterbi algorithm setting.

[0011] According to another aspect of an embodiment of the present invention, a language model training device is also provided, including: an input module, used to input multiple initial sequence data into a preset target sparse predictor respectively, and output prediction parameter subsets corresponding to each of the multiple initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of the multiple levels in a preset original language model, and the original language model is used to generate text; a determination module, used to determine a target parameter subset based on the prediction parameter subsets corresponding to each of the multiple initial sequence data; an adjustment module, used to adjust the parameters included in the corresponding target parameter subset in the preset original language model based on a preset training sample, to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0012] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned language model training methods.

[0013] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above-mentioned language model training methods is executed.

[0014] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which implements any of the above-mentioned language model training methods when executed by a processor.

[0015] In an embodiment of the present invention, a language model training method is adopted, by respectively inputting a plurality of initial sequence data into a preset target sparse predictor, outputting prediction parameter subsets corresponding to the plurality of initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to a plurality of levels in a preset original language model, and the original language model is used to generate text; based on the prediction parameter subsets corresponding to the plurality of initial sequence data, a target parameter subset is determined; based on a preset training sample, the parameters included in the corresponding target parameter subset in the preset original language model are adjusted to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text, thereby achieving the purpose of selecting appropriate parameters for model training, thereby achieving the technical effect of improving the model training efficiency, and further solving the technical problem that the model training process is time-consuming and inefficient due to the large number of parameters when performing language model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing a language model training method is shown;

[0018] Figure 2 is a flow chart of a method for training a language model according to an embodiment of the present invention;

[0019] Figure 3 It is a structural block diagram of a language model training device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] According to an embodiment of the present invention, a method embodiment of a language model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0023] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for training a language model. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0024] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0025] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the training method of the language model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the training method of the language model of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0026] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0027] Figure 2 is a flow chart of a method for training a language model according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0028] Step S202, inputting the multiple initial sequence data into a preset target sparse predictor respectively, and outputting prediction parameter subsets corresponding to the multiple initial sequence data respectively, wherein the prediction parameter subsets include prediction parameter subsets corresponding to the multiple levels in the preset original language model, and the original language model is used to generate text.

[0029] In this step, the process of inputting multiple initial sequence data into the preset target sparse predictor to output the prediction parameter subsets corresponding to each of these sequence data allows the large model (such as the original language model) to adaptively adjust the calculation parameters when processing different sequence data, thereby reducing unnecessary computing resource consumption while ensuring the output quality. Prepare multiple different sequence data, which can be any data set arranged in time or sequence, such as text, speech, time series, etc. Each sequence data represents a different input instance, which can be input from different users or different application scenarios. Preset one or more target sparse predictors, each of which predicts the sparse parameters for different levels or modules in the original language model. For example, a predictor can be set specifically for predicting the sparse parameters of the self-attention layer, and another predictor can be used to predict the sparse parameters of the feedforward neural network (MLP) layer. Each predictor is specially trained to identify which parameters in a specific level or module can be safely ignored for a given input sequence. Input each initial sequence data into the target sparse predictor. The sparse predictor analyzes the characteristics of the input sequence, such as the importance of words, the dynamic changes of the sequence, etc., to predict which parameters can be sparse in the corresponding layers of the original language model. For each input sequence data, the sparse predictor outputs a subset of prediction parameters, which contains the prediction parameters corresponding to multiple levels in the original language model. For example, for the self-attention layer, the prediction parameter subset may include a specific attention head; for the MLP layer, the prediction parameter subset may include a specific subset of neurons or weight matrices. Each prediction parameter subset is then used in the corresponding layer of the original language model to reduce the number of parameters in the actual calculation. This means that when processing each sequence data, the model can dynamically adjust its calculation process according to the output of the sparse predictor, and only use a subset of prediction parameters for calculation, thereby reducing computational complexity and memory usage.

[0030] Through the above process, the processing of multiple initial sequence data is optimized, and each sequence data obtains personalized sparse parameter prediction, which helps to achieve dynamic computing resource management under different inputs and improve the reasoning efficiency and real-time processing capabilities of large models.

[0031] Step S204: determining a target parameter subset based on the prediction parameter subsets corresponding to each of the plurality of initial sequence data.

[0032] In this step, based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, the target parameter subset is determined. The multiple prediction parameter subsets can be analyzed to compare the commonalities and differences between them. This step can be achieved through statistical analysis, pattern recognition or machine learning methods. The purpose is to identify which parameter subsets are frequently predicted to be sparse when processing different sequence data, and their distribution in different levels. Cluster analysis or rule-based grouping methods can be used to group similar prediction parameter subsets into a group to form a category or template of parameter subsets. Each category or template represents a sparseness strategy when processing a specific type of sequence data. From the results of clustering or grouping, parameter subset templates that perform well in processing multiple sequence data can be selected. These templates may require further optimization to ensure that they are not only effective when processing most inputs, but also maintain reasonable performance and efficiency for all inputs. Based on the selected and optimized templates, the target parameter subset is determined. The target parameter subset is a set of parameters that can be used as a default or preferred sparseness strategy when the model processes unknown and new sequence data. They contain parameters that have proven to be efficient in processing various sequence data with minimal impact on model performance.

[0033] Through the above process, based on the prediction parameter subsets corresponding to multiple initial sequence data, a more general and adaptive target parameter subset can be extracted to improve the reasoning efficiency of large models when processing sequence data. This method not only takes into account the flexibility and efficiency of the model when processing different types of sequence data, but also ensures the robustness and universality of the sparsification strategy by analyzing and optimizing multiple prediction results. It has a significant effect on large-scale text generation, speech recognition, time series prediction and other tasks.

[0034] Step S206, based on a preset training sample, adjusting the parameters included in the corresponding target parameter subset in the preset original language model to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0035] In this step, the process of adjusting the corresponding target parameter subset in the original language model based on the preset training sample to obtain the target language model is essentially to optimize the model parameters through specific training data to ensure that the model after sparsification can still maintain or approach the original performance level. The training sample includes a sample text and its corresponding generated text. The sample text can be a sentence or paragraph randomly selected from the corpus, and the generated text is the output of the original language model in the unsparse state. These generated texts represent the expected performance of the model under a specific input for subsequent adjustment and optimization. A sparse model version can be constructed based on the original language model, in which only the parameters in the target parameter subset are retained, while other parameters are ignored or set to zero. The sparse model is trained using the prepared training samples. The goal of the training is to adjust and optimize the parameters in the target parameter subset so that the sparse model can generate generated text close to the output of the original model when processing the sample text. During the training process, the output results of the sparse model and the original model can be compared, and a loss function (such as cross entropy loss) can be used to guide the adjustment of the parameters. After multiple training and optimizations, once the sparse model reaches a satisfactory level in both performance and efficiency, it can be determined as the final target language model.

[0036] Through the above process, based on the preset training samples, the corresponding target parameter subset in the original language model can be adjusted and optimized to obtain a target language model that can still maintain high generation quality after sparsification. This method combines model structure optimization and parameter training, which can significantly improve the model's reasoning efficiency while maintaining generation performance.

[0037] Through the above steps, the purpose of selecting appropriate parameters for model training can be achieved, thereby achieving the technical effect of improving the efficiency of model training, and further solving the technical problem that the model training process is time-consuming and inefficient due to the large number of parameters in the current language model training.

[0038] As an optional embodiment, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to each of the multiple initial sequence data are output, including: obtaining multiple initial texts; inputting the multiple initial texts into multiple levels in the original language model, and outputting output data of multiple levels corresponding to each of the multiple initial texts, wherein the multiple levels include a perceptron layer and an attention mechanism layer; and determining multiple initial sequence data based on the output data of multiple levels corresponding to each of the multiple initial texts.

[0039] Optionally, multiple initial texts are obtained, which can be sentences or paragraphs from different fields, in order to cover a variety of input situations that the model may encounter. These initial texts are input into the original large language model based on the Transformer architecture. This model contains multiple layers, each of which may include an attention mechanism layer (Attention Layer) and a feed-forward neural network layer (Feed-Forward Layer, also known as the perceptron layer MLP). The model processes each input initial text, and each layer generates corresponding output data. For the attention mechanism layer, the output may include an attention weight matrix; for the feed-forward neural network layer, the output may include a feature vector processed by the layer. These output data reflect the details of the original model's processing of the input text. Collect the output data of each initial text at each layer of the model to obtain a set of multiple initial sequence data, where each initial sequence data is composed of the output data of each layer of the model. This step is to prepare for subsequent analysis and prediction of parameter subsets.

[0040] As an optional embodiment, before inputting multiple initial sequence data into a preset target sparse predictor respectively and outputting prediction parameter subsets corresponding to each of the multiple initial sequence data, it includes: obtaining training text; extracting text features from the training text; and training a preset original sparse trainer based on the text features and preset multiple sample parameter subsets to obtain a target sparse predictor.

[0041] Optionally, before multiple initial sequence data are input to a preset target sparse predictor to output their respective predicted parameter subsets, the target sparse predictor needs to be trained. A large amount of training text data can be collected first, which should cover various scenarios and input types that the model may encounter. The amount and diversity of training texts are crucial to the learning effect of the predictor. The training texts are preprocessed to extract features that help predict parameter importance. These features may include word frequency statistics, vocabulary diversity, sentence structure, contextual semantics, etc. In Transformer-based models, special attention is paid to features extracted by the model's own attention mechanism, which reflect the correlation between different parts of the input sequence. A series of parameter subsets can be preset as references during the training process. These sample parameter subsets can be generated by random, heuristic, or rule-based methods, with the goal of providing a diverse set of parameter subsets to guide the learning of the sparse predictor. The preset original sparse trainer is then trained using the extracted text features and the preset sample parameter subsets. This training process involves defining a loss function that measures the difference between the output of the model when the predicted parameter subset is used for calculation and the output when the full parameter set is used. The goal of training is to minimize this difference while maintaining the performance level of the model. After multiple rounds of training and optimization, the original sparse trainer will learn how to predict the most appropriate parameter subset based on the characteristics of the input text. After this step is completed, the original sparse trainer is converted to the target sparse predictor and can be used in the actual reasoning stage.

[0042] As an optional embodiment, a target parameter subset is determined based on prediction parameter subsets corresponding to each of a plurality of initial sequence data, including: determining the occurrence probability of each parameter in the prediction parameter subset corresponding to each of a plurality of initial sequence texts; and determining the target parameter subset based on the parameters whose occurrence probability exceeds a preset threshold.

[0043] Optionally, a common target parameter subset is determined based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, which can ensure that the model can operate in an efficient manner while maintaining its performance as much as possible when processing various inputs. The frequency with which each parameter in the model is selected when predicting multiple initial sequence data can be counted. This involves recording each prediction parameter subset and then counting the number of times each parameter appears in all prediction subsets. The frequency of parameter occurrence can be regarded as a measure of its importance when processing different inputs. The number of times each parameter appears is then divided by the total number of predictions to obtain the probability of occurrence of each parameter. A preset threshold can be set based on the requirements of model performance, computing resources, and inference speed. This threshold is used to filter parameters with a higher probability of occurrence as part of the target parameter subset. The setting of the threshold needs to balance the performance and efficiency of the model. If it is too high, key parameters may be omitted, and if it is too low, the inference efficiency of the model may not be significantly improved. Then, the probability of occurrence of each parameter is compared with the preset threshold, and all parameters with a probability of occurrence exceeding the threshold are selected. These parameters form the basis of the target parameter subset, and they are considered to be key parameters that contribute significantly to the performance of the model in most cases. Finally, all the filtered parameters are combined to form a target parameter subset. In the subsequent reasoning process, the model will mainly rely on this subset for calculation, thereby reducing the amount of calculation and memory usage, improving the reasoning speed, and trying to keep the performance of the model unchanged.

[0044] Through the above steps, a streamlined target parameter subset can be extracted from a large amount of initial sequence data to guide the model to perform efficient and accurate reasoning while minimizing the impact on the original performance of the model.

[0045] As an optional embodiment, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to the multiple initial sequence data are output. It also includes: inputting the initial sequence data into a target sparse predictor and a preset asynchronous predictor respectively, and outputting prediction parameter subsets corresponding to the multiple initial sequence data, wherein the asynchronous predictor is used to predict the prediction parameter subset of the next level corresponding to the level predicted by the target sparse predictor.

[0046] Optionally, multiple initial sequence data are respectively input into a preset target sparse predictor and an asynchronous predictor to output the corresponding prediction parameter subsets. The implementation of this process can further refine the sparse calculation strategy of the model and improve the reasoning efficiency. Multiple initial sequence data can be input into the target sparse predictor. Based on the previous training, the target sparse predictor can predict which parameter subsets of the current layer have the greatest impact on the model output according to the characteristics of the input sequence, thereby outputting the prediction parameter subset of the current layer. The target sparse predictor outputs the prediction parameter subset of the current layer for each initial sequence data. These parameter subsets are used to guide the reasoning process of the model at the current layer, determine which parameters are involved in the calculation and which parameters can be ignored, so as to reduce the amount of calculation and improve the reasoning speed. At the same time, these initial sequence data are input into the preset asynchronous predictor. The purpose of the asynchronous predictor is to predict in advance the parameter subset that the target sparse predictor may output at the next level when processing the current sequence data. In this way, the prediction delay when processing each layer can be avoided, thereby realizing efficient sparse calculation across layers. The asynchronous predictor outputs the prediction parameter subset of the next level for each initial sequence data. These predictions can be made in advance using a fast prediction algorithm based on the output features of the current layer, preparing for the sparse calculations of the next layer. Combine the parameter subsets output by the target sparse predictor and the asynchronous predictor. When processing the current level, use the parameter subset output by the target sparse predictor; at the same time, prepare the sparse calculations of the next level in advance based on the predictions of the asynchronous predictor to ensure the continuity and efficiency of the calculation process. During the model inference process, dynamically adjust and optimize the prediction results of the target sparse predictor and the asynchronous predictor according to the actual output results and performance requirements. This may involve adjusting the prediction parameter subset in real time to adapt to the characteristics of different input sequences, or fine-tuning the predictor based on model performance feedback.

[0047] By implementing this process, the model can achieve more sophisticated inter-layer sparse calculations during the inference process, thereby significantly improving inference efficiency and reducing latency while ensuring model performance. The introduction of asynchronous predictors enables the model to predict the sparse computing requirements of the next layer, avoiding the overhead of real-time predictions layer by layer, and further accelerating the inference process.

[0048] As an optional embodiment, the target sparse predictor is set based on a hidden Markov model and a classic Viterbi algorithm.

[0049] Optionally, the target sparse predictor is set based on a hidden Markov model (HMM) and a classic Viterbi algorithm, wherein the construction of the hidden Markov model (HMM) requires the definition of a state space, treating each layer in the model as a hidden state. For each layer, different states are defined to represent different degrees of sparsity of the parameters in the layer, such as fully connected, partially connected, or completely disconnected. The Viterbi algorithm is then used to find the best path through the HMM state space, that is, the most likely sequence of sparse parameter subsets. An input sequence can be given, and the algorithm recursively calculates the best state sequence from the beginning of the sequence to each moment based on the initial probability, transition probability, and observation probability. Among them, the initial probability reflects the probability that each layer of the model is in a different sparse state when the sequence is processed. The transition probability describes the probability of transitioning from one layer state to the next layer state, which is dynamically calculated based on the output features of the previous layer. The observation probability corresponds to the matching degree of the output features of the given input sequence data and the current layer state, which is calculated by analyzing the relationship between the output data of the previous layer and the parameters of the current layer. The Viterbi algorithm is based on dynamic programming, which compares possible paths of various sparse parameter subset sequences by constructing a possibility matrix. Each element in the matrix represents the probability of a given sequence and the path to that state. The algorithm selects the path with the highest probability at each step, and eventually forms a most likely sparse parameter subset sequence. When the input sequence data passes through the target sparse predictor, the HMM and Viterbi algorithms dynamically predict the optimal sparse parameter subsets for each level. These sparse parameter subsets are then used to guide the model's reasoning process, so that the model only calculates and uses the parameter subsets predicted to be critical during reasoning, thereby improving reasoning efficiency and reducing the consumption of computing resources.

[0050] By setting the target sparse predictor based on HMM and Viterbi algorithm, the sparse reasoning of the model can be effectively achieved while maintaining the model performance, which is especially important for large models that process large-scale data and real-time applications. This method not only considers the importance of the parameters themselves, but also considers the dependencies between the parameters and the dynamic characteristics of the sequence data, so it can provide a more accurate and efficient sparse solution.

[0051] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that the training method of the language model according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0053] According to an embodiment of the present invention, a language model training device for implementing the above-mentioned language model training method is also provided. Figure 3 is a structural block diagram of a language model training device according to an embodiment of the present invention, such as Figure 3 As shown, the language model training device includes: an input module 32, a determination module 34 and an adjustment module 36. The language model training device is described below.

[0054] The input module 32 is used to input multiple initial sequence data into a preset target sparse predictor respectively, and output prediction parameter subsets corresponding to each of the multiple initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of multiple levels in a preset original language model, and the original language model is used to generate text.

[0055] The determination module 34 is connected to the input module 32 and is used to determine the target parameter subset based on the prediction parameter subsets corresponding to each of the multiple initial sequence data.

[0056] The adjustment module 36 is connected to the determination module 34 and is used to adjust the parameters included in the corresponding target parameter subset in the preset original language model based on the preset training sample to obtain the target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0057] Optionally, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to each of the multiple initial sequence data are output, including: obtaining multiple initial texts; inputting the multiple initial texts into multiple levels in the original language model, and outputting output data of multiple levels corresponding to each of the multiple initial texts, wherein the multiple levels include a perceptron layer and an attention mechanism layer; and determining multiple initial sequence data based on the output data of multiple levels corresponding to each of the multiple initial texts.

[0058] Optionally, before inputting multiple initial sequence data into a preset target sparse predictor respectively and outputting prediction parameter subsets corresponding to each of the multiple initial sequence data, it includes: obtaining training text; extracting text features from the training text; and training a preset original sparse trainer based on the text features and preset multiple sample parameter subsets to obtain a target sparse predictor.

[0059] Optionally, based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, a target parameter subset is determined, including: determining the occurrence probability of each parameter in the prediction parameter subsets corresponding to each of the multiple initial sequence texts; and determining the target parameter subset based on the parameters whose occurrence probability exceeds a preset threshold.

[0060] Optionally, multiple initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to the multiple initial sequence data are output. It also includes: inputting the initial sequence data into a target sparse predictor and a preset asynchronous predictor, and outputting prediction parameter subsets corresponding to the multiple initial sequence data, wherein the asynchronous predictor is used to predict the prediction parameter subset of the next level corresponding to the level predicted by the target sparse predictor.

[0061] Optionally, the target sparse predictor is based on a Hidden Markov Model and a classical Viterbi algorithm setting.

[0062] It should be noted that the input module 32, the determination module 34 and the adjustment module 36 correspond to steps S202 to S206 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0063] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0064] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method and device of the language model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the training method of the above-mentioned language model. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: input multiple initial sequence data into a preset target sparse predictor respectively, and output prediction parameter subsets corresponding to each of the multiple initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of the multiple levels in the preset original language model, and the original language model is used to generate text; based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, determine the target parameter subset; based on the preset training samples, adjust the parameters included in the corresponding target parameter subset in the preset original language model to obtain the target language model, wherein the training samples include sample text and generated text corresponding to the sample text.

[0066] According to an embodiment of the present invention, a method for training a language model is provided. A plurality of initial sequence data are respectively input into a preset target sparse predictor, and prediction parameter subsets corresponding to the plurality of initial sequence data are output, wherein the prediction parameter subsets include prediction parameter subsets corresponding to a plurality of levels in a preset original language model, and the original language model is used to generate text; a target parameter subset is determined based on the prediction parameter subsets corresponding to the plurality of initial sequence data; and a parameter included in the corresponding target parameter subset in the preset original language model is adjusted based on a preset training sample to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text, thereby achieving the purpose of selecting appropriate parameters for model training, thereby achieving the technical effect of improving the model training efficiency, and further solving the technical problem that the model training process is time-consuming and inefficient due to a large number of parameters when performing language model training.

[0067] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0068] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the language model training method provided in the above embodiment.

[0069] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0070] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting multiple initial sequence data into a preset target sparse predictor respectively, and outputting prediction parameter subsets corresponding to each of the multiple initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of the multiple levels in a preset original language model, and the original language model is used to generate text; determining a target parameter subset based on the prediction parameter subsets corresponding to each of the multiple initial sequence data; and adjusting the parameters included in the corresponding target parameter subset in the preset original language model based on a preset training sample to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0071] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: inputting multiple initial sequence data into a preset target sparse predictor respectively, and outputting prediction parameter subsets corresponding to each of the multiple initial sequence data, wherein the prediction parameter subsets include prediction parameter subsets corresponding to each of the multiple levels in a preset original language model, and the original language model is used to generate text; based on the prediction parameter subsets corresponding to each of the multiple initial sequence data, determining a target parameter subset; based on a preset training sample, adjusting the parameters included in the corresponding target parameter subset in the preset original language model to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

[0072] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0073] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0076] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0077] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for training a language model, characterized in that: include: Inputting a plurality of initial sequence data into a preset target sparse predictor respectively, and outputting prediction parameter subsets corresponding to the plurality of initial sequence data respectively, wherein the prediction parameter subsets include prediction parameter subsets corresponding to a plurality of levels in a preset original language model respectively, and the original language model is used to generate text; Determining a target parameter subset based on the prediction parameter subsets corresponding to each of the plurality of initial sequence data; Based on a preset training sample, the parameters included in the target parameter subset corresponding to the preset original language model are adjusted to obtain a target language model, wherein the training sample includes a sample text and a generated text corresponding to the sample text.

2. The method according to claim 1, characterized in that The step of inputting a plurality of initial sequence data into a preset target sparse predictor respectively and outputting prediction parameter subsets corresponding to the plurality of initial sequence data respectively comprises: Get multiple initial texts; Input the multiple initial texts into multiple layers in the original language model, and output output data of multiple layers corresponding to each of the multiple initial texts, wherein the multiple layers include a perceptron layer and an attention mechanism layer; The plurality of initial sequence data are determined based on the output data of the plurality of levels corresponding to each of the plurality of initial texts.

3. The method according to claim 1, characterized in that Before the multiple initial sequence data are respectively input into the preset target sparse predictor and the prediction parameter subsets corresponding to the multiple initial sequence data are output, the method includes: Get training text; Extracting text features from the training text; Based on the text features and a plurality of preset sample parameter subsets, a preset original sparse trainer is trained to obtain the target sparse predictor.

4. The method according to claim 1, characterized in that: The determining of the target parameter subset based on the prediction parameter subsets corresponding to the plurality of initial sequence data respectively comprises: Determine the occurrence probability of each parameter in the prediction parameter subsets corresponding to each of the multiple initial sequence texts; The target parameter subset is determined according to the parameters whose occurrence probability exceeds a preset threshold.

5. The method according to claim 1, characterized in that The step of inputting the plurality of initial sequence data into a preset target sparse predictor respectively and outputting prediction parameter subsets corresponding to the plurality of initial sequence data respectively further comprises: The initial sequence data are respectively input into the target sparse predictor and the preset asynchronous predictor, and prediction parameter subsets corresponding to each of the multiple initial sequence data are output, wherein the asynchronous predictor is used to predict the prediction parameter subset of the next level corresponding to the level predicted by the target sparse predictor.

6. The method according to any one of claims 1 to 5, characterized in that: The target sparse predictor is based on a hidden Markov model and a classical Viterbi algorithm setup.

7. A language model training device, characterized in that: include: An input module, used to input a plurality of initial sequence data into a preset target sparse predictor respectively, and output prediction parameter subsets corresponding to the plurality of initial sequence data respectively, wherein the prediction parameter subsets include prediction parameter subsets corresponding to a plurality of levels in a preset original language model respectively, and the original language model is used to generate text; A determination module, configured to determine a target parameter subset based on the prediction parameter subsets corresponding to each of the plurality of initial sequence data; The adjustment module is used to adjust the parameters included in the target parameter subset corresponding to the preset original language model based on the preset training samples to obtain the target language model, wherein the training samples include sample text and generated text corresponding to the sample text.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the language model training method described in any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the language model training method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training the language model described in any one of claims 1 to 6 is implemented.

Images