Method, device, non-volatile storage medium and computer equipment for determining language model

Abstract

The present invention discloses a method, device, non-volatile storage medium and computer equipment for determining a language model. The method comprises: receiving an initial text; inputting the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures; determining a target language model based on the target network architecture and the preset language model. The present invention solves the current technical problem of long model training time and low efficiency when adjusting the language model during training.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a method and device for determining a language model, a non-volatile storage medium and a computer device. Background Art

[0002] During the fine-tuning process of large models, the prompt construction method of large models seriously affects the effect of downstream tasks. The changes of artificially designed prompts are particularly sensitive. Adding or removing a word, or changing the position will cause relatively large changes. At the same time, the current high-efficiency fine-tuning parameter transfer learning methods are all based on artificially designed networks, resulting in low model training efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, non-volatile storage medium and computer device for determining a language model, so as to at least solve the current technical problems of long model training time and low efficiency when training and adjusting a language model.

[0005] According to one aspect of an embodiment of the present invention, a method for determining a language model is provided, comprising: receiving an initial text; inputting the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures; and determining a target language model based on the target network architecture and the preset language model.

[0006] Optionally, based on the continuous vector, a target network architecture in the embedding layer is determined in a preset search space, including: inputting the continuous vector into multiple network architectures respectively to obtain output results corresponding to the multiple network architectures; determining the performance corresponding to the multiple network architectures based on the output results corresponding to the multiple network architectures; and selecting a network architecture that meets preset conditions among the performance corresponding to the multiple network architectures as the target network architecture in the embedding layer.

[0007] Optionally, based on the output results corresponding to each of the multiple network architectures, determining the performance corresponding to each of the multiple network architectures includes: setting multiple evaluation indicators, wherein the evaluation indicators include accuracy and loss value; based on the multiple evaluation indicators, performing performance evaluation on the output results corresponding to each of the multiple network architectures, and determining the performance corresponding to each of the multiple network architectures.

[0008] Optionally, based on the target network architecture and the preset language model, determining the target language model includes: determining the initial language model according to the target network architecture and the preset language model; obtaining a training set, wherein the training set includes sample text matching the length of the initial text and sample generated text corresponding to the sample text; and using the training set to train the initial language model to obtain the target language model.

[0009] Optionally, the multiple network architectures include a combined network architecture of a multilayer perceptron and a long short-term memory network, a combined network architecture of a multilayer perceptron and a recurrent neural network, a combined network architecture of a convolutional neural network and a long short-term memory network, and a combined network architecture of a convolutional neural network and a recurrent neural network.

[0010] Optionally, the initial text is input into a transformer unit in a target language model to obtain a target continuous vector; and the target text is generated based on the continuous vector and the target language model.

[0011] According to another aspect of an embodiment of the present invention, a device for determining a language model is also provided, including: a receiving module for receiving an initial text; an input module for inputting the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; a search module for determining a target network architecture in the embedding layer in a preset search space based on the continuous vector, wherein the search space includes multiple network architectures; a determination module for determining a target language model based on the target network architecture and the preset language model.

[0012] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned methods for determining a language model.

[0013] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above-mentioned methods for determining a language model is executed.

[0014] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above-mentioned methods for determining a language model is implemented.

[0015] In an embodiment of the present invention, a method for determining a language model is adopted, by receiving an initial text; inputting the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes a plurality of network architectures; determining a target language model based on the target network architecture and the preset language model, thereby achieving the purpose of determining a neural network architecture through automatic search, thereby realizing the technical effect of improving the efficiency of model determination, and further solving the current technical problem of long model training time and low efficiency during training and adjustment of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for determining a language model is shown;

[0018] Figure 2 is a flow chart of a method for determining a language model according to an embodiment of the present invention;

[0019] Figure 3 It is a structural block diagram of a device for determining a language model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] According to an embodiment of the present invention, a method embodiment of a method for determining a language model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0023] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining a language model. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0024] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0025] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for determining the language model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for determining the language model of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0026] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0027] Figure 2 is a flow chart of a method for determining a language model according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0028] Step S202, receiving initial text.

[0029] In this step, the initial text is received, where the initial text is the input data of the text generation task. It usually includes the following types:

[0030] Seed text: This is the starting point for text generation, which can be a word, a phrase, or a sentence. The model will expand based on this seed text to generate longer text.

[0031] Prompt: Prompt can be considered as context or instructions to guide text generation. It is not just a simple seed text, but contains more instructions about the type, style, topic, etc. of the generated text.

[0032] Conditional information: In addition to seed text and prompts, the text generation model can also receive additional conditional information, such as the format that the text should follow, the keywords it contains, the author's style preferences, etc. This information can help the model generate more specific and customized text.

[0033] For example, in a text generation task, if you want to generate the beginning of a story, you may provide a short sentence as a seed text, and give the type of the story (such as science fiction, fantasy, realism, etc.) as conditional information or prompt, and the model will generate subsequent text content based on these inputs.

[0034] Step S204, inputting the initial text into the input layer of the transformer unit in the preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer.

[0035] In this step, the initial text is input into the input layer of the transformer unit in the preset language model to obtain a continuous vector. The initial text may be preprocessed, which usually includes tokenization, breaking the text into a series of words or sub-word units. For Chinese text, it may also include character-level or word-level segmentation. The preprocessed text will be sent to the input layer of the transformer model, where the input layer contains an embedding layer. The embedding layer converts each text unit (such as a word or sub-word) into a fixed-dimensional vector representation, which contains the semantic information and contextual relationship of the unit. This vector representation is the so-called continuous vector, which enables the model to process and understand the complex structure and meaning of the text.

[0036] Step S206, based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures.

[0037] In this step, based on the continuous vector, the target network architecture in the embedding layer is determined in the preset search space. The search space can be initialized first. The search space is a set of all possible network architectures. In this set, there are candidates for a variety of network architectures, such as MLP+LSTM, MLP+RNN, CNN+LSTM, CNN+RNN, etc. Each architecture has its own specific parameters and structure, which can be predefined or dynamically generated. The continuous vector, that is, the text representation after conversion by the embedding layer, will be passed as input to each network architecture candidate in the search space. This is because the continuous vector contains the semantic information of the text, which can help the network architecture understand and process the input text data. For each network architecture in the search space, the model will use the continuous vector for training and evaluation, and there is usually an objective function (such as accuracy, loss function or generalization performance) to measure the effect of the architecture. This objective function can help the model determine which architecture can best improve the performance of a specific task. A certain search algorithm can be used to explore in the search space to find the optimal network architecture. Common search algorithms include but are not limited to random search, grid search, evolutionary algorithms, Bayesian optimization, and reinforcement learning-based methods. These algorithms will iteratively select and evaluate network architectures based on the feedback of the objective function until the one with the best performance is found. Through iterative optimization of the search algorithm, one or several network architectures with the best performance in the evaluation will eventually be determined. These architectures will be selected as the target network architectures in the embedding layer for subsequent text processing or generation tasks.

[0038] Once the target network architecture is determined, the model will use this architecture to further encode and fine-tune the continuous vector to optimize the performance of the model on a specific task. This process may also include multi-task learning, which is to automatically search and adopt different network architectures on different tasks to enhance the generalization ability of the model.

[0039] Using continuous vectors as input, an automatic search algorithm is used to find the network architecture that best processes these vectors in the preset search space, thereby improving the performance and efficiency of the model in tasks such as text generation or classification. This method can reduce the workload of manually designing network architectures, while finding more efficient and more adaptable architectures to task requirements through automatic optimization.

[0040] Step S208, determining a target language model based on the target network architecture and the preset language model.

[0041] In this step, the process of determining the target language model based on the target network architecture and the preset language model involves integrating the best network architecture found (i.e., the target network architecture) into the existing preset language model to optimize the performance of the model on a specific task. The target network architecture determined during the search process can be applied to the embedding layer of the preset language model. This means replacing or enhancing the existing embedding layer with the best architecture found. The new embedding layer converts the discrete symbols of the text into a continuous vector to better capture the semantic information and contextual relationships of the text. After integrating the target network architecture into the preset model, the model can be fine-tuned. This includes adjusting the parameters of the new embedding layer, and possibly the parameters of the entire model, to adapt to the new architecture and improve performance. The fine-tuning process is usually performed on the training data of a specific task to optimize the performance of the model on that task. The adjusted model (i.e., the target language model) is trained on the training dataset and the model performance is evaluated on the validation set. This step ensures that the optimization of the model parameters and architecture is effective and can improve the accuracy, efficiency, or generalization ability of the model. The entire process may require multiple iterations, continuously adjusting the target network architecture and model parameters until the model achieves satisfactory performance on all tasks of interest.

[0042] Ultimately, the determined target language model will combine the preset model backbone and the optimized embedding layer network architecture to process and generate text more efficiently and accurately, especially in downstream tasks that are sensitive to prompt structures, the stability and performance of the model will be significantly improved.

[0043] Through the above steps, the purpose of determining the neural network architecture through automatic search can be achieved, thereby realizing the technical effect of improving the efficiency of model determination, and further solving the current technical problems of long model training time and low efficiency when training and adjusting the language model.

[0044] As an optional embodiment, based on the continuous vector, the target network architecture in the embedding layer is determined in a preset search space, including: inputting the continuous vector into multiple network architectures respectively to obtain output results corresponding to the multiple network architectures; determining the performance corresponding to the multiple network architectures based on the output results corresponding to the multiple network architectures; and selecting the network architecture that meets the preset conditions among the performance corresponding to the multiple network architectures as the target network architecture in the embedding layer.

[0045] Optionally, based on the continuous vector, the target network architecture in the embedding layer is determined in a preset search space, and multiple candidate network architectures are initialized based on the preset search space. These architectures may include different network types (such as MLP+LSTM, MLP+RNN, CNN+LSTM, CNN+RNN, etc.) and different parameter configurations. The continuous vector is used as input and sent to each of the candidate network architectures initialized above, and the output results corresponding to each architecture are obtained through the forward propagation of each architecture. These output results reflect the performance of the network architecture when processing a specific text representation. Based on the output results of each network architecture, its performance indicators on specific tasks are calculated. This may include accuracy, loss value, F1 score, AUC-ROC value, etc., depending on the task type of the model (such as text classification, named entity recognition, machine translation, etc.). The performance indicators of each architecture will be used as a criterion for evaluating its pros and cons. The preset conditions can be the minimum requirements for model performance, for example, you want to obtain an accuracy rate that at least reaches a certain threshold, or restrictions on the complexity, training time, resource consumption, etc. of the model. These conditions are used to screen out qualified network architectures. Compare the performance indicators corresponding to all candidate network architectures, and select the network architecture that meets the preset conditions and has the best performance as the target network architecture in the embedding layer. This usually means selecting those architectures that can provide the best results when processing continuous vectors, while also taking into account resource consumption and efficiency. Integrate the selected target network architecture into the preset language model, replacing or enhancing the original embedding layer. In this way, the original language model can use the new architecture to process continuous vectors more effectively and improve the performance of the model on specific tasks. This process avoids the complexity and uncertainty of manually designing and selecting network architectures through automated search and evaluation, and can find network architectures that are more suitable for processing continuous vectors and completing specific tasks, thereby improving the accuracy and efficiency of the model.

[0046] As an optional embodiment, based on the output results corresponding to each of the multiple network architectures, determining the performance corresponding to each of the multiple network architectures, includes: setting multiple evaluation indicators, wherein the evaluation indicators include accuracy and loss value; based on the multiple evaluation indicators, performing performance evaluation on the output results corresponding to each of the multiple network architectures, and determining the performance corresponding to each of the multiple network architectures.

[0047] Optionally, based on the output results corresponding to each of the multiple network architectures, the performance of each of the multiple network architectures can be determined, and a comprehensive evaluation can indeed be performed by setting multiple evaluation indicators, including accuracy and loss values. A set of evaluation indicators can be defined, which will be used to measure the performance of each network architecture. In addition to accuracy and loss values, other indicators can also be included, such as F1 score (especially suitable for binary or multi-class imbalanced data sets), AUC-ROC (to evaluate the ability of the model to distinguish binary samples), mean absolute error (MAE, for regression tasks), perplexity (Perplexity, often used for language model evaluation), etc., and the corresponding indicators are selected according to the nature of the specific task. The continuous vector is input into each candidate network architecture, and the output result of each architecture is obtained by forward propagation. The form of the output result depends on the downstream task, such as the output of the classification task will be the probability distribution of the category prediction, while the output of the generation task may be the generated text sequence. For the generation task, in addition to using perplexity to evaluate the uncertainty of the generated text by the model, indicators such as BLEU (Bilingual Evaluation Understudy) can also be used to evaluate the similarity between the generated text and the reference text. The performance of each network architecture on all evaluation indicators is comprehensively analyzed, which may involve weight assignment to reflect the importance of different indicators in the final performance evaluation. For example, in some tasks, accuracy may be more important than loss value, and vice versa. Based on the above comprehensive evaluation, a performance score is determined for each network architecture. This score can be a weighted average of all evaluation indicators, or a comprehensive ranking to compare the performance of different architectures.

[0048] Through this process, the performance of multiple network architectures in processing continuous vectors can be systematically compared, so as to select the network architecture that performs best under the preset conditions. This performance determination method based on multiple evaluation indicators can ensure that the selected network architecture not only performs well in a single indicator, but also has a good balance in multiple dimensions, which helps to improve the comprehensive ability and generalization performance of the model.

[0049] As an optional embodiment, a target language model is determined based on a target network architecture and a preset language model, including: determining an initial language model according to the target network architecture and the preset language model; obtaining a training set, wherein the training set includes a sample text matching the length of the initial text and a sample generated text corresponding to the sample text; and using the training set to train the initial language model to obtain a target language model.

[0050] Optionally, based on the target network architecture and the preset language model, the process of determining the target language model is as follows: First, according to the target network architecture found in the search space, it is integrated into the embedding layer of the preset language model to replace the original embedding processing method. This integration may involve modifying the input layer of the preset model to adapt to the input requirements of the target network architecture. The modified model, that is, the model containing the optimized embedding layer, becomes the initial language model. In order to train and optimize the initial language model, a training set needs to be prepared. The training set should contain sample texts (input texts) that match the length of the initial text, and sample generated texts (or category labels, depending on the task) corresponding to these sample texts. These sample texts and sample generated texts constitute the basic data for model training. Before inputting the model, the training set can be preprocessed, including converting the sample texts into continuous vectors, which is usually done by the updated embedding layer. At the same time, it is necessary to ensure that the sample generated texts are also properly formatted so that the model can learn and compare correctly. The initial language model is trained using the training set. The training process aims to minimize the difference between the output predicted by the model and the sample generated text (for generation tasks). During the training process, the model adjusts its parameters, especially those in the target network architecture, to improve the performance of the model on the training set. During or after training, use the validation set to evaluate the performance of the model to ensure that the model not only performs well on the training data, but also has strong generalization capabilities. The validation set should be independent of the training set to avoid overfitting. Based on the performance feedback on the validation set, the model may need to be further tuned, including parameter adjustment, learning rate adjustment, and application of regularization methods, to improve the performance of the model on the validation set. When the model achieves satisfactory performance on the training set and validation set, the optimized network architecture and adjusted parameter configuration are integrated, and the model becomes the target language model. The target language model should be able to maintain good performance on new data and handle specific tasks with higher efficiency and accuracy than the original preset language model.

[0051] Through this series of steps, a language model with better performance on a specific task can be determined through training and tuning based on the target network architecture and the preset language model, namely the target language model. This process not only improves the performance of the model, but also reduces the training time and resource consumption of the model, thus improving the efficiency of model development.

[0052] As an optional embodiment, the multiple network architectures include a combined network architecture of a multilayer perceptron and a long short-term memory network, a combined network architecture of a multilayer perceptron and a recurrent neural network, a combined network architecture of a convolutional neural network and a long short-term memory network, and a combined network architecture of a convolutional neural network and a recurrent neural network.

[0053] Optionally, in the task of searching for the optimal network architecture, multiple combined network architectures can indeed be considered, including but not limited to the following:

[0054] 1. Combination of Multilayer Perceptron and Long Short-Term Memory Network (MLP+LSTM): This architecture combines the advantages of fully connected layers (Multilayer Perceptron MLP) and Long Short-Term Memory Network (LSTM). MLP can be used for nonlinear feature mapping, while LSTM is good at processing sequence data and can capture long-term dependencies. The combined network architecture can more comprehensively process and understand the information in continuous vectors, and is particularly suitable for tasks that require understanding the context of text sequences.

[0055] 2. Combination of Multilayer Perceptron and Recurrent Neural Network (MLP+RNN): Similar to MLP+LSTM, but uses a more general recurrent neural network (RNN). RNN can also capture time dependencies in sequences, but may not be as stable as LSTM when processing very long sequences. MLP+RNN architecture is suitable for tasks with high real-time requirements or moderate sequence lengths.

[0056] 3. Combination of Convolutional Neural Network and Long Short-Term Memory Network (CNN+LSTM): This architecture combines the spatial local sensitivity of CNN and the time series analysis capability of LSTM. CNN can be used to extract features, while LSTM handles sequence problems. The CNN+LSTM architecture can better integrate the characteristics of images and text when processing tasks such as image description generation and video text generation.

[0057] 4. Combination of convolutional neural network and recurrent neural network (CNN+RNN): Similarly, CNN is responsible for extracting features, while RNN processes sequence data. This combined architecture is suitable for tasks that need to consider both local features and sequence dependencies, such as speech recognition, handwriting recognition, etc., as well as certain text generation tasks.

[0058] In practical applications, each combined network architecture has its applicable scenarios and advantages. For example, for text tasks that need to analyze long-distance dependencies, MLP+LSTM or CNN+LSTM may be more suitable; while for tasks with high real-time requirements and short sequence lengths, MLP+RNN or CNN+RNN may be more applicable. By trying these combined architectures in the preset search space, you can find the network architecture that best suits the needs of a specific task, thereby improving the performance and efficiency of the model. This search process usually relies on automatic architecture search algorithms, such as neural architecture search (NAS), to reduce manual intervention and time costs. After determining the target network architecture, it is combined with other components of the preset language model to form a target language model to perform tasks such as text generation or classification.

[0059] As an optional embodiment, the initial text is input into a transformer unit in a target language model to obtain a target continuous vector; and the target text is generated based on the continuous vector and the target language model.

[0060] Optionally, the initial text is input into the transformer unit in the target language model to obtain the target continuous vector, and the process of generating the target text based on the continuous vector and the target language model is as follows: First, the initial text needs to be preprocessed, including word segmentation, stop word removal, standardization, etc., so as to convert the text into a form that the model can understand. The preprocessed text will be converted into a sequence of discrete symbols or word IDs. The preprocessed text ID sequence is input into the embedding layer of the target language model, and the embedding layer converts these discrete IDs into a continuous vector representation, namely the initial continuous vector. This continuous vector contains the semantic information and contextual relationship of the text. Next, the obtained initial continuous vector is used as input and sent to the transformer unit in the target language model. The transformer unit consists of a self-attention mechanism and a feed-forward neural network, which can further process and capture the complex relationships and dependencies in the continuous vector. The transformer unit outputs a series of new continuous vectors that have been processed by the self-attention and feedforward networks and contain deeper semantic information and contextual understanding. The target continuous vectors are then fed into the decoder or generation head, which is the part of the language model responsible for generating text. The decoder / generation head generates a new text sequence, the target text, based on the received continuous vectors. The generated target text may require post-processing, such as removing duplicate words, correcting grammatical errors, ensuring semantic coherence, etc., to generate more suitable text.

[0061] The whole process can be summarized as: initial text -> preprocessing -> embedding conversion -> transformer unit processing -> decoding and text generation -> post-processing -> target text. The target language model processes the initial continuous vector through the transformer unit to generate high-quality target text. This process is widely used in natural language generation tasks such as article writing, dialogue generation, code generation, etc. In this way, the understanding and generation capabilities of the target language model can be fully utilized to create new text that meets the task requirements.

[0062] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0063] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method for determining the language model according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0064] According to an embodiment of the present invention, there is also provided a language model determination device for implementing the above-mentioned language model determination method. Figure 3 is a structural block diagram of a device for determining a language model according to an embodiment of the present invention, such as Figure 3 As shown, the language model determination device includes: a receiving module 32, an input module 34, a search module 36 and a determination module 38. The language model determination device is described below.

[0065] The receiving module 32 is used to receive the initial text.

[0066] The input module 34 is connected to the receiving module 32 and is used to input the initial text into the input layer of the transformer unit in the preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer.

[0067] The search module 36 is connected to the input module 34 and is used to determine the target network architecture in the embedding layer in a preset search space based on the continuous vector, wherein the search space includes multiple network architectures.

[0068] The determination module 38 is connected to the search module 36 and is used to determine the target language model based on the target network architecture and the preset language model.

[0069] Optionally, the search module is used to determine the target network architecture in the embedding layer in a preset search space based on the continuous vector, including: an input unit, used to input the continuous vector into multiple network architectures respectively to obtain output results corresponding to each of the multiple network architectures; a determination unit, used to determine the performance corresponding to each of the multiple network architectures based on the output results corresponding to each of the multiple network architectures; and a selection unit, used to select a network architecture that meets preset conditions from the performance corresponding to each of the multiple network architectures as the target network architecture in the embedding layer.

[0070] Optionally, the determination unit is used to determine the performance corresponding to each of the multiple network architectures based on the output results corresponding to each of the multiple network architectures, including: a setting subunit, used to set multiple evaluation indicators, wherein the evaluation indicators include accuracy and loss value; an evaluation subunit, used to perform performance evaluation on the output results corresponding to each of the multiple network architectures based on the multiple evaluation indicators, and determine the performance corresponding to each of the multiple network architectures.

[0071] Optionally, the determination module is used to determine the target language model based on the target network architecture and the preset language model, including: determining the initial language model according to the target network architecture and the preset language model; obtaining a training set, wherein the training set includes sample text matching the length of the initial text and sample generated text corresponding to the sample text; and using the training set to train the initial language model to obtain the target language model.

[0072] Optionally, the multiple network architectures include a combined network architecture of a multilayer perceptron and a long short-term memory network, a combined network architecture of a multilayer perceptron and a recurrent neural network, a combined network architecture of a convolutional neural network and a long short-term memory network, and a combined network architecture of a convolutional neural network and a recurrent neural network.

[0073] Optionally, the input module is used to input the initial text into the transformer unit in the target language model to obtain a target continuous vector; the generation module is used to generate the target text based on the continuous vector and the target language model.

[0074] It should be noted that the receiving module 32, the input module 34, the search module 36 and the determination module 38 correspond to steps S202 to S206 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0075] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0076] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for determining the language model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the above-mentioned method for determining the language model is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0077] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving an initial text; inputting the initial text into the input layer of the transformer unit in the preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures; determining a target language model based on the target network architecture and the preset language model.

[0078] Optionally, the processor may also execute the program code of the following steps: based on the continuous vector, determining the target network architecture in the embedding layer in a preset search space, including: inputting the continuous vector into multiple network architectures respectively to obtain the output results corresponding to each of the multiple network architectures; determining the performance corresponding to each of the multiple network architectures based on the output results corresponding to each of the multiple network architectures; and selecting the network architecture that meets the preset conditions among the performance corresponding to each of the multiple network architectures as the target network architecture in the embedding layer.

[0079] Optionally, the processor may also execute program code of the following steps: determining the performance of each of the multiple network architectures based on the output results corresponding to each of the multiple network architectures, including: setting multiple evaluation indicators, wherein the evaluation indicators include accuracy and loss value; performing performance evaluation on the output results corresponding to each of the multiple network architectures based on the multiple evaluation indicators, and determining the performance of each of the multiple network architectures.

[0080] Optionally, the processor may also execute program code of the following steps: determining a target language model based on a target network architecture and a preset language model, including: determining an initial language model according to the target network architecture and the preset language model; obtaining a training set, wherein the training set includes a sample text matching the length of the initial text and a sample generated text corresponding to the sample text; and training the initial language model using the training set to obtain a target language model.

[0081] Optionally, the processor may also execute program codes of the following steps: the multiple network architectures include a combined network architecture of a multilayer perceptron and a long short-term memory network, a combined network architecture of a multilayer perceptron and a recurrent neural network, a combined network architecture of a convolutional neural network and a long short-term memory network, and a combined network architecture of a convolutional neural network and a recurrent neural network.

[0082] Optionally, the processor may also execute program code of the following steps: inputting the initial text into a transformer unit in a target language model to obtain a target continuous vector; and generating a target text based on the continuous vector and the target language model.

[0083] By adopting the embodiment of the present invention, a method for determining a language model is provided, which receives an initial text; inputs the initial text into the input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, a target network architecture in the embedding layer is determined in a preset search space, wherein the search space includes multiple network architectures; based on the target network architecture and the preset language model, a target language model is determined, thereby achieving the purpose of determining a neural network architecture through automatic search, thereby realizing the technical effect of improving the efficiency of model determination, and further solving the current technical problems of long model training time and low efficiency when training and adjusting the language model.

[0084] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0085] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the method for determining the language model provided in the embodiment.

[0086] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0087] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: receiving initial text; inputting the initial text into the input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures; determining a target language model based on the target network architecture and the preset language model.

[0088] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: receiving an initial text; inputting the initial text into the input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes multiple network architectures; determining a target language model based on the target network architecture and the preset language model.

[0089] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0095] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining a language model, characterized in that: include: receiving the initial text; Inputting the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; Based on the continuous vector, determining a target network architecture in the embedding layer in a preset search space, wherein the search space includes a plurality of network architectures; Based on the target network architecture and the preset language model, a target language model is determined.

2. The method according to claim 1, characterized in that The step of determining a target network architecture in the embedding layer in a preset search space based on the continuous vector includes: Inputting the continuous vectors into the multiple network architectures respectively to obtain output results corresponding to the multiple network architectures respectively; Determine the performance corresponding to each of the multiple network architectures based on the output results corresponding to each of the multiple network architectures; A network architecture that meets preset conditions among the performances corresponding to the multiple network architectures is selected as the target network architecture in the embedding layer.

3. The method according to claim 2, characterized in that The determining, based on the output results corresponding to the multiple network architectures, the performance corresponding to each of the multiple network architectures includes: Setting multiple evaluation indicators, wherein the evaluation indicators include accuracy and loss value; Based on the multiple evaluation indicators, performance evaluation is performed on the output results corresponding to each of the multiple network architectures to determine the performance corresponding to each of the multiple network architectures.

4. The method according to claim 1, characterized in that: The determining of the target language model based on the target network architecture and the preset language model includes: Determining an initial language model according to the target network architecture and the preset language model; Acquire a training set, wherein the training set includes a sample text matching the length of the initial text and a sample generated text corresponding to the sample text; The initial language model is trained using the training set to obtain the target language model.

5. The method according to any one of claims 1 to 4, characterized in that: The multiple network architectures include a combined network architecture of a multilayer perceptron and a long short-term memory network, a combined network architecture of a multilayer perceptron and a recurrent neural network, a combined network architecture of a convolutional neural network and a long short-term memory network, and a combined network architecture of a convolutional neural network and a recurrent neural network.

6. The method according to any one of claims 1 to 4, characterized in that: Also includes: Inputting the initial text into the transformer unit in the target language model to obtain a target continuous vector; A target text is generated based on the continuous vector and the target language model.

7. A device for determining a language model, characterized in that: include: A receiving module, used for receiving an initial text; An input module, configured to input the initial text into an input layer of a transformer unit in a preset language model to obtain a continuous vector, wherein the preset language model is used to generate text, and the input layer of the transformer unit includes an embedding layer; A search module, configured to determine a target network architecture in the embedding layer in a preset search space based on the continuous vector, wherein the search space includes a plurality of network architectures; A determination module is used to determine a target language model based on the target network architecture and the preset language model.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method for determining the language model according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the method for determining the language model according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining a language model according to any one of claims 1 to 6 is implemented.

Images