Generation method and device of industry large model, electronic equipment and storage medium
By using the combination and freezing technology of industry small models and preset large models in industry large model training, the problem of the large parameters in industry large model training is solved, resulting in large-scale graphics cards taking up and long training time in industry large model training, and a more efficient training process is achieved.
Patent Information
- Application Number
- CN202411530703.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-18
AI Technical Summary
During the training process of existing industry big models, due to the large models to be trained, the graphics card takes up a large amount and the training time is long.
By using the industry data of the target industry, training a small industry model with fewer models, combining it with the preset big model, freezing the parameters of the big model, and training only the parameters in the small model to generate the target industry's Industry big model.
It reduces the number of graphics cards that need to be occupied during the training process, improves training efficiency, and reduces the computing power and time required for training.
Smart Images
Figure CN119476366A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model training technology, and in particular to a method, device, electronic device and storage medium for generating a large industry model. Background Art
[0002] Industry large models refer to large language models applied in specific industry fields. Compared with general large language models, industry large models are more focused on a specific industry, such as finance, medical care, law, etc. Industry large models can better understand and process the industry's professional terminology, specifications, and semantics by training and optimizing on the industry's domain data.
[0003] However, the existing large industry models have many model parameters to be trained, which requires more graphics cards to be used during the training process and takes a long time to train, making the training of large industry models face many challenges. Therefore, how to reduce the training parameters of large industry models during the generation process has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a method, device, electronic device and storage medium for generating a large industry model to solve the problem in the prior art that there are many model parameters to be trained in the large industry model, resulting in a large number of graphics cards being required during the training process and a long training time.
[0005] In a first aspect, an embodiment of the present application provides a method for generating a large industry model, the method comprising:
[0006] Using the industry data of the target industry to train the first model to be trained, to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold;
[0007] The small industry model corresponding to the target industry is combined with a preset large model, and the model parameters in the preset large model are frozen to obtain a second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, and the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold;
[0008] The unfrozen model parameters in the second model to be trained are trained using the training data to obtain an industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
[0009] Optionally, the first model to be trained includes an input layer, an encoder layer and a decoder layer;
[0010] The first to-be-trained model is trained using the industry data of the target industry to obtain a small industry model corresponding to the target industry, including:
[0011] Inputting the industry data of the target industry into the input layer for vectorization to obtain a vector matrix corresponding to the industry data of the target industry, wherein the vector matrix includes a word vector corresponding to each text and a position vector corresponding to each text position;
[0012] Input the vector matrix into the encoder layer for encoding to obtain a coding information matrix corresponding to the industry data of the target industry, wherein the coding information matrix carries characteristic information of the industry data of the target industry, and the coding information matrix is a vector matrix compressed by a low-rank matrix;
[0013] The encoded information matrix is input into the decoder layer for decoding to generate a decoded information matrix corresponding to the industry data of the target industry, wherein the decoded information matrix is a decompressed vector matrix obtained based on contextual prediction of the industry data of the target industry.
[0014] Optionally, the encoder layer includes a plurality of sequentially connected sub-encoding layers, wherein the sub-encoding layer includes at least one trainable adapter layer, and the trainable adapter layer is used to compress and reduce the dimension of input data by using a low-rank matrix;
[0015] The decoder layer includes a plurality of sub-decoding layers connected in sequence, wherein the sub-decoding layer includes a masked multi-head attention layer and a feedforward network layer, and the masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when performing data processing, and the trainable prefix is obtained by training by back-propagation parameter update.
[0016] Optionally, combining the small industry model corresponding to the target industry with a preset large model includes:
[0017] The output data of the small industry model corresponding to the target industry and the output data of the preset large model are linearly combined using a fully connected output layer, wherein the combination weight of the output data of the small industry model corresponding to the target industry is greater than the combination weight of the preset large model.
[0018] Optionally, the using of training data to train unfrozen model parameters in the second to-be-trained model to obtain a large industry model corresponding to the target industry includes:
[0019] The small industry model corresponding to the target industry and the model parameters in the fully connected output layer are trained using the preset objective function and the training data to obtain a large industry model corresponding to the target industry, wherein the model parameters in the preset large model do not participate in the training.
[0020] Optionally, the preset objective function is expressed by the following formula:
[0021]
[0022] Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of the training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ0+ΔΦ(Θ) Represents an algorithm model whose number of parameters to be trained is Φ0+ΔΦ(Θ), Φ0 represents the frozen model parameters in the preset large model, Φ0 is a constant, ΔΦ(Θ) represents the number of unfrozen model parameters in the second model to be trained, ΔΦ(Θ) is a variable, and the dimension of ΔΦ(Θ) is smaller than the dimension of Φ0.
[0023] Optionally, after using the training data to train the unfrozen model parameters in the second to-be-trained model to obtain the industry macro model corresponding to the target industry, the method further includes:
[0024] Obtaining frozen model parameters in the preset large model and the trained industry small model corresponding to the target industry and model parameters in the fully connected output layer;
[0025] All model parameters obtained are linearly merged to complete the deployment of the large industry model corresponding to the target industry.
[0026] In a second aspect, the embodiment of the present application further provides a device for generating a large industry model, the device comprising:
[0027] A first training module is used to train a first model to be trained using industry data of a target industry to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold;
[0028] A combination and freezing module, used for combining the small industry model corresponding to the target industry with a preset large model, and freezing the model parameters in the preset large model to obtain a second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold;
[0029] The second training module is used to train the unfrozen model parameters in the second model to be trained using the training data to obtain the industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
[0030] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0031] Memory, used to store computer programs;
[0032] The processor is used to implement the method for generating the industry big model described in any one of the first aspects when executing the program stored in the memory.
[0033] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating an industry big model as described in any one of the first aspects is implemented.
[0034] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, by using the industry data of the target industry to train the first model to be trained, to obtain the small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than the first preset threshold; the small industry model corresponding to the target industry is combined with the preset large model, and the model parameters in the preset large model are frozen to obtain the second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, the number of model parameters of the preset large model is greater than the second preset threshold, and the second preset threshold is greater than the first preset threshold; the unfrozen model parameters in the second model to be trained are trained using the training data to obtain the industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries. Through the above method, the pre-trained small industry model corresponding to the target industry can be combined with the preset large model to train the large industry model corresponding to the target industry. Since there is no need to train the frozen model parameters in the preset large model during the training process, and the model parameters in the small industry model corresponding to the target industry are much smaller than the model parameters in the preset large model, the number of model parameters that need to be trained can be greatly reduced, thereby reducing the number of graphics cards required to be occupied during the training process and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0037] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0038] Figure 1 A schematic diagram of a process for generating a large industry model provided in an embodiment of the present application;
[0039] Figure 2A schematic diagram of the structure of a first model to be trained provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of the structure of an encoder layer provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of a decoder layer provided in an embodiment of the present application;
[0042] Figure 5 A schematic diagram of the structure of a second model to be trained provided in an embodiment of the present application;
[0043] Figure 6 An operation effect diagram of a large industry model provided in an embodiment of the present application;
[0044] Figure 7 It is an operation effect diagram of a large industry model provided in the prior art;
[0045] Figure 8 A schematic diagram of the structure of a device for generating a large industry model provided in an embodiment of the present application;
[0046] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0048] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.
[0049] In order to solve the problem in the prior art that there are many model parameters to be trained in large industry models, which results in more graphics cards being occupied during the training process and longer training time, the present application provides a method, device, electronic device and storage medium for generating large industry models, which can reduce the number of graphics cards required during training and improve training efficiency.
[0050] See also Figure 1 , Figure 1 The following is a flow chart of a method for generating a large industry model provided in an embodiment of the present application. Figure 1 As shown, the method for generating the industry large model may include the following steps:
[0051] Step 101: Use the industry data of the target industry to train the first model to be trained to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold.
[0052] Specifically, the above-mentioned target industry may refer to any industry such as finance, medical care, law, transportation, urban governance, electricity, energy, industry, retail, etc. The above-mentioned first model to be trained may be a Transformer model or other deep learning models. The above-mentioned small industry model refers to an industry model with fewer parameters and shallower layers, which takes up less storage space and computing resources. Compared with the large industry model, the small industry model usually refers to a model with more parameters and deeper layers, which takes up more storage space and computing resources. The above-mentioned first preset threshold can be set according to the actual situation and is not limited here.
[0053] In this step, the first model to be trained can be trained using the industry data of the target industry to obtain a small industry model corresponding to the target industry. For example, the first model to be trained can be trained using the industry data of the automobile field to obtain a small industry model corresponding to the automobile field.
[0054] Step 102: Combine the small industry model corresponding to the target industry with the preset large model, and freeze the model parameters in the preset large model to obtain a second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, and the number of model parameters of the preset large model is greater than the second preset threshold, and the second preset threshold is greater than the first preset threshold.
[0055] Specifically, the preset large model can be a pre-trained large model commonly used in multiple industries, such as the open source Llama2-Chinese-7b-Chat large model, Pangu large model or other large models. The number of model parameters of the preset large model is much larger than the number of model parameters of the small industry model corresponding to the target industry. For example, the number of parameters of the small industry model corresponding to the trained automotive field can be less than one ten-thousandth of the number of parameters of the Llama2-Chinese-7b-Chat model, which is about 700M. The second preset threshold is greater than the first preset threshold, and the second preset threshold can be set according to actual conditions and is not limited here.
[0056] In this step, the small industry model corresponding to the target industry can be combined with the preset large model, and the model parameters in the preset large model can be frozen to obtain the second model to be trained. In this way, the obtained second model to be trained contains both the prediction and generation capabilities of the target industry and the prediction and generation capabilities of other industries, which can better adapt to specific tasks and application scenarios and improve its performance on these tasks.
[0057] Step 103: Use the training data to train the unfrozen model parameters in the second model to be trained to obtain an industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
[0058] Specifically, the training data is a mixture of industry data of the target industry and other industries. The training data can be documents within the company, within the industry, or crawled from the Internet, such as text data, voice, pictures, videos, e-books, news articles on the Internet, and can support data formats such as pdf, doc, ppt, txt, web pages, etc. As an optional implementation, the industry data of the target industry can account for 1 / 5 to 1 / 2 of the total training data.
[0059] In this step, when training the second model to be trained, only the unfrozen model parameters in the second model to be trained (such as the small industry model corresponding to the target industry, etc.) need to be trained, and there is no need to train the frozen model parameters in the preset large model.
[0060] In this embodiment, the pre-trained small industry model corresponding to the target industry can be combined with the preset large model to train the large industry model corresponding to the target industry. Since there is no need to train the frozen model parameters in the preset large model during the training process, and the model parameters in the small industry model corresponding to the target industry are much smaller than the model parameters in the preset large model, the number of model parameters that need to be trained can be greatly reduced, thereby reducing the number of graphics cards required to be occupied during the training process and improving training efficiency.
[0061] Further, the first model to be trained includes an input layer, an encoder layer and a decoder layer;
[0062] The above step 101, using the industry data of the target industry to train the first model to be trained to obtain a small industry model corresponding to the target industry, includes:
[0063] Input the industry data of the target industry into the input layer for vectorization to obtain the vector matrix corresponding to the industry data of the target industry, where the vector matrix includes the word vector corresponding to each text and the position vector corresponding to each text position;
[0064] The vector matrix is input into the encoder layer for encoding to obtain the encoding information matrix corresponding to the industry data of the target industry, wherein the encoding information matrix carries the characteristic information of the industry data of the target industry, and the encoding information matrix is a vector matrix compressed by a low-rank matrix;
[0065] The encoded information matrix is input into the decoder layer for decoding to generate a decoded information matrix corresponding to the industry data of the target industry, wherein the decoded information matrix is a decompressed vector matrix obtained based on the context prediction of the industry data of the target industry.
[0066] In one embodiment, the first model to be trained may include an input layer, an encoder layer, and a decoder layer, such as Figure 2 As shown. The input layer is used to obtain the representation vector X of each word in the input sentence. The representation vector X can be obtained by adding the Input Embedding of the word (Input Embedding is the feature extracted from the original data) and the Positional Encoding of the word position, that is, X = Input Embedding + Positional Encoding. As an optional implementation, the part of the input layer used to extract features from the original input data can be composed of 15 hidden layers, namely 6 convolutional layer blocks, 5 maximum pooling layers, 2 K-Max pooling layers, and 2 fully connected layers. Among them, the number of channels of the convolutional layer is 64, 64, 128, 128, 256, and 256 respectively; the step size of the pooling layer is set to 4, and the k of the K-Max pooling layer is set to 6; the convolution kernel size is 5*5, and each convolutional layer Block includes two convolutional layer sequences, and each convolutional layer is connected to 1 Temporal batch normalization layer and 1 ReLU activation function.
[0067] After obtaining the vector matrix X composed of the representation vector X, the vector matrix X can be passed into the encoder layer for encoding to obtain the encoding information matrix C of all words in the input sentence. It should be noted that in the encoding process, not only can the characteristic information of the industry data of the target industry be extracted, but also the data can be compressed using a low-rank matrix, thereby greatly reducing the number of trainable parameters of downstream tasks.
[0068] After obtaining the coding information matrix C, the coding information matrix C can be input into the decoder layer for decoding to generate the decoding information matrix corresponding to the industry data of the target industry. It should be noted that a variety of trainable prefixes A, prefix B, prefix C, etc. are added to the decoder layer, and multiple prefixes can be used simultaneously with each other, or they can be used independently. In the prior art, the prefix can be designed manually or established using an automated search method. However, the changes in the manually designed prefixes and the design templates are particularly sensitive to the final performance of the model. Adding one more word, one less word or changing the position will cause a great change, and for the template of the automated search, the cost is relatively high. At the same time, the results of this discretized token search may not be optimal. The prefix in the embodiment of the present application is continuously differentiable, and can be trained by back propagation to update parameters. Multiple different prefixes can be trained simultaneously in the same batch, that is, the same question can be asked in a variety of different ways, so that these prefixes can also learn the language features and knowledge in the specific industry, and can be used to guide the generation of subsequent tokens more efficiently.
[0069] In this embodiment, the first model to be trained consisting of an input layer, an encoder layer and a decoder layer can be used for training to obtain a small industry model corresponding to the target industry, thereby reducing the number of model parameters that need to be trained and facilitating the subsequent combination with the preset large model to obtain the second model to be trained.
[0070] Further, the encoder layer includes a plurality of sequentially connected sub-encoding layers, wherein the sub-encoding layer includes at least one trainable adapter layer, and the trainable adapter layer is used to compress and reduce the dimension of input data by using a low-rank matrix;
[0071] The decoder layer includes multiple sub-decoding layers connected in sequence, wherein the sub-decoding layer includes a masked multi-head attention layer and a feedforward network layer. The masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when processing data, and the trainable prefix is obtained by training by back-propagation parameter update.
[0072] In one embodiment, the encoder layer of the first model to be trained includes a plurality of sub-encoding layers connected in sequence, and the number of sub-encoding layers can be 4 to 64. As an optional implementation, the number of the sub-encoding layers is 8. In addition to the multi-head attention layer, the normalization layer, and the feedforward network layer, each sub-encoding layer also includes a trainable adapter layer, wherein the trainable adapter layer is used to compress and reduce the input data using a low-rank matrix, the multi-head attention layer is used to extract features from the input data, the normalization layer is used to normalize the input data, and the feedforward network layer is used to map the input data to a feature space. In addition, the decoder layer of the first model to be trained includes a plurality of sub-decoding layers connected in sequence, and the number of sub-decoding layers can be 4 to 64. As an optional implementation, the number of the sub-decoding layers is also 8. Each sub-decoding layer may include a masked multi-head attention layer, a feedforward network layer, and a normalization layer, wherein the masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when processing data, and the trainable prefix is obtained by back-propagation parameter update training.
[0073] It should be noted that the number of network layers (such as trainable adapter layer, feedforward network layer and normalization layer, etc.) in each sub-encoding layer and sub-decoding layer can be one or more, which is not specifically limited in the embodiment of the present application. As an optional implementation, the structure of the encoder layer can be as follows: Figure 3 As shown, the structure of the decoder layer can be as follows Figure 4 As shown. Figure 3In the example, each sub-encoding layer includes a multi-head attention layer, a first trainable adapter layer, a first normalization layer, a feedforward network layer, a second trainable adapter layer, and a second normalization layer. The multi-head attention layer can focus on different parts of the input sequence at the same time, thereby extracting information from multiple focus points. Each attention head can focus on different feature subspaces and learn different weight allocations, so that the model can perform parallel processing on different focus points, thereby better capturing different features of the input sequence. The first trainable adapter layer and the second trainable adapter layer are composed of a low-rank matrix. Low rank means that the rank of the matrix is relatively small, that is, the matrix contains less information and can be represented by fewer eigenvectors or singular values. In many applications, low-rank matrices can be used to represent common information and rules in data, thereby achieving data compression, dimensionality reduction and analysis, and greatly reducing the number of trainable parameters for downstream tasks. The first normalization layer and the second normalization layer can accelerate the training convergence process of the neural network. By normalizing the input of each layer, the input of each layer can have a similar distribution, reducing the data distribution differences between different layers, thereby facilitating the propagation and update of the gradient and accelerating the convergence of the model. The feedforward network layer can map the input data to a higher-dimensional, more discriminative feature space, approximating any complex nonlinear function. Through the combination of multiple hidden layers and nonlinear activation functions, the feedforward network can fit various complex functional relationships, thereby achieving accurate modeling and prediction of the input data.
[0074] exist Figure 4 In , each sub-decoding layer includes a masked multi-head attention layer, a third normalization layer, a feedforward network layer, and a fourth normalization layer in sequence. The masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when processing data. The trainable prefix here is continuously differentiable and can be trained by updating parameters through back propagation. Multiple different prefixes can be trained at the same time in the same batch, that is, the same question can be asked in multiple different ways. In this way, these prefixes can also learn the language features and knowledge in specific industries and can be used more efficiently to guide the generation of subsequent tokens. The decoder layer will translate the next word i+1 according to the currently translated words 1,2,3,...i in turn. During the forward propagation process, when translating to word i+1, it is necessary to use the Mask operation to cover i+1 and the words after it. The decoder layer receives the encoding information matrix C of the encoder layer, and then first inputs a translation start symbol " <begin>", predict the first word; then enter the translation start symbol" <begin>" and the first word, predict the second word, and so on.
[0075] In this embodiment, a trainable adapter layer can be added to the encoder layer, and the input data can be compressed and reduced in dimension through the trainable adapter layer, thereby achieving the purposes of data compression, dimensionality reduction and analysis, greatly reducing the number of trainable parameters for downstream tasks, and introducing trainable prefixes in the decoder layer, so that more language features and knowledge in specific industries can be learned, which can be used more efficiently to guide the generation of subsequent tokens.
[0076] Furthermore, the above step 102, combining the small industry model corresponding to the target industry with the preset large model, includes:
[0077] The fully connected output layer is used to linearly combine the output data of the small industry model corresponding to the target industry and the output data of the preset large model, wherein the combination weight of the output data of the small industry model corresponding to the target industry is greater than the combination weight of the preset large model.
[0078] In one embodiment, the fully connected output layer can be used to linearly combine the output data of the small industry model corresponding to the target industry and the output data of the preset large model to obtain the second model to be trained. Taking the automotive field as an example of the target industry, the structure of the second model to be trained is as follows: Figure 5 As shown in the figure, we first build a small industry model in the automotive field based on the Transformer design paradigm. The input data for training can be 200B of vertical industry data. We can use convolutional neural networks to extract the feature matrix of the input data and then input it into the encoder and decoder of the Transformer model to train a small industry model with automotive domain knowledge. The number of parameters of this small model is one ten-thousandth of the Llama2-Chinese-7b-Chat model. After training the small model, the output of the small model and the output of the Llama2-Chinese-7b-Chat model are linearly combined. The weights of the output of the small model and the output of the Llama2-Chinese-7b-Chat model can be configured as needed. Generally, the output weight λ of the small model is configured to be greater than or equal to 0.6, so that the prediction results tend to favor the small industry model with domain knowledge, ensuring the professionalism of the model when used later. Then, all model parameters of the Llama2-Chinese-7b-Chat model are added to the fully connected output layer with a parameter number of 5.2M to form a large industry model in the automotive field.
[0079] Through the above method, a second model to be trained with fewer model parameters can be obtained, thereby effectively reducing the computing power and training time required for training.
[0080] Furthermore, the above step 103, using the training data to train the unfrozen model parameters in the second to-be-trained model to obtain the industry macro model corresponding to the target industry, includes:
[0081] The preset objective function and training data are used to train the small industry model corresponding to the target industry and the model parameters in the fully connected output layer to obtain the large industry model corresponding to the target industry, wherein the model parameters in the preset large model do not participate in the training.
[0082] In one embodiment, when training the second model to be trained, the preset objective function and training data can be used to train the model parameters in the small industry model and the fully connected output layer corresponding to the target industry to obtain the large industry model corresponding to the target industry. Continuing with the above example, when fine-tuning the second model to be trained using the vertical industry knowledge data in the automotive field mixed with other data, the gradient update during fine-tuning can only update the small model and the fully connected output layer of the automotive field knowledge, while protecting the model parameters of the frozen Llama2-Chinese-7b-Chat model. It can be seen that the number of parameters that can be trained for the entire large industry model is the parameters of the small industry model plus the parameters of the fully connected output layer, which is greatly reduced compared to the Llama2-Chinese-7b-Chat model, and the computing power and training time required for training are also reduced accordingly.
[0083] Furthermore, the preset objective function is expressed by the following formula:
[0084]
[0085] Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ0+ΔΦ(Θ) Represents an algorithm model with the number of parameters to be trained being Φ0+ΔΦ(Θ), Φ0 represents the frozen model parameters in the preset large model, Φ0 is a constant, ΔΦ(Φ) represents the number of unfrozen model parameters in the second model to be trained, ΔΦ(Θ) is a variable, and the dimension of ΔΦ(Θ) is smaller than the dimension of Φ0.
[0086] In the prior art, when the traditional industry model is fine-tuned, it is assumed that the initial parameter of the traditional industry model is Φ0, and it is eventually changed to Φ0+ΔΦ through repeated gradient updates, where ΔΦ is the updated parameter. Then the objective function is as follows:
[0087]
[0088] Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ Represents the algorithm model with the parameter to be trained as Φ, that is, for the entire training data set, the training update P Φ , so that the prediction of correct context-target pairs is maximized.
[0089] In the embodiment of the present application, the bypass structure proposed is a more effective parameter method, and the parameters changed for different downstream tasks are consistent, wherein the parameter increment ΔΦ of the task to be trained becomes ΔΦ(Θ), and Θ is a low-rank parameter set with a much smaller size, and the dimension of ΔΦ(Θ) is much smaller than the dimension of Φ0. The task of training fine-tuning is to update ΔΦ, so it becomes to optimize Θ, and the amount of parameters to be optimized is greatly reduced, as shown in the following formula:
[0090]
[0091] Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ0+ΔΦ(Θ) Represents an algorithm model with the number of parameters to be trained being Φ0+ΔΦ(Θ), Φ0 represents the frozen model parameters in the preset large model, Φ0 is a constant, ΔΦ(Θ) represents the number of unfrozen model parameters in the second model to be trained, ΔΦ(Θ) is a variable, and the dimension of ΔΦ(Θ) is smaller than the dimension of Φ0.
[0092] In this embodiment, since the parameter quantity to be trained is Φ0 which is frozen and is a constant quantity, and the dimension of ΦΔΦ(Θ) is much smaller than the dimension of Φ0, when training the same question answer pairs, the training time required by the embodiment of the present application is shorter.
[0093] Furthermore, after the above step 103, in which the unfrozen model parameters in the second to-be-trained model are trained using the training data to obtain the industry macro model corresponding to the target industry, the method further includes:
[0094] Obtain the frozen model parameters in the preset large model and the model parameters in the small industry model and the fully connected output layer corresponding to the trained target industry;
[0095] All model parameters obtained are linearly merged to complete the deployment of the large industry model corresponding to the target industry.
[0096] In one embodiment, the design of linearly combining the output of the small industry model corresponding to the target industry and the output of the preset large model allows the frozen model parameters in the preset large model and the model parameters in the small industry model corresponding to the trained target industry and the fully connected output layer to be linearly merged according to the merging formula during deployment. In this way, the need for intermediate calculation parameters can be eliminated, further reducing the parameters used in predictive reasoning, and compared with the fully fine-tuned model, no reasoning delay is introduced.
[0097] The industry big model in this application can not only use Llama2-Chinese-7b-Chat as the preset big model, but also use other preset big models, such as Pangu big model, etc., and use it to build many small modules for industry tasks in different fields. By freezing the preset big model and efficiently switching tasks by replacing the industry small model and the fully connected output layer in the architecture diagram, only a small number of different parameters for different tasks need to be stored and switched, without the need to repeatedly store and load the same preset big model, which greatly reduces storage requirements and task switching overhead. By optimizing the injected, smaller low-rank matrix, rather than calculating the gradients of most parameters or maintaining the optimizer state, training is made more efficient, and the entry threshold of hardware is lowered. Using the solution proposed in the present invention can reduce the computing power requirement by 3 times.
[0098] The industry large model proposed in this invention can be trained and fine-tuned in 8 days on a training server with only 4 Nvidia A100 (80GB) graphics cards, which reduces hardware dependence. After docking with the front-end technology, the running effect is as follows: Figure 6 The model inference of the present invention takes an average of 150ms. In the automotive industry knowledge dialogue, the professionalism and performance of this model far exceed all other existing models. The following figure shows the reasoning results of other industry large models for the same problem. Figure 7 As shown, it can be seen that the industry large model proposed by the present invention has a fast response time and more accurate response content.
[0099] See also Figure 8 , Figure 8 This is a schematic diagram of a device for generating a large industry model provided in an embodiment of the present application. Figure 8 As shown, the device 800 includes:
[0100] A first training module 801 is used to train a first model to be trained using industry data of a target industry to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold;
[0101] The combination and freezing module 802 is used to combine the industry small model corresponding to the target industry with the preset large model, and freeze the model parameters in the preset large model to obtain a second model to be trained, wherein the preset large model is a pre-trained large model commonly used in multiple industries, and the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold;
[0102] The second training module 803 is used to train the unfrozen model parameters in the second model to be trained using the training data to obtain an industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
[0103] Furthermore, the first model to be trained includes an input layer, an encoder layer and a decoder layer; the first training module 801 includes:
[0104] The first input submodule is used to input the industry data of the target industry into the input layer for vectorization, and obtain a vector matrix corresponding to the industry data of the target industry, wherein the vector matrix includes a word vector corresponding to each text and a position vector corresponding to each text position;
[0105] The second input submodule is used to input the vector matrix into the encoder layer for encoding to obtain the encoding information matrix corresponding to the industry data of the target industry, wherein the encoding information matrix carries the characteristic information of the industry data of the target industry, and the encoding information matrix is a vector matrix compressed by a low-rank matrix;
[0106] The third input submodule is used to input the encoded information matrix into the decoder layer for decoding, and generate a decoded information matrix corresponding to the industry data of the target industry, wherein the decoded information matrix is a decompressed vector matrix obtained based on the context prediction of the industry data of the target industry.
[0107] Further, the encoder layer includes a plurality of sequentially connected sub-encoding layers, wherein the sub-encoding layer includes at least one trainable adapter layer, and the trainable adapter layer is used to compress and reduce the dimension of input data by using a low-rank matrix;
[0108] The decoder layer includes multiple sub-decoding layers connected in sequence, wherein the sub-decoding layer includes a masked multi-head attention layer and a feedforward network layer. The masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when processing data, and the trainable prefix is obtained by training by back-propagation parameter update.
[0109] Further, the combining and freezing module 802 includes:
[0110] The linear combination submodule is used to use the fully connected output layer to linearly combine the output data of the small industry model corresponding to the target industry and the output data of the preset large model, wherein the combination weight of the output data of the small industry model corresponding to the target industry is greater than the combination weight of the preset large model.
[0111] Furthermore, the second training module 803 includes:
[0112] The training submodule is used to train the small industry model corresponding to the target industry and the model parameters in the fully connected output layer using the preset objective function and training data to obtain the large industry model corresponding to the target industry, wherein the model parameters in the preset large model do not participate in the training.
[0113] Furthermore, the preset objective function is expressed by the following formula:
[0114]
[0115] Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ0+ΔΦ(Θ) Represents an algorithm model with the number of parameters to be trained being Φ0+ΔΦ(Θ), Φ0 represents the frozen model parameters in the preset large model, Φ0 is a constant, ΔΦ(Θ) represents the number of unfrozen model parameters in the second model to be trained, ΔΦ(Θ) is a variable, and the dimension of ΔΦ(Θ) is smaller than the dimension of Φ0.
[0116] Furthermore, the device 800 further includes:
[0117] An acquisition module is used to obtain the frozen model parameters in the preset large model and the model parameters in the small industry model and the fully connected output layer corresponding to the trained target industry;
[0118] The linear merging module is used to linearly merge all the model parameters obtained to complete the deployment of the large industry model corresponding to the target industry.
[0119] It should be noted that the device 800 can implement the method for generating a large industry model as provided in any of the aforementioned method embodiments, and can achieve the same technical effect, which will not be described in detail here.
[0120] like Fig. 9 As shown, the embodiment of the present application further provides an electronic device, including a processor 911, a communication interface 912, a memory 913 and a communication bus 914, wherein the processor 911, the communication interface 912, and the memory 913 communicate with each other through the communication bus 914.
[0121] Memory 913, used for storing computer programs;
[0122] In one embodiment of the present application, the processor 911 is used to execute the program stored in the memory 913 to implement the method for generating the industry large model provided by any of the above method embodiments, including:
[0123] Using the industry data of the target industry to train the first model to be trained, to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold;
[0124] The small industry model corresponding to the target industry is combined with the preset large model, and the model parameters in the preset large model are frozen to obtain a second model to be trained, wherein the preset large model is a pre-trained large model commonly used in multiple industries, and the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold;
[0125] The unfrozen model parameters in the second model to be trained are trained using the training data to obtain an industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
[0126] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for generating an industry big model as provided in any of the aforementioned method embodiments is implemented.
[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0129] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.
[0130] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.< / begin> < / begin>
Claims
1. A method for generating a large industry model, characterized in that: The method comprises: Using the industry data of the target industry to train the first model to be trained, to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold; The small industry model corresponding to the target industry is combined with a preset large model, and the model parameters in the preset large model are frozen to obtain a second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, and the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold; The unfrozen model parameters in the second model to be trained are trained using the training data to obtain an industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
2. The method according to claim 1, characterized in that The first model to be trained includes an input layer, an encoder layer and a decoder layer; The first to-be-trained model is trained using the industry data of the target industry to obtain a small industry model corresponding to the target industry, including: Inputting the industry data of the target industry into the input layer for vectorization to obtain a vector matrix corresponding to the industry data of the target industry, wherein the vector matrix includes a word vector corresponding to each text and a position vector corresponding to each text position; Input the vector matrix into the encoder layer for encoding to obtain a coding information matrix corresponding to the industry data of the target industry, wherein the coding information matrix carries characteristic information of the industry data of the target industry, and the coding information matrix is a vector matrix compressed by a low-rank matrix; The encoded information matrix is input into the decoder layer for decoding to generate a decoded information matrix corresponding to the industry data of the target industry, wherein the decoded information matrix is a decompressed vector matrix obtained based on contextual prediction of the industry data of the target industry.
3. The method according to claim 2, characterized in that The encoder layer includes a plurality of sequentially connected sub-encoding layers, wherein the sub-encoding layer includes at least one trainable adapter layer, and the trainable adapter layer is used to compress and reduce the dimension of input data by using a low-rank matrix; The decoder layer includes a plurality of sub-decoding layers connected in sequence, wherein the sub-decoding layer includes a masked multi-head attention layer and a feedforward network layer, and the masked multi-head attention layer and the feedforward network layer introduce at least one trainable prefix when performing data processing, and the trainable prefix is obtained by training by back-propagation parameter update.
4. The method according to claim 1, characterized in that: The combining of the small industry model corresponding to the target industry with the preset large model includes: The output data of the small industry model corresponding to the target industry and the output data of the preset large model are linearly combined using a fully connected output layer, wherein the combination weight of the output data of the small industry model corresponding to the target industry is greater than the combination weight of the preset large model.
5. The method according to claim 4, characterized in that The step of training the unfrozen model parameters in the second to-be-trained model using the training data to obtain the industry macro model corresponding to the target industry includes: The small industry model corresponding to the target industry and the model parameters in the fully connected output layer are trained using the preset objective function and the training data to obtain a large industry model corresponding to the target industry, wherein the model parameters in the preset large model do not participate in the training.
6. The method according to claim 5, characterized in that The preset objective function is expressed by the following formula: Where Z = {(x t ,y t )}, t∈[1,|y|], Z represents the training data, |y| represents the total amount of the training data, (x t ,y t ) indicates that the tth question is answered correctly, x t Represents the question statement, y t Indicates the answer sentence, P Φ0+ΔΦ(Θ) Represents an algorithm model whose number of parameters to be trained is Φ0+ΔΦ(Θ), Φ0 represents the frozen model parameters in the preset large model, Φ0 is a constant, ΔΦ(Θ) represents the number of unfrozen model parameters in the second model to be trained, ΔΦ(Θ) is a variable, and the dimension of ΔΦ(Θ) is smaller than the dimension of Φ0.
7. The method according to claim 5, characterized in that After using the training data to train the unfrozen model parameters in the second to-be-trained model to obtain the industry macro model corresponding to the target industry, the method further includes: Obtaining frozen model parameters in the preset large model and the trained industry small model corresponding to the target industry and model parameters in the fully connected output layer; All model parameters obtained are linearly merged to complete the deployment of the large industry model corresponding to the target industry.
8. A device for generating a large industry model, characterized in that: The device comprises: A first training module is used to train a first model to be trained using industry data of a target industry to obtain a small industry model corresponding to the target industry, wherein the number of model parameters of the first model to be trained is less than a first preset threshold; A combination and freezing module, used for combining the small industry model corresponding to the target industry with a preset large model, and freezing the model parameters in the preset large model to obtain a second model to be trained, wherein the preset large model is a pre-trained large model common to multiple industries, the number of model parameters of the preset large model is greater than a second preset threshold, and the second preset threshold is greater than the first preset threshold; The second training module is used to train the unfrozen model parameters in the second model to be trained using the training data to obtain the industry large model corresponding to the target industry, wherein the number of unfrozen model parameters in the second model to be trained is less than the number of model parameters of the preset large model, and the training data is a mixture of industry data of the target industry and other industries.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the method for generating the industry large model as described in any one of claims 1-7 when executing the program stored in the memory.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating an industry macro model described in any one of claims 1 to 7 is implemented.