Model training method and device, electronic equipment and storage medium
By reducing the weight tensor size of the model to be trained, it is trained, and the target weight tensor for large-scale model training is solved, which solves the problem of high resource consumption of large-scale model training and achieves acceleration and cost reduction of model training.
Patent Information
- Application Number
- CN202411805051.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-11
AI Technical Summary
Large-scale model training requires a long time to consume huge computing resources, resulting in high trial and error costs. The model distillation technology has the problem of computing power being unable to be reused, resulting in resource bottlenecks.
By obtaining the first dimension of the weight tensor of the model to be trained and reducing it to obtain the second dimension, building a first model with a size of the second dimension, training it, generating a second target weight tensor, and using it to train the model to be trained to achieve acceleration of model training.
It realizes training of training models with a smaller computing resource occupation, accelerates the large-scale model training process and reduces training costs.
Smart Images
Figure CN119808868A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a model training method and device, electronic equipment, and storage medium. Background Art
[0002] With the rapid development of science and technology, large models are being used more and more widely in various fields.
[0003] A large model is a model with a large number of parameters and complex structures, which is used to process large-scale data sets (e.g., with millions to billions or even more parameters). These parameters are learned from the data through the training process to capture complex patterns and relationships in the data. The training of large models requires a long time and consumes huge computing resources, which also brings higher trial and error costs.
[0004] In related technologies, model distillation technology is used to convert large models into smaller models to save computing resources for model training. However, this solution still has a certain amount of computing power that cannot be reused, causing a bottleneck in computing resources. Summary of the invention
[0005] In view of the above problems, a training method and device, electronic device, and storage medium for a model are proposed to overcome the above problems or at least partially solve the above problems, including:
[0006] In response to a request for model training, obtaining a model to be trained;
[0007] Determine a first size of a weight tensor of the model to be trained, and reduce the first size to obtain a second size;
[0008] constructing a first model whose weight tensor has a size of the second size;
[0009] Acquire sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor;
[0010] Generate a second target weight tensor corresponding to the model to be trained according to the first target weight tensor;
[0011] The model to be trained is trained according to the second target weight to obtain a target model, and the target model is fed back.
[0012] Optionally, generating a second target weight tensor corresponding to the to-be-trained model according to the first target weight tensor includes:
[0013] Splitting the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor;
[0014] Performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors;
[0015] Initializing a weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into a plurality of second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors;
[0016] Expanding the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors;
[0017] The multiple target one-dimensional vectors are combined to obtain the second target weight tensor.
[0018] Optionally, performing weighted processing on each element of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors includes:
[0019] Get the preset upper and lower bounds;
[0020] Determine the horizontal coordinate value of each element, and compare the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result;
[0021] Determine the element weight value corresponding to each element according to the comparison result;
[0022] The plurality of first one-dimensional vectors are weighted according to each element weight value to obtain the plurality of weighted one-dimensional vectors.
[0023] Optionally, determining an element weight corresponding to each element according to the comparison result includes:
[0024] If the horizontal coordinate value of the element is 0, the weight value of the element is 1;
[0025] If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value;
[0026] If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
[0027] Optionally, determining the element weight value according to the element value of the element and the upper limit value includes:
[0028] The element weight value is determined by the following formula:
[0029]
[0030] Wherein, a is the upper bound value, and x is the element value.
[0031] Optionally, reducing the first size to obtain a second size includes:
[0032] The first size is reduced to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size.
[0033] A device for model training, comprising:
[0034] A module for acquiring a model to be trained, used to obtain a model to be trained in response to a request for model training;
[0035] A size processing module, used to determine a first size of a weight tensor of the model to be trained, and reduce the first size to obtain a second size;
[0036] A first model building module, used to build a first model whose weight tensor has a size of the second size;
[0037] A first target weight tensor determination module, used to obtain sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor;
[0038] A second target weight tensor generation module, used to generate a second target weight tensor corresponding to the model to be trained according to the first target weight tensor;
[0039] A target model generation module is used to train the model to be trained according to the second target weight to obtain a target model and feed back the target model.
[0040] Optionally, the second target weight tensor generation module includes:
[0041] A first target weight tensor splitting submodule, used to split the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor;
[0042] A weighted processing submodule, used for performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors;
[0043] A weight tensor splitting submodule of a model to be trained, used for initializing the weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into a plurality of second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors;
[0044] A second one-dimensional vector expansion submodule, configured to expand the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors;
[0045] The second target weight tensor generation submodule is used to merge the multiple target one-dimensional vectors to obtain the second target weight tensor.
[0046] Optionally, the weighted processing submodule includes:
[0047] A threshold value acquisition unit, used to acquire a preset upper limit value and a lower limit value;
[0048] A comparison result acquisition unit, used for determining the horizontal coordinate value of each element, and comparing the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result;
[0049] An element weight value determining unit, used to determine an element weight value corresponding to each element according to the comparison result;
[0050] A weighted processing unit is used to perform weighted processing on the multiple first one-dimensional vectors according to the weight value of each element to obtain the multiple weighted one-dimensional vectors.
[0051] Optionally, the element weight value determining unit is further configured to:
[0052] If the horizontal coordinate value of the element is 0, the weight value of the element is 1;
[0053] If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value;
[0054] If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
[0055] Optionally, the element weight value determining unit is further configured to:
[0056] The element weight value is determined by the following formula:
[0057]
[0058] Wherein, a is the upper bound value, and x is the element value.
[0059] Optionally, the size processing module includes:
[0060] The size processing submodule is used to reduce the first size to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size.
[0061] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for model training as described above is implemented.
[0062] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the model training method as described above when executed by the processor.
[0063] A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the model training method as described above
[0064] The embodiments of the present invention have the following advantages: by responding to a request for model training, obtaining a model to be trained, determining a first size of a weight tensor of the model to be trained, reducing the first size to obtain a second size, constructing a first model whose weight tensor size is the second size, obtaining sample data corresponding to the model to be trained, and training the first model according to the sample data to obtain a first target weight tensor, generating a second target weight tensor corresponding to the model to be trained according to the first target weight tensor, training the model to be trained according to the second target weight to obtain a target model, and feeding back the target model, the training process of the model to be trained is accelerated with less computing power resources occupied, thereby reducing the cost of large-scale model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0066] Figure 1 is a flowchart of a model training method provided by one embodiment of the present invention;
[0067] Figure 2 is a diagram of an implementation process of training a large-size model provided by an embodiment of the present invention;
[0068] Figure 3 is a diagram of an implementation process of upsampling a weight tensor provided by an embodiment of the present invention;
[0069] Figure 4 It is a structural block diagram of a model training device provided by one embodiment of the present invention;
[0070] Figure 5 It is a schematic diagram of the structure of an electronic device for training a model provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0071] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0072] Large language models play an increasingly important role in the field of natural language processing. Large language models usually contain hundreds of millions or even hundreds of billions of parameters, which enables them to capture complex language patterns and relationships, so that they can understand and generate natural language. They are widely used in many fields such as machine translation, text summarization, and question-answering systems. However, the reasoning effect of large models is highly correlated with the scale of model pre-training. The computing resources required to expand the size of the model grow exponentially. The training of large-scale models requires a long time and consumes huge computing resources, which also brings higher trial and error costs.
[0073] Faced with the challenges of training large models, relevant technologies have adopted methods such as increasing the parallelism and stability of calculations, reducing calculation accuracy, and modifying network architecture to reduce computing overhead and improve training quality.
[0074] In addition, the industry may train multiple large models of different sizes for the same training data to adapt to model reasoning in different environments and computing power. In most cases, these models of different sizes are trained separately, or small-scale models are distilled from large-scale models. Since the training cost of large-scale models is much higher than that of small-scale models, even the latter may have a certain amount of computing power that cannot be reused.
[0075] Therefore, the present invention is based on the core idea of using the weights of a small-scale model as a reference before training the large model to achieve a more reasonable initialization weight for the large model, thereby helping the large model training to converge faster, thereby achieving the purpose of reducing the cost of large-scale model training, improving the degree of reuse of computing resources, and accelerating the training of large models.
[0076] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0077] Reference Figure 1, shows a flowchart of a model training method provided by an embodiment of the present invention, which may specifically include the following steps:
[0078] Step 101, in response to a request for model training, obtaining a model to be trained;
[0079] In practical applications, the model to be trained may be a large model, that is, a model with large-scale parameters, such as a large language model, etc. These models require more computing resources and a longer training time to be trained using conventional training methods until the model converges.
[0080] Step 102, determining a first size of a weight tensor of the model to be trained, and reducing the first size to obtain a second size;
[0081] Weight tensor, the weight of the model is the parameter of the model, and the weight tensor is the tensor composed of the weights of the model. The size of the weight tensor refers to the size of the tensor in each dimension. The size of the weight tensor determines the complexity, computing requirements and expressiveness of the model. The larger the size of the weight tensor, the more parameters the model has. For example, the weight tensor (3,4) means that the tensor composed of the weights of the model is a two-dimensional tensor, with a size of 3 in one dimension and a size of 4 in two dimensions.
[0082] In some embodiments of the present invention, reducing the first size to obtain a second size includes:
[0083] The first size is reduced to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size.
[0084] Exemplarily, a weight tensor of a first size (9, 9, 9) is reduced by 1 / 3 to obtain a weight tensor of a second size (3, 3, 3), so that the first size is an integer multiple of the second size.
[0085] Step 103, constructing a first model whose weight tensor has a size of the second size;
[0086] The first model is a smaller model after reducing the size of the weight tensor.
[0087] Step 104, obtaining sample data corresponding to the model to be trained, and training the first model according to the sample data to obtain a first target weight tensor;
[0088] The first model is trained using the same data as the model to be trained until the model converges. The first target weight tensor is the weight tensor corresponding to the first model after the first model converges. Compared with the initial model to be trained, since there are fewer parameters, the computing resources occupied by training until the model converges are also lower, and the time required is also shorter.
[0089] Step 105, generating a second target weight tensor corresponding to the model to be trained according to the first target weight tensor;
[0090] In some examples, a preset interpolation algorithm can be used to expand the size of the first target weight tensor, thereby generating a second target weight tensor that is consistent with the size of the weight tensor of the model to be trained and larger in scale.
[0091] In some embodiments of the present invention, generating a second target weight tensor corresponding to the model to be trained according to the first target weight tensor includes:
[0092] Sub-step 1041, splitting the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor;
[0093] Exemplarily, assuming that the first target weight tensor is an n-dimensional tensor, the first target weight tensor is split to obtain n first one-dimensional vectors.
[0094] Sub-step 1042, performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors;
[0095] Since each element of the first one-dimensional vector has a different degree of influence on the eigenvalue, in order to make the expanded vector elements more accurate, each element needs to be weighted.
[0096] In some embodiments of the present invention, performing weighted processing on each element of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors includes:
[0097] Get the preset upper and lower bounds;
[0098] Determine the horizontal coordinate value of each element, and compare the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result;
[0099] Determine the element weight value corresponding to each element according to the comparison result;
[0100] The plurality of first one-dimensional vectors are weighted according to each element weight value to obtain the plurality of weighted one-dimensional vectors.
[0101] In the specific implementation, the midpoint position of each first one-dimensional vector is set as the coordinate origin, and the horizontal coordinate value of each element on each first one-dimensional vector reflects the importance of the element. The upper and lower bounds can be divided into multiple intervals, representing different degrees of element influence. According to the comparison results of the horizontal coordinate value and the upper and lower bounds, different element weight values can be determined: if the horizontal coordinate value is between the upper and lower bounds, it means that the element weight value can be further divided into fine-grained levels; if the horizontal coordinate value is less than the lower bound, or greater than the upper bound, it means that the influence of the element is small, and the element weight value can also be set smaller.
[0102] In some embodiments of the present invention, determining the element weight corresponding to each element according to the comparison result includes:
[0103] If the horizontal coordinate value of the element is 0, the weight value of the element is 1;
[0104] If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value;
[0105] If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
[0106] In this embodiment, if the horizontal coordinate value of the element is 0, it means that the element has a large influence, and the element weight value is set to 1; if the horizontal coordinate value of the element is less than the lower limit value and the horizontal coordinate value of the element is greater than or equal to the upper limit value, it means that the element has a small influence or almost no influence, and the element weight value is set to 0; if the horizontal coordinate value of the element is less than the lower limit value and the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value can be further determined according to the element value and the upper limit value of the element, for example, the larger the element value and the closer the calculation result of the element value according to the preset algorithm is to the upper limit value, the larger the element weight value.
[0107] In some embodiments of the present invention, determining the element weight value according to the element value of the element and the upper limit value includes:
[0108] The element weight value is determined by the following formula:
[0109]
[0110] Wherein, a is the upper bound value, and x is the element value.
[0111] In some examples, let the upper limit be 3, the lower limit be -3, and each first one-dimensional vector be V1, V2, ... V n, the horizontal coordinate value of each element of the first one-dimensional vector is d with the midpoint of the first one-dimensional vector as the origin, and the element value of each element is x, then the element weight value L(x) corresponding to each element can be determined by the following algorithm L:
[0112]
[0113] Sub-step 1043, initializing the weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into multiple second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors;
[0114] Initializing the weight tensor of the model to be trained means assigning an initial value to each element in the weight tensor.
[0115] For example, let the weight tensor of the model to be trained be T, and there are currently n first one-dimensional vectors V n , then T can be split into n second one-dimensional vectors T n .
[0116] Sub-step 1044, expanding the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors;
[0117] For example, let floor be the rounding function, x n is the element value of the target one-dimensional vector, L(x n ) is a weighted one-dimensional vector, the second one-dimensional vector can be interpolated and expanded by the following algorithm S to obtain the target one-dimensional vector:
[0118]
[0119] Where W(x1...x n ) represents the second one-dimensional vector, S(x1,...,x n ) represents the target one-dimensional vector after interpolation and expansion.
[0120] Sub-step 1045, merge the multiple target one-dimensional vectors to obtain the second target weight tensor.
[0121] The target one-dimensional vectors obtained after splitting, interpolation and expansion are merged again to obtain the required second target weight tensor.
[0122] Step 106: train the model to be trained according to the second target weight to obtain a target model, and feed back the target model.
[0123] By training the model to be trained through the second target weight, it is possible to occupy less computing resources and accelerate the training process of the model to be trained.
[0124] In some embodiments of the present invention, Figure 2 As shown in the figure, an implementation process diagram for accelerating training of large-scale models is also provided, as follows:
[0125] 1) Determine the size of the weight tensor of the large-size model (final model) to be trained;
[0126] 2) Reduce the size of the weight tensor by an integer multiple to construct the first model;
[0127] 3) Training the first model in a conventional training manner;
[0128] 4) Upsampling (i.e., interpolation and expansion) the weight tensor after the first model is trained to convergence to obtain an upsampled weight tensor;
[0129] 5) Use the upsampled weight tensor as the initial weight tensor to train a large-size model;
[0130] 6) Get the final large-size model.
[0131] In some examples, such as Figure 3 As shown, the above step 4) may specifically include the following process:
[0132] Split the weight tensor of the first model after training to convergence into n dimensions (each dimension is a one-dimensional vector);
[0133] Calculate the weight W of the elements for each dimension;
[0134] Initialize the tensor T of the large-size model to be trained and split it into n one-dimensional vectors T n ;
[0135] According to W, the one-dimensional vector T n Perform upsampling;
[0136] After n upsampling, T n Merge to obtain the weight tensor T required to accelerate the training of large-size models.
[0137] The embodiments of the present invention have the following advantages: by responding to a request for model training, obtaining a model to be trained, determining a first size of a weight tensor of the model to be trained, reducing the first size to obtain a second size, constructing a first model whose weight tensor size is the second size, obtaining sample data corresponding to the model to be trained, and training the first model according to the sample data to obtain a first target weight tensor, generating a second target weight tensor corresponding to the model to be trained according to the first target weight tensor, training the model to be trained according to the second target weight to obtain a target model, and feeding back the target model, the training process of the model to be trained is accelerated with less computing power resources occupied, thereby reducing the cost of large-scale model training.
[0138] In addition, the idea of the related technology is to optimize the computational complexity of existing training by pruning. However, in most scenarios, large language models of different scales need to be trained. Therefore, the present invention innovatively proposes to use the weights of a small-scale model that takes up less computing power to initialize the weights of a large-scale target model.
[0139] Moreover, the present invention proposes an upsampling interpolation algorithm for expanding the deep learning weight tensor. This idea and algorithm can be applied to any training acceleration scenario for expanding the scale of deep learning models.
[0140] Furthermore, since the present invention accelerates the convergence speed of large models based on the idea of optimizing initialization weights, the acceleration method of the present invention is fully decoupled from other optimization methods and is not mutually exclusive with other optimization ideas. Therefore, other optimization methods can be adapted without introducing additional workload.
[0141] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0142] Reference Figure 4 , shows a schematic diagram of the structure of a model training device provided by an embodiment of the present invention, which may specifically include the following modules:
[0143] The to-be-trained model acquisition module 401 is used to obtain the to-be-trained model in response to a request for model training;
[0144] A size processing module 402 is used to determine a first size of the weight tensor of the model to be trained, and reduce the first size to obtain a second size;
[0145] A first model building module 403, used to build a first model whose weight tensor has a size of the second size;
[0146] A first target weight tensor determination module 404 is used to obtain sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor;
[0147] A second target weight tensor generating module 405 is used to generate a second target weight tensor corresponding to the to-be-trained model according to the first target weight tensor;
[0148] The target model generation module 406 is used to train the model to be trained according to the second target weight to obtain a target model and feed back the target model.
[0149] In some embodiments of the present invention, the second target weight tensor generation module 405 includes:
[0150] A first target weight tensor splitting submodule, used to split the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor;
[0151] A weighted processing submodule, used for performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors;
[0152] A weight tensor splitting submodule of a model to be trained, used for initializing the weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into a plurality of second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors;
[0153] A second one-dimensional vector expansion submodule, configured to expand the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors;
[0154] The second target weight tensor generation submodule is used to merge the multiple target one-dimensional vectors to obtain the second target weight tensor.
[0155] In some embodiments of the present invention, the weighted processing submodule includes:
[0156] A threshold value acquisition unit, used to acquire a preset upper limit value and a lower limit value;
[0157] A comparison result acquisition unit, used for determining the horizontal coordinate value of each element, and comparing the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result;
[0158] An element weight value determining unit, used to determine an element weight value corresponding to each element according to the comparison result;
[0159] A weighted processing unit is used to perform weighted processing on the multiple first-dimensional vectors according to the weight value of each element to obtain the multiple weighted one-dimensional vectors.
[0160] In some embodiments of the present invention, the element weight value determining unit is further configured to:
[0161] If the horizontal coordinate value of the element is 0, the weight value of the element is 1;
[0162] If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value;
[0163] If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
[0164] In some embodiments of the present invention, the element weight value determining unit is further configured to:
[0165] The element weight value is determined by the following formula:
[0166]
[0167] Wherein, a is the upper bound value, and x is the element value.
[0168] In some embodiments of the present invention, the size processing module 402 includes:
[0169] The size processing submodule is used to reduce the first size to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size.
[0170] Some embodiments of the present invention also provide an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.
[0171] Memory 503, used for storing computer programs;
[0172] The processor 501 is used to execute the program stored in the memory 703, and implements the following steps:
[0173] In response to a request for model training, obtaining a model to be trained;
[0174] Determine a first size of a weight tensor of the model to be trained, and reduce the first size to obtain a second size;
[0175] constructing a first model whose weight tensor has a size of the second size;
[0176] Acquire sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor;
[0177] Generate a second target weight tensor corresponding to the model to be trained according to the first target weight tensor;
[0178] The model to be trained is trained according to the second target weight to obtain a target model, and the target model is fed back.
[0179] In some embodiments of the present invention, generating a second target weight tensor corresponding to the model to be trained according to the first target weight tensor includes:
[0180] Splitting the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor;
[0181] Performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors;
[0182] Initializing a weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into a plurality of second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors;
[0183] Expanding the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors;
[0184] The multiple target one-dimensional vectors are combined to obtain the second target weight tensor.
[0185] In some embodiments of the present invention, performing weighted processing on each element of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors includes:
[0186] Get the preset upper and lower bounds;
[0187] Determine the horizontal coordinate value of each element, and compare the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result;
[0188] Determine the element weight value corresponding to each element according to the comparison result;
[0189] The plurality of first one-dimensional vectors are weighted according to each element weight value to obtain the plurality of weighted one-dimensional vectors.
[0190] In some embodiments of the present invention, determining the element weight corresponding to each element according to the comparison result includes:
[0191] If the horizontal coordinate value of the element is 0, the weight value of the element is 1;
[0192] If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value;
[0193] If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
[0194] In some embodiments of the present invention, determining the element weight value according to the element value of the element and the upper limit value includes:
[0195] The element weight value is determined by the following formula:
[0196]
[0197] Wherein, a is the upper bound value, and x is the element value.
[0198] In some embodiments of the present invention, reducing the first size to obtain a second size includes:
[0199] Reduce the first size to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size
[0200] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the training method of the above model is implemented.
[0201] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above model training method when executed by a processor.
[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0204] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0205] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0206] The communication interface is used for communication between the above terminal and other devices.
[0207] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0208] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0209] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0210] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0211] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0213] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0214] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.
[0215] The training method and device, electronic device, and storage medium of the provided model are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A model training method, characterized in that: The method comprises: In response to a request for model training, obtaining a model to be trained; Determine a first size of a weight tensor of the model to be trained, and reduce the first size to obtain a second size; constructing a first model whose weight tensor has a size of the second size; Acquire sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor; Generate a second target weight tensor corresponding to the model to be trained according to the first target weight tensor; The model to be trained is trained according to the second target weight to obtain a target model, and the target model is fed back.
2. The method according to claim 1, characterized in that Generating a second target weight tensor corresponding to the to-be-trained model according to the first target weight tensor includes: Splitting the first target weight tensor into a plurality of first one-dimensional vectors; wherein the number of the plurality of first one-dimensional vectors is the same as the number of dimensions of the first target weight tensor; Performing weighted processing on the elements of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors; Initializing a weight tensor of the model to be trained, and splitting the weight tensor of the model to be trained into a plurality of second one-dimensional vectors; wherein the number of the second one-dimensional vectors is the same as the number of the first one-dimensional vectors; Expanding the plurality of second one-dimensional vectors according to the plurality of weighted one-dimensional vectors to obtain a plurality of target one-dimensional vectors; The multiple target one-dimensional vectors are combined to obtain the second target weight tensor.
3. The method according to claim 2, characterized in that The step of performing weighted processing on each element of each of the first one-dimensional vectors to obtain a plurality of weighted one-dimensional vectors includes: Get the preset upper and lower bounds; Determine the horizontal coordinate value of each element, and compare the horizontal coordinate value with the upper limit value and the lower limit value to obtain a comparison result; Determine the element weight value corresponding to each element according to the comparison result; The plurality of first one-dimensional vectors are weighted according to each element weight value to obtain the plurality of weighted one-dimensional vectors.
4. The method according to claim 3, characterized in that Determine the element weight corresponding to each element according to the comparison result, including: If the horizontal coordinate value of the element is 0, the weight value of the element is 1; If the horizontal coordinate value of the element is less than the upper limit value, and the horizontal coordinate value of the element is greater than or equal to the lower limit value, then determining the element weight value according to the element value of the element and the upper limit value; If the horizontal coordinate value of the element is less than the lower limit value, or the horizontal coordinate value of the element is greater than or equal to the upper limit value, the element weight value is 0.
5. The method according to claim 4, characterized in that The determining the element weight value according to the element value and the upper limit value of the element includes: The element weight value is determined by the following formula: Wherein, a is the upper bound value, and x is the element value.
6. The method according to claim 1, characterized in that The reducing the first size to obtain a second size includes: The first size is reduced to the second size according to a preset ratio; wherein the first size is an integer multiple of the second size.
7. A model training device, characterized in that: The device comprises: A module for acquiring a model to be trained, used to obtain a model to be trained in response to a request for model training; A size processing module, used to determine a first size of a weight tensor of the model to be trained, and reduce the first size to obtain a second size; A first model building module, used to build a first model whose weight tensor has a size of the second size; A first target weight tensor determination module, used to obtain sample data corresponding to the model to be trained, and train the first model according to the sample data to obtain a first target weight tensor; A second target weight tensor generation module, used to generate a second target weight tensor corresponding to the model to be trained according to the first target weight tensor; A target model generation module is used to train the model to be trained according to the second target weight to obtain a target model and feed back the target model.
8. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method for model training as claimed in any one of claims 1 to 6.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for model training as described in any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for model training according to any one of claims 1 to 6 is implemented.