Deep learning-oriented large model training method and device
By integrating small-scale mapping models and large-scale target models in low-dimensional feature space, the problem of training large-scale deep learning models under data scarcity is solved, and efficient and low-cost model training is achieved.
Patent Information
- Application Number
- CN202311225817.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-21
- Publication Date
- 2025-03-21
AI Technical Summary
In the case of scarcity of data, how to effectively build and train large-scale deep learning models must not only ensure the performance of the model, but also reduce the training cost.
By obtaining labeled data sets in multiple scenarios, the initial mapping model is trained separately to obtain the eigenvector table, combining the downsampling and upsampling parts of the target big model, the coding table and the eigenvector table are used to calculate the embedded eigenvector table, adjust the model parameters, and gradually optimize the target big model.
This method combines small-scale mapping models with large-scale target models in low-dimensional feature space, improves computing efficiency, reduces the computational cost of model training and inference, and effectively utilizes limited training data to achieve the goal of training large models at low cost.
Smart Images
Figure CN119670814A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a training method and device for a large model for deep learning. Background Art
[0002] Today, deep learning models continue to grow in size, often expressed in millions or billions of parameters. This trend stems from the amazing achievements of deep learning models in a wide range of application areas, such as natural language processing, computer vision, and reinforcement learning. However, this model size also leads to the demand for large amounts of data. Large models typically require millions of labeled samples for training in order to be able to capture complex patterns and features. However, in many real-world situations, the amount of data available is limited, which raises an important question: how to efficiently build and train large deep learning models when data is scarce.
[0003] This challenge has led to the development of model expansion techniques, a core idea of which is to start with a relatively small model and then gradually increase its size to be able to make full use of limited data. By gradually introducing more layers, neurons, and complexity, the model can gradually become more powerful while still being able to retain previously learned knowledge.
[0004] Knowledge distillation is a key component of model scaling. It allows the knowledge of one model to be transferred to another, usually from a small model to a large model. This involves transferring information such as activation values, loss functions, and model parameters from a small model to a large model to preserve existing knowledge. Summary of the invention
[0005] In order to better achieve low-cost acquisition of large models, an embodiment of the present invention provides a training method and device for large models for deep learning.
[0006] In a first aspect, an embodiment of the present invention provides a training method for a large model for deep learning, the method comprising:
[0007] Obtain labeled datasets for multiple scenarios;
[0008] Use each labeled data set to train an initial mapping model, and obtain the corresponding trained mapping model and the feature vector table of the corresponding scene;
[0009] Adding the plurality of labeled data sets to a training data set;
[0010] Acquire an initial target large model, and input the sample data in the training data set into the down-sampling part of the initial target large model to obtain a coding table;
[0011] Obtaining an embedded feature vector table according to the encoding table and the feature vector table corresponding to the scene of the sample data;
[0012] Inputting the embedded feature vector table into the up-sampled part of the initial target large model to obtain the target data result;
[0013] Adjusting the parameters of the initial target large model according to the target data result to obtain a trained target large model;
[0014] Repeat the above target large model training process until a target large model that meets the preset conditions is obtained.
[0015] In one or some optional implementations of the embodiment of the present application, obtaining an embedded feature vector table according to the feature vector table corresponding to the scene of the coding table and the sample data includes:
[0016] Expanding the encoding table, maintaining the feature vector dimension, and generating a plurality of first hidden vectors;
[0017] For each of the first hidden vectors, calculating a distance between the first hidden vector and the feature vector table;
[0018] Obtaining a weight vector corresponding to the first hidden vector according to a distance between the first hidden vector and the feature vector table;
[0019] Obtaining an embedded feature vector corresponding to the first hidden vector according to the feature vector table and the weight vector;
[0020] All the obtained embedded feature vectors are concatenated to obtain the embedded feature vector table.
[0021] In one or some optional implementations of the embodiment of the present application, obtaining the embedded feature vector corresponding to the first hidden vector according to the feature vector table and the weight vector includes:
[0022] Multiplying the feature vector table and the weight vector element by element to obtain an intermediate vector;
[0023] The intermediate vectors are summed to obtain an embedded feature vector corresponding to the first hidden vector.
[0024] In one or some optional implementations of the embodiment of the present application, obtaining an embedded feature vector table according to the feature vector table corresponding to the scene of the coding table and the sample data includes:
[0025] Cutting the encoding table into a plurality of first hidden vectors along a channel dimension;
[0026] For each of the first hidden vectors, calculating the distance between the first hidden vector and the feature vector table, and taking the feature vector with the shortest distance as the embedded feature vector corresponding to the first hidden vector;
[0027] All the embedded feature vectors are obtained to obtain the embedded feature vector table.
[0028] In one or some optional implementations of the embodiments of the present application, adjusting the parameters of the initial target large model according to the target data result to obtain the trained target large model includes:
[0029] According to the target data result, a hard target loss is obtained;
[0030] Inputting the sample data in the training data set into a trained mapping model of the scene corresponding to the sample data to obtain a probability distribution;
[0031] Obtaining a soft target loss according to the target data result and the probability distribution;
[0032] The parameters of the initial target large model are adjusted according to the hard target loss and the soft target loss to obtain a trained target large model.
[0033] In one or some optional implementations of the embodiments of the present application, the training data set is obtained by:
[0034] Obtaining new unlabeled data sets in the multiple scenarios;
[0035] Inputting the unlabeled data set into the trained mapping model of the corresponding scene to obtain a corresponding mapping result;
[0036] Using the mapping result as a label of the unlabeled data to obtain a pseudo-labeled data set;
[0037] The plurality of labeled data sets and pseudo-labeled data sets are combined to obtain the training data set.
[0038] In one or some optional implementations of the embodiment of the present application, the step of using the mapping result as a label of the unlabeled data to obtain a pseudo-labeled data set includes:
[0039] Constructing data pairs with part of the unlabeled data and the corresponding mapping results, and ranking the data pairs and the labeled data sets to obtain ranking results;
[0040] According to the ranking results, obtaining an evaluation score by normalization;
[0041] The data pairs with evaluation scores greater than a preset threshold are used as pseudo-annotated data sets;
[0042] Training a scoring model based on the data pairs or the data in the labeled data set and the corresponding evaluation scores;
[0043] Inputting a test data pair consisting of the unlabeled data that has not been ranked and the corresponding mapping result into a scoring model to obtain an evaluation score of the test data pair;
[0044] If the evaluation score of the data pair to be tested is greater than the preset threshold, the data pair to be tested is added to the pseudo-annotated data set.
[0045] In one or some optional implementations of the embodiments of the present application, using each labeled data set to train an initial mapping model respectively to obtain a corresponding trained mapping model and a feature vector table of the corresponding scene includes:
[0046] Use the labeled data sets of each scene to train an initial mapping model respectively, and obtain the trained mapping model corresponding to each labeled data set;
[0047] The multiple labeled data sets are respectively input into the down-sampling part of the trained mapping model corresponding to the scene, so as to obtain a feature vector table corresponding to the labeled data set of each scene.
[0048] In one or some optional implementations of the embodiments of the present application, further comprising:
[0049] Use labeled data sets of multiple different scenarios to train the initial mapping model to obtain a trained mapping model;
[0050] Using the trained mapping model, obtaining a compressed feature vector table corresponding to the labeled data sets of the multiple different scenes;
[0051] Using the compressed feature vector table, a training process of a target large model and / or a new large model is performed.
[0052] In one or some optional implementations of the embodiments of the present application, further comprising:
[0053] If repeating the above-mentioned target large model training process fails to meet the preset requirements, and it is artificially determined that the feature vector table is insufficient to meet the training requirements of the target large model, the step of obtaining the feature vector table corresponding to the labeled data set is re-executed, and the training process of the target large model is re-executed using the obtained new feature vector table.
[0054] In a second aspect, an embodiment of the present invention provides a training device for a large model for deep learning, the device comprising:
[0055] A first acquisition module is used to acquire labeled data sets in multiple scenarios;
[0056] A first training module is used to train an initial mapping model using each labeled data set to obtain a corresponding trained mapping model and a feature vector table of a corresponding scene;
[0057] A first merging module, used to add the plurality of labeled data sets to a training data set;
[0058] A first calculation module is used to obtain an initial target large model, and input the sample data in the training data set into the down-sampling part of the initial target large model to obtain a coding table; and input the sample data in the training data set into the down-sampling part of the trained target large model to obtain a new coding table;
[0059] A second calculation module, used for obtaining an embedded feature vector table according to the coding table and the feature vector table corresponding to the scene of the sample data;
[0060] A third computing module is used to input the embedded feature vector table into the up-sampled part of the initial target large model to obtain a target data result; and input the embedded feature vector table into the up-sampled part of the trained target large model to obtain a new target data result;
[0061] The second training module is used to adjust the parameters of the initial target large model according to the target data results to obtain the trained target large model; and adjust the parameters of the trained target large model according to the target data results to obtain a new trained target large model until the trained target large model meets the preset conditions.
[0062] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a large model for deep learning as described above.
[0063] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for a large model for deep learning as described above is implemented.
[0064] In a fifth aspect, an embodiment of the present invention provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the training method for a large model for deep learning as described above.
[0065] In a sixth aspect, an embodiment of the present invention provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the training method of a large model for deep learning as described above.
[0066] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:
[0067] The training method for a large model for deep learning provided in an embodiment of the present application obtains a feature vector table of a corresponding data set based on the mapping models corresponding to the trained multiple scenarios. In the training of the target model, the data of the training data set is input into the encoding table obtained by the downsampling part of the target model, and the feature vector table is matched and calculated to obtain an embedded feature vector table. The parameters of the target model are adjusted by the model output obtained by the upsampling part to perform training. The fusion step between the trained small mapping model and the large target model is performed in a low-dimensional space. The method of performing fusion in a low-dimensional feature space is more efficient in calculation, which can effectively reduce the computational cost of model training and reasoning, has higher data utilization, and can more effectively use limited training data to train the model, thereby achieving low-cost training to obtain a large target model.
[0068] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0069] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0071] Figure 1 It is a flowchart of a training method for a large model for deep learning provided in an embodiment of the present application;
[0072] Figure 2 It is a schematic diagram of a process of obtaining a feature vector table using an initial mapping model provided in an embodiment of the present application;
[0073] Figure 3 It is a schematic diagram of a process of obtaining a target data result using a target macro model provided in an embodiment of the present application;
[0074] Figure 4It is a schematic diagram of a flow chart for realizing calculation of an embedded feature vector table provided in an embodiment of the present application;
[0075] Figure 5 It is another schematic diagram of a flow chart for calculating an embedded feature vector table provided in an embodiment of the present application;
[0076] Figure 6 It is a structural diagram of a training device for a large model for deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION
[0077] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0078] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0079] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0080] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0081] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0082] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0083] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0084] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.
[0085] The inventors found that in the prior art, insufficient data would result in the inability to train a large model in a specific field application, and it would be impossible to rely on sufficient data for large-scale pre-training. Based on this, the inventors made the present invention through further research, providing a training method and device for a large model for deep learning.
[0086] Embodiment 1
[0087] The first embodiment of the present application provides a training method for a large model for deep learning, referring to Figure 1 As shown, the method includes:
[0088] S101: Obtain a plurality of labeled datasets of scenes.
[0089] In the embodiment of the present application, in the above step S101, it is first necessary to collect multiple data sets of different scenes, which cover multiple application scenarios of the target task, each data set contains sample data corresponding to each scene, and the data in each data set is labeled to obtain labeled data sets corresponding to multiple scenes. Necessary data cleaning and preprocessing are performed on each data set to ensure the quality and consistency of the data, including preprocessing operations such as data denoising and standardization. The target task can be image classification, language generation, semantic generation, natural language processing, reinforcement learning tasks, and multimedia related tasks. For example, if the target task is to achieve road image annotation, the purpose is to map the image area on the road to natural language or other semantic information, then the corresponding multiple application scenarios are multiple road scenes, and the multiple data sets are the images of each road scene and the corresponding annotated semantic information.
[0090] S102: Use each labeled data set to train an initial mapping model respectively, and obtain a corresponding trained mapping model and a feature vector table of the corresponding scene.
[0091] In the embodiment of the present application, in step S102, an initial mapping model is trained using each labeled data set to obtain a corresponding trained mapping model and a feature vector table of the corresponding scene, specifically including:
[0092] An initial mapping model is trained using the labeled data sets of each scene to obtain the trained mapping models corresponding to the labeled data sets; the multiple labeled data sets are respectively input into the down-sampling part of the trained mapping model of the corresponding scene to obtain the feature vector table corresponding to the labeled data set of each scene.
[0093] The steps of training the initial mapping model specifically include: first, according to the specific task, select a suitable deep learning model structure, such as U-Net network, ResNet, etc. The mapping model should have a sampling part and an upsampling part. The sampling part is responsible for reducing the feature dimension, capturing the global features and structure in the data, and obtaining a low-dimensional feature representation. The upsampling part is responsible for expanding the low-dimensional feature representation to the dimension of the output data; then, each obtained data set is divided into a training set, a validation set, and a test set; secondly, according to the specific task, select a suitable loss function and optimizer, perform conventional supervised learning training, and use algorithms such as gradient descent and back propagation to iteratively update the mapping model weights; use indicators such as accuracy and recall rate to evaluate the performance of the mapping model, and ensure that the mapping model can perform well in the corresponding scenario, and then obtain a trained mapping model. In the same way, use the labeled data sets of each scenario to train multiple mapping models separately to obtain multiple trained mapping models. Among them, the mapping models corresponding to different scenarios do not need to have the same model structure.
[0094] After obtaining the trained mapping models corresponding to each scene, multiple labeled data sets are input into the down-sampled part of the trained mapping model of the corresponding scene to obtain the feature vector table corresponding to the labeled data set of each scene. The feature vector table is the low-dimensional feature representation of the labeled data set of each scene in the trained mapping model of the corresponding scene. For example, refer to Figure 2As shown in the figure, the target task in the example above is to realize road image annotation, the purpose is to map the image area on the road to natural language or other semantic information. There is a data set of N scenes, the image size in the data set is H*W, the corresponding annotation is an image with semantic information, the image size is also H*W, and the input of the mapping model is C*H*W data, where C represents the channel (Channel). In the image data, C is generally equal to 3, representing the three color channels of RGB. After the input data of the mapping model passes through the downsampling part, the corresponding feature vector table is obtained, and the vector shape is H*W*C. The vector is reduced in dimension and spliced, and the three-dimensional vector of shape H*W*C is converted into a two-dimensional vector of shape (H*W)*C. The two-dimensional vector is a feature vector table corresponding to the labeled data set of a scene.
[0095] S103: Add the plurality of labeled data sets to a training data set.
[0096] In the embodiment of the present application, in the above step S103, adding the plurality of labeled data sets to the training data set specifically includes: obtaining new unlabeled data sets in a plurality of scenarios;
[0097] Input the unlabeled data set into the trained mapping model of the corresponding scene to obtain the corresponding mapping result;
[0098] The mapping results are used as labels for unlabeled data to obtain a pseudo-labeled dataset;
[0099] Multiple labeled datasets and pseudo-labeled datasets are merged to obtain a training dataset.
[0100] The mapping results are used as labels for unlabeled data to obtain pseudo-labeled datasets, including:
[0101] Construct data pairs with some unlabeled data and corresponding mapping results, and rank the data pairs and labeled data sets to obtain ranking results;
[0102] According to the ranking results, the evaluation score is obtained by normalization;
[0103] The data pairs with evaluation scores greater than the preset threshold are used as pseudo-annotated data sets;
[0104] A scoring model is trained based on the data pairs or the data in the labeled data set and the corresponding evaluation scores;
[0105] Input the unranked unlabeled data and the corresponding mapping results into the scoring model to obtain the evaluation score of the test data pair;
[0106] If the evaluation score of the data pair to be tested is greater than a preset threshold, the data pair to be tested is added to the pseudo-annotated data set.
[0107] In the embodiment of the present application, new unlabeled data sets under multiple scenarios are obtained, and the unlabeled data sets are input into the trained mapping model of the corresponding scenario to obtain the corresponding mapping results, and the unlabeled data and the corresponding mapping results are used to construct data pairs, that is, the corresponding mapping results are used as the labels of the unlabeled data to form pseudo-labeled data. However, there are errors in the accuracy of the mapping model. In order to try to eliminate this inevitable error, it is necessary to screen the mapping results generated by the mapping model and retain the data pairs with better quality.
[0108] The specific steps to obtain and filter the pseudo-label dataset include:
[0109] Ranking some data pairs with the labeled data to obtain the ranking results of the data pairs and the labeled data, obtaining evaluation scores through normalization according to the ranking results, and taking the data pairs with evaluation scores greater than a preset threshold as unlabeled data sets, wherein the ranking operation is to manually rank the data pairs according to the mapping results or the accuracy or quality of the annotations in the data pairs;
[0110] Train a scoring model, the training data set of which is a data pair or labeled data and the corresponding evaluation score. For example, the scoring model can use a convolutional neural network, the structure of which includes a convolution layer, a pooling layer, a batch normalization layer, and a fully connected layer. The specific training steps of the scoring model are: input the data pair or labeled data into the scoring model, obtain the scoring score predicted by the scoring model, compare the scoring score with the evaluation score corresponding to the input data pair or labeled data, calculate the loss function value, use the back propagation algorithm to calculate the gradient of the loss function with respect to the network parameters, use the optimization algorithm (such as the stochastic gradient descent algorithm SGD) to update the network parameters to reduce the value of the loss function, repeat the above steps, iterate multiple times, until the loss function converges or reaches the preset training round, or the accuracy reaches the preset threshold, save the trained model, and obtain the trained scoring model.
[0111] After the training is completed, the unlabeled data and the corresponding mapping results are input into the trained evaluation model, and the evaluation score is output. If the evaluation score of the unlabeled data and the corresponding mapping result is greater than the preset threshold, the unlabeled data and the corresponding mapping result are added to the pseudo-label data set.
[0112] The preset threshold is manually set, and the evaluation score has a value range of 0 to 1 after normalization. In a specific embodiment, the preset threshold can be set to 0.7.
[0113] After obtaining the pseudo-labeled dataset, the labeled dataset and the pseudo-labeled dataset are merged to obtain the training dataset. It should also be noted that new labeled data can also be added to the training dataset.
[0114] In the embodiment of the present application, considering the huge workload of labeling data sets in many application scenarios, the trained mapping model is used to perform the task of automatic data labeling. This method can not only greatly reduce the workload of labelers, but also increase the amount of data available for subsequent training of large models. In this way, the training success rate of large models is improved, and more data helps the model better understand the task and improve performance.
[0115] S104: Obtain an initial target large model, and input the sample data in the training data set into the down-sampling part of the initial target large model to obtain a coding table.
[0116] S105: Obtain an embedded feature vector table according to the coding table and the feature vector table corresponding to the scene of the sample data.
[0117] S106: Input the embedded feature vector table into the up-sampling part of the initial target large model to obtain the target data result.
[0118] In the embodiment of the present application, a suitable deep learning target large model structure is selected according to the specific task, and the target large model should have upsampling and downsampling parts. The training data set of the target large model is the training data set obtained in step S103. Figure 3 As shown, the data calculation sequence of the target large model is: input the sample data in the training data set into the down-sampling part of the target large model to obtain a coding table; calculate the embedded feature vector table according to the coding table and the feature vector table obtained in step S102; input the embedded feature vector table into the up-sampling part of the target large model to obtain the target data result. Among them, the coding table and the embedded feature vector table are both low-dimensional feature representations of the input data.
[0119] In the embodiment of the present application, the process of calculating the embedded feature vector table according to the coding table and the feature vector table includes two methods. The first method includes:
[0120] Expand the coding table, keep the feature vector dimension, and generate multiple first hidden vectors;
[0121] For each first hidden vector, calculating the distance between the first hidden vector and the feature vector table;
[0122] Obtaining a weight vector corresponding to the first hidden vector according to the distance between the first hidden vector and the feature vector table;
[0123] Multiply the feature vector table and the weight vector element by element to obtain the intermediate vector;
[0124] The intermediate vectors are summed to obtain the embedded feature vector corresponding to the first hidden vector.
[0125] All the obtained embedded feature vectors are concatenated to obtain an embedded feature vector table.
[0126] In the embodiment of the present application, after obtaining the coding table corresponding to the sample data of the input target large model, the vector is reduced in dimension and spliced, and the vector is cut horizontally while maintaining the feature vector dimension to obtain multiple first hidden vectors, and calculations are performed for each first hidden vector respectively, and the calculation steps include: calculating the distance between the first hidden vector and the feature vector table; obtaining the weight vector through the softmax operation of the distance vector, as shown in Formula 1; multiplying the weight vector and the feature vector table correspondingly to obtain the intermediate vector, and then summing along the C dimension to obtain the embedded feature vector of the first hidden vector, as shown in Formula 2. Among them, the method of calculating the distance between the first hidden vector and the feature vector table includes, but is not limited to, calculation based on the Euclidean distance formula, but without taking the square root, saving some computing power, and different similarity measurement methods such as COS distance can also be used, which is not specifically limited in this method. Repeat the operation of calculating the embedded feature vector, splice all embedded feature vectors, and obtain the embedded feature vector table.
[0127] Alpha = 1-softmax (distance vector) Formula 1;
[0128] x′=Sum(Alpha*Eigenvector Table) Formula 2;
[0129] Among them, Alpha is the weight information, and x′ is the embedded feature vector corresponding to the first hidden vector x.
[0130] In a specific embodiment, taking the target task of realizing road image annotation as an example, there is a dataset of N scenes, the image size in the dataset is H*W, and the corresponding annotation is an image with semantic information, and the image size is also H*W. Figure 4As shown in the figure, the vector shape of the sample data of the input target large model is C*H*W. After the data passes through the downsampling part, the corresponding encoding table is obtained, and its vector shape is H*W*C. Then, the vector is reduced in dimension and concatenated, that is, the three-dimensional vector with the shape of H*W*C is converted into a two-dimensional vector with the shape of (H*W)*C. Then, the two-dimensional vector with the shape of (H*W)*C is horizontally cut while maintaining the feature vector dimension to obtain (H*W) one-dimensional vectors with the shape of 1*1*C, which is the first hidden vector. Calculation is performed for each first hidden vector, and the calculation steps include: calculating the distance between the first hidden vector (with a shape of 1*1*C) and the feature vector table (with a shape of (H*W)*C), the distance formula is similar to the Euclidean distance formula, but the square root is not taken, and the distance vector obtained is a one-dimensional vector with a shape of (H*W)*1; the distance vector is obtained by softmax operation to obtain a weight vector, as shown in Formula 1, and the weight vector is also a one-dimensional vector with a shape of (H*W)*1; the weight vector (with a shape of (H*W)*1) and the feature vector table (with a shape of (H*W)*C) are multiplied correspondingly to obtain an intermediate vector (with a shape of (H*W)*C), and then summed along the C dimension to obtain the embedded feature vector of the first hidden vector, as shown in Formula 2, and the shape of the embedded feature vector is 1*1*C. Repeat the operation of calculating the embedded feature vector, splice all embedded feature vectors, obtain a vector with a shape of (H*W)*C, and reshape to obtain an embedded feature vector table with a shape of H*W*C.
[0131] In the embodiment of the present application, continuous calculation is implemented to provide better interpretability, avoid the appearance of discrete values in the model, make the model smoother when processing input data, and thus make the training of the neural network more stable.
[0132] In the embodiment of the present application, the process of calculating the embedded feature vector table according to the coding table and the feature vector table includes two methods, and the second method includes:
[0133] Cutting the encoding table into a plurality of first hidden vectors along a channel dimension;
[0134] For each of the first hidden vectors, calculate the distance between the first hidden vector and the feature vector table, and use the feature vector with the shortest distance as the embedded feature vector corresponding to the first hidden vector;
[0135] All the embedded feature vectors are obtained to obtain the embedded feature vector table.
[0136] In the embodiment of the present application, after obtaining the coding table corresponding to the sample data of the input target large model, the vector is reduced in dimension and spliced to obtain multiple first hidden vectors, and calculations are performed for each first hidden vector, and the calculation steps include: calculating the distance between the first hidden vector and all feature vectors in the feature vector table; taking the feature vector closest to the current first hidden vector as the embedded feature vector corresponding to the current first hidden vector. Repeat the operation of calculating the embedded feature vector, splice all the embedded feature vectors, and obtain the embedded feature vector table.
[0137] In a specific embodiment, a schematic diagram of calculating the embedded feature vector table using the second method is shown in FIG. Figure 5 As shown in the figure, the black box on the left represents the sample data x in the training data set of the input model, and the encoding table z is obtained through the encoder Encoder, that is, the downsampling part e (x), the schematic diagram shows three first hidden vectors, one of which has a shape of 1*1*C, where C=4. Figure 5 Below is the feature vector table, which contains 4 feature vectors, and they are encoded in order. The first feature vector is encoded as 0, the second feature vector is encoded as 1, and so on. Then the calculation steps are performed: calculate the distance between the first hidden vector and the four feature vectors respectively. At this time, the Euclidean distance formula can be used directly to calculate that the first hidden vector is closest to the feature vector encoded as 0, record the code 0, and then repeat the operation of calculating the distance between the first hidden vector and the feature vector and recording the code of the nearest feature vector, and get the following Figure 5 The temporary record table in the lower right middle is used to concatenate the corresponding feature vector table to obtain the feature embedding feature vector table z q (x).
[0138] In the embodiment of the present application, since the second method of obtaining the embedded feature vector table will convert the embedded feature vector table into discrete values, back propagation cannot be achieved in subsequent training. In specific implementation, technologies such as STE (Straight-Through Estimator) can be used to process discrete values, so as to use continuous value estimation in back propagation to approximate the gradient for training.
[0139] In an embodiment of the present application, after obtaining an embedded feature vector table from a coding table by the first method or the second method, the embedded feature vector table is then input into an upsampling part of a target large model, and a target data result is obtained as an output.
[0140] S107: Adjust the parameters of the initial target large model according to the target data result to obtain a trained target large model.
[0141] S108: Repeat the target large model training process from step S104 to step S107 above until a target large model that meets the preset conditions is obtained.
[0142] In the embodiment of the present application, in the above step S107, the parameters of the initial target large model are adjusted according to the target data result to obtain the trained target large model, which specifically includes:
[0143] According to the target data results, the hard target loss is obtained;
[0144] Input the sample data in the training data set into the trained mapping model of the scene corresponding to the sample data to obtain the probability distribution;
[0145] According to the target data results and probability distribution, the soft target loss is obtained;
[0146] The parameters of the initial target large model are adjusted according to the hard target loss and the soft target loss to obtain the trained target large model.
[0147] In the embodiment of the present application, in order to speed up the training of the target large model, the knowledge distillation technology can be added to assist the training of the target large model. The trained mapping model will be used as the teacher model in the knowledge distillation, and the target large model will be used as the student model in the knowledge distillation.
[0148] Specifically, in the first step, after the target large model obtains the target data result corresponding to the sample data, the hard target loss is calculated according to the target data result and the corresponding annotation. The setting of the loss function can select the cross entropy loss function or other loss functions that meet the needs of the target task;
[0149] In the second step, the sample data of the input target large model is also input into the corresponding trained mapping model to obtain the probability distribution obtained by the teacher model. The probability distribution includes category probability distribution or feature representation, etc. If the target task is a classification task, the probability distribution is the category probability distribution. If the target task is a mapping task, the probability distribution is the feature representation. According to the target data results and the probability distribution, the soft target loss is calculated, that is, the difference between the output of the teacher model and the output of the student model is calculated.
[0150] In a specific embodiment, the temperature parameter can be used to scale the outputs of the two models and calculate the cross entropy loss; the total target loss of the target large model is the weighted sum of the hard target loss and the soft target loss, and the gradient of the target loss is transmitted back to the target large model through the back propagation algorithm, thereby updating the parameters of the target large model to obtain the trained target large model.
[0151] In the embodiment of the present application, the parameter updating process is iteratively performed until the preset number of training iterations is met or the loss function converges to the minimum value. Through this training process, the parameters of the target large model are gradually optimized so that it can more accurately learn the capabilities of multiple trained mapping models. Finally, after multiple iterative trainings, the target large model that meets the preset conditions is obtained.
[0152] In a specific embodiment, if the amount of data is large, the feature vector table can be compressed, and multiple feature vector tables can be compressed into one feature vector table. Specifically, it can be that the initial mapping model is trained using labeled data sets of multiple different scenes to obtain a trained mapping model; the trained mapping model is used to obtain a compressed feature vector table corresponding to the labeled data sets of the multiple different scenes; and the compressed feature vector table is used to perform the training process of the target large model and / or the new large model.
[0153] In a specific embodiment, if it is found that the loss function does not converge or converges incompletely during the repeated training process of the above-mentioned target large model, and it is artificially determined that the feature vector table is insufficient to meet the training requirements of the target large model, then step S102 can be re-executed to obtain the feature vector table corresponding to the labeled data set, and the training process of the target large model can be re-executed using the obtained new feature vector table until the desired target large model is obtained.
[0154] In an embodiment of the present application, the fusion step between the trained small mapping model and the large target model is performed in a low-dimensional space. The method of fusion in a low-dimensional feature space makes the calculation more efficient, effectively reduces the computational cost of model training and reasoning, and the data utilization rate is higher. The core purpose of the method is to develop a powerful and universal large model that is gradually generated through iterative generation and training in the absence of data. For example, after obtaining the target large model, the target large model can also be used to assist in generating a second target large model according to the above steps. This method can more effectively use limited training data to train the model, and achieve low data cost and low equipment cost training to obtain the target large model.
[0155] Embodiment 2
[0156] Based on the same inventive concept, the embodiment of the present invention also provides a training device for a large model for deep learning, referring to Figure 6 As shown, the device comprises:
[0157] A first acquisition module 101 is used to acquire labeled data sets in multiple scenarios;
[0158] A first training module 102 is used to train an initial mapping model using each labeled data set to obtain a corresponding trained mapping model and a feature vector table of a corresponding scene;
[0159] A first merging module 103, configured to add the plurality of labeled data sets to a training data set;
[0160] The first calculation module 104 is used to obtain an initial target large model, and input the sample data in the training data set into the down-sampled part of the initial target large model to obtain a coding table; and input the sample data in the training data set into the down-sampled part of the trained target large model to obtain a new coding table;
[0161] A second calculation module 105 is used to obtain an embedded feature vector table according to the coding table and the feature vector table corresponding to the scene of the sample data;
[0162] The third calculation module 106 is used to input the embedded feature vector table into the up-sampled part of the initial target large model to obtain the target data result; and input the embedded feature vector table into the up-sampled part of the trained target large model to obtain a new target data result;
[0163] The second training module 107 is used to adjust the parameters of the initial target large model according to the target data results to obtain a trained target large model; and adjust the parameters of the trained target large model according to the target data results to obtain a new trained target large model until the trained target large model meets the preset conditions.
[0164] Embodiment 3
[0165] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the training method of a large model for deep learning as described in the above embodiment 1 is implemented.
[0166] Embodiment 4
[0167] Based on the same inventive concept, an embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for a large model for deep learning as described in the above-mentioned embodiment 1 is implemented.
[0168] Embodiment 5
[0169] Based on the same inventive concept, an embodiment of the present invention also provides a computer program product comprising instructions. When the computer program product is run on a computer device, the computer device executes the training method for a large model for deep learning as described in the above-mentioned embodiment 1.
[0170] Embodiment 6
[0171] Based on the same inventive concept, an embodiment of the present invention also provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the training method of a large model for deep learning as described in the above embodiment one.
[0172] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0176] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A training method for a large model for deep learning, characterized in that: include: Obtain labeled datasets for multiple scenarios; Use each labeled data set to train an initial mapping model, and obtain the corresponding trained mapping model and the feature vector table of the corresponding scene; Adding the plurality of labeled data sets to a training data set; Acquire an initial target large model, and input the sample data in the training data set into the down-sampling part of the initial target large model to obtain a coding table; Obtaining an embedded feature vector table according to the encoding table and the feature vector table corresponding to the scene of the sample data; Inputting the embedded feature vector table into the up-sampled part of the initial target large model to obtain the target data result; Adjusting the parameters of the initial target large model according to the target data result to obtain a trained target large model; Repeat the above target large model training process until a target large model that meets the preset conditions is obtained.
2. The method according to claim 1, characterized in that The step of obtaining an embedded feature vector table according to the feature vector table corresponding to the scene of the sample data and the encoding table includes: Expanding the encoding table, maintaining the feature vector dimension, and generating a plurality of first hidden vectors; For each of the first hidden vectors, calculating a distance between the first hidden vector and the feature vector table; Obtaining a weight vector corresponding to the first hidden vector according to a distance between the first hidden vector and the feature vector table; Obtaining an embedded feature vector corresponding to the first hidden vector according to the feature vector table and the weight vector; All the obtained embedded feature vectors are concatenated to obtain the embedded feature vector table.
3. The method according to claim 2, characterized in that The step of obtaining, according to the feature vector table and the weight vector, an embedded feature vector corresponding to the first hidden vector comprises: Multiplying the feature vector table and the weight vector element by element to obtain an intermediate vector; The intermediate vectors are summed to obtain an embedded feature vector corresponding to the first hidden vector.
4. The method according to claim 1, characterized in that The step of obtaining an embedded feature vector table according to the feature vector table corresponding to the scene of the sample data and the encoding table includes: Cutting the encoding table into a plurality of first hidden vectors along a channel dimension; For each of the first hidden vectors, calculating the distance between the first hidden vector and the feature vector table, and taking the feature vector with the shortest distance as the embedded feature vector corresponding to the first hidden vector; All the embedded feature vectors are obtained to obtain the embedded feature vector table.
5. The method according to claim 1, characterized in that The step of adjusting the parameters of the initial target large model according to the target data result to obtain the trained target large model includes: According to the target data result, a hard target loss is obtained; Inputting the sample data in the training data set into a trained mapping model of the scene corresponding to the sample data to obtain a probability distribution; Obtaining a soft target loss according to the target data result and the probability distribution; The parameters of the initial target large model are adjusted according to the hard target loss and the soft target loss to obtain a trained target large model.
6. The method according to claim 1, characterized in that The training data set is obtained by: Obtaining new unlabeled data sets in the multiple scenarios; Inputting the unlabeled data set into the trained mapping model of the corresponding scene to obtain a corresponding mapping result; Using the mapping result as a label of the unlabeled data to obtain a pseudo-labeled data set; The plurality of labeled data sets and pseudo-labeled data sets are combined to obtain the training data set.
7. The method according to claim 6, characterized in that The step of using the mapping result as a label of the unlabeled data to obtain a pseudo-labeled data set includes: Constructing data pairs with part of the unlabeled data and the corresponding mapping results, and ranking the data pairs and the labeled data sets to obtain ranking results; According to the ranking results, obtaining an evaluation score by normalization; The data pairs with evaluation scores greater than a preset threshold are used as pseudo-annotated data sets; Training a scoring model based on the data pairs or the data in the labeled data set and the corresponding evaluation scores; Inputting a test data pair consisting of the unlabeled data that has not been ranked and the corresponding mapping result into a scoring model to obtain an evaluation score of the test data pair; If the evaluation score of the data pair to be tested is greater than the preset threshold, the data pair to be tested is added to the pseudo-annotated data set.
8. The method according to claim 1, characterized in that The method of using each labeled data set to train an initial mapping model to obtain a corresponding trained mapping model and a feature vector table of the corresponding scene includes: Use the labeled data sets of each scene to train an initial mapping model respectively, and obtain the trained mapping model corresponding to each labeled data set; The multiple labeled data sets are respectively input into the down-sampling part of the trained mapping model corresponding to the scene, so as to obtain a feature vector table corresponding to the labeled data set of each scene.
9. The method according to claim 1, characterized in that Also includes: Use labeled data sets of multiple different scenarios to train the initial mapping model to obtain a trained mapping model; Using the trained mapping model, obtaining a compressed feature vector table corresponding to the labeled data sets of the multiple different scenes; Using the compressed feature vector table, a training process of a target large model and / or a new large model is performed.
10. The method according to claim 1, characterized in that Also includes: If repeating the above-mentioned target large model training process fails to meet the preset requirements, and it is artificially determined that the feature vector table is insufficient to meet the training requirements of the target large model, the step of obtaining the feature vector table corresponding to the labeled data set is re-executed, and the training process of the target large model is re-executed using the obtained new feature vector table.
11. A training device for a large model for deep learning, characterized in that: include: A first acquisition module is used to acquire labeled data sets in multiple scenarios; A first training module is used to train an initial mapping model using each labeled data set to obtain a corresponding trained mapping model and a feature vector table of a corresponding scene; A first merging module, used to add the plurality of labeled data sets to a training data set; A first calculation module is used to obtain an initial target large model, and input the sample data in the training data set into the down-sampling part of the initial target large model to obtain a coding table; And, inputting the sample data in the training data set into the down-sampling part of the trained target large model to obtain a new encoding table; A second calculation module, used for obtaining an embedded feature vector table according to the coding table and the feature vector table corresponding to the scene of the sample data; A third computing module is used to input the embedded feature vector table into the up-sampled part of the initial target large model to obtain a target data result; and input the embedded feature vector table into the up-sampled part of the trained target large model to obtain a new target data result; A second training module is used to adjust the parameters of the initial target large model according to the target data result to obtain a trained target large model; And, adjusting the parameters of the trained target large model according to the target data result to obtain a new trained target large model until the trained target large model meets the preset conditions.
12. A computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal, the terminal executes the training method for a large model for deep learning as described in any one of claims 1 to 10.
13. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for a large model for deep learning as described in any one of claims 1 to 10 is implemented.
14. A computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the training method for a large model for deep learning as described in any one of claims 1 to 10.
15. A chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the training method for a large model for deep learning as described in any one of claims 1-10.