Data generation, model distillation method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides a data generation, model distillation method, device, electronic device and storage medium, which relates to the field of computer technology. The method includes: obtaining first training data, and obtaining a first semantic description set corresponding to a first group, the first semantic description set including multiple semantic descriptions corresponding to the training data in the first group; selecting multiple first semantic descriptions from the first semantic description set; determining data generation conditions based on the first training data, the identifier of the first group and the multiple first semantic descriptions; inputting the data generation conditions into the generation model to obtain second training data. The method can obtain diversified data at a faster speed and lower cost, and solve the problems of overfitting and poor generalization in student model distillation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data generation, model distillation method, device, electronic device and storage medium. Background Art

[0002] In actual applications, there are large differences between data collection scenarios and model application scenarios. Due to the high cost of training data collection and calibration, only a small amount of training data can be collected for some scenarios. In the case of a small amount of training data for some scenarios, large models are widely used to ensure the generalization performance of the model in downstream model applications.

[0003] However, large models require huge computing resources and bandwidth, which makes the cost of applying large models in actual business scenarios too high. In order to reduce the cost of application in actual business scenarios, one of the important ways to apply large models is to transfer the knowledge of large models (i.e., teacher models) to small models (i.e., student models) through model distillation technology, and use the capabilities of large models to guide the performance improvement of small models.

[0004] However, model distillation on a small amount of training data for some scenarios will cause the student model to overfit the training samples and the characteristics of the teacher model, resulting in poor generalization of heterogeneous scenarios. In this case, there is an urgent need for a method to obtain sufficient and diverse data at a faster speed and lower cost to assist in knowledge transfer in model distillation. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data generation and model distillation method, device, electronic device and storage medium to obtain diversified data at a faster speed and lower cost, and solve the problems of overfitting and poor generalization in student model distillation. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a data generation method, the method comprising:

[0007] Acquire first training data, and acquire a first semantic description set corresponding to the first group, wherein the first semantic description set includes a plurality of semantic descriptions corresponding to the training data in the first group;

[0008] Selecting a plurality of first semantic descriptions from the first semantic description set;

[0009] Determining a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions;

[0010] The data generation conditions are input into a generation model to obtain second training data.

[0011] In some embodiments, the step of obtaining a first semantic description set corresponding to the first group includes:

[0012] Using a large multimodal model, semantic analysis is performed on multiple training data to obtain semantic understanding information for each training data.

[0013] According to the category marked by each training data, the plurality of training data are divided into a plurality of groups, each group including at least one category of training data;

[0014] A first semantic description set corresponding to the first group is determined according to the semantic understanding information of each training data included in each group.

[0015] In some embodiments, the step of determining the first semantic description set corresponding to the first group according to the semantic understanding information of each training data included in each group includes:

[0016] Extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtaining an initial semantic description set corresponding to each group;

[0017] Determine the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtain a first semantic description subset corresponding to the first group;

[0018] Determine, from the initial semantic description set corresponding to the first group, semantic descriptions not included in the initial semantic description sets corresponding to other groups, to obtain a second semantic description subset corresponding to the first group;

[0019] Determine, from the initial semantic description sets corresponding to the other groups, semantic descriptions not included in the initial semantic description set corresponding to the first group, to obtain a third semantic description subset corresponding to the first group;

[0020] The first semantic description subset, the second semantic description subset and the third semantic description subset constitute a first semantic description set corresponding to the first group.

[0021] In some embodiments, the step of extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group to obtain an initial semantic description set corresponding to each group includes:

[0022] Extracting a semantic description of each training data included in each group from the semantic understanding information of each training data included in each group;

[0023] From the semantic descriptions of all training data included in each group, multiple semantic descriptions with the highest occurrence frequency are selected to obtain an initial semantic description set corresponding to each group.

[0024] In some embodiments, the step of using a multimodal large model to perform semantic analysis on multiple training data respectively to obtain semantic understanding information of each training data includes:

[0025] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; or,

[0026] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; and perform local semantic analysis on at least one local area of ​​each training data to obtain second semantic understanding information of each training data.

[0027] In some embodiments, the first set of semantic descriptions includes a plurality of subsets of semantic descriptions;

[0028] The step of selecting a plurality of first semantic descriptions from the first semantic description set comprises:

[0029] A plurality of first semantic descriptions are selected from each semantic description subset.

[0030] In some embodiments, the method further comprises:

[0031] Receiving multiple semantic descriptions input by a user to obtain a second semantic description set;

[0032] Selecting a plurality of second semantic descriptions from the second semantic description set;

[0033] The step of determining data generation conditions according to the first training data, the identifier of the first group and the plurality of first semantic descriptions comprises:

[0034] The first training data, the identifier of the first group, the plurality of first semantic descriptions, and the plurality of second semantic descriptions are combined to obtain a data generation condition.

[0035] In some embodiments, before inputting the data generation condition into the generation model, the method further includes:

[0036] Obtain target hyperparameters of the generative model;

[0037] The inference hyperparameters of the generative model are adjusted to the target hyperparameters.

[0038] In some embodiments, the semantic description includes one or more of an entity name, a scene name, an entity description phrase, a scene description phrase, and an entity relationship description phrase.

[0039] In a second aspect, an embodiment of the present application provides a model distillation method, the method comprising:

[0040] Acquire an original data set, wherein the original data set includes a plurality of original training data;

[0041] Using any of the methods provided in the first aspect, a plurality of new training data are generated according to the plurality of original training data to obtain a generated data set;

[0042] The original data set and the generated data set are used to assist the large model in performing knowledge distillation processing to obtain a target small model.

[0043] In a third aspect, an embodiment of the present application provides a data generating device, the device comprising:

[0044] An acquisition module, configured to acquire first training data and acquire a first semantic description set corresponding to a first group, wherein the first semantic description set includes a plurality of semantic descriptions corresponding to the training data in the first group;

[0045] A first selection module, configured to select a plurality of first semantic descriptions from the first semantic description set;

[0046] a determination module, configured to determine a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions;

[0047] An input module is used to input the data generation conditions into the generation model to obtain the second training data.

[0048] In some embodiments, the acquisition module is specifically used to:

[0049] Using a large multimodal model, semantic analysis is performed on multiple training data to obtain semantic understanding information for each training data.

[0050] According to the category marked by each training data, the plurality of training data are divided into a plurality of groups, each group including at least one category of training data;

[0051] A first semantic description set corresponding to the first group is determined according to the semantic understanding information of each training data included in each group.

[0052] In some embodiments, the acquisition module is specifically used to:

[0053] Extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtaining an initial semantic description set corresponding to each group;

[0054] Determine the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtain a first semantic description subset corresponding to the first group;

[0055] Determine, from the initial semantic description set corresponding to the first group, semantic descriptions not included in the initial semantic description sets corresponding to other groups, to obtain a second semantic description subset corresponding to the first group;

[0056] Determine, from the initial semantic description sets corresponding to the other groups, semantic descriptions not included in the initial semantic description set corresponding to the first group, to obtain a third semantic description subset corresponding to the first group;

[0057] The first semantic description subset, the second semantic description subset and the third semantic description subset constitute a first semantic description set corresponding to the first group.

[0058] In some embodiments, the acquisition module is specifically used to:

[0059] Extracting a semantic description of each training data included in each group from the semantic understanding information of each training data included in each group;

[0060] From the semantic descriptions of all training data included in each group, multiple semantic descriptions with the highest occurrence frequency are selected to obtain an initial semantic description set corresponding to each group.

[0061] In some embodiments, the acquisition module is specifically used to:

[0062] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; or,

[0063] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; and perform local semantic analysis on at least one local area of ​​each training data to obtain second semantic understanding information of each training data.

[0064] In some embodiments, the first set of semantic descriptions includes a plurality of subsets of semantic descriptions;

[0065] The first selection module is specifically configured to select a plurality of first semantic descriptions from each semantic description subset.

[0066] In some embodiments, the apparatus further comprises:

[0067] A receiving module, configured to receive multiple semantic descriptions input by a user and obtain a second semantic description set;

[0068] A second selection module, configured to select a plurality of second semantic descriptions from the second semantic description set;

[0069] The determination module is specifically used to combine the first training data, the identifier of the first group, the multiple first semantic descriptions and the multiple second semantic descriptions to obtain data generation conditions.

[0070] In some embodiments, the apparatus further comprises:

[0071] The adjustment module is used to obtain the target hyperparameters of the generation model before the data generation conditions are input into the generation model; and adjust the inference hyperparameters of the generation model to the target hyperparameters.

[0072] In some embodiments, the semantic description includes one or more of an entity name, a scene name, an entity description phrase, a scene description phrase, and an entity relationship description phrase.

[0073] In a fourth aspect, an embodiment of the present application provides a model distillation device, the device comprising:

[0074] An acquisition module, used for acquiring an original data set, wherein the original data set includes a plurality of original training data;

[0075] A generating module, configured to generate a plurality of new training data according to the plurality of original training data by using any one of the devices provided in the third aspect, so as to obtain a generated data set;

[0076] The distillation module is used to use the original data set and the generated data set to assist the large model in performing knowledge distillation processing to obtain a target small model.

[0077] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0078] Memory, used to store computer programs;

[0079] The processor is used to implement any method provided in the first aspect or the second aspect when executing the program stored in the memory.

[0080] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method provided in the first aspect or the second aspect is implemented.

[0081] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods provided in the first aspect or the second aspect.

[0082] Beneficial effects of the embodiments of the present application:

[0083] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, and the diversity of training data can be expanded at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0084] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0086] Figure 1 A schematic diagram of the federal structure;

[0087] Figure 2 A first schematic diagram of a data generation method provided in an embodiment of the present application;

[0088] Figure 3 A second schematic diagram of the data generation method provided in an embodiment of the present application;

[0089] Figure 4a A first schematic diagram of an image provided in an embodiment of the present application;

[0090] Figure 4b A second schematic diagram of an image provided in an embodiment of the present application;

[0091] Figure 5 A schematic diagram of step S304 provided in an embodiment of the present application;

[0092] Figure 6 A schematic diagram of a model distillation method provided in an embodiment of the present application;

[0093] Figure 7 A schematic diagram of the knowledge distillation process provided in the embodiment of the present application;

[0094] Figure 8 A schematic diagram of a data processing flow provided in an embodiment of the present application;

[0095] Fig. 9 A schematic diagram of a data generation device provided in an embodiment of the present application;

[0096] Fig.10 A schematic diagram of a model distillation apparatus provided in an embodiment of the present application;

[0097] Fig.11 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0099] The following is an explanation of the terms that appear in the embodiments of the present application.

[0100] Model distillation: Through a specially designed loss function, the student model imitates the output features of certain network layers in the teacher model, and transfers the teacher model's knowledge of the training samples to the student model to improve the performance of the student model. Generally speaking, the performance of the teacher model is significantly better than that of the student model.

[0101] Large model: A deep neural network model with a large number of parameters and super strong learning ability, mainly referring to large models with transformer structures, but also including similar models with other structures.

[0102] In actual applications, there are large differences between data collection scenarios and model application scenarios. Due to the high cost of training data collection and calibration, only a small amount of training data can be collected for some scenarios. The traditional Convolutional Neural Network (CNN) model trained based on a small number of training samples has poor generalization, and the performance of the CNN model can only be optimized and upgraded by continuously increasing training data, which results in high model application costs.

[0103] With the vigorous development of big model technology, deep learning big models with huge parameter and computational complexity can pre-absorb the knowledge of large-scale training data. Only a small amount of training data from some scenarios is needed to train the big model. The big model can then show good generalization performance in downstream model applications. However, the computing resources and bandwidth required by big models are huge, which makes the cost of applying big models in actual business scenarios too high. In order to reduce the cost of application in actual business scenarios, one of the important ways to apply big models is to transfer the knowledge of big models (i.e., teacher models) to small models (i.e., student models) through model distillation technology, and use the capabilities of big models to guide the performance improvement of small models.

[0104] However, performing model distillation on a small number of training samples in some scenes will cause the student model to overfit the training samples and the characteristics of the teacher model, thereby showing poor generalization in heterogeneous scenes. To solve this problem, in related technologies, model distillation is performed in the following manner.

[0105] Method 1: Use federated learning to perform model distillation.

[0106] like Figure 1 In the federated architecture shown, multiple clients train diffusion models separately and send the trained diffusion models to the server; the server aggregates the diffusion models sent by multiple clients, generates a shared dataset using the aggregated diffusion models, and sends the shared dataset to multiple clients; multiple clients train local small models using the shared dataset and send the trained local small models to the server; the server performs model distillation on the global model based on the local small models sent by multiple clients.

[0107] In this model distillation method, by aggregating diffusion models trained by multiple clients, the aggregated diffusion model can generate diverse data while protecting the privacy of client users. However, this method focuses on the training of generative models for client and local interaction, and the diversity of generated data is achieved by aggregating generative models from multiple clients (multiple application scenarios).

[0108] Method 2: Perform model distillation on a single machine.

[0109] In this model distillation method, the knowledge feedback from the teacher model and the noise vector of the data are used to build a generative model, and the category information is used as the generation condition to input the generative model to obtain new data; the large model is distilled using the new data and the original data. In this model distillation method, the noise vector of the data is used to fine-tune the generative model to generate diverse data for distillation of the large model. The generative model can easily collapse into a specific pattern, resulting in insufficient diversity in the data generated by the generative model. In addition, the generation condition only includes category information, which further leads to insufficient diversity in the data generated by the generative model.

[0110] In the related technology, when generating data, there is still the problem of insufficient data diversity, which causes the student model to overfit the training samples and teacher model characteristics, and the problem of poor generalization in heterogeneous scenarios has not been solved.

[0111] In order to obtain diversified data at a faster speed and lower cost and solve the problems of overfitting and poor generalization in student model distillation, the embodiment of the present application provides a data generation method, such as Figure 2 As shown, the method comprises the following steps:

[0112] Step S201, obtaining first training data, and obtaining a first semantic description set corresponding to a first group, the first semantic description set including a plurality of semantic descriptions corresponding to the training data in the first group;

[0113] Step S202, selecting a plurality of first semantic descriptions from a first semantic description set;

[0114] Step S203, determining a data generation condition according to the first training data, the identifier of the first group and the plurality of first semantic descriptions;

[0115] Step S204: input the data generation conditions into the generation model to obtain second training data.

[0116] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, and the diversity of training data can be expanded at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0117] The data generation method provided in the embodiment of the present application can be applied to electronic devices such as servers, clusters, terminals, etc. For ease of description, the following description is based on electronic devices as the execution subject, which is not limiting.

[0118] In the above step S201, the data of the downstream task acquired by the electronic device is the training data. The downstream tasks may include image recognition, target detection, text recognition, etc., and the training data may be images, texts, tables, etc., which are not limited to this. The electronic device may collect a small amount of training data. When generating data, the electronic device may randomly select a training data from the small amount of training data as the first training data.

[0119] For example, the electronic device collects M training data, which are x 1 、x 2 , …, x M , x 1 、x 2 , …, x M Construct the training data set D train ={x 1 , x 2 , …, x M}. Electronic equipment from D train Randomly select a training data x from i , i = 1, 2, ... or M, x i This is the first training data.

[0120] The electronic device may divide the acquired training data into a plurality of groups, each group including one or more training data, and the electronic device randomly selects a group from the plurality of groups as the first group.

[0121] For example, the electronic device divides the training data into K groups, namely group 1, group 2, ..., group K. The electronic device randomly selects a group j from the K groups, where j=1, 2, ... or K, and group j is the first group.

[0122] In the embodiment of the present application, the electronic device can be grouped according to the category of the training data, that is, the training data of the same category is divided into the same group; the electronic device can also be grouped according to the scene of the training data, that is, the training data of the same scene is divided into the same group, and the specific grouping method is not limited. A group may include one or more categories of training data. In addition, the above-mentioned first training data may belong to the first group or may not belong to the first group.

[0123] The first semantic description set is the semantic description set corresponding to the first group. The first group includes one or more training data, and these training data can be represented by some semantic descriptions. Among them, the semantic description may include one or more of entity names, scene names, entity description phrases, scene description phrases, and entity relationship description phrases. These semantic descriptions are combined together to form the first semantic description set. For example, group 1 includes 10 training data, and these 10 training data can be represented by 20 words (such as word 1 to word 20). The semantic description set corresponding to this group 1 is {word 1, word 2,..., word 20}.

[0124] In the above step S202, after acquiring the first semantic description set, the electronic device selects multiple semantic descriptions from the multiple semantic descriptions included in the first semantic description set as the first semantic description. The number of the first semantic descriptions can be set according to actual needs.

[0125] In an embodiment of the present application, the electronic device may randomly select multiple first semantic descriptions from multiple semantic descriptions included in the first semantic description set. The electronic device may also configure a weight for each semantic description, the greater the weight of the semantic description, the greater the probability of selecting the semantic description; the electronic device selects multiple first semantic descriptions from multiple semantic descriptions included in the first semantic description set according to the weight of the semantic description.

[0126] The electronic device may further define a sampling function R. The electronic device runs the sampling function R to select a plurality of first semantic descriptions from a plurality of semantic descriptions included in the first semantic description set.

[0127] In some embodiments, the first semantic description set may include multiple semantic description subsets, each of which includes one or more semantic descriptions. Based on the multiple semantic description subsets, the electronic device may select multiple first semantic descriptions from each semantic description subset, respectively. The number of first semantic descriptions selected from each semantic description subset may be set according to actual needs.

[0128] For example, the first semantic description set may include three semantic description subsets, namely, semantic description subsets C and j1 , C j2 and C j3 Electronic devices can be powered from C j1 Select n 11 The first semantic description, from C j2 Select n 12 The first semantic description, from C j3 Select n 13 A first semantic description.

[0129] In the embodiment of the present application, the electronic device divides the first semantic description set more finely to facilitate more accurate generation of task-related training data.

[0130] In the embodiment of the present application, the method for selecting the first semantic description is not limited.

[0131] In the above step S203, the electronic device determines the data generation condition by combining the first training data, the identifier of the first group and the plurality of first semantic descriptions.

[0132] In one example, the electronic device may combine the first training data, the identifier of the first group, and the plurality of first semantic descriptions, and the combined data may be used as a condition for a subsequent generation model to generate data, namely, a data generation condition.

[0133] For example, the first training data is x i , the first group is identified as j, and the first semantic descriptions are R (Cj , n 1 ), where n 1 represents the number of the first semantic description, C j represents the first semantic description set, R(C j , n 1 ) indicates that n are selected from the first semantic description set 1 The electronic device combines the first training data, the identifier of the first group and the first semantic descriptions to obtain a data generation condition of {x i ,j,R(C j , n 1 )}.

[0134] In another example, the electronic device may also obtain multiple semantic descriptions received from the user input to obtain a second semantic description set, and select multiple second semantic descriptions from the second semantic description set, wherein the number of the second semantic descriptions may be set according to actual needs.

[0135] In this case, the electronic device may combine the first training data, the identifier of the first group, the plurality of first semantic descriptions, and the plurality of second semantic descriptions to obtain the data generation condition.

[0136] For example, the first training data is x i , the first group is identified as j, and the first semantic descriptions are R (C j , n 1 ), multiple second semantic descriptions are R (C u , n 2 ), where n 1 represents the number of the first semantic description, C j represents the first semantic description set, R(C j , n 1 ) indicates that n are selected from the first semantic description set 1 The first semantic description, n 2 represents the number of the second semantic description, C u represents the second semantic description set, R(C u , n 2 ) indicates that n are selected from the first semantic description set 2 The electronic device combines the first training data, the identifier of the first group, the plurality of first semantic descriptions and the plurality of second semantic descriptions to obtain a data generation condition of {x i ,j,R(C j , n 1 ), R (C u , n 2 )}.

[0137] In the embodiment of the present application, the electronic device allows the user to customize the data characteristics, that is, the user can input the semantic description set C u (any related words, phrases, etc.), control the expected data scene, horizontal or overhead viewing angles, lighting and other characteristics, and achieve generalization for new scenes or applications with special needs of users.

[0138] In the embodiment of the present application, the electronic device may also determine the data generation conditions in other ways, which are not limited to this.

[0139] In the above step S204, the generation model can be any data generation model that includes multimodal condition input, such as a multimodal diffusion model. After determining the data generation condition, the electronic device inputs the data generation condition into the generation model, and the generation model processes the data generation condition and outputs new training data, i.e., the second training data. The grouping or category of the second training data is determined by the teacher model in the model distillation in a pseudo-labeling manner.

[0140] In some embodiments, in order to meet the diverse needs of users, before inputting the data generation conditions into the generation model, the electronic device can also obtain the target hyperparameters of the generation model; and adjust the inference hyperparameters of the generation model to the target hyperparameters. The target hyperparameters are the inference hyperparameters required for this data generation, which may include but are not limited to the information ratio, the degree of refinement of the generation (the number of steps of the generation model inference), etc. The information ratio is the ratio of the first training data and the semantic description (such as the first semantic description and the second semantic description) for this data generation, such as the ratio of the first training data is 0.6 and the ratio of the semantic description is 0.3, or the ratio of the first training data is 0.6 and the ratio of the semantic description is 0.4, etc.

[0141] After adjusting the inference hyperparameters of the generation model, the electronic device can input the data generation conditions into the adjusted generation model to obtain second training data.

[0142] In some embodiments, Figure 3 As shown, a data generation method is also provided, which may include the following steps:

[0143] Step S301, obtaining first training data;

[0144] Step S302, using the multimodal large model, semantically analyze the multiple training data respectively to obtain semantic understanding information of each training data;

[0145] Step S303, dividing the plurality of training data into a plurality of groups according to the category marked by each training data, each group including at least one category of training data;

[0146] Step S304, determining a first semantic description set corresponding to the first group according to the semantic understanding information of each training data included in each group, where the first semantic description set includes multiple semantic descriptions corresponding to the training data in the first group;

[0147] Step S305, selecting a plurality of first semantic descriptions from the first semantic description set;

[0148] Step S306, determining a data generation condition according to the first training data, the identifier of the first group and the plurality of first semantic descriptions;

[0149] Step S307, inputting the data generation conditions into the generation model to obtain second training data.

[0150] The above steps S305 to S307 are the same as the above steps S202 to S204, and will not be repeated here.

[0151] In the technical solution provided in the embodiment of the present application, the multimodal large model of the electronic device performs semantic analysis on the current multiple training data, and then determines the corresponding semantic description set, thereby ensuring that the semantic description set is consistent with the application of the current downstream task and ensuring that the subsequently generated training data is more consistent with the large model distillation under the current task.

[0152] The operation of obtaining the first training data in the above step S301 can refer to the relevant description of the above step S201, which will not be repeated here. The execution order of the above step S303 and the above step S302 is not limited.

[0153] In the above step S302, the multimodal large model can be a visual-language large model such as GPT-4 and Llama, or it can be an open set detection, classification or segmentation model such as SAM and UniDetector.

[0154] The electronic device can obtain multiple training data, and for each training data, use the multimodal large model to perform semantic analysis on the training data to obtain semantic understanding information of the training data. The semantic understanding information can be a text description. For example, Figure 4a The image shown is training data. After the electronic device performs semantic analysis on the image, the semantic understanding information of the image is obtained as "a car is driving on the road, and a person is standing on the side of the road".

[0155] In the embodiment of the present application, the electronic device performs semantic analysis on multiple training data respectively to obtain multiple semantic understanding information. For example, the electronic device collects a training data set D train ={x 1 , x 2 , …, x M}, x irepresents training data, i=1, 2, ..., M, M is the number of training data. The electronic device performs semantic analysis on multiple training data respectively, and obtains a semantic understanding information set {c 1 , c 2 , …, c M},c i Represents the training data x i semantic understanding information.

[0156] In some embodiments, the electronic device may perform global semantic analysis on the training data, that is, perform global semantic analysis on the entirety of each training data to obtain first semantic understanding information of each training data.

[0157] In order to enhance the richness of semantic understanding information and facilitate the subsequent generation of diversified data, electronic devices can perform global and local semantic analysis on the training data, that is: perform global semantic analysis on the entirety of each training data to obtain the first semantic understanding information of each training data; perform local semantic analysis on at least one local area of ​​each training data to obtain the second semantic understanding information of each training data.

[0158] For example, Figure 4a The image shown in FIG. 1 is training data. The electronic device performs semantic analysis on the entire image (i.e., global semantic analysis) and obtains the semantic understanding information of the image as “a car is driving on the road and a person is standing on the side of the road”. After that, the electronic device performs semantic analysis on the entire image (i.e., global semantic analysis). Figure 4a The local semantic analysis of the image shown is performed (local semantic analysis), such as Figure 4b As shown, electronic equipment Figure 4b The semantic understanding information of region 1 is obtained by performing semantic analysis on region 1 as “a moving red car”. Figure 4b The region 2 shown is subjected to semantic analysis, and the semantic understanding information of the region 2 is obtained as “a person wearing a floral dress and short hair”.

[0159] In an embodiment of the present application, the semantic understanding information of a training data includes at least first semantic understanding information (i.e., global semantic parsing results), and may include several second semantic understanding information (i.e., local semantic parsing results). The specific number of parsing results included is related to the recognition capability of the multimodal large model U.

[0160] When the semantic understanding information of a training data includes the second semantic understanding information, the first semantic understanding information may further include the position of the local area corresponding to the second semantic understanding information.

[0161] For example, the electronic device pair set D train ={x 1 , x 2 , …, xM}, and the semantic understanding information set is obtained as {(c 1 , b 1 ), (c 2 , b 2 ),…,(c M , b M )}, c i Represents the training data x i The semantic understanding information of b i Indicates the location corresponding to the semantic understanding information.

[0162] Based on the position, the user can determine the position range of the target in the training data included in each group, and then input the position range into the electronic device in the form of a semantic description; or the electronic device fits the position of the target in the training data included in each group to determine the position range of the target; and then the electronic device adds the position range as a condition to the data generation condition to obtain new data with high task relevance.

[0163] In the above step S303, the training data carries the categories manually labeled according to the current task. The electronic device groups the multiple training data according to the categories labeled by each training data. A group includes one or more categories of training data, and the specific number of categories included in a group is set according to the actual task requirements.

[0164] After the electronic device groups the multiple training data, the semantic understanding information of the training data included in one group is also grouped together.

[0165] In the above step S304, the semantic description can be understood as the phrases or words included in the semantic understanding information.

[0166] After determining multiple groups, for each group, the electronic device can extract the semantic description in the semantic understanding information of each training data included in the group, and after deduplication processing, obtain the semantic description set corresponding to each group, which includes the first semantic description set.

[0167] For example, group 1 includes training data 1 and training data 2. The electronic device extracts words 1, 2, and 3 from the semantic understanding information of training data 1, and extracts words 3, 4, and 5 from the semantic understanding information of training data 2; then the electronic device determines that the semantic description set of group 1 is {word 1, word 2, word 3, word 4, word 5}.

[0168] In some embodiments, to reduce the burden on electronic devices, the electronic devices may count the frequency of occurrence of each semantic description in the semantic understanding information of a group including multiple training data; select multiple semantic descriptions with the highest frequency of occurrence to obtain a set of semantic descriptions corresponding to each group.

[0169] For example, group 1 includes training data 1, training data 2, and training data 3. The electronic device extracts words 1, 2, and 3 from the semantic understanding information of training data 1, extracts words 3, 4, and 5 from the semantic understanding information of training data 2, and extracts words 2, 3, and 4 from the semantic understanding information of training data 3; then the occurrence frequency of words 1 and 5 is 1, the occurrence frequency of words 2 and 4 is 2, and the occurrence frequency of word 3 is 3; the number of semantic descriptions included in the semantic description set is 3, then the electronic device selects words 2, 3, and 4 with the highest occurrence frequency to form the semantic description set corresponding to group 1.

[0170] In some embodiments, Figure 5 As shown, the electronic device may also adopt the following steps to implement the above step S304.

[0171] Step S501, extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtaining an initial semantic description set corresponding to each group;

[0172] In an embodiment of the present application, for each group, the electronic device can extract the semantic description in the semantic understanding information of each training data included in the group, and after deduplication processing, obtain a set of semantic descriptions corresponding to each group.

[0173] In some embodiments, in order to reduce the burden on the electronic device, the electronic device can count the frequency of each semantic description in the semantic understanding information of the group including multiple training data; select multiple semantic descriptions with the highest frequency of occurrence to obtain a semantic description set corresponding to each group. That is, from the semantic understanding information of each training data included in each group, extract the semantic description of each training data included in each group; from the semantic descriptions of all training data included in each group, select multiple semantic descriptions with the highest frequency of occurrence to obtain an initial semantic description set corresponding to each group.

[0174] Among them, the initial semantic description set corresponding to group k can be expressed as H k ={z 1 , z 2 , …, z N}, z j represents the semantic description with high occurrence frequency j, and N represents the number of semantic descriptions included in the initial semantic description set.

[0175] Step S502, determining the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtaining a first semantic description subset corresponding to the first group;

[0176] In the embodiment of the present application, the first semantic description subset can be understood as a set of semantic concepts shared by the semantic description sets of all groups. The electronic device obtains the semantic descriptions shared by the initial semantic description sets corresponding to multiple groups, and the obtained semantic descriptions constitute the first semantic description subset corresponding to the first group. Of course, the first semantic description subset corresponding to the first group is also the first semantic description subset corresponding to other groups. The first semantic description subset can be called a public semantic description set, expressed as:

[0177] C task =H 1 ∩H 2 ∩…∩H K .

[0178] Among them, C task represents the first semantic descriptor set, K represents the number of groups, H j Represents the initial semantic description set corresponding to group j, j=1, 2,…,K.

[0179] Step S503, determining, from the initial semantic description set corresponding to the first group, semantic descriptions not included in the initial semantic description sets corresponding to other groups, to obtain a second semantic description subset corresponding to the first group;

[0180] In the embodiment of the present application, the second semantic description subset can be understood as a set of semantic concepts unique to the semantic description set corresponding to the first group. The electronic device obtains the semantic descriptions included in the initial semantic description set corresponding to the first group but not included in the initial semantic description sets corresponding to other groups, and the obtained semantic descriptions constitute the second semantic description subset corresponding to the first group. The second semantic description subset can be called a positive semantic description set, which is expressed as:

[0181] C j =H j -(H 1 ∪H 2 ∪…H j-1 ∪H j+1 ∪…∪H K )∩H j .

[0182] Among them, C task represents the first semantic descriptor set, K represents the number of groups, H j Represents the initial semantic description set corresponding to group j, j=1, 2,…,K.

[0183] The positive semantic description directly describes the data characteristics of the target group and is used to maintain the group information.

[0184] Step S504: Determine, from the initial semantic description sets corresponding to other groups, the semantic descriptions not included in the initial semantic description set corresponding to the first group, and obtain a third semantic description subset corresponding to the first group.

[0185] In the embodiment of the present application, the third semantic description subset can be understood as a set of semantic concepts unique to the semantic description sets corresponding to other groups. The electronic device obtains semantic descriptions that are not included in the initial semantic description set corresponding to the first group but are included in the initial semantic description sets corresponding to other groups, and the obtained semantic descriptions constitute the third semantic description subset corresponding to the first group. The third semantic description subset can be called a reverse semantic description set, expressed as:

[0186] V j =H 1 ∪H 2 ∪…H j-1 ∪H j+1 ∪…∪H K -H j .

[0187] Among them, C task represents the first semantic descriptor set, K represents the number of groups, H j Represents the initial semantic description set corresponding to group j, j=1, 2,…,K.

[0188] The reverse semantic description describes the data characteristics of mutually exclusive groups and is used to change the data grouping.

[0189] The first semantic description subset, the second semantic description subset and the third semantic description subset constitute a first semantic description set corresponding to the first group. In one example, the electronic device may perform structured processing on the semantic description sets corresponding to the multiple groups, such as {C task ,{C 1 , C 2 , …, C K},{V 1 , V 2 , …, V K}}.

[0190] In real applications, the application scenarios of large models for downstream tasks are often diverse, and the training data for auxiliary distillation should also be as diverse as possible to better transfer the knowledge of the teacher model under various data, thereby improving the generalization performance of the student model after distillation in multiple scenarios.

[0191] The embodiment of the present application does not limit the execution order of step S502, step S503 and step S504.

[0192] In an embodiment of the present application, the electronic device subdivides the semantic description set corresponding to a group into a common semantic description set, a positive semantic description set, and a reverse semantic description set, and uses the subdivided description set to construct data generation conditions. While achieving data diversity, it can effectively control the difference between the generated new training data and the original training data within an appropriate range, in order to ensure that the generated distilled data can be correctly understood by the teacher model, that is, the teacher model can extract the correct knowledge from the distilled data, thereby ensuring the effect of model distillation.

[0193] Based on the above data generation method, the present application embodiment also provides a model distillation method, such as Figure 6 As shown, the method includes the following steps.

[0194] Step S601, obtaining an original data set, where the original data set includes a plurality of original training data;

[0195] Step S602, using any of the above data generation methods to generate multiple new training data according to multiple original training data, to obtain a generated data set;

[0196] Step S603, using the original data set and the generated data set to assist the large model in performing knowledge distillation processing to obtain a target small model.

[0197] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, achieving the diversity of training data at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0198] The model distillation method provided in the embodiment of the present application can be applied to electronic devices such as servers, clusters, and terminals. For ease of description, the following description is based on electronic devices as the execution subject, which does not serve as a limitation. In addition, the electronic device that executes the model distillation method can be the same as or different from the electronic device that executes the data generation method.

[0199] In the above step S601, the original data set D train The training data includes original training data, which is also the data of downstream tasks acquired by the electronic device.

[0200] In the above step S602, the electronic device can cyclically select training data from multiple original training data as the first training data, and generate multiple new training data according to the above data generation method in combination with the corresponding semantic description to obtain a generated data set D g .

[0201] For example, the electronic device may generate a predetermined amount of new training data using the following steps.

[0202] (1) From the original data set D train In the example, randomly sample training data xi, whose group label is y i , i≤M, M represents the original data set D train The amount of training data in ;

[0203] (2) From the common semantic description set C of downstream tasks task In the random selection task A semantic description, namely R(C task , n task ), n task ∈[1,N task ], where N task C task The number of elements included, R(∙,∙) is the random sampling function;

[0204] (3) Randomly select group j to generate data from the K groups;

[0205] (4) From the positive semantic description set C corresponding to group j j In the random selection Cj A semantic description, namely R(C j , n Cj ), n Cj ∈[1,N Cj ], where N Cj C j The number of elements included;

[0206] (5) From the reverse semantic description set V corresponding to group j j In the random selection Vj A semantic description, namely R(V j , n Vj ), n Vj ∈[1,N Vj ], where N Vj V j The number of elements included;

[0207] (6) From the user input condition set C u In the random selectionCu A semantic description, namely R(C u , n Cu ),n Cu ∈[0,N Cu ], where N Cu C u The number of elements included;

[0208] (7) Get a set of data generation conditions (x i ,j,R(C task , n task ), R(C j , n Cj ), R(V j , n Vj ), R(C u , n Cu ));

[0209] (8) Repeat processes (1) to (7) to obtain a given number of data generation conditions.

[0210] At this time, the electronic device obtains a variety of conditions for generating data and realizes the generation of a variety of control conditions.

[0211] Assuming the generation module is G, data generation can be defined as:

[0212] x'=G(x i ,j,R(C task , n task ), R(C j , n Cj ), R(V j , n Vj ), R(C u , n Cu ))

[0213] The electronic device inputs each set of data generation conditions obtained in (1) to (8) into the generation module G to obtain a new training data. After a given number of data generation conditions are input into the generation module G, a given number of new training data can be obtained, and the generated data set D is obtained. g .

[0214] In the above step S603, the electronic device obtains the generated data set D g After that, we can use the original data set D train And generate data set D g , assisting the large model (i.e., teacher model) to perform distillation processing, and transferring the knowledge learned by the large model to the target small model (i.e., student model). The target small model can be a traditional CNN model, a recurrent network model, or other models, without limitation.

[0215] In the embodiment of the present application, before performing knowledge distillation processing on the large model, the electronic device can first use the original data set D train , train the large model f T ( ), and then use the trained large model f T ( ), to generate the data set D g The training data in the inference is used to convert the large model f T The output result of ( ) is used as the generated data set D g Then, the electronic device in D train ∪D g On the training set composed of Figure 7 As shown, it may include intermediate layer feature distillation between the teacher model backbone and the student model backbone, feature relationship distillation, and response output distillation corresponding to the teacher model task head and the student model task head.

[0216] With large model f T The i-th intermediate layer feature of ( ) is Ti∈R B×H×W×C(i,t) , the jth intermediate layer feature of the target small model is S j ∈R B×H×W×C(j,s) , where B is the number of training data in a single batch, H×W is the dimension of the feature space, and C(i,t) and C(j,s) are the number of channels of the corresponding feature. Feature S j Through the function f adap Mapping to feature T i The characteristic dimension of function f adap For a convolutional neural network (such as Conv1x1), the feature S is obtained after conversion j =f adap (S j ), S j '∈R B×H×W×C(i,t) .

[0217] When performing intermediate layer feature distillation, the loss function L of the intermediate layer feature distillation is feature Can be:

[0218] L feature =MSE(LN(S j '), LN(T i ))

[0219] Among them, MSE represents the mean square error loss, and LN represents the feature normalization operation. LN(T i )=(T i -mean(T i )) / std(T i ), mean means the mean, and std means the standard deviation.

[0220] When performing feature relationship distillation, the loss function L of feature relationship distillation is relation Can be:

[0221] L relation =‖d L2 (S j ')-d L2 (T i )‖ 1

[0222]

[0223] in, , , X=S j ' or T i , d L2 represents the L2 distance, and B is the number of training data in a single batch.

[0224] When performing response output distillation, the loss function L of response output distillation is response Can be:

[0225] L response =w×L kd (y s ,y t ).

[0226] Among them, L kd (y s ,y t ) is the loss function, such as KL divergence, MSE loss, BCE loss, etc., w is the weighted coefficient of the response output distillation group (such as category), y s is the output of the student model, y t is the output of the teacher model.

[0227] When the loss function is KL divergence, , N is the feature dimension;

[0228] When the loss function is BCE loss, .

[0229] The total distillation training loss is:

[0230] L total =αL feature +βL response +θL relation +γL task ,

[0231] Among them, α, β, θ and γ are constant coefficients, L task Target task loss for the student model.

[0232] Combine the following Figure 8 The data processing flow shown is a detailed description of the model distillation method provided in the embodiment of the present application.

[0233] Step S801, downstream task data. The electronic device obtains a plurality of downstream task data, wherein the downstream task data is training data.

[0234] Step S802: multimodal large model analysis: The electronic device uses the multimodal large model to perform semantic analysis on the downstream task data to obtain semantic understanding information.

[0235] Step S803: construct a structured semantic description of the downstream task. The electronic device performs structured processing on the semantic understanding information of each downstream task data to construct a structured semantic description of the downstream task, as shown in the above {C task ,{C 1 , C 2 , …, C K},{V 1 , V 2 , …, V K}}. For details, please refer to the above Figure 3 and Figure 5 Description of the part.

[0236] Step S804: User inputs conditions. The user inputs multiple semantic descriptions into the electronic device, and the multiple semantic descriptions are the data generation conditions required by the user.

[0237] Step S805: Generate diversified control conditions. As described in step S602 above, the electronic device executes (1) to (8) in a loop to obtain a predetermined number of data generation conditions, thereby achieving generation of diversified control conditions.

[0238] Step S806, generating model inference: The electronic device inputs a predetermined number of data generation conditions obtained into the generating model respectively, and the generating model infers the data generation conditions to obtain new data.

[0239] The newly generated data is merged with the original downstream task data as distilled data, i.e., training data, to perform distillation on the large model.

[0240] In the embodiment of the present application, the generation of distilled data can be divided into two stages, namely, downstream task semantic description and diversified data generation: (1) downstream task semantic description, such as step S802 to step S803, using a multimodal large model to analyze downstream task data, generate semantic understanding information such as downstream task scenarios, target categories, target locations, entity relationships, and construct a structured task semantic description; (2) diversified data generation, such as step S804 to step S806, according to specific rules, select some combinations from the structured task semantic description, user input conditions and hyperparameters of the generation model, input them into the generation model, and obtain diversified distilled data that is task-related and meets user needs.

[0241] In the technical solution provided in the embodiment of the present application, the electronic device can adaptively expand the training data based on the multimodal large model capability, specifically including the following three aspects:

[0242] (1) Task-adaptive training data generation scheme: Use a multimodal large model to automatically parse downstream task data, combine the parsing results, user input conditions, and original training data in a certain way as the generation conditions of new data, input the generation model, and obtain diversified data to assist distillation;

[0243] (2) Structured semantic description of downstream tasks: Structural processing of the parsing results of downstream task data to form a common semantic description of the target task and positive and negative semantic descriptions of task grouping, which facilitates the diversified expression of the characteristics of the generated data;

[0244] (3) Diversified control condition generation: select data generation conditions from the task structured semantic description, user input conditions, and original training data according to certain rules, and reasonably control the diversity and task relevance of generated data.

[0245] Combining the structured semantic description of downstream tasks and user input conditions to expand the training data of the target task, while ensuring that the new data is relevant to the current task, greatly expands the diversity of training data. Using the newly generated training data for distillation can effectively improve the generalization performance of the student model after distillation in multiple scenarios, especially the performance in user-customized scenarios.

[0246] Corresponding to the above data generation method, the embodiment of the present application also provides a data generation device, such as Fig. 9 As shown, the device comprises:

[0247] An acquisition module 901 is used to acquire first training data and acquire a first semantic description set corresponding to a first group, wherein the first semantic description set includes a plurality of semantic descriptions corresponding to the training data in the first group;

[0248] A first selection module 902, configured to select a plurality of first semantic descriptions from a first semantic description set;

[0249] A determination module 903, configured to determine a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions;

[0250] The input module 904 is used to input the data generation conditions into the generation model to obtain the second training data.

[0251] In some embodiments, the acquisition module 901 may be specifically used to:

[0252] Using a large multimodal model, semantic analysis is performed on multiple training data to obtain semantic understanding information for each training data.

[0253] According to the category marked by each training data, the multiple training data are divided into multiple groups, each group includes at least one category of training data;

[0254] A first semantic description set corresponding to the first group is determined according to the semantic understanding information of each training data included in each group.

[0255] In some embodiments, the acquisition module 901 may be specifically used to:

[0256] Extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtaining an initial semantic description set corresponding to each group;

[0257] Determine the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtain a first semantic description subset corresponding to the first group;

[0258] Determine, from the initial semantic description set corresponding to the first group, semantic descriptions not included in the initial semantic description sets corresponding to other groups, to obtain a second semantic description subset corresponding to the first group;

[0259] Determine, from the initial semantic description sets corresponding to other groups, the semantic descriptions not included in the initial semantic description set corresponding to the first group, to obtain a third semantic description subset corresponding to the first group;

[0260] The first semantic description subset, the second semantic description subset and the third semantic description subset constitute a first semantic description set corresponding to the first group.

[0261] In some embodiments, the acquisition module 901 may be specifically used to:

[0262] Extracting a semantic description of each training data included in each group from the semantic understanding information of each training data included in each group;

[0263] From the semantic descriptions of all training data included in each group, multiple semantic descriptions with the highest occurrence frequency are selected to obtain an initial semantic description set corresponding to each group.

[0264] In some embodiments, the acquisition module 901 may be specifically used to:

[0265] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; or,

[0266] Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; and perform local semantic analysis on at least one local area of ​​each training data to obtain second semantic understanding information of each training data.

[0267] In some embodiments, the first set of semantic descriptions includes a plurality of subsets of semantic descriptions;

[0268] The first selection module is specifically configured to select a plurality of first semantic descriptions from each semantic description subset.

[0269] In some embodiments, the data generating device may further include:

[0270] A receiving module, configured to receive multiple semantic descriptions input by a user and obtain a second semantic description set;

[0271] A second selection module, configured to select a plurality of second semantic descriptions from the second semantic description set;

[0272] In this case, the determination module 903 may be specifically used to combine the first training data, the identifier of the first group, the plurality of first semantic descriptions, and the plurality of second semantic descriptions to obtain the data generation condition.

[0273] In some embodiments, the data generating device may further include:

[0274] The adjustment module is used to obtain the target hyperparameters of the generative model before inputting the data generation conditions into the generative model; and adjust the inference hyperparameters of the generative model to the target hyperparameters.

[0275] In some embodiments, the semantic description includes one or more of an entity name, a scene name, an entity description phrase, a scene description phrase, and an entity relationship description phrase.

[0276] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, and the diversity of training data can be expanded at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0277] Corresponding to the above-mentioned model distillation method, the embodiment of the present application also provides a model distillation device, such as Fig.10 As shown, the device comprises:

[0278] An acquisition module 1001 is used to acquire an original data set, where the original data set includes a plurality of original training data;

[0279] A generating module 1002 is used to generate a plurality of new training data according to a plurality of original training data by using any of the above-mentioned data generating devices to obtain a generated data set;

[0280] The distillation module 1003 is used to utilize the original data set and the generated data set to assist the large model in performing knowledge distillation processing to obtain a target small model.

[0281] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, and the diversity of training data can be expanded at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0282] In the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0283] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0284] Memory 1101, used for storing computer programs;

[0285] The processor 1102 is used to implement the following steps when executing the program stored in the memory 1101:

[0286] Acquire first training data, and acquire a first semantic description set corresponding to the first group, the first semantic description set including a plurality of semantic descriptions corresponding to the training data in the first group;

[0287] Selecting a plurality of first semantic descriptions from the first semantic description set;

[0288] Determining a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions;

[0289] The data generation conditions are input into the generation model to obtain the second training data.

[0290] Or the processor 1102 is used to execute the program stored in the memory 1101, and implement the following steps:

[0291] Obtaining an original data set, where the original data set includes a plurality of original training data;

[0292] Using any of the above data generation methods, generate multiple new training data according to multiple original training data to obtain a generated data set;

[0293] The original data set and the generated data set are used to assist the large model in knowledge distillation to obtain the target small model.

[0294] In the technical solution provided in the embodiment of the present application, the training data included in a group is represented by multiple semantic descriptions. Any training data is used in combination with multiple semantic descriptions corresponding to any group to determine the data generation conditions to generate new training data, thereby ensuring the relevance of the new training data to the current task. In this way, a large amount of new training data related to the current task can be generated, and the diversity of training data can be expanded at a faster speed and lower cost. The use of diverse training data for model distillation solves the problems of overfitting and poor generalization in student model distillation caused by small amounts of training data and limited scenarios.

[0295] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0296] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0297] The communication interface is used for communication between the above electronic device and other devices.

[0298] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0299] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0300] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned data generation methods is implemented, or when the computer program is executed by a processor, any of the above-mentioned model distillation methods is implemented.

[0301] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data generation methods in the above embodiments, or enables the computer to execute any of the model distillation methods in the above embodiments.

[0302] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0303] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0304] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, storage medium, and program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0305] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data generation method, characterized in that: The method comprises: Acquire first training data, and use a multimodal large model to perform semantic analysis on multiple training data respectively to obtain semantic understanding information of each training data; According to the category marked by each training data, the plurality of training data are divided into a plurality of groups, each group including at least one category of training data; Extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtaining an initial semantic description set corresponding to each group; Determine the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtain a first semantic description subset corresponding to the first group; Determine, from the initial semantic description set corresponding to the first group, semantic descriptions not included in the initial semantic description sets corresponding to other groups, to obtain a second semantic description subset corresponding to the first group; Determine, from the initial semantic description sets corresponding to the other groups, semantic descriptions not included in the initial semantic description set corresponding to the first group, and obtain a third semantic description subset corresponding to the first group; wherein the first semantic description subset, the second semantic description subset, and the third semantic description subset constitute a first semantic description set corresponding to the first group, and the first semantic description set includes multiple semantic descriptions corresponding to the training data in the first group; Selecting a plurality of first semantic descriptions from the first semantic description set; Determining a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions; The data generation conditions are input into a generation model to obtain second training data.

2. The method according to claim 1, characterized in that The step of extracting the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group to obtain an initial semantic description set corresponding to each group includes: Extracting a semantic description of each training data included in each group from the semantic understanding information of each training data included in each group; From the semantic descriptions of all training data included in each group, multiple semantic descriptions with the highest occurrence frequency are selected to obtain an initial semantic description set corresponding to each group.

3. The method according to claim 1, characterized in that The step of using the multimodal large model to perform semantic analysis on multiple training data respectively to obtain semantic understanding information of each training data includes: Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; or, Perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; and perform local semantic analysis on at least one local area of ​​each training data to obtain second semantic understanding information of each training data.

4. The method according to claim 1, characterized in that: The first semantic description set includes a plurality of semantic description subsets; The step of selecting a plurality of first semantic descriptions from the first semantic description set comprises: A plurality of first semantic descriptions are selected from each semantic description subset.

5. The method according to claim 1, characterized in that The method further comprises: Receiving multiple semantic descriptions input by a user to obtain a second semantic description set; Selecting a plurality of second semantic descriptions from the second semantic description set; The step of determining data generation conditions according to the first training data, the identifier of the first group and the plurality of first semantic descriptions comprises: The first training data, the identifier of the first group, the plurality of first semantic descriptions, and the plurality of second semantic descriptions are combined to obtain a data generation condition.

6. The method according to claim 1, characterized in that Before inputting the data generation condition into the generation model, the method further includes: Obtain target hyperparameters of the generative model; The inference hyperparameters of the generative model are adjusted to the target hyperparameters.

7. The method according to any one of claims 1 to 6, characterized in that: The semantic description includes one or more of an entity name, a scene name, an entity description phrase, a scene description phrase, and an entity relationship description phrase.

8. A model distillation method, characterized in that: The method comprises: Acquire an original data set, wherein the original data set includes a plurality of original training data; Using the method described in any one of claims 1 to 7, generating multiple new training data according to the multiple original training data to obtain a generated data set; The original data set and the generated data set are used to assist the large model in performing knowledge distillation processing to obtain a target small model.

9. A data generating device, characterized in that: The device comprises: An acquisition module is used to acquire first training data, and use a multimodal large model to perform semantic analysis on multiple training data respectively to obtain semantic understanding information of each training data; divide the multiple training data into multiple groups according to the category annotated by each training data, each group includes at least one category of training data; extract the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group, and obtain an initial semantic description set corresponding to each group; determine the semantic descriptions included in the initial semantic description sets corresponding to the multiple groups, and obtain a first semantic description subset corresponding to the first group; from the initial semantic description set corresponding to the first group, determine the semantic descriptions that are not included in the initial semantic description sets corresponding to other groups, and obtain a second semantic description subset corresponding to the first group; from the initial semantic description sets corresponding to the other groups, determine the semantic descriptions that are not included in the initial semantic description set corresponding to the first group, and obtain a third semantic description subset corresponding to the first group; wherein the first semantic description subset, the second semantic description subset and the third semantic description subset constitute a first semantic description set corresponding to the first group, and the first semantic description set includes multiple semantic descriptions corresponding to the training data in the first group; A first selection module, configured to select a plurality of first semantic descriptions from the first semantic description set; a determination module, configured to determine a data generation condition according to the first training data, the identifier of the first group, and the plurality of first semantic descriptions; An input module is used to input the data generation conditions into the generation model to obtain second training data.

10. The device according to claim 9, characterized in that The acquisition module is specifically used to: extract the semantic description of each training data included in each group from the semantic understanding information of each training data included in each group; select multiple semantic descriptions with the highest occurrence frequency from the semantic descriptions of all training data included in each group to obtain an initial semantic description set corresponding to each group; or, The acquisition module is specifically used to: perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; or perform global semantic analysis on each training data to obtain first semantic understanding information of each training data; and perform local semantic analysis on at least one local area of ​​each training data to obtain second semantic understanding information of each training data; or, The first semantic description set includes a plurality of semantic description subsets; the first selection module is specifically configured to select a plurality of first semantic descriptions from each semantic description subset; or, The device also includes: A receiving module, configured to receive multiple semantic descriptions input by a user and obtain a second semantic description set; A second selection module, configured to select a plurality of second semantic descriptions from the second semantic description set; The determination module is specifically used to combine the first training data, the identifier of the first group, the multiple first semantic descriptions and the multiple second semantic descriptions to obtain a data generation condition; or, The device also includes: An adjustment module, used for obtaining a target hyperparameter of the generation model before inputting the data generation condition into the generation model; and adjusting the inference hyperparameter of the generation model to the target hyperparameter; or, The semantic description includes one or more of an entity name, a scene name, an entity description phrase, a scene description phrase, and an entity relationship description phrase.

11. A model distillation apparatus, characterized in that: The device comprises: An acquisition module, used for acquiring an original data set, wherein the original data set includes a plurality of original training data; A generating module, configured to generate a plurality of new training data according to the plurality of original training data by using the apparatus according to any one of claims 9 to 10, so as to obtain a generated data set; The distillation module is used to use the original data set and the generated data set to assist the large model in performing knowledge distillation processing to obtain a target small model.

12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images