Data set distillation method, computer equipment and storage medium

Abstract

The invention relates to a data set distillation method, computer equipment and a storage medium. The method comprises the following steps: acquiring an original data set, wherein the original data set comprises original data and an original data label; synthetic data in a synthetic data set is determined based on a synthetic data label, the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set is matched with a target network model; updating model parameters of the target network model based on the synthetic data and a first preset rule, and determining an update gradient of the synthetic data; and updating the synthetic data set based on the original data, the updating gradient, the updated target network model and a second preset rule. According to the method, the effectiveness of the post-distillation synthetic data set is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data distillation, and in particular to a data set distillation method, a computer device, and a storage medium. Background Art

[0002] Deep learning has achieved state-of-the-art performance in many fields, using neural networks trained on large-scale datasets, such as e-commerce search recommendation, medical and health diagnosis, and network data classification. However, high-performance models usually rely on their high training costs.

[0003] In traditional techniques, in order to reduce training costs, data selection methods are usually used to compress data sets. Data selection methods can select a subset of valid training samples based on some heuristic metrics, such as cluster center, diversity, and model likelihood.

[0004] However, the performance of data selection methods in traditional technologies is obviously limited by the representative samples in the original dataset used for effective model training. If the original dataset contains insufficient representative samples or the representative samples have low effectiveness, the expression of the compressed dataset to the original dataset will be greatly affected, resulting in low effectiveness of the compressed dataset. Summary of the invention

[0005] Based on this, the present application provides a data set distillation method, a computer device and a storage medium to solve the problem in traditional technology that the representative samples of the original data set are limited, resulting in low effectiveness of the compressed data set.

[0006] In a first aspect, the present application provides a data set distillation method. The method comprises:

[0007] Obtaining an original data set, wherein the original data set includes original data and original data labels;

[0008] Determining synthetic data in a synthetic data set based on a synthetic data label, wherein the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set matches a target network model;

[0009] Based on the synthetic data and a first preset rule, updating the model parameters of the target network model, and determining the update gradient of the synthetic data;

[0010] The synthetic data set is updated based on the original data, the updated gradient, the updated target network model and a second preset rule.

[0011] In one embodiment, determining the synthetic data in the synthetic data set based on the synthetic data label includes:

[0012] Determine a plurality of label categories based on the original data labels;

[0013] determining a plurality of first categories of the synthetic data labels based on a single category among the plurality of label categories;

[0014] Based on the plurality of first categories, the synthesized data is determined.

[0015] In one embodiment, determining the synthesized data based on the plurality of first categories comprises:

[0016] Determine a first preset number of synthetic data matching each first category, wherein the first preset numbers corresponding to different first categories are the same;

[0017] or;

[0018] A second preset number of synthetic data matching each first category is determined, wherein the second preset number is determined based on a proportion of the single category in the plurality of label categories.

[0019] In one embodiment, determining the synthetic data in the synthetic data set based on the synthetic data label includes:

[0020] Determine a plurality of label categories based on the original data labels; determine a plurality of second categories of the synthesized data labels based on at least two of the plurality of label categories, wherein the number of the synthesized data labels is determined based on the number of the original data labels; determine the synthesized data based on the plurality of second categories;

[0021] or;

[0022] Determine multiple label categories based on the original data labels; determine multiple third categories of the synthetic data labels based on random probabilities within a preset probability range, the multiple third categories including the probability that the synthetic data labels belong to each label category; determine the synthetic data based on the multiple third categories.

[0023] In one embodiment, determining the synthesized data based on the plurality of second categories comprises:

[0024] Determine a third preset number of synthetic data matching each second category, wherein the third preset numbers corresponding to different second categories are the same;

[0025] or;

[0026] A fourth preset number of synthetic data matching each second category is determined, wherein the fourth preset number is determined based on a proportion of the at least two categories in the plurality of label categories.

[0027] In one embodiment, determining a plurality of second categories of the synthetic data labels based on at least two of the plurality of label categories comprises:

[0028] Based on a first threshold and a label frequency corresponding to each label category, the original data labels are divided into high-frequency labels and low-frequency labels, wherein the label frequency is determined based on the amount of original data corresponding to the corresponding label category;

[0029] Determine a high-frequency co-occurring binary tag based on the tag frequency corresponding to the tag category of each two high-frequency tags and a second threshold;

[0030] Determine the high-frequency co-occurrence multi-label based on the label frequency corresponding to the label category of multiple high-frequency labels and the third threshold, and determine the co-occurrence multi-label based on the label frequency corresponding to the label category of each low-frequency label, the label frequency corresponding to the label category of the high-frequency co-occurrence multi-label and the fourth threshold;

[0031] The plurality of second categories are determined based on the tag categories of the high-frequency co-occurring binary tags and the tag categories of the co-occurring multi-tags.

[0032] In one of the embodiments, after determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes:

[0033] Determine a first feature vector of the original data corresponding to the first category;

[0034] An average feature vector of the first feature vectors is determined, and a feature vector of the synthetic data is updated based on the average feature vector.

[0035] In one of the embodiments, after determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes:

[0036] determining a synthetic label category of the synthetic data label, the synthetic label category comprising at least one label category;

[0037] In the original data set, determining whether there is original data corresponding to an original data label of the same category as the synthetic label;

[0038] If it exists, the feature vector of the synthetic data is updated based on the feature vector of the original data; if it does not exist, the associated original data corresponding to each label category in the synthetic label category is determined in the original data set respectively, and the feature vector of the synthetic data is updated based on the average vector of the feature vectors corresponding to the associated original data.

[0039] In one embodiment, updating the model parameters of the target network model based on the synthesized data and the first preset rule includes:

[0040] Determining a first update number based on a first preset batch and the amount of the synthesized data;

[0041] Based on the first preset batch of synthetic data and a first gradient descent rule, the model parameters of the target model are updated until the number of updates reaches the first number of updates.

[0042] In one embodiment, updating the synthetic data set based on the original data, the updated gradient, the updated target network model and the second preset rule comprises:

[0043] Based on the second preset batch of raw data, forward propagation is performed in the updated target network model, and the raw data gradient is determined;

[0044] Determining a target loss based on the similarity between the original data gradient and the updated gradient;

[0045] Based on the target loss and the second gradient descent rule, back-propagation is performed in the updated target network model, and distilled synthetic data is determined;

[0046] The synthetic data in the synthetic dataset is updated based on the distilled synthetic data.

[0047] In a second aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the data set distillation methods described in the first aspect are implemented.

[0048] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the data set distillation methods described in the first aspect are implemented.

[0049] The above-mentioned data set distillation method, computer device and storage medium obtain an original data set, which includes original data and original data labels; determine the synthetic data in the synthetic data set based on the synthetic data label, the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set matches the target network model; based on the synthetic data and the first preset rule, update the model parameters of the target network model and determine the update gradient of the synthetic data; update the synthetic data set based on the original data, the update gradient, the updated target network model and the second preset rule.

[0050] The present application determines synthetic data labels based on original data labels and preset label formats, and can effectively compress the information of original data labels into synthetic data labels, and then determine the synthetic data in the synthetic data set based on the synthetic data labels, so that the synthetic data can more accurately reflect the information of the original data. Furthermore, the present application updates the model parameters of the target network model based on the synthetic data and the first preset rule, and determines the update gradient of the synthetic data, and then updates the synthetic data set based on the original data, the update gradient, the updated target network model and the second preset rule, which can further extract the information of the original data and update the information of the original data to the synthetic data set more accurately. The present application does not rely on representative samples of the original data set, and can effectively distill the information in the original data set into the synthetic data set, thereby improving the effectiveness of the synthetic data set after distillation.

[0051] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 A diagram showing an application environment of a data set distillation method in one embodiment;

[0054] Figure 2 is a schematic diagram of a process flow of a data set distillation method in one embodiment;

[0055] Figure 3 A schematic diagram of a process for determining synthetic data by random labeling in one embodiment;

[0056] Figure 4 is a structural block diagram of a data set distillation device in one embodiment;

[0057] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The terms "module", "unit", etc. used below are a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in hardware, the implementation of software, or a combination of software and hardware is also possible and conceivable.

[0060] The dataset distillation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the original data set and sends it to the server 104, and the original data set includes the original data and the original data label. The server 104 determines the synthetic data in the synthetic data set based on the synthetic data label, and the synthetic data label is determined based on the original data label and the preset label format, and the synthetic data set matches the target network model. The server 104 updates the model parameters of the target network model based on the synthetic data and the first preset rule, and determines the update gradient of the synthetic data. The server 104 updates the synthetic data set based on the original data, the update gradient, the updated target network model and the second preset rule, and sends it to the terminal 102. In other embodiments, the steps of the above-mentioned data set distillation method can also be implemented by the server 104 or the terminal 102 alone, or by data interaction between multiple servers in the server group, and this application does not make specific restrictions on this. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0061] In one embodiment, Figure 2 As shown, a dataset distillation method is provided, which is applied to Figure 1 The application scenario in is taken as an example to illustrate, including the following steps:

[0062] S201: Obtain an original data set, where the original data set includes original data and original data labels.

[0063] The dataset distillation method provided in the embodiment of the present application can be used to distill the training set dataset of the image and text recognition model, and can also be directly used in application scenarios that require compression and distillation of the dataset. Specifically, the image recognition model may include a medical image recognition model, an intelligent security image recognition model, etc., and correspondingly, the original data may include medical image data, monitoring image data, etc. Application scenarios that require compression and distillation of datasets include network text data distillation, etc., and correspondingly, the original data may include network text data. In other embodiments, the dataset distillation method provided in the present application can also be used in scenarios with a large amount of multi-label data, such as e-commerce search recommendation engines, image classification, text classification, etc., to achieve distillation of multi-label data.

[0064] In an embodiment of the present application, the original data set includes original data and original data labels. Among them, the original data set includes the data set to be distilled. In the original data set, the original data label is used to label the original data. In some embodiments, each original data can correspond to one original data label. In other embodiments, each original data can also correspond to multiple original data labels. The above embodiment can be understood as that, in the original data set, each original data label can be used to label only one original data, or can be used to label multiple original data. Among them, a certain original data corresponds to a certain data label, including that the original data belongs to the label category represented by the data label.

[0065] S203: Determine the synthetic data in the synthetic data set based on the synthetic data label, the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set matches the target network model.

[0066] In an embodiment of the present application, the synthetic data set includes a target data set for distillation. In some embodiments, the format of the synthetic data in the synthetic data set can be determined based on the original data in the original data set. Specifically, if the original data is a picture, and the length, width, and number of channels of the picture are 224, 224, and 3, respectively, then the synthetic data is also a picture with a length, width, and number of channels of 224, 224, and 3, respectively. If the original data is text, that is, a discrete sequence of words, the synthetic data can be a continuous vector sequence, and the format of the vector sequence is [127,768], where 127 represents the sequence length and 768 represents the vector dimension. In some embodiments, both the original data in the original data set and the synthetic data in the synthetic data set can be represented as feature vectors.

[0067] In the embodiment of the present application, the synthetic data in the synthetic data set is generated based on the synthetic data label, and the synthetic data label is determined based on the original data label and the preset label format. It can be understood that based on the preset label format, the label category and other information of the original data label are integrated into the synthetic data label, and then the synthetic data of the corresponding label category is generated according to the synthetic data label. In some embodiments, the preset label format may include a first category determined according to a single category of the original data label, a second category determined according to multiple categories of the original data label, a third category randomly determined according to the category of the original data label, etc.

[0068] In the embodiments of the present application, the synthetic data set matches the target network model. In some embodiments, the size and dimension of the synthetic data set can be determined according to the target network model. Reasonable setting of the synthetic data set can improve the learning efficiency of the original data set information, and reasonable setting of the size of the synthetic data set can also improve the efficiency of the distillation process and the performance after distillation.

[0069] S205: Based on the synthesized data and the first preset rule, update the model parameters of the target network model, and determine the update gradient of the synthesized data.

[0070] In the embodiment of the present application, the data set distillation needs to compress the knowledge in the original data set into the set synthetic data set, and ensure the integrity and effectiveness of the knowledge transfer as much as possible, so that the synthetic data set obtained by distillation can also be used for effective model training, or can effectively express the information of the original data. Specifically, the data set distillation can include two processes, the inner loop and the outer loop. First, the inner loop process of updating the target network model based on the synthetic data is explained.

[0071] In the embodiment of the present application, the parameters of the target network model can be updated according to the synthetic data and the first preset rule, and the update gradient of the synthetic data can be determined. The first preset rule can include a first preset batch and a first gradient descent rule.

[0072] In some embodiments, based on the synthesized data and the first preset rule, updating the model parameters of the target network model includes:

[0073] S301: Determine a first update number based on a first preset batch and the amount of synthesized data.

[0074] S303: Based on a first preset batch of synthetic data and a first gradient descent rule, updating model parameters of the target model until the number of updates reaches a first number of updates.

[0075] In the embodiment of the present application, the first preset batch includes the number of synthetic data required each time the target network model is updated. According to the first preset batch and the number of synthetic data, the first update number can be determined. Specifically, if the first preset batch is 32 and the number of synthetic data is 3200, the first update number is 100 times.

[0076] Based on the first preset batch of synthetic data and the first gradient descent rule, the model parameters of the target model are updated until the number of updates reaches the first update number. The first gradient descent rule may include stochastic gradient descent (SGD). In some specific embodiments, all parameters of the target network model may be updated according to the batch size (=32) according to stochastic gradient descent (SGD), and after traversing the synthetic data set to update the network model for the first update number, the update gradient of all synthetic data is determined and recorded.

[0077] S207: Update the synthetic data set based on the original data, the updated gradient, the updated target network model and the second preset rule.

[0078] In an embodiment of the present application, the outer loop process may include a process of updating a synthetic data set based on the original data, an updated gradient, an updated target network model, and a second preset rule. In each batch of the outer loop, the gradient of the synthetic data set is fitted based on a small batch of data of the original data set, thereby updating the synthetic data set. Specifically, the gradient fitting-based method believes that when the target network model is updated consistently using the synthetic data set S' and the original data set S (the same updated gradient), it is believed that S' can be a good substitute for S to learn the model, which means that the synthetic data set S' has effectively aggregated the discriminative feature information in the original data set S.

[0079] In some embodiments, updating the synthetic data set based on the original data, the updated gradient, the updated target network model, and the second preset rule includes:

[0080] S401: Based on the second preset batch of original data, forward propagation is performed in the updated target network model, and the original data gradient is determined.

[0081] S403: Determine the target loss based on the similarity between the original data gradient and the updated gradient.

[0082] S405: Based on the target loss and the second gradient descent rule, back propagation is performed in the updated target network model, and the distilled synthetic data is determined.

[0083] S407: Update the synthetic data in the synthetic data set based on the distilled synthetic data.

[0084] In an embodiment of the present application, for the updated target network model, forward propagation is performed using batch processing of the original data based on a second preset batch to determine the original data gradient of the original data set. The similarity between the original data gradient and the updated gradient is determined as the loss for updating the synthetic data set. Among them, the similarity can be determined based on cosine similarity or mean square error. Then, according to the second gradient descent rule and the target loss, optimization is performed, the target loss is back-propagated along the computational graph of the updated target network model, the distilled synthetic data is determined, and the synthetic data in the synthetic data set is updated based on the distilled synthetic data. Among them, the second gradient descent rule may include stochastic gradient descent.

[0085] In other embodiments, the second preset rule may also include determining the end condition of the forward propagation of the original data in the updated target network model during the original data gradient process. The end condition may include the maximum number of training times or the model convergence target, etc.

[0086] The present application determines synthetic data labels based on original data labels and preset label formats, and can effectively compress the information of original data labels into synthetic data labels, and then determine the synthetic data in the synthetic data set based on the synthetic data labels, so that the synthetic data can more accurately reflect the information of the original data. Furthermore, the present application updates the model parameters of the target network model based on the synthetic data and the first preset rule, and determines the update gradient of the synthetic data, and then updates the synthetic data set based on the original data, the update gradient, the updated target network model and the second preset rule, which can further extract the information of the original data and update the information of the original data to the synthetic data set more accurately. The present application does not rely on representative samples of the original data set, and can effectively distill the information in the original data set into the synthetic data set, thereby improving the effectiveness of the synthetic data set after distillation.

[0087] Compared with traditional data selection methods, dataset distillation, as an alternative method to reduce training datasets, can generate fewer data samples to effectively train models. However, dataset distillation methods in traditional technologies usually only focus on the distillation of single-label data and do not take into account the characteristics of multi-label data. In addition, since each generated data instance belongs to a single category, a large number of samples still need to be retained when the label space is large.

[0088] Based on the problem that the amount of data after distillation in the above-mentioned traditional technology is still large, this application takes into account that multi-label data can be annotated with multiple related labels at the same time, so the amount of data can be further compressed by generating samples by sharing multiple labels. The following embodiment of the present application describes the specific method of determining the synthetic data in the synthetic data set based on the synthetic data label in the case of multi-label data, so as to set a corresponding label for each synthetic data (that is, the information category that needs to be fused in the distillation process).

[0089] In some embodiments, determining the synthetic data in the synthetic data set based on the synthetic data label includes:

[0090] S501: Determine multiple label categories based on original data labels.

[0091] S503: Determine multiple first categories of synthetic data labels based on a single category among multiple label categories.

[0092] S505: Determine synthetic data based on multiple first categories.

[0093] In an embodiment of the present application, before determining the synthetic data, the original data set can be analyzed to statistically analyze the distribution of the original data, thereby providing a data basis for the subsequent synthetic data generation and data set distillation process, and ensuring the effectiveness and rationality of the data set distillation process. In an embodiment of the present application, by traversing the original data set, the number of original labels can be determined to clarify the size of the label space, and multiple label categories can be determined based on the original data labels. For example, the label space of the original data label can be determined to be {A, B, C, D, E}, where A, B, C, D, and E represent different label categories, respectively.

[0094] Based on a single category among multiple label categories, multiple first categories of synthetic data labels are determined. Specifically, similar to single-label data, any one of the multiple label categories is determined as the label category of each synthetic sample. For example, if the original data set includes 5 label categories A, B, C, D, and E, label category A is determined as the first category of the synthetic data label, and label categories B, C, D, and E are determined as the first category of the synthetic data label. Then, based on a first category among the multiple first categories, the synthetic data is determined. The first category can be determined based on One-hot encoding.

[0095] In some specific embodiments, a single label category is set for each synthetic data, i.e., One-hot encoding. For example, if multiple label categories are determined to be {A, B, C, D, E} based on the original data labels, and the label category of a synthetic data is set to A, then in the label vector, only the dimension corresponding to label A is set to 1, and the remaining dimensions are set to 0, i.e., y = [1, 0, 0, 0, 0].

[0096] In some embodiments, based on the plurality of first categories, determining the synthesized data includes: determining a first preset number of synthesized data matching each first category, wherein the first preset numbers corresponding to different first categories are the same.

[0097] In an embodiment of the present application, the number of synthetic data can also be set. Determine a first preset number of synthetic data that matches each first category, wherein the first preset numbers corresponding to different first categories are the same. That is, it is ensured that different first categories have the same number of samples. For example, if the first preset number is 3, then for the first category A, 3 synthetic data are set accordingly, and for the first category B, 3 synthetic data are also set, and so on. The embodiment of the present application can generate respective synthetic data for each label category, covering all label categories, and improving the accuracy and effectiveness of data set distillation.

[0098] In some embodiments, based on multiple first categories, determining the synthesized data further includes: determining a second preset number of synthesized data matching each first category, wherein the second preset number is determined based on a proportion of the single category in the multiple label categories.

[0099] In an embodiment of the present application, the same number of synthetic data may no longer be set for each first category. A second preset number of synthetic data matching each first category is determined, wherein the second preset number is determined based on the proportion of a single category in multiple label categories. Specifically, the proportion of a single category in multiple label categories includes the annotation ratio of the corresponding original labels in the original data set. For example, if the number of original data in the original data set is 100, the label space of the original data label is {A, B, C, D, E}, and the number of original data annotated with label category A is 50, then the annotation ratio is 0.5. When generating synthetic data, if 50 synthetic data need to be generated, the ratio of synthetic data corresponding to the first category A is also 0.5 of the total number of synthetic data, that is, 25 synthetic data corresponding to the first category A are set.

[0100] The embodiment of the present application sets the corresponding amount of synthetic data for different first categories according to the ratio, so that the synthetic data can reflect the data information distribution in the original data set, thereby further improving the accuracy and effectiveness of the data set distillation.

[0101] The embodiment of the present application also provides a method for generating synthetic data based on multi-label categories.

[0102] In some embodiments, determining the synthetic data in the synthetic data set based on the synthetic data label includes:

[0103] S601: Determine multiple label categories based on original data labels.

[0104] S603: Determine a plurality of second categories of synthesized data labels based on at least two of the plurality of label categories, wherein the number of the synthesized data labels is determined based on the number of the original data labels.

[0105] S605: Determine synthetic data based on the multiple second categories.

[0106] In the embodiment of the present application, the method of determining multiple label categories based on the original data label is similar to the above embodiment and will not be repeated here. Based on at least two categories of the multiple label categories, multiple second categories of the synthetic data label are determined. Still taking the label space of the original data label as {A, B, C, D, E} as an example, the second category can be determined in at least two label categories of A, B, C, D, E. For example, the second category can be A, B or B, C, D, etc.

[0107] In some specific embodiments, multiple second categories are set for each synthetic data, i.e., Multi-hot encoding. At the same time, a random combination of label categories corresponding to the second category is set, and the random combination may not conform to the label combination in the original data set. Specifically, even if there is no original data in the original data set that is labeled with label categories A and B at the same time, the second label category of the synthetic data can also be set to A and B. For example, if the label of a synthetic data is set to {A, B}, then the dimensions corresponding to label categories A and B are set to 1 in the label vector of the synthetic data, and the rest are set to 0, i.e., y = [1, 1, 0, 0, 0].

[0108] The number of synthetic data labels is determined based on the number of original data labels. Specifically, the number of synthetic data labels is determined based on the average number of labels of the original data labels. For example, the number of synthetic data labels Num∈[1,2×η], where η is the average number of labels of the original data in the original data set. If the labels corresponding to the label categories A, B, C, D, and E in the original data set are respectively annotated with 20, 30, 50, 30, and 20 original data, then the average number of labels is (20+30+50+30+20) / 5=30.

[0109] The present application also provides a method for determining synthetic data by random annotation. In some embodiments, Figure 3 As shown, determining the synthetic data in the synthetic data set based on the synthetic data label includes:

[0110] S701: Determine multiple label categories based on original data labels.

[0111] S703: Determine multiple third categories of the synthesized data label based on random probabilities within a preset probability range, where the multiple third categories include probabilities that the synthesized data label belongs to each label category.

[0112] S705: Determine synthetic data based on multiple third categories.

[0113] In the embodiment of the present application, the method of determining multiple tag categories based on the original data tag can refer to the above embodiment. Based on the random probability within the preset probability range, multiple third categories of the synthesized data tag are determined, and the multiple third categories include the probability that the synthesized data tag belongs to each tag category.

[0114] In some specific embodiments, random labeling can be performed based on the label frequency in the original data set to determine the synthetic data label. Specifically, if the preset probability range is a real number in [0,1], for each synthetic data, a real number between [0,1] is randomly set for each dimension of the label vector of its composite data label, while ensuring that the distribution of all synthetic data labels conforms to the distribution of the original data set labels. For example, the synthetic data label vector is set to y = [0.9, 0.7, 0.4, 0.2, 0.1].

[0115] In some embodiments, when the model parameters of the target network model are updated based on the synthetic data and the first preset rule, the random probability can also be updated synchronously to further improve the validity of the synthetic data set.

[0116] The embodiments of the present application can more flexibly set the synthetic data by presetting the probability range, and can aggregate the feature information of different label categories in the original data set according to certain weights, which can not only reduce the amount of generated data, but also make the synthetic data in the synthetic data set closer to the distribution of the original data, thereby further improving the accuracy and effectiveness of the data set distillation.

[0117] After determining the synthetic data in the synthetic data set based on the synthetic data label, the present application also provides a method for initializing and updating the synthetic data set to further improve the effectiveness of the synthetic data set.

[0118] In some embodiments, based on the plurality of second categories, determining the synthesized data includes: determining a third preset number of synthesized data matching each second category, wherein the third preset numbers corresponding to different second categories are the same.

[0119] In an embodiment of the present application, a third preset number of synthetic data matching each second category can be determined, wherein the third preset numbers corresponding to different second categories are the same. For example, if the third preset number is 3, then for the second categories A and C, 4 synthetic data are set accordingly, and for the second categories B, C, and D, 4 synthetic data are also set, and so on. The embodiment of the present application sets synthetic data based on multiple label categories, which can further reduce the amount of data generated and improve the compression effect of data set distillation.

[0120] In some embodiments, based on the multiple second categories, determining the synthesized data includes: determining a fourth preset number of synthesized data matching each second category, wherein the fourth preset number is determined based on a proportion of at least two categories in the multiple label categories.

[0121] In the embodiment of the present application, the number of synthetic data may not be set based on equal-number tags, and the fourth preset number of synthetic data matching each second category may be determined, wherein the fourth preset number is determined based on the ratio of at least two categories in multiple tag categories. The method for determining the ratio of at least two categories in multiple tag categories can refer to the method for determining the ratio of a single category in multiple tag categories in the above embodiment, and will not be repeated here.

[0122] The embodiments of the present application set synthetic data based on multiple label categories and set the number of labels according to a ratio. This can further compress the amount of data generated while taking into account data distribution information, while improving the compression effect of data set distillation and the effectiveness and accuracy of the synthetic data set.

[0123] In an embodiment of the present application, the second category can also be determined by determining the real label co-occurrence relationship in the original data set, thereby determining the synthetic data. In some embodiments, based on at least two categories of the multiple label categories, determining multiple second categories of synthetic data labels includes:

[0124] S801: Based on a first threshold and a label frequency corresponding to each label category, the original data labels are divided into high-frequency labels and low-frequency labels, and the label frequency is determined based on the amount of original data corresponding to the corresponding label category.

[0125] S803: Determine high-frequency co-occurring binary tags based on the tag frequencies corresponding to the tag categories of every two high-frequency tags and a second threshold.

[0126] S805: Determine high-frequency co-occurring multi-tags based on the tag frequencies corresponding to the tag categories of multiple high-frequency tags and the third threshold, and determine the co-occurring multi-tags based on the tag frequencies corresponding to the tag categories of each low-frequency tag, the tag frequencies corresponding to the tag categories of the high-frequency co-occurring multi-tags and the fourth threshold.

[0127] S807: Determine a plurality of second categories based on the tag categories of high-frequency co-occurring binary tags and the tag categories of co-occurring multi-tags.

[0128] In the embodiment of the present application, before determining the second category, the original data set may be further analyzed. Based on the first threshold and the label frequency corresponding to each label category, the original data labels are divided into high-frequency labels and low-frequency labels, and the label frequency is determined based on the amount of original data corresponding to the corresponding label category.

[0129] In some specific embodiments, the number of raw data corresponding to different label categories is determined in the raw data, and the label frequency of each raw label is calculated. The label frequency can be determined by dividing the frequency of the raw label by the total number of raw data. The raw data label whose label frequency is greater than the first threshold is determined as a high-frequency label, and the raw data label whose label frequency is not greater than the first threshold is determined as a low-frequency label. Specifically, the first threshold may include 80%.

[0130] Based on the label frequency corresponding to the label category of each two high-frequency labels and the second threshold, a high-frequency co-occurrence binary label is determined. In some specific embodiments, a binary label co-occurrence matrix is ​​established, the frequency of co-occurrence between different original data labels is counted, and the high-frequency co-occurrence labels are recorded. Specifically, the second threshold may include 1 / 2. If the frequency of co-occurrence of label category A and label category B is greater than 1 / 2 of the frequency of label category A and label category B at the same time, the two labels corresponding to label category A and label category B are determined as high-frequency co-occurrence binary labels.

[0131] Based on the label frequencies corresponding to the label categories of multiple high-frequency labels and the third threshold, high-frequency co-occurring multi-labels are determined, and based on the label frequencies corresponding to the label categories of each low-frequency label, the label frequencies corresponding to the label categories of high-frequency co-occurring multi-labels and the fourth threshold, co-occurring multi-labels are determined. Based on the label categories of high-frequency co-occurring binary labels and the label categories of co-occurring multi-labels, multiple second categories are determined.

[0132] In some specific embodiments, in the high-frequency tags, all co-occurring tag combinations are determined and the corresponding quantity is determined. Co-occurring tags include multiple original data tags that annotate one original data at the same time. For example, if the third threshold is 1 / 2, for the high-frequency tag set SHEAD = {A, B, C, D, E}, if the frequency of co-occurrence of the original data tags {A, B, C} is greater than 1 / 2 of the frequency of A, B, and C at the same time, {A, B, C} is determined as a high-frequency co-occurring multi-tag. Secondly, considering the sparsity of low-frequency tags, the low-frequency tags are combined with the high-frequency tags to determine the co-occurring multi-tags. This process is calculated independently on each low-frequency tag one by one. Specifically, if the fourth threshold is 1 / 2, for the low-frequency tag Z, check each element in the high-frequency tag set SHEAD = {A, B, C, D, E} to see if the co-occurrence frequency with the low-frequency tag Z is greater than 1 / 2 of the frequency of the tag Z appearing alone. If it is satisfied, it is considered that the high-frequency tag co-occurs with the low-frequency tag Z, and finally all high-frequency tags that co-occur with the low-frequency tag Z are determined together with the low-frequency tag Z as co-occurring multi-tags. Then, the tag category of the high-frequency co-occurring binary tag and the tag category of the co-occurring multi-tag are taken as the second category.

[0133] The embodiment of the present application determines the real label co-occurrence relationship in the original data set, determines the second category, and then determines the synthetic data, so that the synthetic data can learn more realistic original data information, further improving the accuracy and effectiveness of the data set distillation.

[0134] In some embodiments, after determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes:

[0135] S901: Determine a first eigenvector of original data corresponding to a first category.

[0136] S903: Determine an average feature vector of the first feature vectors, and update a feature vector of the synthetic data based on the average feature vector.

[0137] In an embodiment of the present application, the first feature vector of the original data corresponding to the first category can be determined. The average feature vector of the first feature feature vector is determined, and the feature vector of the synthetic data is updated based on the average feature vector. Specifically, according to the corresponding first category, the synthetic data is initialized to the feature vector of the corresponding label category in the original data set. For example, for label category A, the average feature vector of the first feature vector of all original data marked with label category A is calculated, and then used as the feature vector of label category A to update the feature vector of the synthetic data corresponding to the first category A.

[0138] The embodiment of the present application updates the synthetic data set by averaging the feature vector in the original data set, which can better utilize the discriminant information of different categories in the original data set for data distillation, and further improve the effectiveness and accuracy of the synthetic data set.

[0139] In some embodiments, after determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes:

[0140] S1001: Determine a synthetic label category of a synthetic data label, where the synthetic label category includes at least one label category.

[0141] S1003: In the original data set, determine whether there is original data corresponding to an original data label of the same category as the synthesized label.

[0142] S1005: If it exists, update the feature vector of the synthesized data based on the feature vector of the original data.

[0143] S1007: If not, then in the original data set, respectively determine the associated original data corresponding to each label category in the synthetic label category, and update the feature vector of the synthetic data based on the average vector of the feature vectors corresponding to the associated original data.

[0144] In an embodiment of the present application, a synthetic label category of a synthetic data label is determined, and the synthetic label category includes at least one label category. For example, the synthetic label category of the synthetic data X' may be {A, B, D}. In the original data set, it is determined whether there is original data corresponding to the same original data label as the synthetic label category {A, B, D}. If so, the feature vector of the original data, that is, the feature vector of the random original data labeled with the {A, B, D} label in the original data set, is used as the feature vector of the synthetic data X' to update the feature vector of the synthetic data.

[0145] If it does not exist, then in the original data set, the associated original data corresponding to each label category in the synthetic label category is determined respectively, and the feature vector of the synthetic data is updated based on the average vector of the feature vectors corresponding to the associated original data. Specifically, three feature vectors of original data labeled A, B, and D are randomly determined in the original data set, and the average feature vector of the three feature vectors is used as the feature vector of the synthetic data X' to achieve the update of the feature vector of the synthetic data.

[0146] In some embodiments, before updating the model parameters of the target network model based on the synthetic data and the first preset rule, it also includes: updating the feature vector of the synthetic data based on the normal distribution initialization rule. The embodiment of the present application can also randomly initialize the synthetic data based on the normal distribution initialization rule.

[0147] The following is a specific example to illustrate the dataset distillation method provided by this application.

[0148] Take the dataset distillation of the text multi-label dataset AAPD (Arxiv Academic Paper Dataset, a large dataset in the field of computer science with 54 labels in total) as an example.

[0149] First, analyze the original data set, traverse the original data set, and count the label space size as 54. Record the number of original data related to each original data label, and sort the original data labels in descending order according to the corresponding number of original data. Calculate the frequency of the original data label by dividing the number of original data related to the original data label by the total number of original data, and set the original data labels with the top 20% of the original data label frequency as high-frequency labels, and the remaining labels as low-frequency labels. Record the binary co-occurrence relationship and multivariate co-occurrence relationship between the original data labels, and determine the high-frequency co-occurrence binary labels and co-occurrence multivariate labels.

[0150] Set a synthetic data set. Considering that this embodiment processes a text data set, the target network model adopts a network model based on the Transformer architecture, and sets the format of each synthetic data in the synthetic data set to a vector sequence of [128,768]. Set synthetic data labels based on random probabilities within a preset probability range, and generate synthetic data so that it can be dynamically adjusted during the distillation process. Initialize the synthetic data and use a random normal distribution to initialize and update the synthetic data.

[0151] The BERT-base model based on the Transformer architecture is selected as the target network model for the distillation of the text dataset. The synthetic dataset is learned based on the gradient matching method, and the termination times of the inner loop and outer loop are set to 50 and 30 respectively. In the inner loop, the gradient update calculation graph on the entire synthetic dataset is recorded, and the target network model is updated; in the outer loop, the gradient on each batch of original data is calculated, and the synthetic data is updated by gradient matching. Iterate by batch and based on the entire original data to complete the update of the synthetic dataset until the termination condition is met.

[0152] Test the performance of synthetic datasets. Retrain a BERT-base model based on the synthetic dataset obtained by final distillation, and perform performance tests on the AAPD test set. You can also test the migration effect of different models. Based on the synthetic dataset obtained by final distillation, use different network models for training and performance tests, such as XL-Net, RoBERTa, etc.

[0153] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0154] Based on the same inventive concept, the embodiment of the present application also provides a data set distillation device 1200 for implementing the data set distillation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more data set distillation device 1200 embodiments provided below can refer to the limitations of the data set distillation method above, and will not be repeated here.

[0155] In one embodiment, Figure 4 As shown, a data set distillation device 1200 is provided, comprising:

[0156] The original data set acquisition module 1201 is used to acquire the original data set, where the original data set includes the original data and the original data label;

[0157] A synthetic data determination module 1202 is used to determine the synthetic data in the synthetic data set based on the synthetic data label, the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set matches the target network model;

[0158] A model updating module 1203 is used to update the model parameters of the target network model based on the synthetic data and the first preset rule, and determine the update gradient of the synthetic data;

[0159] The synthetic data set updating module 1204 is used to update the synthetic data set based on the original data, the updated gradient, the updated target network model and the second preset rule.

[0160] In some embodiments, the synthetic data determination module 1202 is further used to determine multiple label categories based on the original data label; determine multiple first categories of the synthetic data label based on a single category among the multiple label categories; and determine the synthetic data based on the multiple first categories.

[0161] In some embodiments, the synthesized data determination module 1202 is further configured to determine a first preset number of synthesized data matching each first category, wherein the first preset numbers corresponding to different first categories are the same.

[0162] In some embodiments, the synthetic data determination module 1202 is further configured to determine a second preset number of synthetic data matching each first category, wherein the second preset number is determined based on a proportion of a single category in a plurality of label categories.

[0163] In some embodiments, the synthetic data determination module 1202 is also used to determine multiple label categories based on the original data labels; determine multiple second categories of synthetic data labels based on at least two label categories among the multiple label categories, and the number of synthetic data labels is determined based on the number of original data labels; and determine the synthetic data based on the multiple second categories.

[0164] In some embodiments, the synthesized data determination module 1202 is further configured to determine a third preset number of synthesized data matching each second category, wherein the third preset numbers corresponding to different second categories are the same.

[0165] In some embodiments, the synthetic data determination module 1202 is further used to determine a fourth preset number of synthetic data matching each second category, wherein the fourth preset number is determined based on a proportion of at least two categories in the plurality of label categories.

[0166] In some embodiments, the synthetic data determination module 1202 is also used to divide the original data labels into high-frequency labels and low-frequency labels based on a first threshold and the label frequency corresponding to each label category, and the label frequency is determined based on the amount of original data corresponding to the corresponding label category; determine high-frequency co-occurring binary labels based on the label frequency corresponding to the label category of every two high-frequency labels and the second threshold; determine high-frequency co-occurring multi-label labels based on the label frequency corresponding to the label category of multiple high-frequency labels and the third threshold, and determine co-occurring multi-label labels based on the label frequency corresponding to the label category of each low-frequency label, the label frequency corresponding to the label category of the high-frequency co-occurring multi-label and the fourth threshold; determine multiple second categories based on the label categories of high-frequency co-occurring binary labels and the label categories of co-occurring multi-label labels.

[0167] In some embodiments, the synthetic data determination module 1202 is also used to determine multiple label categories based on the original data labels; determine multiple third categories of the synthetic data labels based on random probabilities within a preset probability range, the multiple third categories including the probability that the synthetic data labels belong to each label category; and determine the synthetic data based on the multiple third categories.

[0168] In some embodiments, the data set distillation device 1200 also includes a synthetic data initialization module, which is used to determine a first feature vector of the original data corresponding to the first category; determine an average feature vector of the first feature vectors, and update the feature vector of the synthetic data based on the average feature vector.

[0169] In some embodiments, the synthetic data initialization module is also used to determine the synthetic label category of the synthetic data label, and the synthetic label category includes at least one label category; in the original data set, determine whether there is original data corresponding to the original data label with the same synthetic label category; if so, update the feature vector of the synthetic data based on the feature vector of the original data; if not, determine the associated original data corresponding to each label category in the synthetic label category in the original data set, and update the feature vector of the synthetic data based on the average vector of the feature vectors corresponding to the associated original data.

[0170] In some embodiments, the synthetic data initialization module is further used to update the feature vector of the synthetic data based on the normal distribution initialization rule.

[0171] In some embodiments, the model update module 1203 is also used to determine a first update number based on a first preset batch and the amount of synthetic data; based on the first preset batch of synthetic data and the first gradient descent rule, update the model parameters of the target model until the update number reaches the first update number.

[0172] In some embodiments, the synthetic data set update module 1204 is further used to perform forward propagation in the updated target network model based on the second preset batch of original data, and determine the original data gradient; determine the target loss based on the similarity between the original data gradient and the updated gradient; perform back propagation in the updated target network model based on the target loss and the second gradient descent rule, and determine the distilled synthetic data; and update the synthetic data in the synthetic data set based on the distilled synthetic data.

[0173] Each module in the above-mentioned data set distillation device 1200 can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0174] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data set distillation method is implemented.

[0175] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the data set distillation method in any of the above embodiments are implemented.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data set distillation method in any of the above embodiments are implemented.

[0178] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the data set distillation method in any of the above embodiments.

[0179] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0182] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A dataset distillation method, characterized in that: The method comprises: Obtaining an original data set, wherein the original data set includes original data and original data labels; Determining synthetic data in a synthetic data set based on a synthetic data label, wherein the synthetic data label is determined based on the original data label and a preset label format, and the synthetic data set matches a target network model; Based on the synthetic data and a first preset rule, updating the model parameters of the target network model, and determining the update gradient of the synthetic data; The synthetic data set is updated based on the original data, the updated gradient, the updated target network model and a second preset rule.

2. The method according to claim 1, characterized in that Determining the synthetic data in the synthetic data set based on the synthetic data label includes: Determine a plurality of label categories based on the original data labels; determining a plurality of first categories of the synthetic data labels based on a single category among the plurality of label categories; Based on the plurality of first categories, the synthesized data is determined.

3. The method according to claim 2, characterized in that The determining the synthesized data based on the plurality of first categories comprises: Determine a first preset number of synthetic data matching each first category, wherein the first preset numbers corresponding to different first categories are the same; or; A second preset number of synthetic data matching each first category is determined, wherein the second preset number is determined based on a proportion of the single category in the plurality of label categories.

4. The method according to claim 1, characterized in that: Determining the synthetic data in the synthetic data set based on the synthetic data label includes: Determine a plurality of label categories based on the original data labels; determine a plurality of second categories of the synthesized data labels based on at least two of the plurality of label categories, wherein the number of the synthesized data labels is determined based on the number of the original data labels; determine the synthesized data based on the plurality of second categories; or; Determine multiple label categories based on the original data labels; determine multiple third categories of the synthetic data labels based on random probabilities within a preset probability range, the multiple third categories including the probability that the synthetic data labels belong to each label category; determine the synthetic data based on the multiple third categories.

5. The method according to claim 4, characterized in that The determining the synthesized data based on the plurality of second categories comprises: Determine a third preset number of synthetic data matching each second category, wherein the third preset numbers corresponding to different second categories are the same; or; A fourth preset number of synthetic data matching each second category is determined, wherein the fourth preset number is determined based on a proportion of the at least two categories in the plurality of label categories.

6. The method according to claim 4, characterized in that The determining of a plurality of second categories of the synthetic data labels based on at least two of the plurality of label categories comprises: Based on a first threshold and a label frequency corresponding to each label category, the original data labels are divided into high-frequency labels and low-frequency labels, wherein the label frequency is determined based on the amount of original data corresponding to the corresponding label category; Determine a high-frequency co-occurring binary tag based on the tag frequency corresponding to the tag category of each two high-frequency tags and a second threshold; Determine the high-frequency co-occurrence multi-label based on the label frequency corresponding to the label category of multiple high-frequency labels and the third threshold, and determine the co-occurrence multi-label based on the label frequency corresponding to the label category of each low-frequency label, the label frequency corresponding to the label category of the high-frequency co-occurrence multi-label and the fourth threshold; The plurality of second categories are determined based on the tag categories of the high-frequency co-occurring binary tags and the tag categories of the co-occurring multi-tags.

7. The method according to claim 2 or 3, characterized in that: After determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes: Determine a first feature vector of the original data corresponding to the first category; An average feature vector of the first feature vectors is determined, and a feature vector of the synthetic data is updated based on the average feature vector.

8. The method according to any one of claims 2 to 6, characterized in that: After determining the synthetic data in the synthetic data set based on the synthetic data label, the method further includes: determining a synthetic label category of the synthetic data label, the synthetic label category comprising at least one label category; In the original data set, determining whether there is original data corresponding to an original data label of the same category as the synthetic label; If it exists, updating the feature vector of the synthesized data based on the feature vector of the original data; If not, then in the original data set, the associated original data corresponding to each label category in the synthetic label category is determined respectively, and the feature vector of the synthetic data is updated based on the average vector of the feature vectors corresponding to the associated original data.

9. The method according to claim 1, characterized in that: The updating of the model parameters of the target network model based on the synthesized data and the first preset rule includes: Determining a first update number based on a first preset batch and the amount of the synthesized data; Based on the first preset batch of synthetic data and a first gradient descent rule, the model parameters of the target model are updated until the number of updates reaches the first number of updates.

10. The method according to claim 1, characterized in that The updating of the synthetic data set based on the original data, the updated gradient, the updated target network model and the second preset rule comprises: Based on the second preset batch of raw data, forward propagation is performed in the updated target network model, and the raw data gradient is determined; Determining a target loss based on the similarity between the original data gradient and the updated gradient; Based on the target loss and the second gradient descent rule, back-propagation is performed in the updated target network model, and distilled synthetic data is determined; The synthetic data in the synthetic dataset is updated based on the distilled synthetic data.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images