Unification of multiple models with individual target classes using distillation

Abstract

A computer-implemented method for generating unified soft annotations is disclosed. In the computer-implemented method, a set of samples is prepared. Then, for each sample, a plurality of predictions generated by a plurality of individually trained models are obtained, wherein each individually trained model has an individual class set to at least partially form a unified class set including a plurality of target classes. A unified soft annotation for each sample on the target class in the unified class set is estimated from the plurality of predictions using a relationship connecting a first output of each individually trained model and a second output of the unified model. The unified soft annotation is output to train a unified model having a unified class set.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning and, more particularly, to computer-implemented methods, computer systems, and computer program products for unifying multiple models having a set of individual target classes. Background Art

[0002] The success of machine learning in classification tasks has been largely enabled by the availability of large datasets. Having more training data helps improve the accuracy of classification tasks. As the technology becomes more widespread, data collection is shifting towards a more distributed setting, where data originates from multiple entities and is then combined to train a classifier in a central node.

[0003] However, in many cases, data transfer between entities is restricted due to privacy concerns or resource limitations such as network resources. In addition, due to the availability of different data, each data source may not be able to collect enough training data for all target categories. These limitations hinder the unification of knowledge from different sources.

[0004] This situation has led to multiple works proposing to learn classifiers without directly sharing data. Such works include distributed optimization, consensus-based training, federated learning, knowledge distillation (GE Hinton et al., "Distilling the knowledge in a neural network", in NIPS (Neural Information Processing Systems) Deep Learning and Representation Learning Workshop, 2015), data-free knowledge distillation for deep neural networks (RG Lopes et al., "Data-free knowledge distillation for deep neural networks", NIPS, Workshop on learning with limited labeled data, 2017), etc. However, these conventional methods assume that the target class of all models is the same. However, there are many cases where it is impossible to have the same target class for each model, for example due to data availability. Summary of the invention

[0005] Technical issues

[0006] Therefore, there is a need for a technology that can train a unified model from multiple individual models even when the individually trained models have respective partial sets of target categories that are different from the set of unified categories of the unified model.

[0007] Solution to the problem

[0008] According to an embodiment of the present invention, a computer-implemented method for generating unified soft annotations is provided. The computer-implemented method includes preparing a sample set. The computer-implemented method also includes obtaining multiple predictions generated by multiple individually trained models for each sample, wherein each individually trained model has an individual category set to at least partially form a unified category set including multiple target categories. The computer-implemented method further includes: for each sample, using a relationship connecting the first output of each individually trained model and the second output of the unified model to estimate a unified soft annotation on the target category in the unified category set from the multiple predictions. The computer-implemented method also includes outputting the unified soft annotation to train a unified model with the unified category set.

[0009] According to the computer-implemented method of an embodiment of the present invention, since the unified soft annotations to be used to train the unified model are estimated from the predictions generated by the individually trained models in a manner consistent with the output of the unified model, the knowledge of the individually trained models can be transferred to the unified model even if the individually trained models have different partial sets of target categories in the unified category set of the unified target model.

[0010] In a preferred embodiment, the computer-implemented method includes feeding each sample into a unified model to infer a predicted value on a target class in a unified set of classes as a second output. The computer-implemented method also includes updating the unified model based on a loss function between the unified soft annotation and the predicted value of each sample. The computer-implemented method also includes storing the updated unified model. Thus, the unified model trained by the computer-implemented method according to a preferred embodiment of the present invention exhibits high performance, particularly comparable to a model directly trained in a centralized and supervised manner using training data used to train individual training models.

[0011] In another preferred embodiment, the relationship indicates the equivalence between each predicted value in the first output of an individually trained model and the corresponding predicted value in the second output of the unified model normalized by the individual class set of the individually trained model. Thus, the provision of inappropriate supervision for the missing target class when predicting the individually trained model is avoided, and the performance degradation of the unified model due to such inappropriate supervision can be prevented.

[0012] In other preferred embodiments, the loss function is weighted by weighting the target classes in a unified set of classes. Each weight for one target class is calculated in a manner based on the statistics of the unified soft annotation on the one target class of the set. Thus, biases in the training data of individual training models that may be more biased towards some classes than other classes are prevented from being transferred to the unified model.

[0013] In a particular embodiment, the unified soft label is estimated by solving a problem of optimizing an objective function with respect to a distribution q corresponding to the second output of the unified model. The objective function measures the multiple reference distributions p corresponding to the multiple predictions. i and multiple normalized distributions q i ^The error between each normalized distribution q i ^ is obtained by normalizing the distribution q over the target categories in each set of individual categories.

[0014] In other specific embodiments, the objective function measures the multiple reference distributions p corresponding to the multiple predictions. i and multiple normalized distributions q i ^The error between each normalized distribution q i ^ is obtained by normalizing the distribution q over the target categories in each set of individual categories. The unified soft annotations are expressed in the form of probabilities or logits.

[0015] In another preferred embodiment, the computer-implemented method includes receiving, for each of a plurality of individually trained models, (i) the content of the individually trained model itself or (ii) a soft annotation set obtained by feeding each sample into the individually trained model and the content of the individual category set. Thus, no data transmission of training data for training the individually trained model is required. The content of the individually trained model and the soft annotation set obtained from the individually trained model are not raw data and are typically more compact than their training data. In addition, there are fewer restrictions on the model architecture of the individually trained model. Therefore, this is preferred in the presence of at least one of model architecture, computation, data availability, and confidentiality constraints.

[0016] According to another embodiment of the present invention, a computer-implemented method for training a unified model is provided. The computer-implemented method includes preparing a sample set. The computer-implemented method also includes obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, wherein each individually trained model has an individual class set to at least partially form a unified class set including the plurality of target classes. The computer-implemented method further includes updating the unified model with the unified class set using the plurality of predictions for each sample and a relationship connecting a first output of each individually trained model with a second output of the unified model on the target class in the unified class set.

[0017] According to a computer-implemented method of another embodiment of the present invention, since the unified model is updated using outputs generated by multiple individually trained models in a manner consistent with the output of the unified model, the knowledge of the individually trained models can be transferred to the unified model even if the individually trained models have different partial sets of target categories in the unified category set of the unified model. The unified model trained by the computer-implemented method of another embodiment of the present invention shows high performance, especially comparable to models that have been directly trained in a centralized and supervised manner using the training data of the individual models. In addition, it is possible to prevent the accumulation of errors when estimating unified soft annotations from propagating toward the training of the unified model.

[0018] In a specific embodiment, the computer-implemented method further includes feeding each sample into the unified model to infer a predicted value on a target class in the unified class set as a second output, wherein the predicted value is used to update the unified model. The computer-implemented method further includes storing the updated unified model.

[0019] In a further specific embodiment, the unified model comprises a neural network, and the unified model is updated by back-propagating a loss throughout the neural network, wherein the loss measures the plurality of reference distributions p corresponding to the plurality of predictions. i and the errors between a plurality of normalized distributions qi^ obtained by normalizing the distribution q of the target category in each individual category set, and the distribution q is obtained as the predicted value inferred by the unified model.

[0020] In a specific embodiment, the unified model includes a neural network, and the unified model is updated by back-propagating a loss through the neural network, wherein the loss is obtained by solving a problem of optimizing an objective function having an output vector u representing the second output of the unified model and other variables in a matrix factorization-based manner, the output vector u being obtained as the predicted value inferred by the unified model and being fixed in the problem.

[0021] Computer systems and computer program products related to one or more aspects of the present invention are also described and claimed herein.

[0022] Additional features and advantages are realized through the techniques of the present invention.Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of the invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A training scheme with a knowledge distillation system according to an exemplary embodiment of the present invention is shown;

[0025] Figure 2 A block diagram showing a knowledge distillation system and related components around the knowledge distillation system according to an exemplary embodiment of the present invention;

[0026] Figure 3 A schematic diagram intuitively depicts a probability relationship connecting the outputs of individual classifiers with the output of a unified classifier according to an exemplary embodiment of the present invention;

[0027] Figure 4 is a flow chart depicting a knowledge distillation process for training a unified classifier having multiple individual classifiers according to an exemplary embodiment of the present invention;

[0028] Figure 5 A block diagram of a knowledge distillation system according to another exemplary embodiment of the present invention is shown;

[0029] Figure 6 A block diagram showing a knowledge distillation system according to another exemplary embodiment of the present invention;

[0030] Figure 7 is a flowchart describing a knowledge distillation method for training a unified classifier having individual classifiers according to still other exemplary embodiments of the present invention;

[0031] Figure 8 shows the results of sensitivity analysis: (A) size of the unlabeled group, (B) temperature, and (C) accuracy of individual classifiers; and

[0032] Fig. 9 A schematic diagram of a computer system in accordance with one or more embodiments of the present invention is depicted. DETAILED DESCRIPTION

[0033] Now, the present invention will be described using specific embodiments, and the embodiments described hereinafter are to be understood as being merely examples and are not intended to limit the scope of the present invention.

[0034] According to one or more embodiments of the present invention, a computer-implemented method, a computer system, and a computer program product for generating unified soft annotations are provided. The unified soft annotations can be used to train a student unified model having a set of target categories (referred to as a unified category set) by using data obtained from multiple individual teacher models, wherein the multiple individual teacher models may respectively have different sets of target categories (referred to as individual category sets).

[0035] One or more other embodiments according to the present invention are directed to computer-implemented methods, computer systems, and computer program products for training a student unified model having a unified set of target categories by using data obtained from multiple individual teacher models having corresponding sets of individual categories.

[0036] First, refer to Figure 1-5 The present invention will describe a computer system and method for training a student unified model using multiple individual teacher models in a framework of knowledge distillation according to an exemplary embodiment of the present invention, wherein the student and the teacher models are classifiers with corresponding target categories, and the data obtained from the individual teacher classifiers are used to estimate a unified soft annotation for the student unified classifier.

[0037] Then, refer to Figure 6 and Figure 7 The present invention will describe a computer system and method for training a student unified model using multiple individual teacher models in the framework of knowledge distillation according to other exemplary embodiments of the present invention, wherein the student and the teacher models are classifiers with corresponding target categories, and the data obtained from the individual teacher classifiers are used to update the student unified classifier without first estimating the unified soft annotation.

[0038] In addition, referring to FIG8 , an experimental study on image classification with new knowledge distillation according to an exemplary embodiment of the present invention will be described. Fig. 9 , the hardware configuration of a computer system according to one or more embodiments of the present invention will be described.

[0039] refer to Figure 1 and Figure 2 , describes a knowledge distillation system 130 for image classification according to an exemplary embodiment of the present invention.

[0040] Figure 1 FIG. 1 shows a training scheme using the knowledge distillation system 130. Figure 1As shown, there is a central computing system 120 and a plurality of data sources 110 - 1 through 110 -N.

[0041] exist Figure 1 , the components of each data source 110 are depicted. In each data source 110, there is a training data set 112, a training module 114, and an individual classifier 116. In each data source 110, the individual classifier 116 is trained by the training module 114 using the training data set 112. Typically, each data source 110 is managed by an entity with its rights and obligations, and such entities may include private individuals, companies, enterprises, educational institutions, medical institutions, to name a few.

[0042] Each training data set 112 stores a set of training data, each of which includes an image and a category label assigned to the image. The training data stored in the training data set 112 may include privacy and / or confidential information to be stored within the data source 110. The diversity and amount of the training data stored in the training data sets 112-1 to 112-N are generally different from each other due to their data availability. Note that in the described embodiment, each training data includes an image to which a category label is assigned. However, in other embodiments with different classification tasks, the data included in each training data is not limited to images, and may be other data types, such as audio recordings for speech recognition tasks.

[0043] Each individual classifier 116 can be any one of the known classification models, including but not limited to, a classification model based on a neural network (e.g., DNN (deep neural network), CNN (convolutional neural network), RNN (recurrent neural network) and its variants), a classification model based on a decision tree or forest (e.g., random forest), a classification model based on a nearest neighbor, and a classification model based on a support vector machine (SVM) (a standard SVM can be modified to serve as an individual classifier 116). Moreover, each individual classifier 116 may not be limited to a single classification model, and may be a cluster of multiple classification models (e.g., bagging, boosting, expert mixture, etc.). The individual classifier 116 is trained by the training module 114 to predict the probability that the image belongs to each target category. These individual classifiers 116-1 to 116-N may have different architectures and may be trained to classify different sets of target categories (individual category sets), and are therefore referred to as "heterogeneous classifiers".

[0044] Each training module 114 implements a learning algorithm suitable for the corresponding architecture of the individual classifier 116 to be trained. Each training module 114 is configured to train the individual classifier 116 by using the training data stored in the training data set 112.

[0045] See also Figure 1 , components of a central computing system 120 are also depicted. The central computing system 120 includes a transition set 124, a knowledge distillation system 130, and a unified classifier 126. In the central computing system 120, the unified classifier 126 is trained by the knowledge distillation system 130 using the transition set 124. The central computing system 120 is managed by an entity that may be different from the entity of the data sources 110-1 to 110-N, and therefore, data transfer from the training data sets 112-1 to 112-N to the central computing system 120 is generally restricted.

[0046] The purpose of the novel knowledge distillation according to an exemplary embodiment of the present invention is to unify the knowledge of data sources 110-1~110-N into a single unified model without transmitting training data distributed across multiple data sources 110-1~110-N, which may cause privacy and / or confidentiality issues.

[0047] In this training scheme, in order to integrate the knowledge distributed on multiple data sources 110-1 to 110-N, each entity is only requested to forward its trained individual classifiers 116-1 to 116-N and their class names to the central computing system 120. In the central computing system 120, the knowledge distillation system 130 receives the transferred individual classifiers 122-1 to 122-N and runs them on its resources. Then, the knowledge distillation system 130 trains the unified classifier 126 using the transferred individual classifiers 122-1 to 122-N by utilizing the transferred collection 124.

[0048] Since the knowledge of the data sources 110-1~110-N is accumulated in the trained individual classifiers 122-1~122-N, the distributed knowledge can be unified into a single unified classifier 126 so that it can classify all target categories of all individual classifiers 116-1~116-N.

[0049] Typically, the content of an individual classifier 116 is more compact than the entire training data stored in the training data set 112. Therefore, this training approach is preferred in situations where there are resource constraints, such as bandwidth constraints (eg, very large data sets).

[0050] Figure 2 1 shows a block diagram of the knowledge distillation system 130 and related components around the knowledge distillation system 130. Figure 2 As shown, the knowledge distillation system 130 includes a soft annotation generation module 132 and a unified model training module 134. There is a unified classifier 126 to be trained as a student model and a plurality of individual classifiers 122-1 to 122-N that have been trained as teacher models.

[0051] The transfer set 124 stores a set of transfer data, each of which may include an image when classified for an image. Transfer data is data used to transfer knowledge to a unified classifier 126. Transfer data (or samples) are extracted from the transfer set 124 and fed as input to each of the teacher and student classifiers 122, 126. In a specific embodiment, the images stored in the transfer set 124 do not need to be assigned hard annotations and may not include privacy and confidential information. Any image in a public image dataset and / or other image datasets collected for new knowledge distillation can be used as transfer data. Note that in the described embodiment, each transfer data includes an image. However, in other embodiments with different classification tasks, each transfer data may include other data types, such as audio recordings for speech recognition tasks.

[0052] The unified classifier 126 can be any of the following: a known neural network-based classification model (e.g., DNN, CNN, RNN, and variants thereof), a decision tree or forest-based classification model (e.g., random forest), a nearest neighbor-based classification model, and an SVM-based classification model (a standard SVM can be modified to be used as the unified classifier 126). The unified classifier 126 is trained by the knowledge distillation system 130 to predict the probability that an image belongs to each target category. The unified classifier 126 can have an architecture different from that of the individual classifiers 122-1 to 122-N, and can be trained to classify a target category set (unified category set) different from the classifiers in the individual classifiers 122-1 to 122-N. The unified category set can be the union of the individual category sets of the individual classifiers 122-1 to 122-N, and thus includes all members in the individual category sets.

[0053] The knowledge distillation system 130 is configured to prepare a transfer set 124 and feed each transfer data extracted from the transfer set 124 into individual classifiers 122-1 to 122-N. Each individual classifier 122 infers the probability that the image of the transfer data belongs to each target category and generates a prediction as an output. The prediction may include a probability or a logit on the target category in its individual category set.

[0054] The soft annotation generation module 132 is configured to obtain predictions generated by individual classifiers 122-1 to 122-N for each transferred data (or sample). The multiple predictions obtained from multiple individual classifiers 122-1 to 122-N are used as primary data to generate unified soft annotations for knowledge distillation. The soft annotation generation module 132 is also configured to estimate, for each transferred data (or sample), a unified soft annotation on a target category in a unified category set based on the predictions using the probabilistic relationship newly derived herein. The probabilistic relationship connects or associates the output of each individual classifier 122 and the output of the unified classifier 126. More details about the probabilistic relationship will be described later. After the soft annotation generation module 132 obtains the unified soft annotation, it outputs the unified soft annotation to the unified model training module 134. The unified soft annotation is called a "soft" annotation because the category identification is not as determined as a hot hard annotation.

[0055] The unified model training module 134 implements a learning algorithm suitable for the corresponding architecture of the unified classifier 126, and is configured to train the unified classifier 126 by using the unified soft annotation estimated for each transmission data in a supervised manner. When a neural network-based model is adopted as the unified classifier 126, a back propagation algorithm can be used.

[0056] More specifically, the unified model training module 134 is configured to feed each transfer data (or sample) into the unified classifier 126 to infer the probability that the image of the transfer data belongs to each target category and obtain an output. The output of the unified classifier 126 may include the probability or logit of the target category in the unified category set. The unified model training module 134 is configured to further iteratively update the unified classifier 126 (its parameters) based on the loss function between the unified soft annotation of each sample and the output of the unified classifier 126. When the training process is completed, the unified model training module 134 stores the unified classifier 126 in an appropriate storage device.

[0057] In the following, further reference is made to Figure 2 , describes more details about new knowledge distillation. Here, new knowledge distillation using heterogeneous classifiers is defined as follows. Let U be the transfer set 124 with the set of unlabeled images. Let C = {C 1 , C 2 , ..., C N} is a set of N individual classifiers 122-1 to 122-N. The input image x is pulled from the transition set 124 (U) and input to the individual classifiers 122-1 to 122-N {C 1 , C 2 , ..., C N}, where each individual classifier 122 (C i) has been trained to predict that an image belongs to a set of individual categories L i Class I j The probability p i (Y=l j ).

[0058] Given U and C, the goal is to train an estimate of U Class I j The probability q(Y=l j ) unified classifier 126 (C U ),in:

[0059] Formula 1

[0060]

[0061] Note that individual classifier 122 (C i ) can be trained to classify different sets of target categories, i.e., when i is not j, so an individual category set L i May not be equal to other individual category sets L j and / or even a set of individual classes |L i The number of categories in | may not be equal to the number of other individual category sets |L j The number of categories in |.

[0062] Generally, knowledge distillation is a method for transferring multiple teacher models C from an unlabeled dataset U. i Compressed into a single model C U The standard distillation corresponds to the same set of individual categories. The case and student model C U The teacher model C can be minimized i Output C i and Student Model C i The cross entropy between the output q is trained as follows:

[0063] Formula 2

[0064]

[0065] Essentially, the teacher model C i The output p i In training the student model C U is used as the soft label of the unlabeled dataset U. For neural networks, the probability is usually calculated using the softmax function as follows:

[0066] Equation 3

[0067]

[0068] where z l is the logit of class l (activation before softmax), and T represents an adjustable temperature parameter. When T is suitably high, minimizing the cross entropy shown in Equation (1) is similar to minimizing p i The L2 error between the logits of q and q, thus associating cross entropy minimization with logit matching.

[0069] However, the main problem with standard distillation stems from its inability to handle the more general case where one set of individual categories Li may not be equal to another set of individual categories L j The above formula (1) assumes that the student model C U and teacher model C i share the same set of target categories (L i =L j ,=L U ).

[0070] However, in heterogeneous classifiers, each teacher model C i are trained to predict the set of individual categories L i So for the categories included in the set L- i l, p i (Y = l) is undefined, the set L- i Defined as a unified class set L U Intrinsic but in the individual category set and L i For the teacher model C i and Student Model C U A simple solution to the inconsistency between the outputs is to simply include the i lSet p in i (Y = l) = 0. However, this obvious solution may lead to errors, for example, when the teacher model C i Without classifying cats, one might consider the set of “cat” images p i (Y=cat) is set to zero, which would be an inappropriate supervision.

[0071] To overcome the limitation of standard distillation, we introduce a i )'s output p i and unified classifier 126 (C U )’s output q. Since the individual category set L i is a unified category set L U Subset of, assuming that Y cannot be in L- i Then the probability p i(Y=1) is regarded as the probability q that Y=1. This leads to the following derivation:

[0072] Formula 4

[0073]

[0074] Formula (6) represents an individual classifier 122 (C i ) output for each predicted value p i (Y=1) and the unified classifier 126 (C U The equivalence between the corresponding predicted values ​​q(Y=l) in the output of ) is determined by the individual training model 122 (C i ) of the individual category set L i Normalization.

[0075] Figure 3 Describes the connection of individual classifiers 122 (C i ) and the output of the unified classifier 126 (C U ) is a schematic diagram of the probability relationship of the output. Figure 3 As shown, we can transform the individual classifier 122 (C i ) is regarded as the output unified classifier 126 (C U ) part class.

[0076] In the new knowledge distillation according to one or more embodiments of the present invention, based on this probability relationship, two types of techniques are provided, including a technique based on cross entropy minimization and a technique based on matrix factorization with missing entries. Both techniques allow us to U A unified soft label q is estimated from the unlabeled samples x on all categories in , and the unified soft label q is used to train a unified classifier 126. More details about the cross-entropy based technique and the matrix factorization based technique will be described later.

[0077] In a specific embodiment, Figure 1 Each of the modules 122, 126, 130 of the central computing system 120 described in Figure 2 Each of the sub-modules 132 and 134 of the knowledge distillation system 130 shown in can be, but is not limited to being implemented as a software module including program instructions and / or data structures in combination with hardware components such as processing circuits (e.g., CPU (central processing unit), GPU (graphics processing unit), FPGA (field programmable gate array)), memory, etc.; as a hardware module including electronic circuits (e.g., neuromorphic chips); or as a combination thereof.

[0078] Figure 1 and Figure 2These modules 122, 126, 130, 132, and 134 described in the can be implemented on a single computer system (such as a personal computer and a server machine) or on a computer system distributed across multiple computing devices (such as a computer cluster of computing devices). The contents of the transfer set 124 and the individual classifiers 122-1 to 122-N can be stored in any internal or external storage device or medium accessible to the central computing system 120.

[0079] same, Figure 1 Each of the modules 114, 116 in the data source 110 described in the accompanying drawings may be implemented, but is not limited to, as a software module; as a hardware module; or as a combination thereof. Figure 1 These modules 114, 116 described in can be implemented on a single computer system or a computer system distributed across multiple computing devices. The training data set 112 can be stored in any internal or external storage device or medium accessible to the computer system implementing the module 114.

[0080] Note that the individual classifier 122 does not have to be local to the central computing system 120. It is sufficient if the individual classifier 122 is available over a network. Thus, preparing the individual classifier 122 means making the individual classifier 122 available by reading the individual classifier 122 onto the memory space of the local computer system; or establishing a connection with the individual classifier 122 operating on a remote computer system so that the transition data (sample) can be fed into the individual classifier 122 and the resulting prediction of the transition data (sample) can be received from it.

[0081] Furthermore, in one or more embodiments, the individual classifiers 122 do not even need to be accessed from the central computing system 120. In this case, the contents of the transfer set 124 are given to each data source 110, and the entity is requested to forward the obtained soft annotation set and its class name by feeding each sample in the transfer set 124 into the individual classifier 122. Then, the knowledge distillation system 130 receives the soft annotation set and the class name.

[0082] refer to Figure 4 , depicts a flow chart depicting the knowledge distillation process for training the unified classifier 126. Figure 4 As shown, the method may start at step S100 in response to receiving a request from an operator to start the knowledge distillation process. The individual classifiers 122-1 to 122-N may be specified in the request. 1 ,C 2 ,...,C N}, unified classifier 126 (C U ) configuration and transmission set 124 (U). Note that Figure 4The processes shown in may be performed by a processing circuit, such as one or more processing units in central computing system 120 .

[0083] In step S101, the processing circuit may prepare the transfer set 124 (U) and the trained individual classifiers 122-1 to 122-N {C 1 ,C 2 ,...,C N In step S102, the processing circuit may initialize the unified classifier 126. Each individual classifier 122 prepared in step S101 has a corresponding individual category set L i The unified classifier 126 prepared in step S102 has as its individual class set L i The unified category set L of the union of U .

[0084] In step S103, the processing circuit may read sample x from the transition set 124 (U). In step S104, the processing circuit may feed sample x to each individual classifier 122 (C i ) to obtain multiple predictions p i In step S105, the processing circuit may use the above probability relationship to predict p i Estimate the uniform category set L U The unified soft annotation q on all object categories in .

[0085] Cross-entropy based methods

[0086] In the following, a first method for estimating a unified soft annotation q based on cross entropy minimization is described. Based on the probability relationship (6), the aforementioned cost function (1) can be summarized as the following cost function:

[0087] Formula 5

[0088]

[0089] in

[0090]

[0091] The difference between the above formulas (1) and (7) lies in the normalization of the probability q. Specifically, in the individual category set L i The probability p of the target class in i (Y = l) and the normalized probability q i ^(Y=l) between which each individual classifier 122 (C i ). According to formula (7), for the missing target category L -i , there is no need to arbitrarily define p i (Y=l) value.

[0092] Let us assume that the uniform category set L U All categories in have probability p - ,in:

[0093] Formula 6

[0094]

[0095] Then q=p - is the global minimum of the cost function (7). This establishes the form of the global minimum of the cost function (7), if it exists, minimizing the cost function (7) can give the true potential probability p - .

[0096] When a probability distribution q is given, the above cost function (7) measures multiple reference probability distributions p i (Y=1), each of which corresponds to an individual classifier 122 (C i ) and multiple normalized distributions q i ^(Y=l), where each is obtained by i The optimal distribution q is obtained by normalizing the given probability distribution q on the target category in L. By minimizing the above cost function (7) on the probability distribution q, the optimal distribution q is obtained as U In the described embodiment, a cost function is used as the objective function, and thus the problem of optimizing the objective function is to minimize the cost function.

[0097] Minimizing the above cost function (7) can be transformed into a geometric program, which can then be converted into a convex problem and solved efficiently. In short, the new temporary variable u l (∈R) is given to be included in the unified class set L U For each class l in , and each temporary variable u l By probability q(Y=l) (i.e., u l =log q(Y=1)). Therefore, in a specific embodiment, the above formula (7) can be transformed into the following formula:

[0098] Formula 7

[0099]

[0100] In the temporary variable {u l} l is convex in the case of l} lThe scaled and exponential sums are summed to find the logarithms. Equation (9) can be minimized by using an appropriate method such as gradient descent. Since the probability distribution q is determined by the temporary variables u in the convex problem l (i.e. exp(u l )) is a set representation of the exponential function, so once the optimal temporary variable {u l} l , the optimal temporary variable {u l} l The soft maximum function (2) is transformed into q in the form of probability. The probability distribution q, and the corresponding unified soft labeling, can be obtained by solving this convex problem. Although the cross entropy function is used as the objective function in the described embodiment, in other embodiments, the Kullback-Leibler (KL) divergence can also be used as the objective function, because the cross entropy and KL divergence are equivalent in the sense that they will provide the same solution when they are optimized, even though the KL divergence and the cross entropy are generally different.

[0101] Methods based on matrix factorization

[0102] In the following, a second class of methods for estimating unified soft annotations q based on matrix factorization with missing entries is described. The problem of unifying heterogeneous classifiers can be cast as the problem of filling in an incomplete soft annotation matrix. In the following, a method for estimating unified soft annotations q based on matrix factorization with missing entries is described. i} i Three ways to recover the unified soft annotation q from the set of .

[0103] Matrix Factorization in Probability Space

[0104] Let us consider the probability matrix P∈[0,1] LxN , where if l is included in the individual category set L i In, P li (element in row l and column i) is set to p i (Y = 1), otherwise zero (0). The probability matrix P represents the prediction p in the form of probabilities with missing entries i To account for these missing predictions, we define a mask matrix M∈{0,1} LxN , where M li (element in row l and column i) if l is included in the individual category set L i The mask matrix M represents the individual category set L. i There are missing categories in .

[0105] Using p in formula (6) iThe probability relationship between q and q, the probability matrix P can be factorized into the masked product of vectors as follows:

[0106] Formula 8

[0107] M⊙P=M⊙(uv T )(10)

[0108]

[0109] One of the circled points represents the Hadamard product. Here, vector u is the output vector containing the distribution q expressed in probability form, and each element in vector v contains the probability of each individual training model (C i ), so the vector v is called the normalized vector v.

[0110] In this form, the probability output vector u can be estimated by solving the following rank-1 matrix completion problem:

[0111] Formula 9

[0112]

[0113] subject to you T 1 L =1 (13)

[0114] v≥0 N ,u≥0 L (14)

[0115] where ||.|| F represents the Frobenius norm, and 0 k and 1 k represents the zero vector and a unit vector of magnitude k.

[0116] Here, formula (12) represents the problem of optimizing an objective function with respect to an output vector u and a normalized vector v as variables, wherein the objective function measures the error between a probability matrix P with missing entries and the product of the output vector u and the normalized vector v using a mask matrix M. Constraint (14) ensures that the normalized vector V is non-negative. Constraints (13, 14) ensure that the output vector u is a probability vector. This formula can be considered as a non-negative matrix factorization, which can be solved by using, for example, the least squares (ALS) method, where the output vector u is normalized to sum to 1 in each iteration. Due to the gauge degrees of freedom, this normalization in the vector u does not affect the cost function.

[0117] Matrix Factorization in Logarithmic Space

[0118] As described above, in minimizing the cross entropy with the individual classifier 122 (C i ) and unified classifier 126 (C U ) between the L2 distances of the logarithms of z. Therefore, low-rank matrix factorization can be applied to logarithms. This formula is actually a generalization of bisection logit matching. Let z l i is the individual classifier 122 (C i ), and u is the logit output vector to be estimated. For algorithms other than neural networks, z l i =log p i (Y = l) to obtain logit from the probability. Let us consider the logit matrix Z∈R LxN , where if l is included in the individual category set L i In, Z li (Line Z li and the elements in column i) are z l i , otherwise zero (0). Estimated logit output vector u∈R L The problem can be formulated as follows:

[0119] Formula 10

[0120]

[0121] subject to v≥0 N (16)

[0122] where the vector c∈R N The shift in the logit is then called the logit shift vector c, and r∈R is a hyperparameter that controls the regularization.

[0123] The shift of the logit values ​​has no effect on the probability output, but the different shifts of the individual classifiers are handled to model it as a matrix factorization. Here, the optimized scaling vector v∈R N Similar to optimizing the classifiers 122 (C i ), and the logit scaling vector v is constrained to be non-negative to prevent logit sign flipping that could affect the probability.

[0124] Formula (15) represents the problem of optimizing the objective function with respect to the output vector u represented in logit form, the logit scaling vector v and the logit shift vector c as variables, wherein the objective function uses the mask matrix M to measure the error between the logit matrix Z with missing entries and the product of the output vector u and the logit scaling vector v shifted by the logit shift vector c. The problem defined by the above formula (15) can be solved by using ALS on the vectors u, v and c. Here, there is no constraint on the output vector u, so the output vector u is not normalized. The unified soft annotation q in the probability space can be obtained from the output vector u using the above softmax function (2), where u l and q l Considered as z l and p(Y=l). By recovering q from u, the obtained soft annotation q can be used to train a unified classifier that accepts soft annotations. In other embodiments, the obtained output vector u can be directly used to train a unified classifier that can output logits (e.g., a neural network with the final softmax layer removed) as other types of soft annotations. Although the logits in the output vector u are different from the probabilities q in the representation, the unified soft annotations described herein contain values ​​in the logit space.

[0125] Matrix factorization in logarithmic space with fixed scaling vector V

[0126] Although setting the vector v as a variable allows the above formula (15) to handle different scaling of the logit. However, uv T The gauge degrees of freedom of can lead to arbitrary scaling in u as follows:

[0127] Formula 11

[0128] uv T =(u / α)(αv T )for a≠0.

[0129] Next, although regularization helps prevent the norms of vectors u and v from being too large, it is difficult to set a single regularization hyperparameter r that is effective for all data in the transmission set 124 (U). To address these issues, an alternative formulation of (15) can be proposed, in which the scaling vector v is fixed to 1 N . When the scaling factor v is fixed, there is no need to regularize the vector u since its scale is determined by Z. Moreover, the new formula is convex and can be solved to a global optimum. Estimating the logit output vector u∈R with a fixed scaling vector L The problem can be stated as follows:

[0130] Formula 12

[0131]

[0132] Formula (17) represents the problem of optimizing an objective function with respect to an output vector u and a logit shift vector c expressed in logit form, wherein the objective function measures the error between a logit matrix Z with missing entries and the product of an output vector u and a fixed scaling vector shifted by the logit shift vector c using a mask matrix M. The problem defined by the aforementioned formula (17) can be solved using an appropriate method, such as a gradient descent method on vectors u and c. In addition, a unified soft annotation q in a probability space can be obtained from the output vector u in the logit space using a softmax function (2). Alternatively, the obtained output vector u can be directly used to train a unified classifier that can be used as a soft-annotated output logit of other types.

[0133] Return to reference Figure 4 At step S106, the processing circuit may feed the same sample into the current unified classifier 126 to infer the output p U In step S107, the processing circuit may calculate the value of the unified soft label q based on the output p of the current unified classifier 126. U The loss function between is used to update the current unified classifier 126.

[0134] In a particular embodiment, the loss function used to train the unified classifier 126 is expressed as the following cross entropy function:

[0135] Formula 13

[0136]

[0137] Where q represents the unified soft annotation used as pseudo annotation estimated in step S105, and p U represents the output probability of the categories in the unified category set of the current unified classifier 126 with parameters θ. In an exemplary embodiment, since the unlabeled data is used as transfer data to unify the individual classifiers 122, no labor is required to label any data at the central computing system 120. However, when hard labels assigned to the transfer data are available, the hard labels can be used to update the parameters θ of the unified classifier 126 during the training process in an alternating or simultaneous manner with the soft labels.

[0138] At step S108, the processing unit may determine whether an exit condition is satisfied. For example, when a predetermined number of time periods have been completed, it is determined that the exit condition is satisfied. If at step S108 the processing unit determines that the exit condition is not yet satisfied, the processing may loop back to step S103 for further iterations. On the other hand, if at step S108 the processing unit determines that the exit condition has been satisfied, the processing may proceed to step S109. At step S109, the processing unit may store the current unified classifier 126, and the process may end at step S110. Note that iterations of training may be performed for each transfer example (e.g., online training) or a predetermined set of transfer examples (e.g., batch training, mini-batch training).

[0139] In the following, reference is made to Figure 5 , describes a knowledge distillation system according to other exemplary embodiments of the present invention. Figure 5 A block diagram of a knowledge distillation system according to other exemplary embodiments of the present invention is described. Figure 2 The embodiment shown is Figure 5 The difference between the illustrated embodiments is the presence of a soft annotation memory 136 and a balancing module 138 in the knowledge distillation system 130 .

[0140] As described above, the unified soft label q is a set of multiple predictions {p i} i estimated and used to train the unified classifier 126 (C U ). However, in some cases, the set of estimated soft labels q from the entire set of transitions 124 (U) may be unbalanced. That is, the estimated unified labels q may be more biased towards some classes than other classes, which may result in training the unified classifier 126 (C U ) tends to be biased towards the same set of categories. To counteract this bias, when using a unified soft annotation q to train the unified classifier 126 (C U ), a weight may be applied to the cross entropy loss, wherein the weight of each class l is calculated in a static manner based on the probability q(Y=l) over all data from the transition set 124(U). The inverse of the average of the probability q(Y=l) over all data may be used as the weighted static.

[0141] In the present exemplary embodiment, the soft annotation generation module 132 is further configured to store the unified soft annotation for each transmission data in the soft annotation storage 136. After obtaining the soft annotation for all transmission data in the transmission set 124 (U), the balancing module 138 is configured to calculate the static of the probability q (Y = 1) on all data from the transmission set 124 (U) and save the static as a weight. The unified model training module 134 is used to weight the loss function by weighting the target class in the unified class set when training the unified classifier 126.

[0142] According to the above embodiment, the unified soft label q is first estimated and used to train the unified classifier 126 in a supervised manner. U ) is a neural network, there is an alternative approach in which the loss of estimating q is directly back-propagated throughout the neural network without having to first estimate q.

[0143] In the following, reference Figure 6 and Figure 7 , describes knowledge distillation according to yet other exemplary embodiments of the present invention, in which the unified classifier 126 is updated using data obtained from the individual classifiers 122 - 1 to 122 -N without first estimating a unified soft label q.

[0144] Figure 6 A block diagram of a knowledge distillation system according to another exemplary embodiment of the present invention is described. Figure 2 The embodiment shown is Figure 6 The differences between the embodiments shown are Figure 2 The soft annotation generation module 132 is shown omitted and the updating module 140 is incorporated into the unified model training module 134. Also note that the unified classifier 126 is a neural network based model.

[0145] The updating module 140 is configured to obtain the prediction p generated by the individual classifiers 122-1 to 122-N for each transfer data (or sample). i The update module 140 is also configured to feed each sample into the unified classifier 126 to infer the predicted value of the target class in the unified class set. The update module 140 is further configured to update the unified classifier 126 (its parameters θ) using the predicted value of the unified classifier 126 of each sample having the above-mentioned probability relationship and the predictions of the individual classifiers 122-1 to 122-N, which is obtained by the unified class set L U The target category in connects the outputs of the individual classifiers 122-1 to 122-N and the output of the unified classifier 126. When the training process is completed, the unified model training module 134 stores the unified classifier 126 in an appropriate storage device.

[0146] refer to Figure 7 , depicts a flow chart for describing a knowledge distillation process according to another exemplary embodiment. Figure 7 As shown, the method may start at step S200 in response to receiving a request from an operator to initiate a knowledge distillation process. Figure 7 The processes shown in may be performed by a processing circuit, such as one or more processing units in central computing system 120 .

[0147] In step S201, the processing circuit may prepare the transfer set 124 (U) and the individual classifiers 122-1 to 122-N {C 1 ,C 2 ,...,C N}. In step S202, the processing circuit may initialize the unified classifier 126. In step S203, the processing circuit may read the sample x from the transition set 124 (U). At step S204, the processing circuit may feed the sample x to each individual classifier 122 (C i ) to obtain multiple predictions p i At step S205 , the processing circuit may feed the same sample into the current unified classifier 126 to infer the output q.

[0148] At step S206, the processing circuit may generate a prediction result based on the prediction result from each classifier 122-1 to 122-N. i The current unified classifier 126 is updated based on the above probability relationship and the loss function of the output q of the current unified classifier 126 on all target categories.

[0149] In the case of the cross-entropy based approach, q in formula (7) is considered as the sum of the current unified classifier 126 (C U ) output probability, then the loss can be directly back-propagated through. In step S206, the loss is propagated back through the unified classifier 126 (C U ) neural network to update the current unified classifier 126, the loss function (7) measures the reference distribution p i The normalized distribution q obtained by normalizing the distribution q on the target category in each set of individual categories is i The error between ^, where the distribution q is obtained in step 205 as the predicted value inferred by the unified classifier 126. The loss function (7) is used to update the parameter θ by back propagation using the form of the gradient of the loss function.

[0150] In the case of the matrix factorization method, the output vector u is considered to be from the unified classifier 126 (C U) is a probability vector or logit output of the unified model. Once the output vector u is obtained, the output vector u is inserted into each formula (12), (15) or (17), and the problem is solved for the other variables (e.g., v and c) using the fixed output vector u, and then the loss is back-propagated via the output vector u. The unified classifier 126 is updated by back-propagating the loss throughout the neural network, where the loss is obtained by solving the problem of optimizing the objective function with the output vector u and other variables (e.g., v and c) of the unified model in a matrix factorization-based manner. The output vector u is obtained as the predicted value inferred by the unified classifier 126 and fixed in the problem. For example, let us consider formula (15). In this case, after obtaining the output vector u from the unified classifier 126 (C U ) After obtaining the output vector u, the value of the output vector u in formula (15) is fixed, and then the objective function represented by formula (15) is optimized for the logarithmic scaling vector v and the logarithmic shift vector c until their optimal values ​​are obtained. Then, the vectors v and c are fixed, and the gradient of formula (15) is calculated with respect to the output vector u. Then, the gradient can be used to back-propagate the unified classifier 126 (C U ).

[0151] In both approaches, directly back-propagating the loss helps separate the step of estimating q and using the estimated q to train a unified classifier 126 (C U ) steps into a single step. When moving from the unified classifier 126 (C U )’s training propagation estimates q, which prevents the accumulation of errors.

[0152] In step S207, the processing unit may determine whether an exit condition is satisfied. If, in step S207, the processing unit determines that the exit condition is not satisfied, the process may loop back to step S203 for further iterations. On the other hand, if, in step S207, the processing unit determines that the exit condition is satisfied, the process may proceed to step S208. In step S208, the processing unit may store the current unified classifier 126, and the process may end in step S209.

[0153] According to one or more embodiments of the present invention, the knowledge of the individually trained models (individual classifiers 122-1 to 122-N) can be transferred to a unified model (unified classifier 126) even if the individually trained models have different partial sets of target categories. The unified soft labels are estimated from the predictions generated by the individually trained models in a manner consistent with the output of the unified model.

[0154] Furthermore, a unified model trained using unified soft annotations shows high performance, in particular, comparable to models that have been trained directly with training data for training individual training models in a centralized and supervised manner, as will be described in the following experimental section. In contrast to ensemble methods, individual training models used to train the unified model are no longer required for inference, whereas ensemble methods require storing and running all models, which may lead to scalability issues when using complex models. Training new models allows for reduced computational costs compared to ensemble methods. Furthermore, by introducing probabilistic relationships, the provision of inappropriate supervision for missing target categories when predicting the individually trained models is avoided, and degradation of the performance of the unified model due to such inappropriate supervision can be prevented.

[0155] The cross entropy method and matrix factorization method described above provide a method for obtaining the predictions p from the individual classifiers 122. i Find a practical solution to the unified soft annotation q.

[0156] Furthermore, in order to perform new knowledge distillation, no data transfer of the training data used to train the individual training models is required. The content of the individual trained models and the soft annotation sets obtained from the individual trained models are not raw data and are generally more compact than their training data. Furthermore, there are fewer restrictions on the model architecture of the individual trained models. Therefore, this is preferred in the presence of at least one of model architecture, computation, data availability, and confidentiality constraints.

[0157] There may be practical applications for situations where it is not possible to force every entity to use the same model / architecture, to collect sufficient training data for all categories, or to send data to the central node due to computation, data availability, and confidentiality constraints.

[0158] According to the above embodiments, the new knowledge distillation has been described in the context of image classification. However, the classifier to be trained by the new knowledge distillation according to one or more embodiments of the present invention is not limited to image classifiers. In one or other embodiments, many applications can be conceived, including speech recognition, handwriting recognition, medical image analysis, pattern recognition, to name a few.

[0159] Having described advantages achieved with respect to one or more specific embodiments according to the present invention, it should be understood that some embodiments may not have these potential advantages, and that these potential advantages are not necessarily required of all embodiments.

[0160] Experimental studies

[0161] Reference to exemplary embodiments Figures 1 to 7 The series describes the implementation of the knowledge distillation system and the procedures for knowledge distillation process are coded and executed.

[0162] The main experiments on ImageNet, LSUN and Places 365 datasets will be described. In the following, the following abbreviations are used to represent the above methods: SD stands for the first extension of standard distillation (Comparative Example 1), in which the images contained in the set L -i . In l, simply set p i (Y=l)=0, set CE-X for the cross-entropy based method (Examples 1-3); set MF-PX for the matrix factorization on probability (Examples 4-6); and set MF-L for the matrix factorization on logit for using non-fixed (Examples 7-9) and fixed v (Examples 10-12), respectively. U -X and MF-LF-X. In the above abbreviations, if the unified soft annotation q is used to train the unified classifier 126 (C U ) before estimating the unified soft label q, the suffix 'X' is replaced by "E" (instances 1, 4, 7, 10); if direct backpropagation is performed from the loss function, it is replaced by "BP" (instances 2, 5, 8, 11); and when training the unified classifier 126 (C U ), if the unified soft annotation q is balanced to be static by the inverse of the average value of the probability q (Y=l), it is replaced by "BS" (Examples 3, 6, 9, 12). In addition to the above methods, SD-BS is included as an SD method with balanced soft annotation (Comparative Example 2), and as a method using all individual classifiers 122 (C i ), the SPV of the method directly trained in a supervised manner (Comparative Example 3) is included as a baseline. For the MF-LU-X method, the hyperparameter r=0.01 is used. All methods use temperature T=3 to smooth the soft annotations and logits.

[0163] First, we will describe the experimental protocol, details about the dataset, and the classifiers used as individual classifiers 122 (C i ) and unified classifier 126 (C U ) architecture and individual classifier 122 (C i Then, the results will be discussed.

[0164] Experimental protocol:

[0165] Datasets: Three datasets are used. The first dataset is ImageNet (ILSVRC2012). This dataset contains 1k classes, each with approximately 700 to 1300 training and 50 validation images, and 100k unlabeled test images. In this experiment, the training images are used as the individual classifiers 122-1 to 122-N (C i), unlabeled test images are used as transfer set 124 (U), and validation images are used as test set to evaluate accuracy. The second dataset is LSUN, which contains 10 classes with approximately 100k to 3M training images (depending on the class) and 300 validation images per class with 10k unlabeled test images. Here, a set of 1k training images is randomly sampled per class to train individual classifiers 122 (C i ), a second randomly sampled set of 20k images per class also from the training data is used as the transfer set 124 (U), and the validation data is used as the test set. The third dataset is Places365. This dataset contains 365 categories, each with approximately 3k to 5k training and 100 validation images, and 329k unlabeled test images. In this experiment, the training images are used as individual classifiers 122-1 to 122-N (C i ), unlabeled test images are used as transfer set 124 (U), and validation images are used as test set to evaluate accuracy. All images are preprocessed by center cropping and scaling to 64x64 pixels.

[0166] Individual classifier configuration:

[0167] The proposed method is tested under two configurations of individual classifiers (summarized in Table 1). (i) Random categories: For the ImageNet dataset, 20 to 50 categories are sampled as L in each trial. U And train 10 to 20 individual classifiers 122 (C i ), each of which is trained to classify 5 to 15 categories. For the LSUN dataset, in each trial, 5 to 10 categories are sampled as L U , and train 3 to 7 individual classifiers 122, where each classifier is trained to classify 2 to 5 categories. This configuration is used as the individual classifier 122 (C i ) is the main test for classifying different sets of categories. For the Places365 dataset, in each trial, 20 to 50 categories are sampled as L U , and train 10 to 20 individual classifiers 122 (C i ), where each classifier is trained to classify between 5 and 15 categories. (ii) Completely overlapping categories: Here, the same configuration as in (i) is used, except that all individual classifiers 122 (C i ) to convert L U This situation is used to classify all categories in all classifiers C i and C U The proposed method is tested on two common configurations that share the same category. In both configurations, the transition set 124(U) consists of LU In other words, most of the images in the transfer set 124(U) do not fall into the L U in any category.

[0168] Table 1

[0169]

[0170] Model:

[0171] Each individual classifier 122 (C i ) are randomly selected from one of the following four architectures with ImageNet pre-trained weights: AlexNet, VGG16, ResNet18, and ResNet34. For AlexNet and VGG16, the weights of their feature extractor parts are fixed, their fc (fully connected) layers are replaced by two fc layers with 256 hidden nodes (with BatchNorm and ReLu), and the fc layers are trained with their training data. Similarly, in the ResNet model, their fc layers are replaced by two fc layers with 256 hidden nodes as described above. In addition, the last residual block is fine-tuned. As for the unified classifier 126 (C U ), two models (including VGG16 and ResNet34) are used with similar settings as above.

[0172] For all datasets and configurations, each individual classifier 122 (C i ) were used with 50 to 200 samples per class for training; in the same experiment, in any individual classifier 122 (C i ) are not shared between them. Then these individual classifiers 122 (C i ) together with the transfer set 124 (U) to train the unified classifier 126 (C U All models were trained for 20 epochs using the SGD (Stochastic Gradient Descent) optimizer (with a momentum of 0.9 and a step size of 0.1 and 0.015 for the first and last 10 epochs). To control for the variation in the results, in each experiment, the same architecture of the unified classifier 126 (C U ), and use the same batch command to train the unified classifier 126 (C U ). A unified classifier 126 (C U ) are evaluated on the test data for all categories in L for their corresponding trials. 50 trials are run for each combination of dataset, model, and individual classifier configuration.

[0173] result:

[0174] The evaluated accuracy results are summarized in Tables 2 and 3 below:

[0175] Table 2

[0176]

[0177] Table 3

[0178]

[0179]

[0180] Tables 2 and 3 show the accuracy of the methods on different combinations of individual classifier configurations, datasets, and unified classifier models. Each column shows the average accuracy of each method under each experimental setting, where the best result excluding the SPV of each setting is shown in bold. To test statistical significance, the Wilcoxon signed rank test over standard deviations was selected to cater for very different settings across the experiments (e.g., model structure, number of categories, HC, etc.). The test was performed between the best performing method and the rest in each experiment. Methods that were not statistically significantly different from the best method at α=0.01 are shown in bold.

[0181] In the following, results for the random class case are described, which addresses the main scenario of the proposed new knowledge distillation when each individual classifier 122 is trained to classify a different set of classes.

[0182] As shown in Table 2, all the proposed methods (Examples 1-12) performed significantly better than SD (Comparative Example 1). All the methods in Table 2, including the q estimation method using the suffix "E" (Examples 1, 4, 7, 10), the direct back propagation method using the suffix "BP" (Examples 2, 5, 8, 11), and the balanced method using the suffix "BS" (including SD-BS) (Examples 3, 6, 9, 12 and Comparative Example 2), performed better than SD (Comparative Example 1) by a large margin of 9%-15%. It has been shown that simply setting the probability of the undefined category of each individual classifier to 0 may significantly reduce the accuracy. On the other hand, the proposed methods (Examples 1-12) achieved significantly better results and almost reached the same accuracy as SPV (Comparative Example 3), with a gap of 1%-4%. This shows that the individual classifiers 122 (C i ) can be used for unsupervised training at a minimal cost in accuracy, even if the transfer set 124(U) contains a significant fraction of images that are not part of the target class. Still, there are issues that may affect the unified classifier 126(C U ) to avoid several factors that affect the accuracy of SPV, such as the individual classifiers 122 (C i )’s accuracy, their architecture, etc.

[0183] It can be seen that different algorithms perform best in different settings, however, matrix factorization on logits with fixed v (Examples 10-12) always performs best or is not statistically different from the best method. This suggests that matrix factorization on logits with fixed v can be the best approach for solving the unified heterogeneous classifier problem.

[0184] Although the improvement may be minor (less than 1.5%), the annotation-balanced methods with the suffix "BS" (Examples 3, 6, 9, 12) consistently outperform their counterparts with the suffix "E" (Examples 1, 4, 7, 10). These results indicate that using balanced soft annotations to train the unified classifier 126 (C U ) is a good idea. Note that while SD-BS (Comparative Example 2) is significantly improved, it still generally underperforms the other CE and MF methods (Examples 1-12), indicating that {p i} i The relationship between q and q is important to incorporate into training.

[0185] The methods with suffixes 'E' and 'BS' are based on training a unified classifier 126 (C U ) before estimating q, while methods with the suffix "BP" directly perform backpropagation from the loss function. The losses of cross entropy (CE) and matrix factorization of logits with fixed v (MF-LF) are convex in their variants, while matrix factorization of probability (MF-P) and matrix factorization of logits with non-fixed v (MF-L U ) is non-convex. Here, a small but interesting effect is observed where the method with convex loss performs better than directly backpropagating the loss when q is first estimated, while the non-convex loss has the opposite result. This is likely due to the accumulation of errors when estimating q for non-convolutional methods, which is passed down to the unified classifier 126 (C U ), and then train the unified classifier 126 (C U ), while non-convolutional "BP" methods may be able to avoid such local error accumulation. On the other hand, convex methods can estimate the global optimal q, so it is used to train the unified classifier 126 (C U ) can avoid bad local optimality.

[0186] Next, the results for the case of complete overlap are discussed by referring to Table 3. All methods (Examples 1-12) perform quite well. It can be seen that all methods achieve about the same accuracy as SD (Comparative Example 1). This shows that the proposed method can also perform well in the common case where all individual classifiers are trained to classify all categories, and enhances the overview that the proposed method is knowledge distillation.

[0187] It can be noted that balanced soft labeling tends to slightly reduce accuracy. Here, even SD-BS (Comparative Example 2), which receives improved accuracy in the random class case, performs worse than its counterpart SD (Comparative Example 1). This suggests that unbalanced soft labeling may be a better option in this case.

[0188] For the LSUN and Places365 datasets, it can be seen that all methods (Examples 1-12) outperform SPV (Comparison Example 3). Especially for the case of VGG16, SPV consistently performs 1% to 3% worse than other methods in most of the 50 trials. This suggests that distillation-based methods may outperform their supervised counterparts.

[0189] Sensitivity analysis

[0190] In addition, the size of the transition set, the temperature parameter T, and the accuracy (C i We used the same settings as the ImageNet random category experiment, with VGG16 as the unified classifier126 (C U ). 50 trials were run for each test. As a representative set of SD and top performing methods from the previous section, the following five methods were evaluated: SD (Comparative Example 4), SD-BS (Comparative Example 5), MF-P-BP (Example 13), MF-LF-BS (Example 14), and CE-BS (Example 15).

[0191] Size of the transfer set: A test was used to evaluate the impact of the number of unlabeled samples in the transfer set U. The number of samples was increased from 10 3 Change to 10 5 The results are shown in Fig. 8A As expected, it can be seen that all methods degrade as the size of the transmission set decreases. In this test, MF-P-BP (Example 13) is the most affected by this reduction, as its accuracy drops the fastest. Nevertheless, all other methods (Examples 13-15) perform better than SD over the entire test range, demonstrating robustness to transmissions with groups of different sizes.

[0192] Temperature: In this test, used to smooth the probability {p i} i The temperature T used to estimate q or train C U The values ​​evaluated were T = 1, 3, 6; and 10. The results are shown in Figure 8BIt can be seen that when T is set to high and low values, respectively, the accuracy of SD and SD-BS (Comparative Examples 4-5) is significantly reduced. On the other hand, the other three methods (Examples 13-15) are less affected by different T values.

[0193] Accuracy of individual classifiers: In this test, the accuracy of individual classifiers 122 (C i ) to evaluate the robustness of the method. The test scheme is as follows: In each experiment, all individual classifiers 122 (C i ) varies from 40% to 80%, p i From the individual classifier 122 (C i ) is obtained and used to perform the method. In order to change each individual classifier 122 (C i ), 50 samples from each category in the training data are taken as the adjustment set, and each individual classifier is fully trained from the remaining training data 122 (C i ), and then inject increasing Gaussian noise into the last fc layer until its accuracy on the adjusted set drops to the expected value. i ) is lower than the expected value, then simply use the initial C i . Figure 8C The results of this evaluation are shown in FIG. 1 . It can be seen that the accuracy of all methods (Examples 13-15) increases with the individual classifier 122 (C i ) increases and performs better, thus indicating that the individual classifier 122 (C i ) is a significant factor in performance. It can also be seen that MF-P-BP (Example 13) is most affected by the single classifier 122 (C i ), while MF-LF-BS (Example 14) is the most robust.

[0194] Based on sensitivity analysis, MF-LF-BS (Example 14) proved to be the most robust method with respect to the number of samples in the transfer set, temperature, and the accuracy of the individual classifiers 122. This result provides further evidence that MF-LF-BS is our proposed method.

[0195] Computer hardware components

[0196] See now Fig. 9 , shows a schematic diagram of an example of a computer system 10 that can be used for a central computing system 120 and other computing systems of a data resource 110 . Fig. 9The computer system 10 shown in is implemented as a computer system. The computer system 10 is only one example of a suitable processing device and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the invention described herein. In any case, the computer system 10 can be implemented and / or perform any of the functions set forth above.

[0197] The computer system 10 may operate with numerous other general or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the computer system 10 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, vehicle-mounted devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, and the like.

[0198] Computer system 10 may be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally speaking, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types.

[0199] like Fig. 9 As shown, computer system 10 is shown in the form of a general-purpose computing device. Components of computer system 10 may include, but are not limited to, processor (or processing circuit) 12 and memory 16, memory 16 coupled to processor 12 via a bus including a memory bus or memory controller and a processor or local bus using any of a variety of bus architectures.

[0200] Computer system 10 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system 10, and it includes both volatile and nonvolatile media, removable and non-removable media.

[0201] Memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM). Computer system 10 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 18 may be provided for reading from and writing to non-removable, non-volatile magnetic media. As will be further depicted and described below, storage system 18 may include at least one program product having a collection (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0202] As an example and not limitation, a program / utility having a set (at least one) of program modules, as well as an operating system, one or more application programs, other program modules, and program data may be stored in storage system 18. Each or some combination of the operating system, one or more application programs, other program modules, and program data may include an embodiment of a networking environment. The program modules generally perform the functions and / or methods of the embodiments of the present invention described herein.

[0203] The computer system 10 may also communicate with one or more peripheral devices 24 (such as a keyboard, a pointing device, a car navigation system, an audio system, etc.), a display 26, one or more devices that enable a user to interact with the computer system 10, and / or any device that enables the computer system 10 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. In addition, the computer system 10 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system 10 via a bus. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system 10. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems, etc.

[0204] Computer program implementation

[0205] The present invention may be a computer system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructing a processor to perform various aspects of the present invention.

[0206] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the above. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0207] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0208] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA)) can execute a computer-readable program instruction to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform various aspects of the present invention.

[0209] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0210] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that is executed by a processor of a computer or other programmable data processing device to create a device for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing device, and / or other device that functions in a specific manner, such that a computer-readable storage medium having instructions stored therein includes an artifact that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0211] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device that causes a series of operating steps to be performed on a computer, other programmable device, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0212] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the box may not occur in the order marked in the figure. For example, depending on the function involved, the two boxes shown in succession can actually be performed substantially at the same time, or these boxes can sometimes be performed in the opposite order. It will also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a system based on special-purpose hardware, and the system based on special-purpose hardware performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0213] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an", and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will be further understood that when the terms "comprise" and / or "comprising" are used in this specification, they specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0214] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements (if any) in the following claims are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of one or more aspects of the present invention has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the invention in the form disclosed.

[0215] Without departing from the scope and spirit of the described embodiments, many modifications and variations will be apparent to those of ordinary skill in the art. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the market, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for generating unified soft annotations, the unified soft annotations being used to train a unified model, the computer-implemented method include: Prepare sample collection; Obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, each of the individually trained models being used to classify an image or audio and having a set of individual categories to form at least a portion of a unified set of categories of a unified model, the unified set of categories comprising a plurality of target categories; for each sample, estimating a unified soft label on the plurality of target classes in the unified set of classes from the plurality of predictions using a relationship connecting a first output of each individually trained model and a second output of the unified model, wherein each sample is fed into the unified model to infer predicted values ​​on the plurality of target classes as the second output, and wherein the relationship indicates an equivalence between each predicted value in the first output of one individually trained model and a corresponding predicted value in the second output of the unified model normalized by the set of individual classes of the one individually trained model; and The unified soft annotation is output to train a unified model with the unified category set.

2. The computer-implemented method of claim 1, in, The computer-implemented method comprises: updating the unified model based on a loss function between the unified soft label and the predicted value of each sample; and The updated unified model is stored.

3. A computer-implemented method as described in claim 2, wherein the loss function is weighted by weighting the target categories in the unified category set, and each weight for a target category is calculated in a manner based on a statistic of the unified soft annotation on the one target category in the set.

4. The computer-implemented method of claim 1, in, The unified soft label is estimated by solving a problem of optimizing an objective function with respect to a distribution q corresponding to the second output of the unified model, the objective function measuring a plurality of reference distributions p corresponding to the plurality of predictions i and multiple normalized distributions qi ^ The error between the multiple normalized distributions qi ^ Each of is obtained by normalizing the distribution q over the target category in each set of individual categories.

5. The computer-implemented method of claim 4, in, The distribution q is obtained by the following steps: Use temporary variables ul to solve the convex problem. Each temporary variable ul is given a uniform category set L. U For each target category l in , the distribution q is represented by a set of exponential functions of the temporary variables ul in the convex problem; and The solved temporary variable ul is converted into the distribution q.

6. The computer-implemented method of claim 4, wherein the objective function is a cross entropy function.

7. The computer-implemented method of claim 1, in, The unified soft label is estimated by solving a problem of optimizing an objective function for at least an output vector u representing the second output of the unified model as a variable in a matrix factorization-based manner, the unified soft label being represented in the form of probability or logit.

8. The computer-implemented method of claim 7, in, The output vector u is represented in the form of probabilities, and the problem of optimizing the objective function, which measures the probability of having missing entries, is further solved for a normalized vector v representing a normalization factor for the model trained on the individual. i The error between the probability matrix P and the product of the output vector u and the normalized vector v is calculated, and the mask matrix M is used to represent the presence of missing classes in each class set.

9. The computer-implemented method of claim 7, in, The output vector u is represented in logit form, and the problem of optimizing the objective function is further solved for a logit scaling vector v and a logit shift vector c, the objective function measuring the logit representation of the plurality of predictions p with missing entries i The error between the logit matrix Z and the product of the output vector u and the logit scaling vector v shifted by the logit shift vector c is calculated, and the mask matrix M is used to represent the presence of missing categories in the individual category set.

10. The computer-implemented method of claim 7, in, The output vector u is represented in logit form, and the problem of optimizing the objective function, which measures the logit-based representation of the plurality of predictions p with missing entries, is further solved for the logit-based shift vector c. i The error between the logit matrix Z and the product of the output vector u and the fixed scaling vector v shifted by the logit shift vector c is calculated, using the mask matrix M to represent the presence of missing categories in the set of individual categories.

11. The computer-implemented method of claim 1, in, Each of the unified model and the individually trained models is selected from the group consisting of a neural network-based classification model, a decision tree or forest-based classification model, and a support vector machine-based classification model.

12. The computer-implemented method of claim 1, in, The computer-implemented method comprises: For each of the multiple individually trained models, receive (i) the content of the individually trained model itself or (ii) a soft annotation set obtained by feeding each sample into the individually trained model, and the content of the individual category set.

13. A computer-implemented method for training a unified model, the computer-implemented method include: Prepare sample collection; Obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, each of the individually trained models being used to classify an image or audio and having a set of individual categories to form at least a portion of a unified set of categories of a unified model, the unified set of categories comprising a plurality of target categories; as well as The unified model with the unified set of categories is updated using the multiple predictions for each sample and a relationship connecting the first output of each individually trained model and the second output of the unified model, wherein each sample is fed into the unified model to infer predicted values ​​on the multiple target categories in the unified set of categories as the second output, and wherein the relationship indicates an equivalence between each predicted value in the first output of an individually trained model and a corresponding predicted value in the second output of the unified model normalized by the individual set of categories of the one individually trained model.

14. The computer-implemented method of claim 13, in, The unified model includes a neural network, and the unified model is updated by back-propagating a loss through the neural network, the loss measuring a plurality of reference distributions pi corresponding to the plurality of predictions and a plurality of normalized distributions qi obtained by normalizing a distribution q over a target class in each set of individual classes. ^ The error between , the distribution q is obtained as the predicted value inferred by the unified model.

15. The computer-implemented method of claim 13, in, The unified model includes a neural network, and the unified model is updated by back-propagating a loss through the neural network, the loss being obtained by solving a problem of optimizing an objective function having an output vector u representing the second output of the unified model and other variables in a matrix factorization-based manner, the output vector u being obtained as the predicted value inferred by the unified model and being fixed in the problem.

16. A computer system for generating unified soft annotations, wherein the unified soft annotations are used to train a unified model, wherein the computer system include: Memory for storing program instructions; a processing circuit in communication with the memory and configured to execute the program instructions, wherein the processing circuit is configured to: Prepare sample collection; Obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, wherein each individually trained model is used to classify an image or audio and has a set of individual categories to form at least a portion of a unified set of categories of a unified model, the unified set of categories including a plurality of target categories; for each sample, estimating a unified soft label on the plurality of target classes in the unified set of classes from the plurality of predictions using a relationship connecting a first output of each individually trained model and a second output of the unified model, wherein each sample is fed into the unified model to infer predicted values ​​on the plurality of target classes as the second output, and wherein the relationship indicates an equivalence between each predicted value in the first output of one individually trained model and a corresponding predicted value in the second output of the unified model normalized by the set of individual classes of the one individually trained model; and The unified soft annotation is output to train a unified model with the unified category set.

17. The computer system of claim 16, in, in, The unified soft label is estimated by solving a problem of optimizing an objective function with respect to a distribution q corresponding to the second output of the unified model, the objective function measuring a plurality of reference distributions p corresponding to the plurality of predictions i and multiple normalized distributions qi ^ The error between the multiple normalized distributions qi ^ Each of the errors between is obtained by normalizing the distribution q on the target category in each individual category set.

18. The computer system of claim 16, in, The unified soft label is estimated by solving a problem of optimizing an objective function for at least an output vector u representing the second output of the unified model as a variable in a matrix factorization-based manner, the unified soft label being represented in the form of probability or logit.

19. A computer system for training a unified model, the computer system include: Memory for storing program instructions; A processing circuit, in communication with the memory, for executing the program instructions, wherein the processing circuit is configured to: Prepare sample collection; Obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, each of the individually trained models being used to classify an image or audio and having a set of individual categories to form at least a portion of a unified set of categories of a unified model, the unified set of categories comprising a plurality of target categories; and The unified model with the unified set of categories is updated using the multiple predictions for each sample and a relationship connecting the first output of each individually trained model and the second output of the unified model, wherein each sample is fed into the unified model to infer predicted values ​​on the multiple target categories in the unified set of categories as the second output, and wherein the relationship indicates an equivalence between each predicted value in the first output of an individually trained model and a corresponding predicted value in the second output of the unified model normalized by the individual set of categories of the one individually trained model.

20. The computer system of claim 19, in, The unified model includes a neural network, and the unified model is updated by back-propagating a loss through the neural network, the loss measuring a plurality of reference distributions pi corresponding to the plurality of predictions and a plurality of normalized distributions qi obtained by normalizing a distribution q over a target category in each set of individual categories ^ The error between , the distribution q is obtained as the predicted value inferred by the unified model.

21. The computer system of claim 19, in, The unified model includes a neural network, and the unified model is updated by back-propagating a loss through the neural network, the loss being obtained by solving a problem of optimizing an objective function having an output vector u representing the second output of the unified model and other variables in a matrix factorization-based manner, the output vector u being obtained as the predicted value inferred by the unified model and being fixed in the problem.

22. A computer program product for generating unified soft annotations, the unified soft annotations being used to train a unified model, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computer to cause the computer to perform a method, the method include: Prepare sample collection; Obtaining, for each sample, a plurality of predictions generated by a plurality of individually trained models, each of the individually trained models being used to classify an image or audio and having a set of individual categories to form at least a portion of a unified set of categories of a unified model, the unified set of categories comprising a plurality of target categories; for each sample, estimating a unified soft label on the plurality of target classes in the unified set of classes from the plurality of predictions using a relationship connecting a first output of each individually trained model and a second output of the unified model, wherein each sample is fed into the unified model to infer predicted values ​​on the plurality of target classes as the second output, and wherein the relationship indicates an equivalence between each predicted value in the first output of one individually trained model and a corresponding predicted value in the second output of the unified model normalized by the set of individual classes of the one individually trained model; and The unified soft annotation is output to train a unified model with the unified category set.

Images