Multi-heterogeneous model fusion classification model training method, classification method and device

Abstract

The application provides a multi-heterogeneous model fusion classification model training method, a classification method and a device, the multi-heterogeneous model fusion classification model training method comprises the following steps: fusing the output representation of a plurality of teacher models based on DS evidence theory to obtain fused output representation, and determining the probability distribution vector of each class in the class set corresponding to the plurality of teacher models according to the fused output representation; training a generative adversarial network (GAN) based on the probability distribution vector to obtain a trained GAN, and distilling the student model according to the probability distribution vector and the trained GAN to obtain a target student model for performing an image classification task. The method disclosed by the application realizes the fusion and distillation of the image classification model by using the DS evidence theory and the generative model, improves the prediction performance of the image classification model, and further improves the accuracy of the image classification result.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, classification method and apparatus for training a classification model that integrates multiple heterogeneous models. Background Technology

[0002] In recent years, with the development and maturation of artificial intelligence (AI) technology, its application in the field of computer vision (CV) has become increasingly widespread; in particular, the development of deep learning (DL) methods in the fields of image recognition and classification has achieved remarkable results.

[0003] Image recognition and classification is one of the core tasks in computer vision. Models need to identify specific objects or scenes from an input image and then classify them into one of predefined categories. With the rise of artificial intelligence, deep learning methods such as Convolutional Neural Networks (CNNs) and Transformers have greatly improved the performance of image recognition and classification.

[0004] In related technologies, image classification models require a large amount of effective sample data for training. However, the high cost of acquiring and labeling sample data, as well as the uneven distribution of data, often lead to a significant performance degradation when the trained models are applied to real-world scenarios, resulting in a decrease in the accuracy of image inference results such as image recognition, image classification, and image segmentation. Summary of the Invention

[0005] This invention provides a classification model training method, classification method, and apparatus that integrates multiple heterogeneous models, thereby addressing the shortcomings of existing technologies, such as high costs of sample data acquisition and annotation, and low predictive performance of trained image classification models due to uneven data distribution, and improving image classification accuracy.

[0006] This invention provides a method for training a classification model that integrates multiple heterogeneous models, comprising:

[0007] Based on the DS evidence theory, the output representations of multiple teacher models are fused to obtain a fused output representation. The probability distribution vectors of each category in the category set corresponding to the multiple teacher models are then determined based on the fused output representation. The fused output representation is used to represent the probability and out-of-domain probability of each category in the category set. Different teacher models are used to perform different image classification tasks.

[0008] The Generative Adversarial Network (GAN) is trained based on the probability distribution vector to obtain the trained GAN. The student model is then distilled and trained according to the probability distribution vector and the trained GAN to obtain the target student model, which is used to perform image classification tasks.

[0009] According to the present invention, a classification model training method for multi-heterogeneous model fusion, wherein training a generative adversarial network (GAN) based on the probability distribution vector to obtain the trained GAN includes:

[0010] The probability distribution vector is mapped to the image space based on the GAN, and the GAN is iteratively trained according to the first joint loss function. When the GAN converges or reaches the maximum number of iterations, the trained GAN is obtained. The first joint loss function is determined based on the probability value of the class with the highest probability after fusion, the random sampling vector determined based on the probability space after fusion, and the unlabeled image samples.

[0011] According to the classification model training method of multi-heterogeneous model fusion provided by the present invention, the student model is distilled and trained based on the probability distribution vector and the trained GAN to obtain the target student model, including:

[0012] The KL divergence loss function is determined based on the KL divergence between the output representation of the student model and the probability distribution vector; the distance loss function is determined based on the distance between the output representation of the generator and the mapped representation in the trained GAN; wherein, the mapped representation is obtained by mapping the output representation of the generator to the image space based on the generator;

[0013] A second joint loss function is determined based on the KL divergence loss function and the distance loss function, and the student model is iteratively trained according to the second joint loss function. When the student model converges or reaches the maximum number of iterations, the target student model is obtained.

[0014] According to the classification model training method of multi-heterogeneous model fusion provided by the present invention, the KL divergence loss function is expressed by the following formula:

[0015]

[0016] Among them, L KL For the KL divergence loss, x ′(i) The image is generated by the trained GAN based on the image after sampling noise, where β is the weighting coefficient, m is the number of samples in a single training iteration, and T(x′) is the image generated by the trained GAN. (i) S(x′) represents the fused output. (i) ) represents the output of the student model;

[0017] The distance loss function is expressed by the following formula:

[0018]

[0019] Among them, L F Let τ be the distance loss function, τ be the temperature coefficient, and G be the generator. F This is a normal form operation.

[0020] According to the classification model training method of multi-heterogeneous model fusion provided by the present invention, the step of determining the probability distribution vector of each category in the category set corresponding to the multiple teacher models based on the fused output representation includes:

[0021] The probability distribution vector is determined based on the fused output representation and the number of categories in the union of multiple category sets; the data length of the probability distribution vector is the number of categories in the union plus one.

[0022] The present invention also provides a classification method, comprising:

[0023] Obtain the image data to be classified;

[0024] The image data to be classified is processed based on the target student model to obtain the target classification result; wherein the target student model is trained based on the classification model training method of multi-heterogeneous model fusion.

[0025] The present invention also provides a classification model training device for multi-heterogeneous model fusion, comprising:

[0026] The model fusion module is used to fuse the output representations of multiple teacher models based on the DS evidence theory to obtain a fused output representation, and to determine the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation; wherein, the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks;

[0027] The model distillation training module is used to train the Generative Adversarial Network (GAN) based on the probability distribution vector to obtain the trained GAN, and to distill the student model according to the probability distribution vector and the trained GAN to obtain the target student model for performing image classification tasks.

[0028] The present invention also provides a sorting device, comprising:

[0029] The image acquisition module is used to acquire image data to be classified.

[0030] The classification module is used to process the image data to be classified based on the target student model to obtain the target classification result; wherein the target student model is trained based on the classification model training method of multi-heterogeneous model fusion.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a classification model training method or a classification method as described above for multi-heterogeneous model fusion.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a classification model training method or classification method for multi-heterogeneous model fusion as described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a classification model training method or classification method for multi-heterogeneous model fusion as described above.

[0034] The present invention provides a multi-heterogeneous model fusion classification model training method, classification method, and apparatus. It fuses the output representations of multiple teacher models using DS evidence theory to obtain a fused output representation. Based on the fused output representation, it determines the probability distribution vectors of each category in the category set corresponding to the multiple teacher models. These probability distribution vectors are then used to guide the training of a generative adversarial network (GAN). Finally, based on the probability distribution vectors and the trained GAN, it distills and trains a student model to obtain a target student model. This achieves the fusion and distillation of image classification models using DS evidence theory and generative models, improving the predictive performance of image classification models and thus increasing the accuracy of image classification results. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the classification model training method for multi-heterogeneous model fusion provided by the present invention.

[0037] Figure 2 This is the second flowchart of the classification model training method for multi-heterogeneous model fusion provided by the present invention.

[0038] Figure 3This is a schematic diagram of the mapping relationship of the GAN network provided by the present invention.

[0039] Figure 4 This is a flowchart illustrating the GAN network training method provided by the present invention.

[0040] Figure 5 This is the third flowchart of the classification model training method for multi-heterogeneous model fusion provided by the present invention.

[0041] Figure 6 This is a flowchart illustrating the classification method provided by the present invention.

[0042] Figure 7 This is a schematic diagram of the structure of the classification model training device for multi-heterogeneous model fusion provided by the present invention.

[0043] Figure 8 This is a schematic diagram of the classification device provided by the present invention.

[0044] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] The following is combined Figures 1-8 This invention describes a classification model training method, classification method, and apparatus for multi-heterogeneous model fusion.

[0047] Figure 1 This is one of the flowcharts illustrating the multi-heterogeneous model fusion classification model training method provided by the present invention, such as... Figure 1 As shown, the training method for this multi-heterogeneous model fusion classification model includes the following steps:

[0048] Step 110: Based on the DS evidence theory, fuse the output representations of multiple teacher models to obtain the fused output representation, and determine the probability distribution vectors of each category in the category set corresponding to the multiple teacher models according to the fused output representation; wherein, the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks.

[0049] It should be noted that the teacher model can be defined as the model being integrated. The student model acquires the task capabilities of each teacher model through the knowledge of each teacher model, and at the same time achieves the ability to generalize to real-world application scenarios, making its task performance in application scenarios superior to that of the teacher model.

[0050] In this step, a pre-trained model capable of recognizing and classifying images is used as the teacher model, such as a commercially available large model hosted on a cloud server or other open-source models; however, its training data, internal structure and composition, connections between layers, parameters of the teacher model, and gradients used for backpropagation are all invisible.

[0051] For example, teacher models include, but are not limited to, Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), or Convolutional Recurrent Neural Networks (CRNN).

[0052] In this embodiment, when selecting a teacher model, there are no restrictions on the similarities or differences in the internal structure of the models. Multiple models with different tasks (category sets) can be selected as teacher models. These models can all provide valuable probability outputs for real-world application scenarios. For example, if the category set of model A contains some categories of the real-world application scenario, then model A will be used as the teacher model to ensure that the teacher model has good accuracy and robustness.

[0053] In this embodiment, the multiple teacher models can be deep learning models with completely identical model architectures; for example, all the teacher models can be convolutional neural networks.

[0054] In this embodiment, the two teacher models can be deep learning models with different model architectures, such as a convolutional neural network and a recurrent neural network.

[0055] In this embodiment, when performing image recognition or image segmentation tasks, the corresponding teacher model also includes a trained image recognition model or a trained image segmentation model.

[0056] In this embodiment, the results of image reasoning from multiple teacher models are used as input to train the final student model. The reasoning results are represented as follows:

[0057] output = teacher(x);

[0058] Here, output is a normalized probability distribution containing out-of-domain probabilities, represented by a vector of length equal to the number of classes plus 1, representing the probability of each class corresponding to the input image x and the probability of not belonging to that class set. teacher(x) represents the reasoning process of the teacher model for the input image x.

[0059] It should be noted that when performing the classification model training method that integrates multiple heterogeneous models, it is not necessary to load the teacher model into the local machine. It is only necessary to ensure that the output of the teacher model can be obtained before proceeding with the subsequent steps.

[0060] In this embodiment, after selecting a teacher model, the outputs of the teacher model are first integrated to obtain a correct probability representation; for example, the probability of the teacher output is fused using DS evidence theory. The specific fusion process is as follows:

[0061] Suppose there are n teacher models, Let be the category set of the i-th model. The fusion formula is as follows:

[0062]

[0063] Where K is the normalization coefficient, used to ensure the normalized class probabilities are obtained, k∈(1,2,...,n). U c =U1∪U2...∪U n ;U i ={ω1,ω2,...,ω M} represents the category set of the i-th model, Δ i This represents the category not included in the i-th model. Indicated by U i and Δ i The power set that constitutes a set, m c (C) represents the confidence level, or probability, assigned to class C after fusion. A i express A certain element; K and K -1 They are reciprocals of each other.

[0064] Based on actual image classification scenarios, the above formula can be simplified as follows:

[0065]

[0066]

[0067] Where, ω j ∈U c M c U cThe number of categories included; the above formula ultimately yields a basic probability assignment for an incomplete recognition framework Θ, i.e., for... The basic probability assignment (elements not in the set are assigned a value of zero); h is a conditional variable.

[0068] Finally, the fused probability distribution is obtained through the above calculation process, which is the probability m of each category within the category set. c (ω j ) and out-of-domain probability m c (Δ c ).

[0069] In this embodiment, a vector P with a length equal to the number of categories plus 1 can be used. merged Let represent the probability distribution after fusion, where the number of categories is the number of categories in the union of all teacher model category sets.

[0070] Specifically, determining the probability distribution vectors for each category in the category set corresponding to multiple teacher models based on the fused output representation includes: determining the probability distribution vector based on the number of categories in the union of the fused output representation and multiple category sets; the data length of the probability distribution vector is the number of categories in the union plus one.

[0071] In this embodiment, vector P merged With the above probability m c (ω j ) and out-of-domain probability m c (Δ c The relationship between them is expressed by the following formula:

[0072]

[0073] Among them, P merged It is composed of the in-class probability vector m c (ω j ) and out-of-class probability m c (Δ c The concatenated vector, M C This represents the number of categories after fusion.

[0074] Step 120: Train the Generative Adversarial Network (GAN) based on the probability distribution vector to obtain the trained GAN. Then, distill the student model according to the probability distribution vector and the trained GAN to obtain the target student model to perform the image classification task.

[0075] In this step, to address the differences between real-world application scenarios and teacher training scenarios, as well as the difficulty in collecting high-quality labeled data in real-world application scenarios, Generative Adversarial Networks (GANs) are used to augment the data and generalize the domain.

[0076] In this embodiment, the application scene image and probability distribution vector P are used. merged By training a Generative Adversarial Network (GAN), unlabeled data augmentation and deprivation of the teacher model were achieved, resulting in better generalization performance for real-world applications.

[0077] In this embodiment, the student model includes, but is not limited to, one of a convolutional neural network, a recurrent neural network, a long short-term memory network, or a convolutional recurrent neural network.

[0078] In this embodiment, after obtaining the trained GAN, the student model is distilled using the trained GAN network and probability distribution vector to obtain a target student model suitable for the actual application scenario; then, a small amount of labeled data is used to test the target student model to verify the model's prediction accuracy.

[0079] Specifically, after the test image is input into the trained target student model, the model will output the corresponding probability distribution, select the category with the highest probability as the classification answer, compare it with the labeled answer, and thus calculate the accuracy of the target student model.

[0080] Figure 2 This is the second flowchart illustrating the multi-heterogeneous model fusion classification model training method provided by this invention. Figure 2 In the embodiment shown, the classification model training method of multi-heterogeneous model fusion is further implemented through the following steps: S1, obtaining the teacher model; S2, integrating the teacher output, training the generation model and training the student model; S3, evaluating the student model.

[0081] This invention provides a multi-heterogeneous model fusion classification model training method. It fuses the output representations of multiple teacher models using DS evidence theory to obtain a fused output representation. Based on the fused output representation, it determines the probability distribution vectors of each category in the category set corresponding to the multiple teacher models. These probability distribution vectors are then used to guide the training of a Generative Adversarial Network (GAN). Finally, based on the probability distribution vectors and the trained GAN, it distills and trains a student model to obtain the target student model. This method achieves the fusion and distillation of image classification models using DS evidence theory and generative models, improving the predictive performance of image classification models and thus increasing the accuracy of image classification results.

[0082] In some embodiments, training a generative adversarial network (GAN) based on a probability distribution vector to obtain a trained GAN includes: mapping the probability distribution vector to an image space based on the GAN, and iteratively training the GAN according to a first joint loss function. When the GAN converges or reaches the maximum number of iterations, the trained GAN is obtained. The first joint loss function is determined based on the probability value of the class with the highest probability after fusion, a random sampling vector determined based on the probability space after fusion, and unlabeled image samples.

[0083] In this embodiment, random sampling is performed from the probability distribution vector to map the fused probability distribution back to the image space, thereby training the GAN network to generate corresponding application scenario images. This ensures that the images generated by the GAN network are more consistent with the classification results of the fused probability, thus improving the quantity and quality of the samples used.

[0084] In this embodiment, during the training process, the probability distribution vector can be obtained based on the output of the pre-trained teacher model; the loss function corresponding to the pre-trained teacher model is expressed by the following formula:

[0085]

[0086] in, z is the probability value of the category with the highest probability after fusion. (i) z is a random sampling vector determined based on the fused probability space. (i) Length and P merged Same; x (i) For unlabeled image samples in real-world application scenarios, D and G represent the discriminator and generator of the trained GAN, respectively; G(z) (i) ) is the generator based on x (i) The generated pseudo-image.

[0087] The loss function of the discriminator in a GAN is expressed by the following formula:

[0088]

[0089] Where D(x) (i) The discriminator is based on x (i) A definitive judgment result.

[0090] The loss function of the generator in a GAN is expressed by the following formula:

[0091]

[0092] In this embodiment, the first joint loss function L is expressed by the following formula:

[0093] L1=a1*LT +b1*L D +c*L G ;

[0094] Among them, a1, b1 and c are all weight factors no greater than 1, and the values ​​of the weight factors can be set according to user needs.

[0095] In this embodiment, the GAN is iteratively trained using the first joint loss function described above. In each iteration, the parameters of the generator and discriminator are updated until the GAN converges or reaches the maximum number of iterations, resulting in the trained GAN.

[0096] Figure 3 This is a schematic diagram of the mapping relationship of the GAN network provided by the present invention. Figure 3 In the embodiment shown, f G ,f T ,f S Let GAN, the fused teacher model, and the student model represent the GAN, the fused teacher model, and the student model, respectively. The fused teacher model and student model calculate the probability distribution of the category to which the image belongs through the input image. In other words, the fused teacher model and student model realize the mapping from the image space to the probability space of the category set. The GAN is trained according to the determined probability distribution to generate more realistic images. That is, the GAN realizes the mapping from the probability space of the category set to the image space.

[0097] Figure 4 This is a flowchart illustrating the GAN network training method provided by the present invention. Figure 4 In the illustrated embodiment, a generator G generates a pseudo-image based on the input image Z, and a discriminator D discriminates the pseudo-image to expand the sample. During this process, the output representations of multiple teacher models (T1, ..., Tn) are fused using the DS evidence theory to obtain a probability distribution vector. The GAN is then iteratively trained according to the DS guidance to obtain the trained GAN. The loss function of each teacher model is L. T The loss function of GAN is L GAN .

[0098] This invention provides a classification model training method that integrates multiple heterogeneous models. By mapping the probability distribution vector to the image space through GAN, and iteratively training the GAN according to the first joint loss function, the trained GAN is obtained, which improves the robustness and accuracy of the image samples generated by GAN, thereby improving the sample quality and generation efficiency.

[0099] In some embodiments, the student model is distilled and trained based on the probability distribution vector and the trained GAN to obtain the target student model, including:

[0100] (1) Determine the KL divergence loss function based on the KL divergence between the output representation of the student model and the probability distribution vector; determine the distance loss function based on the distance between the output representation of the generator and the mapping representation in the trained GAN; wherein, the mapping representation is obtained by mapping the output representation of the generator to the image space based on the generator.

[0101] In this embodiment, the KL divergence loss function is expressed by the following formula:

[0102]

[0103] Among them, L KL For KL divergence loss, x′ (i) The image is generated by the trained GAN based on the image after sampling noise, where β is the weighting coefficient, m is the number of samples in a single training iteration, and T(x′) is the weighting factor. (i) S(x′) represents the fused output. (i) ) represents the output of the student model.

[0104] The distance loss function is expressed by the following formula:

[0105]

[0106] Among them, L F Let be the distance loss function, τ be the temperature coefficient, and G be the generator. F This is a normal form operation.

[0107] (2) Determine the second joint loss function based on the KL divergence loss function and the distance loss function, and iteratively train the student model according to the second joint loss function. When the student model converges or reaches the maximum number of iterations, the target student model is obtained.

[0108] In this embodiment, the second joint loss function L2 is expressed by the following formula:

[0109] L2=a2*L KL +b2*L F ;

[0110] Where a2 and b2 are both weight factors no greater than 1, the values ​​of which can be set according to user needs.

[0111] In this embodiment, the student model is iteratively trained using the second joint loss function described above. In each iteration, the network parameters of the student model are updated until the student model converges or reaches the maximum number of iterations, thus obtaining the target student model.

[0112] Figure 5 This is the third flowchart illustrating the multi-heterogeneous model fusion classification model training method provided by this invention. Figure 5 In the illustrated embodiment, the probability distribution vector (used to represent the probability of each category) is obtained by fusing the output representations of multiple teacher models (T1, ..., Tn) using DS evidence theory, and the KL divergence is calculated by combining it with the output representation of the student model S to determine the KL divergence loss function L. KL The distance function L is obtained by inversely mapping the output representation of the generator G to the image space. F Finally, using L KL and L F Determine the second joint loss to complete the iterative training of the student model.

[0113] This invention provides a classification model training method that integrates multiple heterogeneous models. The KL divergence loss function is determined by the KL divergence between the output representation of the student model and the probability distribution vector; the distance loss function is determined by the distance between the output representation of the generator and the mapping representation; and a second joint loss function is obtained. The student model is then iteratively trained to obtain the target student model, which improves the prediction performance of the image classification model and further improves the image classification accuracy.

[0114] The classification method provided by this invention is described below. The classification method described below can be referred to in correspondence with the classification model training method of multi-heterogeneous model fusion described above.

[0115] Figure 6 This is a flowchart illustrating the classification method provided by the present invention, as shown below. Figure 6 As shown, the method includes the following:

[0116] Step 610: Obtain the image data to be classified.

[0117] In this step, the image data to be classified can be images captured in real time or existing images obtained from an image database.

[0118] Step 620: Process the image data to be classified based on the target student model to obtain the target classification result; wherein, the target student model is trained based on the classification model training method of multi-heterogeneous model fusion.

[0119] In this step, the target student model is trained through the following steps:

[0120] (1) Based on the DS evidence theory, the output representations of multiple teacher models are fused to obtain the fused output representation, and the probability distribution vectors of each category in the category set corresponding to the multiple teacher models are determined according to the fused output representation; wherein, the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks;

[0121] (2) The Generative Adversarial Network (GAN) is trained based on the probability distribution vector to obtain the trained GAN. The student model is then trained by distillation based on the probability distribution vector and the trained GAN to obtain the target student model for performing image classification tasks.

[0122] It should be noted that the implementation of each training step of the target student model corresponds one-to-one with steps 110-120 above, and will not be repeated in this embodiment.

[0123] In this embodiment, the target student model obtained through the above training processes the image data to be classified, which can obtain more accurate classification results to meet the needs of image classification tasks in different scenarios.

[0124] The classification method provided in this invention uses a target student model trained by a multi-heterogeneous model fusion classification model training method to process the image data to be classified, thereby improving the image classification accuracy in various complex scenarios.

[0125] The following describes the multi-heterogeneous model fusion classification model training device provided by the present invention. The multi-heterogeneous model fusion classification model training device described below and the multi-heterogeneous model fusion classification model training method described above can be referred to in correspondence.

[0126] Figure 7 This is a schematic diagram of the structure of the classification model training device for multi-heterogeneous model fusion provided by the present invention, as shown below. Figure 7 As shown, the device includes a model fusion module 710 and a model distillation training module 720.

[0127] The model fusion module 710 is used to fuse the output representations of multiple teacher models based on the DS evidence theory to obtain a fused output representation, and to determine the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation; wherein, the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks;

[0128] The model distillation training module 720 is used to train the generative adversarial network (GAN) based on the probability distribution vector to obtain the trained GAN. The student model is then distilled and trained based on the probability distribution vector and the trained GAN to obtain the target student model for performing image classification tasks.

[0129] This invention provides a multi-heterogeneous model fusion classification model training device. It fuses the output representations of multiple teacher models using DS evidence theory to obtain a fused output representation. Based on the fused output representation, it determines the probability distribution vectors of each category in the category set corresponding to the multiple teacher models. Then, it uses the probability distribution vectors to guide the training of a generative adversarial network (GAN). Finally, it distills the student model based on the probability distribution vectors and the trained GAN to obtain the target student model. This device achieves the fusion and distillation of image classification models using DS evidence theory and generative models, thereby improving the predictive performance of image classification models and thus increasing the accuracy of image classification results.

[0130] The classification device provided by the present invention is described below. The classification device described below and the classification method described above can be referred to in correspondence.

[0131] Figure 8 This is a schematic diagram of the classification device provided by the present invention, as shown below. Figure 8 As shown, the device includes an image acquisition module 810 and a classification module 820.

[0132] Image acquisition module 810 is used to acquire image data to be classified;

[0133] The classification module 820 is used to process the image data to be classified based on the target student model to obtain the target classification result; wherein, the target student model is trained based on the classification model training method of multi-heterogeneous model fusion.

[0134] The classification device provided in this invention processes the image data to be classified using a target student model trained by a classification model training method that integrates multiple heterogeneous models, thereby improving the image classification accuracy in various complex scenarios.

[0135] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a multi-heterogeneous model fusion classification model training method. This method includes: fusing the output representations of multiple teacher models based on DS evidence theory to obtain a fused output representation, and determining the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation; wherein the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks; training a generative adversarial network (GAN) based on the probability distribution vector to obtain a trained GAN, and distilling the student model according to the probability distribution vector and the trained GAN to obtain a target student model to perform the image classification task.

[0136] Alternatively, a classification method may be performed, which includes: acquiring image data to be classified; processing the image data to be classified based on a target student model to obtain a target classification result; wherein the target student model is trained based on a classification model training method that integrates multiple heterogeneous models.

[0137] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-heterogeneous model fusion classification model training method provided by the above methods. The method includes: fusing the output representations of multiple teacher models based on DS evidence theory to obtain a fused output representation, and determining the probability distribution vectors of each category in the category set corresponding to the multiple teacher models according to the fused output representation; wherein the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks; training a generative adversarial network (GAN) based on the probability distribution vector to obtain a trained GAN, and distilling the student model according to the probability distribution vector and the trained GAN to obtain a target student model to perform the image classification task.

[0139] Alternatively, a classification method may be performed, which includes: acquiring image data to be classified; processing the image data to be classified based on a target student model to obtain a target classification result; wherein the target student model is trained based on a classification model training method that integrates multiple heterogeneous models.

[0140] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a classification model training method for multi-heterogeneous model fusion provided by the above methods. The method includes: fusing the output representations of multiple teacher models based on DS evidence theory to obtain a fused output representation, and determining the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation; wherein the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks; training a generative adversarial network (GAN) based on the probability distribution vector to obtain a trained GAN, and distilling a student model based on the probability distribution vector and the trained GAN to obtain a target student model for performing the image classification task.

[0141] Alternatively, a classification method may be performed, which includes: acquiring image data to be classified; processing the image data to be classified based on a target student model to obtain a target classification result; wherein the target student model is trained based on a classification model training method that integrates multiple heterogeneous models.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a classification model by fusing multiple heterogeneous models, characterized in that, include: Based on the DS evidence theory, the output representations of multiple teacher models are fused to obtain a fused output representation. The probability distribution vectors of each category in the category set corresponding to the multiple teacher models are then determined based on the fused output representation. The fused output representation is used to represent the probability and out-of-domain probability of each category in the category set. Different teacher models are used to perform different image classification tasks. The Generative Adversarial Network (GAN) is trained based on the probability distribution vector to obtain a trained GAN. A student model is then trained using distillation techniques based on the probability distribution vector and the trained GAN to obtain a target student model for performing image classification tasks. The process of training the GAN based on the probability distribution vector to obtain the trained GAN includes: Based on the GAN, the probability distribution vector is mapped to the image space, and the GAN is iteratively trained according to the first joint loss function. When the GAN converges or reaches the maximum number of iterations, the trained GAN is obtained. The first joint loss function is determined based on the probability value of the class with the highest probability after fusion, the random sampling vector determined by the fused probability space, and unlabeled image samples. The student model is distilled and trained according to the probability distribution vector and the trained GAN to obtain the target student model, which includes: The KL divergence loss function is determined based on the KL divergence between the output representation of the student model and the probability distribution vector; the distance loss function is determined based on the distance between the output representation of the generator and the mapped representation in the trained GAN; wherein, the mapped representation is obtained by mapping the output representation of the generator to the image space based on the generator; A second joint loss function is determined based on the KL divergence loss function and the distance loss function, and the student model is iteratively trained according to the second joint loss function. When the student model converges or reaches the maximum number of iterations, the target student model is obtained.

2. The classification model training method based on the fusion of multiple heterogeneous models according to claim 1, characterized in that, The KL divergence loss function is expressed by the following formula: Among them, L KL For the KL divergence loss, x ′(i) The image is generated by the trained GAN based on the image after sampling noise, where β is the weighting coefficient, m is the number of samples in a single training iteration, and T(x) is the image generated by the trained GAN. ′(i) S(x) represents the fused output. ′(i) ) represents the output of the student model; The distance loss function is expressed by the following formula: Among them, L F Let τ be the distance loss function, τ be the temperature coefficient, and G be the generator. F This is a normal form operation.

3. The classification model training method based on the fusion of multiple heterogeneous models according to claim 1, characterized in that, The step of determining the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation includes: The probability distribution vector is determined based on the fused output representation and the number of categories in the union of multiple category sets; the data length of the probability distribution vector is the number of categories in the union plus one.

4. A classification method, characterized in that, include: Obtain the image data to be classified; The image data to be classified is processed based on the target student model to obtain the target classification result; wherein the target student model is trained based on the classification model training method of multi-heterogeneous model fusion as described in any one of claims 1-3.

5. A classification model training device for multi-heterogeneous model fusion, employing the multi-heterogeneous model fusion classification model training method as described in claim 1, characterized in that, include: The model fusion module is used to fuse the output representations of multiple teacher models based on the DS evidence theory to obtain a fused output representation, and to determine the probability distribution vectors of each category in the category set corresponding to the multiple teacher models based on the fused output representation; wherein, the fused output representation is used to represent the probability and out-of-domain probability of each category in the category set; different teacher models are used to perform different image classification tasks; The model distillation training module is used to train the Generative Adversarial Network (GAN) based on the probability distribution vector to obtain the trained GAN, and to distill the student model according to the probability distribution vector and the trained GAN to obtain the target student model for performing image classification tasks.

6. A sorting device, characterized in that, include: The image acquisition module is used to acquire image data to be classified. The classification module is used to process the image data to be classified based on the target student model to obtain the target classification result; wherein the target student model is trained based on the classification model training method of multi-heterogeneous model fusion as described in any one of claims 1-3.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images