Information processing method, device, computer readable storage medium and computer equipment
Through screening and convergence techniques, the information processing efficiency during knowledge distillation is improved, the problems of slow training speed and high cost in the existing technology are solved, and more efficient model training is achieved.
Patent Information
- Application Number
- CN202111031124.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-03
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-03
AI Technical Summary
The existing knowledge distillation technology is slow to train on large-scale corpus, resulting in high cost of model compression deployment and low information processing efficiency, which in turn affects the model training efficiency.
By obtaining the target sample, using the preset classification model for classification processing, calculating the difficulty coefficient for screening the samples, using the training depth classification model for further processing, converging the preset model based on the probability distribution difference, reducing the number of training samples and calculation amount.
It improves the information processing efficiency during model training, shortens training time, and improves the model training efficiency.
Smart Images

Figure CN114357152B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to an information processing method, apparatus, computer-readable storage medium, and computer equipment. Background Art
[0002] Natural Language Processing (NLP) is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. With the development of computer technology and artificial intelligence technology, people's requirements for natural language processing technology are constantly increasing, and correspondingly, the performance requirements for models are also increasing. Among them, in order to obtain a model with better performance and smaller model size, the model can be trained through knowledge distillation (KD).
[0003] However, in the existing technology, the knowledge distillation technology used for model compression has a slow training speed on large-scale corpus, which makes the cost of model compression deployment high and the information processing efficiency low during model training, which in turn leads to low model training efficiency. Summary of the invention
[0004] The embodiments of the present application provide an information processing method, apparatus, computer-readable storage medium, and computer equipment, which can improve the efficiency of information processing during model training, thereby improving the efficiency of model training.
[0005] The present application provides an information processing method, including:
[0006] Obtain target samples;
[0007] Using a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample;
[0008] Calculating the difficulty coefficient of the target sample according to the first category probability distribution, and screening the target sample based on the difficulty coefficient to obtain a screened target sample, wherein the difficulty coefficient represents the uncertainty degree of each category probability corresponding to the target sample output by the preset classification model;
[0009] Using a trained deep classification model to classify the screened target samples to obtain a second category probability distribution corresponding to the screened target samples, wherein the network depth of the trained deep classification model is greater than the network depth of the preset classification model;
[0010] The difference between the second category probability distribution and the first category probability distribution is calculated, and based on the difference, a preset classification model is converged to obtain a trained classification model, where the trained classification model is used to classify the information to be processed.
[0011] Accordingly, an embodiment of the present application provides an information processing device, including:
[0012] An acquisition unit, used for acquiring a target sample;
[0013] A first classification unit, configured to classify the target sample using a preset classification model to obtain a first category probability distribution corresponding to the target sample;
[0014] A screening unit, configured to calculate a difficulty coefficient of the target sample according to the first category probability distribution, and screen the target sample based on the difficulty coefficient to obtain a screened target sample, wherein the difficulty coefficient represents a degree of uncertainty of each category probability corresponding to the target sample output by a preset classification model;
[0015] A second classification unit is used to classify the screened target samples using a trained deep classification model to obtain a second category probability distribution corresponding to the screened target samples, wherein the network depth of the trained deep classification model is greater than the network depth of the preset classification model;
[0016] A calculation unit is used to calculate the difference between the second category probability distribution and the first category probability distribution, and converge the preset classification model based on the difference to obtain a trained classification model, and the trained classification model is used to classify the information to be processed.
[0017] In one embodiment, the screening unit comprises:
[0018] A first sorting subunit, used to sort each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result;
[0019] The screening subunit is used to screen the target samples that meet the preset conditions according to the first sorting result to obtain screened target samples.
[0020] In one embodiment, the screening unit comprises:
[0021] A first acquisition subunit, configured to acquire a category probability of each category corresponding to each target sample according to the first category probability distribution;
[0022] A second sorting subunit is used to sort the category probability of each category corresponding to each target sample in descending order to obtain a second sorting result;
[0023] A first determining subunit, configured to determine the two top-ranked category probabilities among the category probabilities corresponding to each target sample according to the second sorting result;
[0024] A second determination subunit is used to determine the classification interval difference of each target sample according to the difference between the two most ranked class probabilities;
[0025] The third determination subunit is used to determine the difficulty coefficient of each target sample based on the classification interval difference.
[0026] In one embodiment, the screening unit comprises:
[0027] A second acquisition subunit, configured to acquire a category probability of each category corresponding to each target sample according to the first category probability distribution;
[0028] A third sorting subunit is used to sort the category probability of each category corresponding to each target sample in descending order to obtain a third sorting result;
[0029] A fourth determining subunit, configured to determine the highest ranking category probability among the category probabilities corresponding to each target sample according to the third sorting result;
[0030] A first calculation subunit, configured to calculate a confusion score of each target sample according to the top-ranked category probability;
[0031] The fifth determining subunit is configured to determine a difficulty coefficient of each target sample based on the confusion score.
[0032] In one embodiment, the screening unit comprises:
[0033] A second calculation subunit, configured to calculate the entropy value of the target sample according to the first category probability distribution;
[0034] The sixth determining subunit is configured to determine a difficulty coefficient of each target sample based on the entropy value.
[0035] In one embodiment, the computing unit comprises:
[0036] A relative entropy calculation subunit, used to calculate the relative entropy between the second category probability distribution and the first category probability distribution;
[0037] A difference determination subunit is used to determine the difference between the second category probability distribution and the first category probability distribution according to the relative entropy.
[0038] In one embodiment, the information processing device further includes:
[0039] A model acquisition unit, used to acquire a preset classification model and a trained deep classification model;
[0040] A network depth layer data acquisition unit, used for acquiring a first network depth layer number of the preset classification model and a second network depth layer number of the trained deep classification model when the preset classification model and the trained deep classification model have the same model architecture, wherein the first network depth layer number is less than the second network depth layer number;
[0041] The initialization model parameter determination unit is used to determine the model parameters of the layers of the first network depth layer number of the trained deep classification model as the initialization model parameters of the preset classification model.
[0042] In one embodiment, the acquisition unit includes:
[0043] An initial sample acquisition subunit, configured to acquire an initial sample data set, wherein the initial sample data set includes at least one initial sample;
[0044] A word segmentation subunit, used to perform word segmentation processing on the initial sample to obtain an initial sample word set;
[0045] A feature extraction subunit, used to extract features from the initial sample word set to obtain an initial sample word vector set;
[0046] The data expansion subunit is used to calculate the synonyms of each word in the initial sample word vector, and perform data expansion on the initial sample according to the synonyms to obtain a target sample.
[0047] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any information processing method provided in the embodiment of the present application.
[0048] In addition, an embodiment of the present application also provides a computer device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the information processing method provided in the embodiment of the present application.
[0049] The embodiment of the present application also provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the information processing method provided in the embodiment of the present application.
[0050] The embodiment of the present application obtains a target sample; classifies the target sample using a preset classification model to obtain a first category probability distribution corresponding to the target sample; calculates the difficulty coefficient of the target sample according to the first category probability distribution, and screens the target sample based on the difficulty coefficient to obtain a screened target sample; classifies the screened target sample using a trained deep classification model to obtain a second category probability distribution corresponding to the screened target sample; calculates the difference between the second category probability distribution and the first category probability distribution, and converges the preset classification model based on the difference to obtain a trained classification model, which is used to classify the information to be processed. In this way, the difficulty coefficient of each target sample is calculated by the first category probability distribution corresponding to the target sample obtained by classifying the target sample using the preset classification model, and the target sample is screened according to the difficulty coefficient to obtain the screened target sample, and then the preset classification model is trained based on the screened target sample and the trained classification model. This method of specifically screening the training samples according to the difficulty coefficient of each target sample reduces the number of training samples, thereby reducing the amount of calculation and shortening the training time, thereby improving the information processing efficiency in the model training process, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0052] Figure 1 This is a schematic diagram of an implementation scenario of an information processing method provided in an embodiment of the present application;
[0053] Figure 2 It is a flowchart of an information processing method provided by an embodiment of the present application;
[0054] Figure 3 This is a schematic diagram of target sample screening of an information processing method provided in an embodiment of the present application;
[0055] Figure 4 It is a schematic diagram of a model training of an information processing method provided in an embodiment of the present application;
[0056] Figure 5 is another flowchart of an information processing method provided by an embodiment of the present application;
[0057] Figure 6 is a schematic diagram of the structure of an information processing device provided in an embodiment of the present application;
[0058] Figure 7 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0060] The embodiments of the present application provide an information processing method, apparatus, computer-readable storage medium and computer equipment. The information processing apparatus can be integrated in a computer equipment, which can be a server or a terminal.
[0061] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc. that can perform information processing, but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
[0062] In order to better illustrate the embodiments of the present application, please refer to the following terms for reference:
[0063] Knowledge Distillation (KD): A method for model compression that uses the output of a teacher model to guide a smaller student model to learn in order to achieve knowledge transfer.
[0064] Teacher model: also called teacher model, a model used for knowledge distillation, which is generally more complex, has a larger model capacity and better performance.
[0065] Student model: The student model to be compressed generally has a small model capacity and a simpler model structure.
[0066] Relative entropy: also known as Kullback-Leibler divergence or information divergence, is an asymmetric measure of the difference between two probability distributions. In information theory, relative entropy is equivalent to the difference in the Shannon entropy of two probability distributions. Relative entropy is the loss function of some optimization algorithms, such as the Expectation-Maximization algorithm (EM). In this case, one of the probability distributions involved in the calculation is the true distribution, and the other is the theoretical (fitted) distribution. The relative entropy represents the information loss generated when the theoretical distribution is used to fit the true distribution.
[0067] See also Figure 1 , taking the example of an information processing device integrated into a computer device, Figure 1 A schematic diagram of an implementation scenario of the information processing method provided in the embodiment of the present application, wherein a computer device can obtain a target sample; a preset classification model is used to classify the target sample to obtain a first category probability distribution corresponding to the target sample; the difficulty coefficient of the target sample is calculated according to the first category probability distribution, and the target sample is screened based on the difficulty coefficient to obtain a screened target sample; a trained deep classification model is used to classify the screened target sample to obtain a second category probability distribution corresponding to the screened target sample; the difference between the second category probability distribution and the first category probability distribution is calculated, and the preset classification model is converged based on the difference to obtain a trained classification model, which is used to classify the information to be processed. Among them, information processing can be understood as the processing of information such as training samples in the process of building a model with better performance to improve the efficiency of model training.
[0068] It should be noted that Figure 1 The implementation environment scenario diagram of the information processing method shown is only an example. The implementation environment scenario of the information processing method described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of information processing and the emergence of new business scenarios, the technical solution provided by the present application is also applicable to similar technical problems.
[0069] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0070] This embodiment will be described from the perspective of an information processing device. The information processing device may be integrated into a computer device. The computer device may be a server, and this application does not limit this.
[0071] See also Figure 2 , Figure 2 : is a flow chart of an information processing method provided in an embodiment of the present application. The information processing method includes:
[0072] In step 101, a target sample is obtained.
[0073] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. With the development of computer technology and artificial intelligence technology, people's requirements for natural language processing technology are constantly increasing, and correspondingly, the performance requirements for models are also increasing. The models obtained by model training are generally large in size. These large models have a long execution time and a large amount of calculation, which makes the deployment cost of the model high and cannot be widely used in various devices, such as resource-constrained devices.
[0074] In order to obtain a model with better performance and smaller size, the trained model with better performance and larger size can be compressed through model compression technology to obtain a model with better performance and smaller size, so that the trained model can be more applied to devices with limited resources such as memory, such as mobile phones, robots and other mobile devices.
[0075] Among them, the model can be trained by knowledge distillation (KD). In the prior art, a large and high-performance teacher model is generally trained on the training set, and the training set is augmented, for example, the training set is expanded to about 10 times, and then knowledge distillation can be performed on the expanded training set. However, this training method requires the use of a large-scale augmented training set to train the model, and the training speed is slow. If the student model has gradually mastered the characteristics of certain input samples during the learning process, these samples still need to be repeatedly learned, and the teacher model with a large amount of calculation needs to be repeatedly executed, resulting in redundancy in training, increasing the amount of calculation of unnecessary model compression, and making the cost of model compression deployment high. Therefore, the existing training method has low information processing efficiency during the model training process, which leads to low efficiency of model training.
[0076] In order to solve the above problems, the present application provides an information data method, which considers the dynamic changes in the performance of the student model during the knowledge distillation process, and determines the sample difficulty measurement index to screen samples that are difficult to predict for the student model. The sample difficulty measurement index is used to measure the difficulty of the student model in predicting a certain sample, that is, the uncertainty of the student model in the results output based on the sample. Then, the teacher model can be compressed based on these targeted screened samples to obtain a trained student model, avoiding the student model from repeatedly learning the samples it has already mastered, reducing the amount of calculation for unnecessary model compression, thereby improving the information processing efficiency in the model training process, and then improving the efficiency of model training. The information processing method provided by the present application is described in detail below.
[0077] First, a target sample may be obtained, wherein the target sample may be at least one training sample for model training.
[0078] In one embodiment, the target sample can be obtained by performing data augmentation on the initial training data set to enhance the performance of the student model obtained by knowledge distillation. Specifically, for the acquired initial training data set, for any word in each target sample therein, synonym replacement can be performed by using a context word vector based on a pre-trained model or a static word vector (GloVe), and each sample can obtain D augmented samples. These samples obtained by data augmentation and the samples in the initial training data set constitute an augmented data set, that is, multiple target samples are obtained. Among them, D can be a value greater than 1, and the size of D can be set according to the needs of the specific business, for example, it can be set to a constant such as 10, 15 or 20.
[0079] Specifically, an initial sample data set can be obtained, which includes at least one initial sample; an initial sample word set is obtained by performing word segmentation on the initial sample; feature extraction is performed on the initial sample word set to obtain an initial sample word vector set; synonyms of each word in the initial sample word vector are calculated, and data expansion is performed on the initial sample according to the synonyms to obtain a target sample.
[0080] Among them, the obtained initial samples can be segmented through text preprocessing, wherein the initial samples of different languages or different fields can have different segmentation processing methods. For example, for English, the words can be segmented by the spaces between words, and for Chinese, the word segmentation tool can be used for processing. For the initial samples of professional fields, the word segmentation algorithm can be designed according to the characteristics of the samples in different professional fields, so as to achieve word segmentation processing, etc., and obtain the initial sample word set after the word segmentation processing. For example, for the initial sample "I ate a sandwich", the word segmentation processing can obtain "I, ate, a, sandwich", etc. By performing word segmentation processing on each initial sample in this way, the initial sample word set can be obtained, and then the feature extraction of the initial sample word set can be performed through the synonym prediction model to obtain the initial sample word vector set, and the synonyms of each word in the initial sample word vector are calculated. According to the synonyms, the data of the initial sample is expanded to obtain the target sample.
[0081] Among them, by performing part-of-speech analysis on the words in the initial sample word set, the words in the vocabulary set that do not need to be expanded with synonyms can be filtered. The parts of speech of the words that do not need to be expanded with synonyms can be prepositions, articles, etc. For example, for the words "you", "of" and "what" in the vocabulary set "you, lunch, eat, of, what" are prepositions, articles, etc., since replacing these words cannot enhance the diversity of the data and may even destroy the original semantics of the sample, the words of these parts of speech can be filtered, and then the synonyms of each word in the filtered initial sample word set are calculated to achieve data expansion of the initial sample and obtain the target sample.
[0082] The training sample data such as the initial sample can be obtained from a memory connected to the information processing device, or from other data storage terminals. It can also be obtained from the memory of a physical terminal, or from a virtual storage space such as a data set or a corpus. In some embodiments, the training sample data can be obtained from one storage location or from multiple storage locations. For example, the training sample data can be stored on a blockchain, and the information processing device obtains the above training sample data from the blockchain. The information processing device can centrally obtain the training sample data in a time period in response to a training sample data acquisition instruction, or it can continuously obtain the training sample data according to a certain data acquisition logic.
[0083] In step 102, a preset classification model is used to classify the target sample to obtain a first category probability distribution corresponding to the target sample.
[0084] Among them, the preset classification model can be a model to be trained, and the model structure is relatively simple, for example, it can be a student model in the knowledge distillation architecture. In order to perform targeted screening of target samples to improve the efficiency of model compression training, the target samples can be first input into the preset classification model, and the difficulty of the student model predicting each target sample is determined according to the output result of the preset classification model, and the target samples are screened according to the difficulty. Specifically, the target samples can be classified by the preset classification model to obtain the first category probability distribution corresponding to the target samples. Optionally, the target samples can be feature extracted by the preset classification model, and the features extracted by the feature extraction can be classified by the classifier to obtain the first category probability distribution corresponding to the target samples. The classifier can be a multilayer perceptron (MLP for short).
[0085] The first category probability distribution can be a probability distribution composed of the probability of a certain target sample corresponding to each category predicted by the preset classification model. For example, assuming that the preset classification model is used to predict whether the emotional polarity of the target sample is positive or negative, and assuming that the preset classification model predicts that the probability of the emotional polarity of the target sample being positive is 0.7, and the probability of being negative is 0.3, then the first category probability distribution can be (positive 0.7, negative 0.3); for another example, assuming that the preset classification model is used to predict what news the target sample belongs to, and assuming that the preset classification model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1, then the first category probability distribution can be (category A news For example, assuming that the preset classification model is used to predict whether the semantics of target sample 1 and target sample 2 are the same, and assuming that the preset classification model predicts that the probability that the semantics of target sample 1 and target sample 2 are the same is 0.9, and the probability that they are different is 0.1, then the first category probability distribution can be (same 0.9, different 0.1); for example, assuming that the preset classification model is used to predict whether target sample A contains target sample B, and assuming that the preset classification model predicts that the probability that target sample A contains target sample B is 0.2, and the probability that target sample A does not contain target sample B is 0.8, then the first category probability distribution can be (contains 0.2, does not contain 0.8), and so on.
[0086] Optionally, before using the preset classification model to classify the target sample, the preset classification model can be initialized. Specifically, the preset classification model and the trained deep classification model can be obtained. The trained deep classification model is a trained high-performance model. Compared with the preset classification model, the model structure is more complex and the model capacity is larger. For example, it can be a teacher model in the knowledge distillation architecture. When the preset classification model and the trained deep classification model have the same model architecture, in order to improve the speed of model training, the first network depth layer number of the preset classification model and the second network depth layer number of the trained deep classification model can be obtained. The first network depth layer number is less than the second network depth layer number. The model parameters of the layers before the first network depth layer number of the trained deep classification model are determined as the initialization model parameters of the preset classification model. For example, it can be assumed that the network depth of the preset classification model is L s , that is, the first network depth level is L s Layer, the network depth of the trained deep classification model is L t , that is, the second network depth level is L t , and L s < <L t , then the front L of the deep classification model can be trained s The parameters of the layer are used as the parameters of the preset classification model.
[0087] Optionally, when the preset classification model and the model architecture of the trained deep classification model are different, the preset classification model can be randomly initialized.
[0088] In step 103, the difficulty coefficient of the target sample is calculated according to the first category probability distribution, and the target sample is screened based on the difficulty coefficient to obtain the screened target sample.
[0089] Among them, the difficulty coefficient can be an indicator to measure the difficulty of the sample, that is, it characterizes the uncertainty of the probability of each category corresponding to the target sample output by the preset classification model, and is used to characterize the difficulty of the target sample. The more uncertain the output result of the preset classification model based on the target sample, the greater the difficulty of the target sample, and the greater the difficulty coefficient. The difficulty coefficient can be a score, a ratio, or other forms of parameters, which are not limited here.
[0090] Please refer to Figure 3 , Figure 3 A schematic diagram of target sample screening of an information processing method provided in an embodiment of the present application, after the target samples in the target sample data set are input into a preset classification model to obtain the first category probability, the difficulty coefficient of the target samples can be calculated according to the first category probability distribution, and the target samples can be screened based on the difficulty coefficient to obtain the screened target samples.
[0091] In one embodiment, the classification interval difference (Margin) of each target sample can be calculated according to the first category probability distribution, and the difficulty coefficient of the target sample can be calculated according to the classification interval difference. Specifically, the category probability of each category corresponding to each target sample can be obtained according to the first category probability distribution, and the category probability of each category corresponding to each target sample can be sorted in order from large to small to obtain a second sorting result. The two top category probabilities in the category probability corresponding to each target sample are determined according to the second sorting result, and the classification interval difference of each target sample is determined according to the difference between the two top category probabilities, and the difficulty coefficient of each target sample is determined based on the classification interval difference. Among them, the difficulty coefficient can be determined by the classification interval difference, can be equal to the classification interval difference, and can also be transformed into other values according to the classification interval difference, such as weighted processing and other transformation methods, which are not limited here. For the sake of convenience, the classification interval difference is equal to the difficulty coefficient as an example for specific description, wherein it can be assumed that the difficulty coefficient is u x , that is, the classification interval difference is u x , then the specific calculation formula of the difficulty coefficient can be
[0092]
[0093] Among them, x can be any target sample, It can be the category that the preset classification model believes the target sample x is most likely to belong to. is the probability of the category to which the target sample X most likely belongs, It can be the category that the preset classification model believes the target sample x is the second most likely to belong to. is the probability of the second most likely category of the target sample X, and the probability difference u between the most likely category and the second most likely category xThe smaller it is, the more the preset classification model tends to give ambiguous predictions, which means that this target sample is more difficult for the preset classification model. Then, the difficulty coefficient of the target sample can be determined according to the classification interval difference. For example, it can be assumed that the preset classification model is used to predict what category of news the target sample x belongs to. At the same time, it is assumed that the preset classification model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1. Then the first category probability distribution can be (category A news 0.3, category B news 0.6, category C news 0.1). The category probabilities of each category corresponding to each target sample are sorted in order from large to small, and the second sorting results are obtained as 0.6, 0.3, and 0.1. According to the second sorting result, it can be obtained that the preset classification model believes that the category that the target sample x is most likely to belong to is category B news, P(B|x) is 0.6, and the preset classification model believes that the second most likely category to which the target sample x belongs is category A news, P(A|x) is 0.3, then the classification interval difference u x =0.6-0.3=0.3, that is, the difficulty coefficient is 0.3.
[0094] In one embodiment, the confusion score (Confusion Score) of each target sample can be calculated according to the first category probability distribution, and the difficulty coefficient of the target sample can be calculated according to the confusion score. Specifically, the category probability of each category corresponding to each target sample can be obtained according to the first category probability distribution, and the category probability of each category corresponding to each target sample can be sorted in order from large to small to obtain a third sorting result. The category probability with the highest ranking in the category probability corresponding to each target sample is determined according to the third sorting result, and the confusion score of each target sample is calculated according to the first category probability distribution, and the difficulty coefficient of each target sample is determined based on the confusion score. Among them, the difficulty coefficient can be determined by the confusion score, can be equal to the confusion score, and can also be transformed into other values according to the confusion score, such as weighted processing and other transformation methods, which are not limited here. For the sake of convenience, the confusion score is equal to the difficulty coefficient as an example for specific description. Therefore, the difficulty coefficient u x The specific calculation formula can be
[0095]
[0096] in, It means that when P(y|x) reaches its maximum value, the value of y is It can be the category with the largest probability value in the first category probability distribution of a target sample. It can be the probability of the category with the largest probability value in the probability distribution of the first category of a target sample, so that the confusion score can be obtained as The larger the confusion score, the less confident the preset classification model is in its prediction of the category corresponding to the target sample, which indicates that the target sample is more difficult for the preset classification model.
[0097] For example, it can be assumed that the preset classification model is used to predict what category of news the target sample x belongs to. At the same time, it can be assumed that the preset classification model predicts that the probability that the target sample C belongs to category A news is 0.3, the probability that it belongs to category B news is 0.6, and the probability that it belongs to category C news is 0.1. Then the first category probability distribution can be (category A news 0.3, category B news 0.6, category C news 0.1). The category probabilities of each category corresponding to the target sample C are sorted in order from large to small, and a third sorting result of 0.6, 0.3, and 0.1 can be obtained. According to the third sorting result, it can be determined that the category probability with the highest ranking in the category probability corresponding to the target sample C is 0.6. The confusion score of the target sample C is calculated based on the category probability with the highest ranking, and the confusion score is 1-0.6=0.4, that is, the difficulty coefficient is u x =0.4. At the same time, it can also be assumed that the preset classification model predicts that the probability of the target sample D belonging to category A news is 0.1, the probability of belonging to category B news is 0.2, and the probability of belonging to category C news is 0.7. Then the first category probability distribution can be (category A news 0.1, category B news 0.2, category C news 0.7). The category probabilities of each category corresponding to the target sample D are sorted in order from large to small, and the third sorting result 0.7, 0.2, 0.1 can be obtained. According to the third sorting result, it can be determined that the category probability with the highest ranking in the category probability corresponding to the target sample D is 0.7. The confusion score of the target sample D is calculated based on the category probability with the highest ranking, and the confusion score is 1-0.7=0.3, that is, the difficulty coefficient is u x =0.3. Comparing the confusion scores of target samples C and D, it can be found that the confusion score of target sample C is larger, that is, the difficulty coefficient of target sample C is larger, indicating that target sample C is more difficult for the preset classification model.
[0098] In one embodiment, the entropy value (Entropy) of each target sample can be calculated according to the first category probability distribution, and the difficulty coefficient of the target sample can be calculated according to the entropy value. Specifically, the entropy value of the target sample can be calculated according to the first category probability distribution, and the difficulty coefficient of each target sample can be determined based on the entropy value. The entropy value represents the expected value of the amount of information and represents the degree of uncertainty. The difficulty coefficient of each target sample can be determined by the entropy value of the target sample, which can be equal to the entropy value, or can be transformed into other values according to the entropy value, such as weighted processing and other transformation methods, which are not limited here. For the sake of convenience, the entropy value is used as the difficulty coefficient as an example for specific description. Therefore, the difficulty coefficient u x The specific calculation formula can be
[0099]
[0100] Among them, P(y|x) represents the probability that the preset classification model presets the target sample x corresponding to the category y, logP(y|x) represents the result of the logarithmic operation of P(y|x), and ∑ represents the symbol used in mathematics to find the sum of multiple numbers. The entropy value reflects the uncertainty of the preset classification model in predicting the target sample. The larger the entropy value, the higher the degree of uncertainty, indicating that this target sample is more difficult for the current preset classification model. For example, assuming that the entropy value of target sample 1 is 5, that is, the difficulty coefficient is 5, assuming that the entropy value of target sample 2 is 7, that is, the difficulty coefficient is 7, then the difficulty coefficient of target sample 2 is greater than that of target sample 1, then relative to target sample 1, target sample 2 is more difficult for the preset classification model.
[0101] Because the model structure of the trained deep classification model is relatively complex and has many model parameters, the execution time and computational consumption of the trained deep classification model are much greater than those of the preset classification model. Therefore, in order to improve the efficiency of model training, the target samples can be selectively screened for knowledge distillation. Therefore, the trained deep classification model can be selectively executed on the more difficult target samples according to the difficulty coefficient of the target samples to obtain the corresponding outputs, and then these outputs can be used to guide the training of the preset classification model. At the same time, since the generalization performance of difficult samples is better, the preset classification model can achieve better performance by learning from the difficult samples obtained by screening. In one embodiment, the step of screening the target samples based on the difficulty coefficient to obtain the screened target samples may include:
[0102] Sort each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result;
[0103] The target samples meeting the preset conditions are screened according to the first sorting result to obtain screened target samples.
[0104] Among them, each target sample can be sorted in order from large to small according to the difficulty coefficient of the target sample, so as to obtain a first sorting result, and the target samples that meet the preset conditions are screened according to the first sorting result to obtain the screened target samples. Among them, there are many ways to determine the screened target samples. For example, when each target sample is sorted in order from large to small according to the difficulty coefficient of the target sample, the preset condition can be that the ranking in the first sorting result is greater than the preset ranking, that is, the target sample that meets the preset condition can be a target sample whose ranking is greater than the preset ranking. Among them, the preset ranking can be determined according to the actual training needs. For example, when 50,000 samples are required for training, 50,000 target samples ranked in the top 50,000 can be obtained according to the first sorting result. For another example, assuming that the total number of target samples in the target sample training set is N, the preset ranking can be a target sample whose ranking is greater than pN in the first sorting result, wherein p can be a value from 0 to 1, and the specific value can be set according to the actual situation, for example, it can be set to a constant between 0.1 and 0.5. For example, it can be assumed that the total number N of target samples in the target sample training set is 10000, p is 0.2, then pN is 10000×0.2=2000, and the preset ranking is 2000. The target sample that meets the preset conditions can be a target sample ranked greater than 2000.
[0105] In addition, the target samples can also be sorted in ascending order. When each target sample is sorted in ascending order according to the difficulty coefficient of the target sample, the preset condition can be that the ranking in the first sorting result is less than the preset ranking, that is, the target sample that meets the preset condition can be a target sample with a ranking less than the preset ranking.
[0106] In one embodiment, the target sample that meets the preset conditions may refer to a target sample whose difficulty coefficient is greater than a preset threshold, and the preset threshold may be a critical value. When the difficulty coefficient of the target sample is greater than the critical value, the target sample may be determined as a screened target sample for subsequent training of a preset classification model. The specific value of the preset threshold may be determined based on actual applications and is not limited here.
[0107] In step 104, the trained deep classification model is used to classify the screened target samples to obtain a second category probability distribution corresponding to the screened target samples.
[0108] Among them, the network depth of the trained deep classification model can be greater than the network depth of the preset classification model.
[0109] Specifically, the trained deep classification model can be used to extract features of the screened target samples, and then the features obtained by the feature extraction can be classified to obtain the second category probability distribution corresponding to the screened target samples.
[0110] The second category probability distribution can be a probability distribution composed of the probabilities of a target sample corresponding to each category predicted by the trained deep classification model. For example, assuming that the trained deep classification model is used to predict whether the emotional polarity of the target sample is positive or negative, and assuming that the trained deep classification model predicts that the probability of the target sample's emotional polarity being positive is 0.7 and the probability of being negative is 0.3, then the second category probability distribution can be (positive 0.7, negative 0.3); for another example, assuming that the trained deep classification model is used to predict what news the target sample belongs to, and assuming that the trained deep classification model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1, then the second category probability distribution can be (A For example, assuming that the trained deep classification model is used to predict whether the semantics of target sample 1 and target sample 2 are the same, and assuming that the trained deep classification model predicts that the probability that the semantics of target sample 1 and target sample 2 are the same is 0.9, and the probability that they are different is 0.1, then the second category probability distribution can be (same 0.9, different 0.1); for another example, assuming that the trained deep classification model is used to predict whether target sample A contains target sample B, and assuming that the trained deep classification model predicts that the probability that target sample A contains target sample B is 0.2, and the probability that target sample A does not contain target sample B is 0.8, then the second category probability distribution can be (contains 0.2, does not contain 0.8), and so on.
[0111] In step 105, the difference between the second category probability distribution and the first category probability distribution is calculated, and the preset classification model is converged based on the difference to obtain a trained classification model.
[0112] For details, please refer to Figure 4 , Figure 4 It is a model training schematic diagram of an information processing method provided in an embodiment of the present application. The difference between the probability distribution of the second category and the probability distribution of the first category can be calculated through a loss function, and the model parameters of the preset classification model can be updated according to the difference. When the preset classification model does not meet the convergence condition, the target samples can continue to be dynamically screened, so that the screened target samples can be input into the trained deep classification model, and then the difference is calculated according to the output result. The model parameters of the preset classification model are iteratively trained according to the difference. When the preset classification model meets the convergence condition, the trained classification model can be obtained.
[0113] Among them, the trained classification model can be used to classify the information to be processed. For example, the trained classification model can be used to predict whether the sentiment polarity of the information to be processed is positive or negative; for another example, the trained classification model can be used to predict what category the information to be processed belongs to, wherein the information to be processed can be a piece of news information, for example, the news information can be a news text, and the trained classification model can be used to predict what category of news this news information belongs to, such as sports news, financial news, entertainment news, etc.; for another example, it can be used to determine whether the semantics of the information to be processed 1 and the information to be processed 2 are the same, for example, it can be used to determine whether the semantics of text a is the same as the semantics of text b; for another example, it can be used to predict whether the semantics of the information to be processed c contains the semantics of the information to be processed d and other natural language understanding and semantic analysis tasks.
[0114] In one embodiment, the relative entropy between the second category probability distribution and the first category probability distribution of the target sample can be calculated, and the difference between the second category probability distribution and the first category probability distribution can be determined based on the relative entropy. The relative entropy between the second category probability distribution and the first category probability distribution can be used to measure the difference between the two probability distributions. The first category probability distribution calculated by the preset classification model can be used as the true distribution, and the second probability distribution calculated by the trained deep classification model can be used as the theoretical (fitting) distribution. The relative entropy represents the information loss generated when the theoretical distribution is used to fit the true distribution. The difference between the preset classification model and the trained deep classification model can be obtained based on the relative entropy. The specific calculation formula can be
[0115]
[0116] Among them, X can be the target sample after screening, It indicates that x can be a target sample in X, P(x) represents the first probability distribution, and Q(x) represents the second probability distribution. It should be noted that the relative entropy can be calculated in many different ways, that is, the specific calculation formula of the relative entropy can be in many forms, which is not limited here.
[0117] From the above, it can be seen that the embodiment of the present application obtains a target sample; uses a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample; calculates the difficulty coefficient of the target sample according to the first category probability distribution, and screens the target sample based on the difficulty coefficient to obtain a screened target sample; uses a trained deep classification model to classify the screened target sample to obtain a second category probability distribution corresponding to the screened target sample; calculates the difference between the second category probability distribution and the first category probability distribution, and converges the preset classification model based on the difference to obtain a trained classification model, which is used to classify the information to be processed. In this way, the difficulty coefficient of each target sample is calculated by the first category probability distribution corresponding to the target sample obtained by classifying the target sample with the preset classification model, and the target sample is screened according to the difficulty coefficient to obtain the screened target sample, and then the preset classification model is trained based on the screened target sample and the trained classification model. Compared with the existing technology, this method of specifically screening the training samples according to the difficulty coefficient of each target sample reduces the number of training samples, thereby reducing the amount of calculation in the training process and shortening the training time, thereby improving the information processing efficiency in the model training process, thereby improving the efficiency of model training.
[0118] According to the method described in the above embodiment, the following is further described in detail with examples.
[0119] In this embodiment, the information processing device is specifically integrated in a computer device as an example for explanation. The information processing method is executed by a server, and the preset classification model is used as a student model and the trained deep classification model is used as a teacher model for specific description.
[0120] For a better description of the embodiments of this application, please refer to Figure 5 .like Figure 5 As shown, Figure 5 This is another flow chart of the information processing method provided in the embodiment of the present application. The specific process is as follows:
[0121] In step 201, the server obtains an initial sample data set, performs word segmentation processing on the initial sample to obtain an initial sample word set, performs feature extraction on the initial sample word set to obtain an initial sample word vector set, calculates the synonyms of each word in the initial sample word vector, and performs data expansion on the initial sample according to the synonyms to obtain a target sample.
[0122] In order to enhance the performance of the student model obtained by knowledge distillation, the target sample can be obtained by performing data augmentation on the initial training data set. Specifically, the server can obtain an initial sample data set, which includes at least one initial sample, perform word segmentation on the initial sample to obtain an initial sample word set, perform feature extraction on the initial sample word set to obtain an initial sample word vector set, and then calculate the synonyms of each word in the initial sample word vector, perform data augmentation on the initial sample according to the synonyms, and obtain the target sample.
[0123] Among them, the server can perform word segmentation processing on the acquired initial samples through text preprocessing, wherein the initial samples of different languages or different fields can have different word segmentation processing methods, for example, for English, the words can be segmented by the spaces between words, for Chinese, the word segmentation tool can be used for processing, and for the initial samples of professional fields, the word segmentation algorithm can be designed according to the characteristics of the samples in different professional fields, so as to achieve word segmentation processing, etc., to obtain the initial sample word set after the word segmentation processing. For example, for the initial sample "I ate a sandwich", the word segmentation processing can obtain "I, ate, a, sandwich", etc. By performing word segmentation processing on each initial sample in this way, the initial sample word set can be obtained, and then the feature extraction of the initial sample word set can be performed through the synonym prediction model to obtain the initial sample word vector set, and the synonyms of each word in the initial sample word vector are calculated, and the data of the initial sample is expanded according to the synonyms to obtain the target sample.
[0124] Among them, the server can perform part-of-speech analysis on the words in the initial sample word set, and filter out the words in the vocabulary set that do not need to be expanded with synonyms. The parts of speech of the words that do not need to be expanded with synonyms can be prepositions, articles, etc. For example, for the words "you", "of", and "what" in the vocabulary set "you, lunch, eat, of, what" are prepositions, articles, etc., since replacing these words cannot enhance the diversity of the data and may even destroy the original semantics of the sample, the words of these parts of speech can be filtered, and then the synonyms of each word in the filtered initial sample word set are calculated to achieve data expansion of the initial sample and obtain the target sample.
[0125] The training sample data such as the initial sample can be obtained from a memory connected to the information processing device, or from other data storage terminals. It can also be obtained from the memory of a physical terminal, or from a virtual storage space such as a data set or a corpus. In some embodiments, the training sample data can be obtained from one storage location or from multiple storage locations. For example, the training sample data can be stored on a blockchain, and the information processing device obtains the above training sample data from the blockchain. The information processing device can centrally obtain the training sample data in a time period in response to a training sample data acquisition instruction, or it can continuously obtain the training sample data according to a certain data acquisition logic.
[0126] In step 202, the server obtains a preset classification model and a trained deep classification model. When the preset classification model has the same model architecture as the trained deep classification model, the server obtains the first network depth layer number of the preset classification model and the second network depth layer number of the trained deep classification model, and determines the model parameters of the layers before the first network depth layer number of the trained deep classification model as the initialization model parameters of the preset classification model.
[0127] Before using the student model to classify the target sample, the server can initialize the student model. Specifically, the server can obtain the student model and the teacher model. The teacher model is a trained high-performance model. Compared with the student model, the model structure is more complex and the model capacity is larger. For example, it can be the teacher model in the knowledge distillation architecture. When the student model has the same model architecture as the teacher model, in order to improve the speed of model training, the server can obtain the first network depth layer number of the student model and the second network depth layer number of the teacher model. The first network depth layer number is less than the second network depth layer number. The server determines the model parameters of the first network depth layer number of the teacher model as the initialization model parameters of the student model. For example, it can be assumed that the network depth of the student model is L s , that is, the first network depth level is L s Layer, the network depth of the teacher model is L t , that is, the second network depth level is L t , and L s < <L t , then the front L of the teacher model can be s The parameters of the layer are used as the parameters of the student model.
[0128] Optionally, when the model architecture of the student model is different from that of the teacher model, the student model can be randomly initialized.
[0129] In step 203, the server uses a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample, obtains the category probability of each category corresponding to each target sample based on the first category probability distribution, and sorts the category probability of each category corresponding to each target sample in order from large to small to obtain a second sorting result.
[0130] Please continue to refer to Figure 3 In order to perform targeted screening of target samples to improve the efficiency of model compression training, the server can first input the target samples into the student model, determine the difficulty of the student model predicting each target sample based on the output results of the student model, and screen the target samples based on the difficulty. Specifically, the student model can be used to classify the target samples to obtain the first category probability distribution corresponding to the target samples. Optionally, the student model can be used to extract features from the target samples, and then the features extracted can be classified by a classifier to obtain the first category probability distribution corresponding to the target samples. The classifier can be a multilayer perceptron (MLP).
[0131] The first category probability distribution can be a probability distribution composed of the probability of the student model predicting that a certain target sample corresponds to each category. For example, assuming that the student model is used to predict whether the emotional polarity of the target sample is positive or negative, and assuming that the student model predicts that the probability of the emotional polarity of the target sample being positive is 0.7, and the probability of being negative is 0.3, then the first category probability distribution can be (positive 0.7, negative 0.3); for another example, assuming that the student model is used to predict what news the target sample belongs to, and assuming that the student model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1, then the first category probability distribution can be (category A news 0.3 for Class A news, 0.6 for Class B news, and 0.1 for Class C news); for another example, assuming that the student model is used to predict whether the semantics of target sample 1 and target sample 2 are the same, and assuming that the student model predicts that the probability of target sample 1 and target sample 2 being the same is 0.9, and the probability of being different is 0.1, then the first category probability distribution can be (same 0.9, different 0.1); for another example, assuming that the student model is used to predict whether target sample A contains target sample B, and assuming that the student model predicts that the probability of target sample A containing target sample B is 0.2, and the probability of predicting that target sample A does not contain target sample B is 0.8, then the first category probability distribution can be (contains 0.2, does not contain 0.8), and so on.
[0132] Furthermore, the server can calculate the classification interval difference (Margin) of each target sample according to the first category probability distribution, and calculate the difficulty coefficient of the target sample according to the classification interval difference. Specifically, the category probability of each category corresponding to each target sample can be obtained according to the first category probability distribution, and the category probability of each category corresponding to each target sample is sorted in order from large to small to obtain a second sorting result. For example, it can be assumed that the student model is used to predict what category of news the target sample x belongs to, and it is assumed that the student model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1. Then the first category probability distribution can be (category A news 0.3, category B news 0.6, category C news 0.1), and the category probability of each category corresponding to each target sample is sorted in order from large to small to obtain a second sorting result of 0.6, 0.3, and 0.1.
[0133] In step 204, the server determines the two top-ranked category probabilities among the category probabilities corresponding to each target sample based on the second sorting result, determines the category interval difference of each target sample based on the difference between the two top-ranked category probabilities, and determines the difficulty coefficient of each target sample based on the category interval difference.
[0134] Among them, the server can determine the two most ranked category probabilities in the category probabilities corresponding to each target sample according to the second sorting result, determine the classification interval difference of each target sample according to the difference between the two most ranked category probabilities, and determine the difficulty coefficient of each target sample based on the classification interval difference. Among them, the difficulty coefficient can be determined by the classification interval difference, can be equal to the classification interval difference, or can be transformed into other values according to the classification interval difference, such as weighted processing and other transformation methods, which are not limited here. For the sake of convenience, a specific description is given by taking the classification interval difference equal to the difficulty coefficient as an example, wherein it can be assumed that the difficulty coefficient is u x , that is, the classification interval difference is u x , then the specific calculation formula of the difficulty coefficient can be
[0135]
[0136] Among them, x can be any target sample, It can be the category that the student model thinks the target sample x is most likely to belong to, is the probability of the category to which the target sample X most likely belongs, It can be the category that the student model thinks the target sample x is the second most likely to belong to, is the probability of the second most likely category of the target sample X, and the probability difference u between the most likely category and the second most likely categoryx The smaller it is, the more ambiguous the prediction of the student model is, which means that the target sample is more difficult for the student model. Then, the difficulty coefficient of the target sample can be determined according to the classification interval difference. For example, it can be assumed that the student model is used to predict what category of news the target sample x belongs to. At the same time, it is assumed that the student model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1. Then the probability distribution of the first category can be (category A news 0.3, category B news 0.6, category C news 0.1). The category probabilities of each category corresponding to each target sample are sorted in descending order to obtain the second sorting results of 0.6, 0.3, and 0.1. According to the second sorting result, it can be obtained that the student model believes that the category that the target sample x is most likely to belong to is category B news, P(B|x) is 0.6, and the student model believes that the second most likely category of the target sample x is category A news, P(A|x) is 0.3, then the classification interval difference u x =0.6-0.3=0.3, that is, the difficulty coefficient is 0.3.
[0137] In step 205, the server sorts each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result, and screens the target samples that meet the preset conditions according to the first sorting result to obtain screened target samples.
[0138] In order to improve the efficiency of model training, the target samples can be selectively screened for knowledge distillation. Therefore, the teacher model can be selectively executed on the more difficult target samples according to the difficulty coefficient of the target samples to obtain the corresponding output, and then these outputs can be used to guide the training of the student model. Among them, the server can sort each target sample in order from large to small according to the difficulty coefficient of the target sample to obtain a first sorting result, and screen the target samples that meet the preset conditions according to the first sorting result to obtain the screened target samples. Among them, there are many ways to determine the screened target samples. For example, when each target sample is sorted in order from large to small according to the difficulty coefficient of the target sample, the preset condition can be that the ranking in the first sorting result is greater than the preset ranking, that is, the target sample that meets the preset condition can be a target sample whose ranking is greater than the preset ranking. Among them, the preset ranking can be determined according to the actual training needs. For example, when 50,000 samples are needed for training, 50,000 target samples ranked in the top 50,000 can be obtained according to the first sorting result. For another example, assuming that the total number of target samples in the target sample training set is N, the preset ranking can be the target samples ranked greater than pN in the first sorting result, wherein p can be a value from 0 to 1, and the specific value can be set according to the actual situation, for example, it can be set to a constant between 0.1 and 0.5. For example, it can be assumed that the total number of target samples in the target sample training set N is 10,000, and p is 0.2, then pN is 10,000×0.2=2000, then the preset ranking is 2000, and the target sample that meets the preset conditions can be a target sample ranked greater than 2000.
[0139] In addition, the target samples can also be sorted in ascending order. When each target sample is sorted in ascending order according to the difficulty coefficient of the target sample, the preset condition can be that the ranking in the first sorting result is less than the preset ranking, that is, the target sample that meets the preset condition can be a target sample with a ranking less than the preset ranking.
[0140] In step 206, the server uses the trained deep classification model to classify the screened target sample, obtains the second category probability distribution corresponding to the screened target sample, and calculates the relative entropy between the second category probability distribution and the first category probability distribution.
[0141] Among them, the server can use the teacher model to extract features of the screened target samples, and then classify the features obtained by the feature extraction to obtain the second category probability distribution corresponding to the screened target samples.
[0142] The second category probability distribution can be a probability distribution composed of the probability of the teacher model predicting that a target sample corresponds to each category. For example, assuming that the teacher model is used to predict whether the emotional polarity of the target sample is positive or negative, and assuming that the teacher model predicts that the probability of the emotional polarity of the target sample being positive is 0.7 and the probability of being negative is 0.3, then the second category probability distribution can be (positive 0.7, negative 0.3); for another example, assuming that the teacher model is used to predict what news the target sample belongs to, and assuming that the teacher model predicts that the probability of the target sample belonging to category A news is 0.3, the probability of belonging to category B news is 0.6, and the probability of belonging to category C news is 0.1, then the second category probability distribution can be (category A news 0.3 for Class A news, 0.6 for Class B news, and 0.1 for Class C news); for another example, assuming that the teacher model is used to predict whether the semantics of target sample 1 and target sample 2 are the same, and assuming that the teacher model predicts that the probability of target sample 1 and target sample 2 being the same is 0.9, and the probability of being different is 0.1, then the second category probability distribution can be (same 0.9, different 0.1); for another example, assuming that the teacher model is used to predict whether target sample A contains target sample B, and assuming that the teacher model predicts that the probability of target sample A containing target sample B is 0.2, and the probability of predicting that target sample A does not contain target sample B is 0.8, then the second category probability distribution can be (contains 0.2, does not contain 0.8), and so on.
[0143] In addition, in order to calculate the difference between the student model and the teacher model and train the student model based on the difference, the server can determine the difference by calculating the relative entropy between the second category probability distribution of the target sample and the first category probability distribution. The relative entropy between the second category probability distribution and the first category probability distribution can be used to measure the difference between the two probability distributions. The first category probability distribution calculated by the student model can be used as the true distribution, and the second probability distribution calculated by the teacher model can be used as the theoretical (fitting) distribution. The relative entropy represents the information loss generated when the theoretical distribution is used to fit the true distribution. The difference between the student model and the teacher model can be obtained based on the relative entropy. The relative entropy The specific calculation formula can be
[0144]
[0145] Among them, X can be the target sample after screening, It indicates that x can be a target sample in X, P(x) represents the first probability distribution, and Q(x) represents the second probability distribution. It should be noted that the relative entropy can be calculated in many different ways, that is, the specific calculation formula of the relative entropy can be in many forms, which is not limited here.
[0146] In step 207, the server determines the difference between the second category probability distribution and the first category probability distribution according to the relative entropy, and converges the preset classification model based on the difference to obtain a trained classification model.
[0147] Please continue to refer to Figure 4 The server can determine the difference between the probability distribution of the second category and the probability distribution of the first category according to the relative entropy, so as to update the model parameters of the student model according to the difference. When the student model does not meet the convergence condition, the target samples can continue to be dynamically screened, so that the screened target samples can be input into the teacher model, and then the difference is calculated according to the output result. The model parameters of the student model are iteratively trained according to the difference. When the student model meets the convergence condition, the trained student model can be obtained.
[0148] Among them, the trained student model can be used to classify the information to be processed. For example, the trained student model can be used to predict whether the sentiment polarity of the information to be processed is positive or negative; for another example, the trained student model can be used to predict what category the information to be processed belongs to, wherein the information to be processed can be a piece of news information, for example, the news information can be a news text, and the trained student model can be used to predict what category of news this news information belongs to, such as sports news, financial news, entertainment news, etc.; for another example, it can be used to determine whether the semantics of the information to be processed 1 and the information to be processed 2 are the same, for example, it can be used to determine whether the semantics of text a is the same as the semantics of text b; for another example, it can be used to predict whether the semantics of the information to be processed c contains the semantics of the information to be processed d and other natural language understanding and semantic analysis tasks.
[0149] From the above, it can be seen that the embodiment of the present application obtains a preset classification model and a trained deep classification model through a server. When the preset classification model has the same model architecture as the trained deep classification model, the first network depth layer number of the preset classification model and the second network depth layer number of the trained deep classification model are obtained, and the model parameters of the layers before the first network depth layer number of the trained deep classification model are determined as the initialization model parameters of the preset classification model; the server uses the preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample, obtains the category probability of each category corresponding to each target sample according to the first category probability distribution, and sorts the category probability of each category corresponding to each target sample in order from large to small to obtain a second sorting result; the server determines the category probability corresponding to each target sample according to the second sorting result. The two top-ranked category probabilities, determine the classification interval difference of each target sample based on the difference between the two top-ranked category probabilities, and determine the difficulty coefficient of each target sample based on the classification interval difference; the server sorts each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result; according to the first sorting result, the target samples that meet the preset conditions are screened to obtain screened target samples; the server uses the trained deep classification model to classify the screened target samples to obtain the second category probability distribution corresponding to the screened target samples, and calculates the relative entropy between the second category probability distribution and the first category probability distribution; the server determines the difference between the second category probability distribution and the first category probability distribution based on the relative entropy, and converges the preset classification model based on the difference to obtain the trained classification model. In this way, the target samples are classified by a preset classification model to obtain the first category probability distribution corresponding to the target samples, and the classification interval difference of each target sample is calculated according to the first category probability distribution to obtain the difficulty coefficient of each target sample, and the target samples are screened according to the difficulty coefficient to obtain the screened target samples, and then the preset classification model is trained based on the screened target samples and the trained classification model. Compared with the prior art, this method of specifically screening training samples according to the difficulty coefficient of each target sample reduces the number of training samples, thereby reducing the amount of calculation and shortening the training time, thereby improving the information processing efficiency in the model training process, and thereby improving the efficiency of model training.
[0150] In order to better implement the above method, an embodiment of the present invention further provides an information processing device, which can be integrated in a computer device, and the computer device can be a server.
[0151] For example, Figure 6As shown, it is a schematic diagram of the structure of the information processing device provided in an embodiment of the present application. The information processing device may include an acquisition unit 301, a first classification unit 302, a screening unit 303, a second classification unit 304 and a calculation unit 305, as follows:
[0152] An acquisition unit 301 is used to acquire a target sample;
[0153] The first classification unit 302 is used to classify the target sample using a preset classification model to obtain a first category probability distribution corresponding to the target sample;
[0154] A screening unit 303 is used to calculate the difficulty coefficient of the target sample according to the first category probability distribution, and screen the target sample based on the difficulty coefficient to obtain a screened target sample, wherein the difficulty coefficient represents the uncertainty degree of each category probability corresponding to the target sample output by the preset classification model;
[0155] The second classification unit 304 is used to classify the screened target sample using a trained deep classification model to obtain a second category probability distribution corresponding to the screened target sample, wherein the network depth of the trained deep classification model is greater than the network depth of the preset classification model;
[0156] The calculation unit 305 is used to calculate the difference between the second category probability distribution and the first category probability distribution, and converge the preset classification model based on the difference to obtain a trained classification model, and the trained classification model is used to classify the information to be processed.
[0157] In one embodiment, the screening unit 303 includes:
[0158] A first sorting subunit, used to sort each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result;
[0159] The screening subunit is used to screen the target samples that meet the preset conditions according to the first sorting result to obtain screened target samples.
[0160] In one embodiment, the screening unit 303 includes:
[0161] A first acquisition subunit, configured to acquire a category probability of each category corresponding to each target sample according to the first category probability distribution;
[0162] A second sorting subunit is used to sort the category probability of each category corresponding to each target sample in descending order to obtain a second sorting result;
[0163] A first determining subunit is used to determine the two most highly ranked category probabilities among the category probabilities corresponding to each target sample according to the second sorting result;
[0164] A second determination subunit is used to determine the classification interval difference of each target sample according to the difference between the two most ranked class probabilities;
[0165] The third determination subunit is used to determine the difficulty coefficient of each target sample based on the classification interval difference.
[0166] In one embodiment, the screening unit 303 includes:
[0167] A second acquisition subunit, used to acquire the category probability of each category corresponding to each target sample according to the first category probability distribution;
[0168] A third sorting subunit is used to sort the category probability of each category corresponding to each target sample in descending order to obtain a third sorting result;
[0169] A fourth determining subunit is used to determine the highest ranking category probability among the category probabilities corresponding to each target sample according to the third sorting result;
[0170] A first calculation subunit, configured to calculate a confusion score of each target sample according to the probability of the top-ranked category;
[0171] The fifth determining subunit is configured to determine a difficulty coefficient of each target sample based on the confusion score.
[0172] In one embodiment, the screening unit 303 includes:
[0173] A second calculation subunit, used to calculate the entropy value of the target sample according to the first category probability distribution;
[0174] The sixth determination subunit is used to determine the difficulty coefficient of each target sample based on the entropy value.
[0175] In one embodiment, the calculation unit 305 includes:
[0176] A relative entropy calculation subunit, used to calculate the relative entropy between the second category probability distribution and the first category probability distribution;
[0177] The difference determination subunit is used to determine the difference between the second category probability distribution and the first category probability distribution according to the relative entropy.
[0178] In one embodiment, the information processing device further includes:
[0179] A model acquisition unit, used to acquire a preset classification model and a trained deep classification model;
[0180] A network depth layer data acquisition unit, used for acquiring a first network depth layer number of the preset classification model and a second network depth layer number of the trained deep classification model when the preset classification model and the trained deep classification model have the same model architecture, wherein the first network depth layer number is less than the second network depth layer number;
[0181] The initialization model parameter determination unit is used to determine the model parameters of the layers before the first network depth layer number of the trained deep classification model as the initialization model parameters of the preset classification model.
[0182] In one embodiment, the acquisition unit 301 includes:
[0183] An initial sample acquisition subunit, used to acquire an initial sample data set, where the initial sample data set includes at least one initial sample;
[0184] The word segmentation subunit is used to perform word segmentation processing on the initial sample to obtain an initial sample word set;
[0185] A feature extraction subunit is used to extract features from the initial sample word set to obtain an initial sample word vector set;
[0186] The data expansion subunit is used to calculate the synonyms of each word in the initial sample word vector, and perform data expansion on the initial sample according to the synonyms to obtain a target sample.
[0187] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0188] From the above, it can be seen that the embodiment of the present application obtains the target sample through the acquisition unit 301; the first classification unit 302 adopts the preset classification model to classify the target sample to obtain the first category probability distribution corresponding to the target sample; the screening unit 303 calculates the difficulty coefficient of the target sample according to the first category probability distribution, and screens the target sample based on the difficulty coefficient to obtain the screened target sample; the second classification unit 304 adopts the trained deep classification model to classify the screened target sample to obtain the second category probability distribution corresponding to the screened target sample; the calculation unit 305 calculates the difference between the second category probability distribution and the first category probability distribution, and converges the preset classification model based on the difference to obtain the trained classification model, which is used to classify the information to be processed. In this way, the difficulty coefficient of each target sample is calculated by using the first category probability distribution corresponding to the target sample obtained by classifying the target sample with the preset classification model, and the target sample is screened according to the difficulty coefficient to obtain the screened target sample, and then the preset classification model is trained based on the screened target sample and the trained classification model. Compared with the existing technology, this method of specifically screening the training samples according to the difficulty coefficient of each target sample reduces the number of training samples, thereby reducing the amount of calculation and shortening the training time, thereby improving the information processing efficiency in the model training process, and thus improving the efficiency of model training.
[0189] The present application also provides a computer device, such as Figure 7 As shown, it shows a schematic diagram of the structure of a computer device involved in an embodiment of the present application, and the computer device may be a server, specifically:
[0190] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 7 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0191] The processor 401 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.
[0192] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and information processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0193] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0194] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0195] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:
[0196] Obtain a target sample; use a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample; calculate the difficulty coefficient of the target sample according to the first category probability distribution, and screen the target sample based on the difficulty coefficient to obtain a screened target sample; use a trained deep classification model to classify the screened target sample to obtain a second category probability distribution corresponding to the screened target sample; calculate the difference between the second category probability distribution and the first category probability distribution, and converge the preset classification model based on the difference to obtain a trained classification model, which is used to classify the information to be processed.
[0197] The specific implementation of each of the above operations can be found in the previous embodiments, which will not be described in detail here. It should be noted that the computer device provided in the embodiment of the present application and the information processing method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above method embodiment, which will not be described in detail here.
[0198] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0199] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the information processing methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0200] Obtain a target sample; use a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample; calculate the difficulty coefficient of the target sample according to the first category probability distribution, and screen the target sample based on the difficulty coefficient to obtain a screened target sample; use a trained deep classification model to classify the screened target sample to obtain a second category probability distribution corresponding to the screened target sample; calculate the difference between the second category probability distribution and the first category probability distribution, and converge the preset classification model based on the difference to obtain a trained classification model, which is used to classify the information to be processed.
[0201] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0202] Since the instructions stored in the computer-readable storage medium can execute the steps in any information processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any information processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0203] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations provided in the above embodiments.
[0204] The above is a detailed introduction to an information processing method, apparatus, computer-readable storage medium and computer device provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An information processing method, characterized in that: include: Get the preset classification model and the trained deep classification model; When the preset classification model and the model architecture of the trained deep classification model are the same, obtaining a first network depth layer number of the preset classification model and a second network depth layer number of the trained deep classification model, wherein the first network depth layer number is less than the second network depth layer number; determining the model parameters of the layers before the first network depth layer number of the trained deep classification model as the initialization model parameters of the preset classification model; Obtain target samples; Using a preset classification model to classify the target sample to obtain a first category probability distribution corresponding to the target sample; Calculating the difficulty coefficient of the target sample according to the first category probability distribution, and screening the target sample based on the difficulty coefficient to obtain a screened target sample, wherein the difficulty coefficient represents the uncertainty degree of each category probability corresponding to the target sample output by the preset classification model; The difficulty coefficient is determined by a classification interval difference, a confusion score or an entropy value; Using a trained deep classification model to classify the screened target samples to obtain a second category probability distribution corresponding to the screened target samples, wherein the network depth of the trained deep classification model is greater than the network depth of the preset classification model; The difference between the second category probability distribution and the first category probability distribution is calculated, and the preset classification model is converged based on the difference to obtain a trained preset classification model, and the trained preset classification model is used to classify the information to be processed.
2. The information processing method according to claim 1, characterized in that: The step of screening the target sample based on the difficulty coefficient to obtain the screened target sample includes: Sorting each target sample in descending order according to the difficulty coefficient of the target sample to obtain a first sorting result; According to the first sorting result, target samples that meet preset conditions are screened to obtain screened target samples.
3. The information processing method according to claim 1, wherein: The calculating the difficulty coefficient of the target sample according to the first category probability distribution includes: Obtaining the category probability of each category corresponding to each target sample according to the first category probability distribution; Sort the category probabilities of each category corresponding to each target sample in descending order to obtain a second sorting result; Determine the top two category probabilities among the category probabilities corresponding to each target sample according to the second sorting result; Determine the classification interval difference of each target sample based on the difference between the two top-ranked class probabilities; Determine a difficulty coefficient for each target sample based on the classification interval difference; The calculation formula of the difficulty coefficient is: Where x is any target sample, is the category that the preset classification model believes the target sample x is most likely to belong to. is the probability of the category to which the target sample X most likely belongs, is the category that the preset classification model believes the target sample x is the second most likely to belong to. is the probability of the second possible category to which the target sample X belongs.
4. The information processing method according to claim 1, wherein: The calculating the difficulty coefficient of the target sample according to the first category probability distribution includes: Obtaining the category probability of each category corresponding to each target sample according to the first category probability distribution; Sort the category probabilities of each category corresponding to each target sample in descending order to obtain a third sorting result; Determine the highest ranking category probability among the category probabilities corresponding to each target sample according to the third sorting result; Calculating a confusion score for each target sample based on the top-ranked class probability; Determining a difficulty coefficient for each target sample based on the confusion score; Among them, the difficulty coefficient u x The calculation formula is in, It means that when P(y|x) reaches its maximum value, the value of y is is the category with the largest probability value in the first category probability distribution of a target sample. It is the probability of the category with the largest probability value in the first category probability distribution of a target sample.
5. The information processing method according to claim 1, wherein: The calculating the difficulty coefficient of the target sample according to the first category probability distribution includes: Calculating the entropy value of the target sample according to the first category probability distribution; Determining a difficulty coefficient for each target sample based on the entropy value; Among them, the difficulty coefficient u x The calculation formula is Among them, P(y|x) represents the probability that the preset target sample x corresponds to the category y of the preset classification model, logP(y|x) represents the result of the logarithmic operation of P(y|x), and ∑ represents the symbol used in mathematics to find the sum of polynomials.
6. The information processing method according to any one of claims 1 to 5, characterized in that: The calculating the difference between the second category probability distribution and the first category probability distribution comprises: Calculating the relative entropy between the second category probability distribution and the first category probability distribution; A difference between the second class probability distribution and the first class probability distribution is determined based on the relative entropy.
7. The information processing method according to claim 1, wherein: The obtaining of the target sample comprises: Acquire an initial sample data set, where the initial sample data set includes at least one initial sample; Performing word segmentation processing on the initial sample to obtain an initial sample word set; Performing feature extraction on the initial sample word set to obtain an initial sample word vector set; The synonyms of each word in the initial sample word vector are calculated, and the initial sample is expanded according to the synonyms to obtain a target sample.
8. An information processing device, characterized in that: include: A model acquisition unit, used to acquire a preset classification model and a trained deep classification model; A network depth layer data acquisition unit, used for acquiring a first network depth layer number of the preset classification model and a second network depth layer number of the trained deep classification model when the preset classification model and the trained deep classification model have the same model architecture, wherein the first network depth layer number is less than the second network depth layer number; An initialization model parameter determination unit, used to determine the model parameters of the layers of the first network depth layer number of the trained deep classification model as the initialization model parameters of the preset classification model; An acquisition unit, used for acquiring a target sample; A first classification unit, configured to classify the target sample using a preset classification model to obtain a first category probability distribution corresponding to the target sample; A screening unit, configured to calculate a difficulty coefficient of the target sample according to the first category probability distribution, and screen the target sample based on the difficulty coefficient to obtain a screened target sample, wherein the difficulty coefficient represents a degree of uncertainty of each category probability corresponding to the target sample output by a preset classification model; The difficulty coefficient is determined by a classification interval difference, a confusion score or an entropy value; A second classification unit is used to classify the screened target samples using a trained deep classification model to obtain a second category probability distribution corresponding to the screened target samples, wherein the network depth of the trained deep classification model is greater than the network depth of the preset classification model; A calculation unit is used to calculate the difference between the second category probability distribution and the first category probability distribution, and converge the preset classification model based on the difference to obtain a trained preset classification model, and the trained preset classification model is used to classify the information to be processed.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the information processing method according to any one of claims 1 to 7.
10. A computer device, characterized in that: The information processing method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the information processing method according to any one of claims 1 to 7 when executing the computer program.
11. A computer program, characterized in that The computer program includes computer instructions, which are stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer device executes the information processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Text classification model training method and device, equipment and readable medium
CN112883193A
Model distillation method and device, equipment and storage medium
CN112949818A
Unification of models having respective target classes with distillation
US20210034985A1
Model generation based on model compression
WO2021002968A1
