A method and system for improving model training effect

Abstract

The embodiment of the specification discloses a method and system for improving model training effect, comprising: acquiring a plurality of training samples, the training samples comprising training sample data and sample labels; inputting the training samples into a student model and at least one teacher model to obtain a first processing result and at least one second processing result respectively; the teacher model is from a teacher model set; dynamically determining the importance of each of the at least one teacher model to the training sample, and calculating the at least one second processing result based on the importance to determine a fusion processing result; determining a first loss function of the student model based on the first processing result, the sample label and the fusion processing result; updating the parameters of the student model based on the first loss function, and adding the student model with the updated parameters to the teacher model set as a new teacher model; iteratively completing the training of the student model for multiple times; and using the student model after the training to process speech, image or text data.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method and system for improving model training performance. Background Technology

[0002] With the development of machine learning technology, it has been widely applied in various scenarios, such as facial recognition, natural language processing, data mining, and intelligent customer service dialogue. Machine learning technology refers to the technique of learning the inherent patterns and characteristics of sample data through machine learning models, enabling trained models to have analytical and learning capabilities and to recognize text, images, and speech. As machine learning technology becomes more widely used, users are placing increasingly higher demands on the capabilities of machine learning models.

[0003] Therefore, it is necessary to provide a method that can improve the training effect of the model. Summary of the Invention

[0004] One aspect of this specification provides a method for improving model training performance. The method includes: acquiring multiple training samples, wherein the training samples include training sample data and sample labels; inputting the training samples into a student model to obtain a first processing result, and inputting the training samples into at least one teacher model to obtain at least one second processing result; the teacher models are from a set of teacher models; dynamically determining the importance of each of the at least one teacher model to the training samples, and calculating the at least one second processing result based on the importance to determine a fusion processing result; determining a first loss function of the student model based on the first processing result, the sample labels, and the fusion processing result; updating the parameters of the student model based on the first loss function, and adding the parameter-updated student model to the set of teacher models as a new teacher model; iterating multiple times to complete the training of the student model; the trained student model is used to process speech, image, or text data.

[0005] One aspect of this specification provides a system for improving model training performance. The system includes: an acquisition module for acquiring multiple training samples, wherein the training samples include training sample data and sample labels; a processing module for inputting the training samples into a student model to obtain a first processing result, and inputting the training samples into at least one teacher model to obtain at least one second processing result; the teacher models are from a set of teacher models; a determination module for dynamically determining the importance of each of the at least one teacher model to the training samples, and calculating the at least one second processing result based on the importance to determine a fusion processing result; determining a first loss function of the student model based on the first processing result, the sample labels, and the fusion processing result; a parameter update module for updating the parameters of the student model based on the first loss function, and adding the parameter-updated student model to the set of teacher models as a new teacher model; and an iteration module for iterating multiple times to complete the training of the student model; the trained student model is used to process speech, image, or text data.

[0006] One aspect of this specification provides an apparatus for improving model training performance, comprising at least one storage medium and at least one processor, wherein the at least one storage medium is used to store computer instructions; and the at least one processor is used to execute the computer instructions to implement the method described above. Attached Figure Description

[0007] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0008] Figure 1 These are schematic diagrams illustrating exemplary application scenarios of systems for improving model training performance, as shown in some embodiments of this specification.

[0009] Figure 2 This is an exemplary flowchart illustrating a method for improving model training performance according to some embodiments of this specification;

[0010] Figure 3 This is an exemplary flowchart illustrating the determination of the importance of the teacher model to the training samples, according to some embodiments of this specification;

[0011] Figure 4 This is an exemplary schematic diagram illustrating the determination of a first loss function for a student model according to some embodiments of this specification. Detailed Implementation

[0012] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0013] It should be understood that the terms "system," "device," "unit," and / or "module" as used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0014] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0015] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0016] Figure 1 These are schematic diagrams illustrating exemplary application scenarios of systems for improving model training performance, based on some embodiments of this specification.

[0017] The system 100 for improving model training performance can make the model training process smoother by implementing the methods and / or processes disclosed in this specification, thereby solving the problem of training oscillations and enabling the trained model to achieve better results.

[0018] In some embodiments, the system 100 can train any model. For example, it can be used to train an intent recognition model in the field of intelligent customer service, which can obtain the user's intent based on the user's input question, i.e., what aspect of content the user wants to inquire about, and further, the intelligent customer service robot provides answers to the user based on the recognized intent. As another example, it can be used to train a face recognition model in the field of image recognition, which can recognize the user's face to verify the user's identity through the face image. Yet another example, it can be used to train a speech recognition model in the field of speech processing, which can recognize the user's speech, obtain the user's speech-text, and analyze the speech-text.

[0019] like Figure 1 As shown, the system 100 for improving model training performance may include a processing device 110, a network 120, a user terminal 130, and a storage device 140.

[0020] In some embodiments, the processing device 110 can access information and / or data stored in the database 140 and / or the user terminal 130 via the network 120. For example, the processing device can retrieve training samples or teacher models from the storage device 140 via the network 120. As another example, the processing device 110 can retrieve user input data from the user terminal via the network 120; for instance, in the field of intelligent customer service, this might involve retrieving user-inputted questions.

[0021] In some embodiments, the processing device 110 can be used to process information and / or data related to data processing. In some embodiments, the processing device 110 can be used to process information and / or data associated with model training to perform one or more functions disclosed in this specification. For example, the processing device 110 can input training samples into a student model to obtain a first processing result, and input training samples into at least one teacher model to obtain at least one second processing result. As another example, the processing device 110 can determine a fusion processing result based on at least one second processing result. As another example, the processing device 110 can determine a first loss function for the student model based on the first processing result, sample labels, and the fusion processing result. As another example, the processing device 110 can update the parameters of the student model based on the first loss function. In some embodiments, the processing device 110 can be used to process information and / or data associated with model execution to perform one or more functions disclosed in this specification. For example, the processing device 110 can input data obtained from the user terminal 130 into a trained student model to obtain an output result; for example, in the field of intelligent customer service, the output is the user's intent.

[0022] In some embodiments, the processing device 110 may include one or more processing engines (e.g., a single-core processing engine or a multi-core processor). By way of example only, the processing device 110 may include one or more combinations of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a special-purpose instruction set processor (ASIP), a graphics processing unit (GPU), a microprocessor, etc.

[0023] In some embodiments, network 120 may facilitate the exchange of data and / or information, which may include data input by user terminal 130, training samples stored in database 140, parameters of student models trained by processing device 110, teacher models, etc. In some embodiments, one or more components in system 100 (e.g., processing device 110, user terminal 130, database 140) may send data and / or information to other components in system 100 via network 120.

[0024] In some embodiments, network 120 can be any type of wired or wireless network. For example, network 120 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), and any combination thereof. In some embodiments, network 120 may include one or more network access points. For example, network 120 may include wired or wireless network access points, such as base stations and / or Internet exchange points 120-1, 120-2, ..., through which one or more components of the access point system 100 can be connected to network 120 to exchange data and / or information.

[0025] In some embodiments, the user terminal 130 may be a device with data acquisition, storage, and / or transmission functions. The computing device may include one or any combination of a mobile phone 130-1, a tablet computer 130-2, a laptop computer 130-3, a desktop computer 130-4, etc. In some embodiments, the user terminal 130 may send the acquired data to one or more devices in the system 100. For example, the user terminal 130 may transmit the acquired data to a processing device 110 or a storage device 140. As an example only, in the field of intelligent customer service, the data acquired by the user terminal 130 may be a question raised by the user.

[0026] Storage device 140 can store data and / or instructions. In some embodiments, storage device 140 can store data acquired from user terminal 130. In some embodiments, storage device 140 can store the model trained in each training cycle. In some embodiments, storage device 140 can store training data. In some embodiments, storage device 140 can be connected to network 120 to enable communication with one or more components in system 100 (e.g., processing device 110, user terminal 130, etc.).

[0027] In some embodiments, storage device 140 may be part of processing device 110. In some embodiments, storage device 120 may include mass storage, removable storage, volatile read-write storage, read-only storage (ROM), etc., or any combination thereof. In some embodiments, storage device 120 may be implemented on a cloud platform. By way of example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-tiered cloud, etc., or any combination thereof.

[0028] In some embodiments, the system 100 may include an acquisition module, a processing module, a determination module, a parameter update module, and an iteration module.

[0029] The acquisition module can be used to acquire multiple training samples, wherein the training samples include training sample data and sample labels.

[0030] The processing module can be used to input the training samples into a student model to obtain a first processing result, and to input the training samples into at least one teacher model to obtain at least one second processing result; the teacher model comes from a set of teacher models.

[0031] The determining module can be used to determine a fusion processing result based on the at least one second processing result; and to determine a first loss function of the student model based on the first processing result, the sample labels, and the fusion processing result. In some embodiments, the determining module can be used to determine the weight of each of the at least one second processing result based on the training samples, and to perform a weighted operation on the at least one second processing result to obtain the fusion processing result. In some embodiments, the determining module can be used to determine the weight of each of the at least one second processing result based on the similarity between the first processing result and each of the at least one second processing result. In some embodiments, the determining module can be used to determine the weight of each of the at least one second processing result using a weight model.

[0032] In some embodiments, the determining module is further configured to determine the weight of each of the at least one second processing result, and perform a weighted operation on the at least one second processing result to obtain the fusion processing result.

[0033] In some embodiments, the determining module is further configured to determine the weight of each of the at least one second processing result based on the similarity between the first processing result and each of the at least one second processing result.

[0034] In some embodiments, the determining module is further configured to determine the weight of each of the at least one second processing result using a weight model. In some embodiments, the weight model is a neural network model, and the student model and the weight model are jointly trained through an end-to-end learning approach.

[0035] In some embodiments, the determining module is further configured to: generate a second loss function based on the difference between the first processing result and the sample label; generate a third loss function based on the difference between the first processing result and the fusion processing result; and perform a weighted summation of the second loss function and the third loss function as the first loss function of the student model.

[0036] The parameter update module can be used to update the parameters of the student model based on the first loss function, and add the updated student model to the teacher model set as a new teacher model.

[0037] The iteration module can be used to iterate multiple times to complete the training of the student model; after training, the student model is used to process speech, image, or text data.

[0038] For a detailed description of each module of the system for improving model training performance, please refer to the flowchart section of this manual, for example... Figures 2 to 4 The relevant explanations will not be repeated here.

[0039] It should be understood that Figure 1The systems and modules shown can be implemented in various ways. For example, in some embodiments, the systems and modules can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in processor control code, for example, on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The systems and modules of this specification can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., but also by software, for example, executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0040] It should be noted that the above description of the system 100 and its modules for improving model training performance is for convenience only and should not be construed as limiting this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. For example, Figure 1 The acquisition module, processing module, determination module, parameter update module, and iteration module disclosed herein can be different modules within a single system, or a single module can implement the functions of two of the aforementioned modules. For example, in the system 100 for improving model training performance, various modules can share a single storage module, or each module can have its own independent storage module. Such variations are all within the scope of protection of this specification.

[0041] Figure 2 This is an exemplary flowchart illustrating a method for improving model training performance according to some embodiments of this specification. The method may include:

[0042] Step 202: Obtain multiple training samples, wherein the training samples include training sample data and sample labels. Specifically, step 202 can be performed by the acquisition module.

[0043] Training samples refer to the data used for model training. In some embodiments, training samples include training sample data and sample labels.

[0044] In some embodiments, training sample data can be various types of data such as images, text, or speech. In some embodiments, the corresponding training sample data can be selected according to different application scenarios of the model. For example, if the model is applied to image recognition, the training sample data is image data.

[0045] Sample labels are used to represent certain real-world information about the training sample data. In some embodiments, sample labels are related to the function of the model. For example, if the model is trained to classify input data, then the sample labels are the categories of the training sample data, such as images being cats, dogs, trees, or the sky.

[0046] In some implementations, the form of the sample labels can match the output values ​​of the model. For example, if the output value is a probability distribution, then the label annotation can also be a probability distribution. If the model is used to determine the category of an image, including cats, dogs, and birds, the output is a probability distribution of the form [X, Y, Z], where X represents the probability that the image is a cat, Y represents the probability that the image is a dog, and Z represents the probability that the image is a bird. When the training sample data is an image of a cat, the label of that image is [1, 0, 0]; when the training sample data is an image of a dog, the label of that image is [0, 1, 0].

[0047] In some embodiments, the sample labels of the training sample data can be manually labeled offline or labeled online through automatic identification, statistics, or other methods.

[0048] In some embodiments, the acquisition module can obtain the training sample by directly reading it from the storage device 140, or it can obtain multiple training samples by calling relevant interfaces or other means.

[0049] Step 204 involves inputting the training samples into a student model to obtain a first processing result, and inputting the training samples into at least one teacher model to obtain at least one second processing result; the teacher models are from a set of teacher models. Specifically, step 204 can be executed by a processing module.

[0050] The student model can refer to any model that needs to be trained. In some embodiments, the student model can be a machine learning model, such as a neural network model, a deep learning model, etc.

[0051] A teacher model set can be a collection of at least one teacher model. A teacher model can refer to a model used to guide the training of a student model. It can be understood that a student model is the student model in a teacher-student framework, and a teacher model is the teacher model in a teacher-student framework.

[0052] In some embodiments, the teacher model and the student model are the same type of model, differing only in their parameters. In some embodiments, the teacher model can be a model obtained from different historical training periods prior to the current training. For example, if the student model is the model obtained in the third training period, and the current training is in the fourth training period, then the teacher model could be the model obtained in the first, second, and third training periods. It can be understood that if the student model is a model that has not undergone any training periods, the teacher model set is empty; if the student model is the model obtained after the first training period, the teacher model set contains one teacher model, which is the model obtained in the first training period; if the student model is the model obtained after the second training period, the teacher model set contains two teacher models, which are the models obtained in the first and second training periods respectively; and so on.

[0053] The first processing result can be the prediction result of the student model on the training data of the training samples. In some embodiments, the output form of the first processing result can be a probability distribution represented by a vector. For example, continuing with the previous example, the first processing result can be [0.3, 0.4, 0.3], where the values ​​in the vector correspond to the probabilities that the image belongs to a cat, a dog, and a bird, respectively.

[0054] The second processing result can be the prediction result of the teacher model on the training data of the training samples. In some embodiments, the output form of the second processing result is the same as that of the first processing result, that is, the first processing result is a probability distribution represented by a vector, and the second processing result can also be a probability distribution represented by a vector.

[0055] In some embodiments, training samples can be input into all teacher models in the teacher model set to obtain the corresponding second processing result; alternatively, training samples can be input into a subset of teacher models in the teacher model set to obtain the corresponding second processing result.

[0056] Step 206: Based on the at least one second processing result, determine the fusion processing result. Specifically, step 206 can be executed by the determining module.

[0057] In some embodiments, the fusion processing result is associated with the second processing result, and the fusion processing result can be obtained by performing calculations on the second processing result. The calculation methods include, but are not limited to, weighted calculation, average calculation, variance calculation, etc.

[0058] In some embodiments, the importance of each of the at least one teacher model to the training samples can be determined, and at least one second processing result can be calculated based on the importance to determine the fusion processing result.

[0059] In some embodiments, the importance of the teacher model to the training samples can be determined dynamically. For example, the importance of the teacher model that outputs the second processing result to the training samples can be determined based on the similarity between the first processing result and the second processing result. The higher the similarity, the greater the importance; the lower the similarity, the smaller the importance.

[0060] In some embodiments, similarity can be determined by calculating the distance between the first processing result and the second processing result. This distance may include, but is not limited to, cosine distance, Euclidean distance, etc. In some embodiments, a neural network model can be used to calculate the similarity between the first processing result and the second processing result, and to determine the importance of the second processing result. For a more detailed description of determining the importance of the second processing result, please refer to [link to relevant documentation]. Figure 3 The details and related descriptions will not be repeated here.

[0061] As mentioned above, the importance of each teacher model to the training samples is determined by the processing results of each teacher model with that training sample. For the same training sample, different teacher models will produce different processing results, thus affecting its importance; similarly, for different training samples, the same teacher model will produce different processing results, further affecting its importance. This can be understood as a dynamically determined process, related to both the teacher model and the training data.

[0062] In some embodiments, the importance of the teacher model to the training samples can also be determined statically, and this specification does not impose any limitations. For example, the importance to the training samples is positively correlated with the training period of the teacher model. For instance, if teacher model A is the model obtained in the first training period, teacher model B is the model obtained in the second training period, and the third model C is the model obtained in the third training period, then the importance of teacher model A < the importance of teacher model B < the importance of teacher model C.

[0063] Furthermore, in some embodiments, the weights of the second processing results output by the teacher model can be determined based on the importance of the teacher model to the training samples, and the fusion processing result is determined based on each second processing result and its weight. The teacher model with greater importance to the training samples receives a greater weight for its output second processing result. For example, the importance of the teacher model to the training samples can be directly used as the weight of the second processing result output by the teacher model. Alternatively, the importance can be transformed, for example, by normalization or multiplication by the same coefficient, and the transformed result can be used as the weight.

[0064] In some embodiments, at least one second processing result can be weighted according to the weights corresponding to different second processing results to obtain a fusion processing result. In some embodiments, the weighting operation can be a weighted summation. For example, assuming there are three second processing results with corresponding weights of 0.2, 0.4, and 0.4, the second processing results are multiplied by their corresponding weights and then summed. The summation process is 0.2*[0.3, 0.4, 0.3]+0.4*[0.3, 0.3, 0.4]+0.4*[0.3, 0.3, 0.4]. After calculation, the fusion processing result is [0.3, 0.32, 0.38].

[0065] The importance (or weight) of the teacher model to the training samples reflects the degree of influence of the teacher model in guiding the training of the student model when using the training samples. By weighting at least one second processing result according to different weights, the influence of multiple teacher models trained at different times on the training samples can be considered simultaneously, thereby better utilizing the teacher model to guide the student model's training and improving the student model's training effect. Step 208: Determine the first loss function of the student model based on the first processing result, the sample labels, and the fusion processing result. Specifically, step 208 can be executed by the determination module.

[0066] In some embodiments, the first loss function of the student model can be determined based on the difference between the first processing result and the fusion processing result, and the difference between the first processing result and the sample labels. For example, if the outputs of the student model and the teacher model are in the form of probability distributions, the difference between the first processing result and the fusion processing result can refer to the difference between the probability distribution of the first processing result and the probability distribution of the fusion processing result, and the difference between the first processing result and the sample labels can refer to the difference between the probability distribution of the first processing result and the sample labels.

[0067] In some embodiments, a second loss function can be generated based on the difference between the first processing result and the sample labels, and a third loss function can be generated based on the difference between the first processing result and the fusion processing result. Further, a first loss function for the student model is determined based on the second and third loss functions. For more details on determining the first loss function for the student model, please refer to [link to relevant documentation]. Figure 4 The details and related descriptions will not be repeated here.

[0068] Step 210: Update the parameters of the student model based on the first loss function. Specifically, step 210 can be performed by the parameter update module.

[0069] In some embodiments, the student model can be trained by minimizing a first loss function. This involves updating the parameters of the student model during the minimization of the first loss function. These parameters can be the connection weights w between layers and the bias b of the output layer. In some embodiments, the model can be trained by minimizing the first loss function using common methods such as gradient descent.

[0070] Step 212 involves adding the updated student model to the teacher model set as a new teacher model. Specifically, step 212 can be executed by the parameter update module.

[0071] In some embodiments, the updated student model is added to the teacher model set as a new teacher model.

[0072] In some embodiments, the student model can be trained by iterating through steps 202-212 multiple times, where each execution of steps 202-212 can be considered a training cycle. Specifically, the student model is trained by iterating through multiple training cycles until the trained student model meets a preset condition, at which point the iteration ends (i.e., training ends), and the trained student model is obtained. The preset condition can be that the first loss function converges or is less than a preset threshold, or that the number of iterations reaches a preset number, such as 100, 1000, or 10000.

[0073] In some embodiments, the training samples used in different iteration processes, i.e. different training cycles, may be the same or different.

[0074] In some embodiments, the trained student model can be used to process speech, image, or text data. For example, the trained student model can be used for speech recognition, image recognition, text semantic analysis, etc.

[0075] As mentioned earlier, the teacher model is the model obtained from previous training cycles, specifically the model from the first N iterations or training cycles. Incorporating the second processing result of the teacher model into the determination of the first loss function of the student model allows the teacher model to influence or guide the training of the student model. Training the student model in this way ensures that its output closely approximates the teacher model's output, preventing significant oscillations during training, resulting in a more stable training process and better performance.

[0076] Figure 3 This is an exemplary flowchart illustrating the determination of the importance of the teacher model to the training samples, according to some embodiments of this specification. (Combined with...) Figure 3 As shown, determining the importance of the teacher model to the training samples can include:

[0077] Step 302 involves concatenating the first processing result with each of the at least one second processing result to obtain a concatenated result. Specifically, step 302 can be performed by the determining module.

[0078] The splicing result can be the splicing result of the vector representation of the first processing result and the vector representation of the second processing result. For example, the vector representation of the first processing result is [1, 0, 0], and the vector representation of the second processing result is [0, 0, 1]. The splicing result [1, 0, 0, 0, 0, 1] is obtained by concatenating the first and second processing results end to end.

[0079] In some embodiments, when there are multiple second processing results, there are also multiple splicing processing results. For example... Figure 3 As shown, multiple splicing results can be obtained by splicing the first processing result with the second processing result 1, the second processing result 2, ..., the second processing result n, respectively.

[0080] In some embodiments, the weight model is a neural network model, which can be obtained by jointly training the student model and the weight model through an end-to-end learning approach. Using an end-to-end learning approach, the weight model can be trained together with the student model, eliminating the need for separate training of the weight model, thereby simplifying the model training process and improving training efficiency. The type of neural network model may include DNN, CNN, RNN, etc.

[0081] Step 304: Input the splicing result into the weight model to obtain the importance of the teacher model outputting the second processing result to the training samples. Specifically, step 304 can be executed by the determination module.

[0082] In some embodiments, after the concatenation result is input into the weight model, the output of the weight model represents the importance of the corresponding teacher model to the training samples. Here, the corresponding teacher model is the model that outputs the second processing result used for concatenation. For example... Figure 3 The concatenated processing result obtained by concatenating the first processing result and the second processing result 1 is input into the weight model, and the importance 1 of teacher model 1 to training sample 1 is output. Here, the first processing result is the result obtained by inputting training sample 1 into student model, and the second processing result 1 is the result obtained by inputting training sample 1 into teacher model 1. The concatenated processing result obtained by concatenating the first processing result and the second processing result 2 is input into the weight model, and the importance 2 of teacher model 2 to training sample 1 is output. Here, the second processing result 2 is the result obtained by inputting training sample 1 into teacher model 2.

[0083] In some embodiments, the importance of the teacher model to the training samples can be determined based on the similarity between the first processing result and the second processing result. In some embodiments, the greater the similarity, the greater the importance.

[0084] In some embodiments, the weighted model can measure the similarity between the first processing result and the second processing result by calculating the distance between the first processing result and the second processing result. The distance may include, but is not limited to, cosine distance, Euclidean distance, etc.

[0085] Figure 4 This is an exemplary schematic diagram illustrating the determination of the first loss function for a student model according to some embodiments of this specification. Combined with... Figure 4 As shown, determining the first loss function of the student model may include:

[0086] Step 402: Based on the difference between the first processing result and the sample label, a second loss function is generated. Specifically, step 402 can be executed by the determination module.

[0087] The loss function can be used to evaluate the difference between the predicted value and the target value obtained by the neural network being trained, where the predicted value refers to the output value obtained by inputting training sample data into the neural network being trained. In some embodiments, the difference can be measured by a distance metric, such as cross-entropy, KL distance, JS distance, etc.

[0088] The second loss function is the loss function determined based on the difference between the first processing result and the sample labels. In other words, the predicted value in the second loss function can be the first processing result, and the target value can be the sample labels of the training sample data.

[0089] Taking the second loss function, determined by the distance and cross-entropy, as an example, the second loss function can be expressed as formula (1):

[0090]

[0091] Where L represents the second loss function; L i The second loss function represents a single training sample; i represents the i-th training sample; N represents the number of training samples; c represents the c-th class; M represents the number of classes; y ic This indicates an indicator variable (0 or 1), which is 1 if the label of the i-th training sample is the same as that of the c-th class, and 0 otherwise; p ic Let be the probability value of the c-th category in the first processing result of the i-th training sample.

[0092] For example, taking image three-class classification as an example, the sample label of the i-th training sample is [1, 0, 0], representing that the true category of the image is cat. The first processing result of this training sample is [0.4, 0.3, 0.3], where this probability distribution indicates that the probability of the training sample belonging to cat is 0.4, the probability of belonging to dog is 0.3, and the probability of belonging to bird is 0.3. Then the second loss function L of this training sample is... i = -(1×log0.4+0×log0.3+0×log0.3).

[0093] Step 404: Based on the difference between the first processing result and the fusion processing result, a third loss function is generated. Specifically, step 404 can be executed by the determination module.

[0094] The third loss function is determined based on the difference between the first processing result and the fused processing result. In other words, the predicted value in the third loss function can be the first processing result, and the target value can be the fused processing result.

[0095] Continuing with the example of establishing a third loss function based on distance as cross-entropy, the third loss function can be expressed as formula (2):

[0096]

[0097] Where L′ represents the third loss function; L i ' represents the third loss function for a single training sample; i represents the i-th training sample; N represents the number of training samples; c represents the c-th class; M represents the number of classes; y ic p represents the probability of the c-th class in the fusion processing result of the i-th training sample; ic Let be the probability value of the c-th category in the first processing result of the i-th training sample.

[0098] For example, taking image three-class classification as an example, the fusion processing result of the i-th training sample is [0.6, 0.3, 0.1]. If the first processing result of this training sample is [0.4, 0.3, 0.3], then the third loss function L of this training sample... i = -(0.6×log0.4+0.3×log0.3+0.1×log0.3).

[0099] Step 406 involves a weighted sum of the second and third loss functions to obtain the first loss function for the student model. Specifically, step 406 can be executed by the determination module.

[0100] In some embodiments, the second loss function and the third loss function can be weighted and summed according to different weight parameters. The weight parameter of the second loss function can be λ, and the weight parameter of the third loss function can be 1-λ. λ can be 0.5, 0.6, etc.

[0101] In some embodiments, the weights of the second and third loss functions can be determined empirically. For example, the weight of the second loss function can be greater than the weight of the third loss function.

[0102] It is understandable that during the training of the student model, minimizing the first loss function of the student model constructed in this embodiment can simultaneously minimize the second and third loss functions.

[0103] As mentioned earlier, the second loss function represents the difference between the true labels of the training samples and the prediction results of the student model. It can be understood that minimizing the second loss function can bring the prediction results of the trained student model as close as possible to the true labels. The third loss function represents the difference between the prediction results of the student model and the fusion processing result, where the fusion processing result is obtained from the prediction results of the teacher model. It can be understood that minimizing the third loss function can bring the prediction results of the trained student model as close as possible to the prediction results of the teacher model, thereby enabling the teacher model to guide the training of the student model, preventing large fluctuations in the training of the student model, making the training process more stable, and achieving better results.

[0104] This specification also provides an apparatus, which includes at least a processor and a memory. The memory is used to store instructions. When the instructions are executed by the processor, the apparatus causes the aforementioned method for improving model training performance. The method may include: acquiring multiple training samples, wherein the training samples include training sample data and sample labels; inputting the training samples into a student model to obtain a first processing result, and inputting the training samples into at least one teacher model to obtain at least one second processing result; the teacher models are from a set of teacher models; dynamically determining the importance of each of the at least one teacher model to the training samples, and calculating the at least one second processing result based on the importance to determine a fusion processing result; determining a first loss function for the student model based on the first processing result, the sample labels, and the fusion processing result; updating the parameters of the student model based on the first loss function, and adding the parameter-updated student model to the set of teacher models as a new teacher model; training the student model iteratively multiple times; the trained student model is used to process speech, image, or text data.

[0105] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0106] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0107] Furthermore, those skilled in the art will understand that various aspects of this specification can be described and illustrated in several patentable ways or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Accordingly, various aspects of this specification can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. All of the above hardware or software may be referred to as a “data block,” “module,” “engine,” “unit,” “component,” or “system.” Furthermore, various aspects of this specification may be represented as a computer product located on one or more computer-readable media, including computer-readable program code.

[0108] Computer storage media may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and suitable combinations thereof. Computer storage media can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer storage medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0109] The computer program code required for the operation of each part of this manual can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.; conventional procedural programming languages ​​such as C, Visual Basic, Fortran2003, Perl, COBOL2002, PHP, ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or processing device. In the latter case, the remote computer can be connected to the user's computer through any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0110] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented by hardware devices, they can also be implemented solely by software solutions, such as installing the described system on existing processing devices or mobile devices.

[0111] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0112] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0113] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0114] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for improving model training performance, comprising: Multiple training samples are acquired, wherein the training samples include training sample data and sample labels; the training sample data includes image, text, or voice data; and the sample labels are used to characterize the true information of the training sample data. The training samples are input into a student model to obtain a first processing result, and the training samples are input into at least one teacher model to obtain at least one second processing result; the teacher models come from a set of teacher models; the first processing result is the prediction result of the student model on the training sample data, and the second processing result is the prediction result of the teacher model on the training sample data; the prediction result includes face recognition result, image data classification result, predicted speech text, or predicted user intent; The importance of each of the at least one teacher model to the training samples is dynamically determined, and the at least one second processing result is calculated based on the importance to determine the fusion processing result; The first loss function of the student model is determined based on the first processing result, the sample labels, and the fusion processing result; The parameters of the student model are updated based on the first loss function, and the updated student model is added to the teacher model set as a new teacher model. The student model is trained through multiple iterations; after training, the student model is used to process speech, image, or text data. The step of dynamically determining the importance of each of the at least one teacher model to the training samples, and calculating the at least one second processing result based on the importance to determine the fusion processing result, includes: Based on the similarity between the first processing result and each of the at least one second processing result, the importance of each of the at least one teacher model to the training sample is determined; The weight of the second processing result is determined based on the aforementioned importance; The at least one second processing result is weighted and summed based on the weights to obtain the fusion processing result.

2. The method according to claim 1, wherein determining the importance of each of the at least one teacher model to the training sample based on the similarity between the first processing result and each of the at least one second processing result comprises: The importance of each of the at least one teacher model to the training samples is determined by a weighted model.

3. The method according to claim 2, wherein the weight model is a neural network model, and the student model and the weight model are jointly trained through an end-to-end learning method.

4. The method according to claim 1, wherein determining the first loss function of the student model based on the first processing result, the sample labels, and the fusion processing result comprises: Based on the difference between the first processing result and the sample label, a second loss function is generated; Based on the difference between the first processing result and the fusion processing result, a third loss function is generated; The second loss function and the third loss function are weighted and summed to form the first loss function of the student model.

5. A system for improving model training performance, comprising: An acquisition module is used to acquire multiple training samples, wherein the training samples include training sample data and sample labels; the training sample data includes image, text, or voice data; and the sample labels are used to characterize the true information of the training sample data. The processing module is used to input the training samples into a student model to obtain a first processing result, and to input the training samples into at least one teacher model to obtain at least one second processing result; the teacher model comes from a set of teacher models. A determination module is configured to dynamically determine the importance of each of the at least one teacher model to the training samples, and calculate the at least one second processing result based on the importance to determine a fusion processing result; and determine a first loss function for the student model based on the first processing result, the sample labels, and the fusion processing result; the first processing result is the prediction result of the student model on the training sample data, and the second processing result is the prediction result of the teacher model on the training sample data; the prediction result includes face recognition results, image data classification results, predicted speech text, or predicted user intent; and, based on the similarity between the first processing result and each of the at least one second processing result, determine the importance of each of the at least one teacher model to the training samples; determine the weight of the second processing result based on the importance; and perform a weighted summation of the at least one second processing result based on the weight to obtain the fusion processing result; The parameter update module is used to update the parameters of the student model based on the first loss function, and add the updated student model to the teacher model set as a new teacher model. The iteration module is used to iterate multiple times to complete the training of the student model; after training, the student model is used to process speech, image, or text data.

6. The system according to claim 5, wherein the determining module is further configured to: The importance of each of the at least one teacher model to the training samples is determined by a weighted model.

7. The system according to claim 6, wherein the weight model is a neural network model, and the student model and the weight model are jointly trained through an end-to-end learning method.

8. The system according to claim 5, wherein the determining module is further configured to: Based on the difference between the first processing result and the sample label, a second loss function is generated; Based on the difference between the first processing result and the fusion processing result, a third loss function is generated; The second loss function and the third loss function are weighted and summed to form the first loss function of the student model.

9. An apparatus for improving model training performance, comprising at least one storage medium and at least one processor, wherein the at least one storage medium is used to store computer instructions; and the at least one processor is used to execute the computer instructions to implement the method as described in any one of claims 1 to 4.

Images