Language model training method and device, equipment, storage medium and product

Abstract

The invention discloses a language model training method and device, equipment, a storage medium and a product, and relates to the technical field of language model training, and the language model training method comprises the steps: obtaining training data; a deep convolutional neural network model is trained according to the training data, a trained deep convolutional neural network model is obtained, and the deep convolutional neural network model is established according to a preset statistical pooling layer; based on a knowledge distillation technology, training a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model; and performing quantization processing on the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model. According to the method, the technical problem that the language model performance and the computing resource demand are difficult to balance when the language model is processed in the field of training data scarcity can be solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a language model training method, apparatus, device, storage medium and product. Background Art

[0002] In the prior art, since training data in some fields is relatively rare, such as audio data containing information about babies crying or the calls of rare animals, audio data with corresponding meaning annotations is even more difficult to obtain. Therefore, when training language models, there is a problem of data scarcity, which limits the training effect of the model, resulting in insufficient performance of the training model and the inability to accurately identify data in the corresponding field.

[0003] In order to improve the accuracy of language model training in areas where training data is scarce, existing methods include methods that use large neural network models to identify data in areas where training data is scarce. However, large neural network models have a large number of parameters and high requirements for computing resources, which makes them inconvenient to deploy on mobile devices. In addition, there are methods that reduce the number of model parameters and the number of layers to make them suitable for mobile applications. However, the performance of this lightweight model is lower than that of large neural network models. Therefore, in mobile deployment applications, when identifying data in areas where training data is scarce, it is difficult to balance the recognition performance and computing resource requirements.

[0004] Therefore, how to balance the performance of language models and computing resource requirements in areas where training data is scarce is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a language model training method, apparatus, device, storage medium and product, aiming to solve the technical problem of the difficulty in balancing language model performance and computing resource requirements when the language model is processed in a field where training data is scarce.

[0006] To achieve the above objectives, the present application provides a language model training method, the language model training method comprising:

[0007] Get training data;

[0008] Training a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer;

[0009] Performing knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model;

[0010] The trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model.

[0011] In one embodiment, the step of training the deep convolutional neural network model according to the training data to obtain the trained deep convolutional neural network model includes:

[0012] Performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data;

[0013] Get domain data;

[0014] The deep convolutional neural network model is fine-tuned according to the fused feature data and the domain data to obtain a trained deep convolutional neural network model.

[0015] In one embodiment, the step of performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data includes:

[0016] Performing sliding window processing on the training data through the statistical pooling layer to obtain the average value of the training data and the variance of the training data;

[0017] The mean value of the training data and the variance of the training data are subjected to feature fusion through the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data.

[0018] In one embodiment, the step of performing knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model includes:

[0019] Get domain data;

[0020] Predicting the domain data using the trained deep convolutional neural network model to obtain a target label and a probability distribution of the target label;

[0021] Training a shallow convolutional neural network model using the trained deep convolutional neural network model and the domain data to obtain a target label loss and a probability distribution loss of the target label;

[0022] A loss function is calculated according to the target label loss and the probability distribution loss of the target label, and the shallow convolutional neural network model is optimized according to the loss function to obtain a trained shallow convolutional neural network model.

[0023] In one embodiment, the step of calculating the loss function according to the target label loss and the probability distribution loss of the target label, and optimizing the shallow convolutional neural network model according to the loss function to obtain the trained shallow convolutional neural network model includes:

[0024] Performing a weighted operation according to the target label loss and the probability distribution loss of the target label to obtain the loss function;

[0025] According to the loss function, back propagation is used to optimize the shallow convolutional neural network model to obtain a trained shallow convolutional neural network model.

[0026] In one embodiment, the step of quantizing the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model includes:

[0027] Performing forward propagation processing on the trained shallow convolutional neural network model to obtain output statistical information and weight statistical information;

[0028] Calculate a quantization parameter according to the output statistical information and the weight statistical information to obtain a quantization parameter;

[0029] According to the quantization parameters and the preset quantization strategy, the trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a language model training device, the language model training device comprising:

[0031] An acquisition module is used to acquire training data;

[0032] A first model training module is used to train a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer;

[0033] A second model training module is used to perform knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model;

[0034] The quantization module is used to perform parameter quantization processing on the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a language model training device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the language model training method described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing the language model training method is stored on the computer-readable storage medium. The program for implementing the language model training method is executed by a processor to implement the steps of the language model training method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a program for implementing the language model training method, and when the computer program is executed by a processor, it implements the steps of the language model training method described above.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] By acquiring training data; training a deep convolutional neural network model according to the training data, obtaining a trained deep convolutional neural network model, the deep convolutional neural network model is established according to a preset statistical pooling layer; the deep convolutional neural network model usually has higher performance and can capture complex features in the training data. By training the deep convolutional neural network model, it can be ensured that as much information as possible is learned from the training data. The preset statistical pooling layer helps the deep convolutional neural network model to more effectively learn the common features and difference information data in the training data, thereby improving the training effect of the deep convolutional neural network model. The trained deep convolutional neural network model is used to perform knowledge distillation training on the preset shallow convolutional neural network model to obtain a trained shallow convolutional neural network model; the knowledge distillation technology allows the knowledge in the deep convolutional neural network model to be transferred to the shallow convolutional neural network model, and the shallow convolutional neural network model can learn the data structure and pattern learned by the deep convolutional neural network model without a large amount of training or complex architecture design, which can reduce the computational cost, improve the performance of the shallow convolutional neural network model, and make it close to the performance of the deep convolutional neural network model. The trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model; the quantization process can significantly reduce the storage requirements and computing costs of the shallow convolutional neural network model, and under the premise of ensuring the performance of the shallow convolutional neural network model, the data type in the calculation process is reduced, thereby reducing the amount of calculation. The quantized shallow convolutional neural network model can speed up the model reasoning speed, making it easier to deploy the shallow convolutional neural network model on resource-constrained devices, such as mobile devices, embedded systems, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A flowchart of the first embodiment of the language model training method of the present application is provided;

[0043] Figure 2 This is a flowchart of the language model training method of this application;

[0044] Figure 3 This is a schematic diagram of the module structure of the language model training device according to an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the language model training method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the present application is: obtaining training data; training a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer; performing knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model; and quantizing the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

[0050] In the prior art, since training data in some fields is relatively rare, such as audio data containing information about crying babies or the calls of rare animals, audio data with corresponding meaning annotations is even more difficult to obtain. Therefore, when training language models, the problem of data scarcity is faced, which limits the training effect of the model, resulting in insufficient performance of the training model and inability to accurately identify data in the corresponding field. In order to improve the accuracy of language model training in fields with scarce training data, there is a method in the existing method to identify data in fields with scarce training data through a large neural network model. However, the large neural network model has a large number of parameters and high requirements for computing resources, which is not convenient for deployment and application on mobile terminals. In addition, there is also a method of reducing the number of model parameters and the number of layers to make it suitable for mobile terminal applications. However, the performance of this lightweight model is lower than that of the large neural network model. Therefore, in mobile terminal deployment applications, when identifying data in fields with scarce training data, it is difficult to balance between recognition performance and computing resource requirements. Therefore, how to balance the training performance of language models in fields with scarce training data and computing resource requirements is a problem that needs to be solved urgently.

[0051] This application can ensure that as much information as possible is learned from the training data by training the deep convolutional neural network model. The preset statistical pooling layer helps the deep convolutional neural network model to learn the common features and difference information data in the training data more effectively, and improves the training effect of the deep convolutional neural network model. The knowledge distillation technology allows the knowledge in the deep convolutional neural network model to be transferred to the shallow convolutional neural network model. The shallow convolutional neural network model can learn the data structure and pattern learned by the deep convolutional neural network model without a lot of training or complex architecture design, which can reduce the computing cost and improve the performance of the shallow convolutional neural network model, making it close to the performance of the deep convolutional neural network model. Quantization processing can significantly reduce the storage requirements and computing costs of the shallow convolutional neural network model. Under the premise of ensuring the performance of the shallow convolutional neural network model, the data type in the calculation process is reduced, the amount of calculation is reduced, and the quantized shallow convolutional neural network model can speed up the model reasoning speed, making the shallow convolutional neural network model easier to deploy on resource-constrained devices, such as mobile devices, embedded systems, etc.

[0052] It should be noted that the execution subject of this embodiment can be a language model training system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a language model training device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the language model training system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, this application proposes a language model training method of the first embodiment, please refer to Figure 1 , the language model training method comprises steps S10 to S40:

[0054] Step S10, obtaining training data;

[0055] It should be noted that training data provides basic data for subsequent neural network model training, and collects data sets related to the current task. For example, image recognition tasks require the collection of image data from various fields, and sound recognition tasks require the collection of sound data from various fields. Using training data from multiple fields of data can enable the trained model to learn more diverse features and patterns, so that the model can make more accurate predictions when facing new data from different fields in actual use.

[0056] Furthermore, the training data can be preprocessed, including data cleaning, normalization, enhancement and other operations, to improve the training effect of the model.

[0057] Step S20, training a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer;

[0058] It should be noted that at least one deep convolutional neural network model is established based on the preset statistical pooling layer. A deep convolutional neural network model usually contains multiple convolutional layers, pooling layers, activation functions, fully connected layers and other components. When designing a deep convolutional neural network model, it is necessary to select a suitable network structure according to the task requirements.

[0059] The statistical pooling layer is usually used to capture information in the training data, such as the standard deviation, mean, maximum value, minimum value, etc. of the training data features. Furthermore, the preset statistical pooling layer can calculate the average value of the training data, and then calculate the variance based on the obtained average value. The average value reflects the common characteristics of all training data, and the variance reflects the differences between the training data. The fully connected layer of the deep convolutional neural network model combines the two. The statistical pooling layer can help the deep convolutional neural network model capture more feature information and improve the robustness and generalization ability of the deep convolutional neural network model to a certain extent.

[0060] Step S30, performing knowledge distillation training on the preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model;

[0061] It should be noted that a shallow convolutional neural network model can be designed according to the complexity of the processing task, the characteristics of the data, or the limitations of the computing resources of the deployment terminal. After being trained with training data, the deep convolutional neural network model can achieve high performance and accuracy. As a teacher model for knowledge distillation, it provides predictions or intermediate layer representations of the input data for the shallow convolutional neural network model. During the knowledge distillation training process, the goal of the shallow convolutional neural network model as a student model is to imitate the prediction behavior or intermediate layer representation of the teacher model, thereby acquiring the knowledge of the teacher model. The difference between the predictions of the teacher model and the student model is measured by setting the loss function. When training the student model, the predicted probability distribution of the teacher model is used as a soft label to help the student model learn more about the relationship between categories and the uncertainty of the model. The predicted category results of the teacher model are used as hard labels to provide direct supervision information for the student model and clarify the expected output results. The soft and hard labels of the teacher model are used as the training targets of the student model. During the training process, the parameters of the student model are continuously adjusted through back-propagation algorithms or gradient descent optimizers to minimize the difference between the predictions of the teacher model and the student model. Multiple iterations are repeated until the performance of the student model on the validation set reaches the preset stopping condition.

[0062] Step S40, quantizing the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

[0063] It should be noted that quantization processing is to convert the weight parameters (such as the filter weights of the convolutional layer, the weight matrix of the fully connected layer, etc.) and activation values ​​(i.e., the output of the network layer) in the trained shallow convolutional neural network model from the original high-precision floating-point numbers to a low-precision representation. According to the specific application requirements, accuracy requirements, or support of the model deployment terminal hardware, the quantization method can be fixed-point quantization, floating-point quantization, symmetric quantization, or asymmetric quantization. The parameters and activation values ​​of the quantized shallow convolutional neural network model will be represented using fewer bits, significantly reducing the storage requirements of the model. As the data type used in the calculation process becomes smaller, the computational cost will also be reduced, thereby speeding up the model inference. The quantized shallow convolutional neural network model can be directly used for deployment on resource-constrained devices, such as mobile devices or embedded systems.

[0064] For example, refer to Figure 2 In the field of baby cry recognition, audio data containing baby crying information is relatively rare. When training the baby cry recognition model, a large-scale sound classification dataset including sounds from multiple fields is used to preliminarily train two deep convolutional neural network models. The two preliminarily trained deep convolutional neural network models are fine-tuned using the domain data of baby cry detection and the domain data of baby cry meaning classification, respectively, to obtain a deep convolutional neural network model for baby cry detection and a deep convolutional neural network model for cry meaning classification. Two new shallow convolutional neural network models are established, and one of the shallow convolutional neural network models is trained using the domain data of baby cry detection and the deep convolutional neural network model for baby cry detection using the knowledge distillation technology, to obtain a shallow convolutional neural network model for baby cry detection. At the same time, another shallow convolutional neural network model is trained using the domain data of baby cry meaning classification and the deep convolutional neural network model for cry meaning classification, to obtain a shallow convolutional neural network model for baby cry meaning classification. Through quantization processing, the number of parameters of the shallow convolutional neural network model for detecting infant crying and the shallow convolutional neural network model for classifying the meaning of infant crying is further reduced, thereby reducing the computing resource usage of the model. In the process of model application, the shallow convolutional neural network model for detecting infant crying after quantization detects the sound data in the environment to determine whether there is infant crying in the sound data; if there is infant crying in the sound data, the shallow convolutional neural network model for classifying the meaning of infant crying after quantization analyzes the meaning of crying on the sound data to realize the recognition of the meaning of infant crying.

[0065] The present embodiment provides a language model training method, which specifically obtains training data; trains a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, and the deep convolutional neural network model is established according to a preset statistical pooling layer; the deep convolutional neural network model usually has higher performance and can capture complex features in the training data. By training the deep convolutional neural network model, it can be ensured that as much information as possible is learned from the training data. The preset statistical pooling layer helps the deep convolutional neural network model to more effectively learn the common features and difference information data in the training data, thereby improving the training effect of the deep convolutional neural network model. The trained deep convolutional neural network model is used to perform knowledge distillation training on the preset shallow convolutional neural network model to obtain a trained shallow convolutional neural network model; the knowledge distillation technology allows the knowledge in the deep convolutional neural network model to be transferred to the shallow convolutional neural network model, and the shallow convolutional neural network model can learn the data structure and pattern learned by the deep convolutional neural network model without a large amount of training or complex architecture design, which can reduce the computing cost and improve the performance of the shallow convolutional neural network model, making it close to the performance of the deep convolutional neural network model. The trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model; the quantization process can significantly reduce the storage requirements and computing costs of the shallow convolutional neural network model, and under the premise of ensuring the performance of the shallow convolutional neural network model, the data type in the calculation process is reduced, thereby reducing the amount of calculation. The quantized shallow convolutional neural network model can speed up the model reasoning speed, making it easier to deploy the shallow convolutional neural network model on resource-constrained devices, such as mobile devices, embedded systems, etc.

[0066] In a feasible implementation, step S20 may include steps S201 to S203:

[0067] Step S201, performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data;

[0068] Step S202, obtaining domain data;

[0069] Step S203, fine-tuning the deep convolutional neural network model according to the fused feature data and the domain data to obtain a trained deep convolutional neural network model.

[0070] It should be noted that the statistical pooling layer is usually used to capture information in the training data, such as the standard deviation, mean, maximum value, minimum value, etc. of the training data features. Furthermore, the preset statistical pooling layer can calculate the average value of the training data, and then calculate the variance based on the obtained average value. The average value reflects the common characteristics of all training data, and the variance reflects the differences between the training data. The statistical pooling layer can help deep convolutional neural network models extract more comprehensive features, especially when dealing with complex or changeable data. In the feature fusion stage, the fully connected layer fuses the average value of the training data and the variance of the training data to obtain the fused feature data of the training data.

[0071] Domain data refers to data related to a specific problem or domain that the model is trying to solve. This data may come from different sources and have different formats and characteristics. For example, in the field of baby cry recognition, domain data can be baby cry detection data and baby cry meaning classification data. Using fused feature data and domain data as input to fine-tune the deep convolutional neural network model can help the model better adapt to the corresponding domain and improve the performance of processing domain-related tasks.

[0072] In this embodiment, the statistical pooling layer can extract richer statistical information from the training data, provide a more comprehensive feature representation for the model, and help improve the performance of the model. The fully connected layer can integrate features from different layers or different pooling methods, further enriching the diversity of features, so that the model can better learn the complex structures and patterns in the training data. By fusing feature data and domain data to fine-tune the model training deep convolutional neural network model, the model can learn specific features related to specific fields, further optimize the performance in specific fields, and make the model more suitable for specific application scenarios.

[0073] In a feasible implementation, step S201 may include steps S2011 to S2012:

[0074] Step S2011, performing sliding window processing on the training data through the statistical pooling layer to obtain the average value of the training data and the variance of the training data;

[0075] Step S2012: performing feature fusion on the mean value of the training data and the variance of the training data through the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data.

[0076] It should be noted that the sliding window is a rectangular area that moves on the training data according to a certain step length and window size. At each movement, the statistical pooling layer calculates the statistical characteristics of the data in the window. At each position in the sliding window, the statistical pooling layer calculates the average of all feature values ​​in the area. This average reflects the central trend of the data in the area. In addition to the average, the statistical pooling layer also calculates the variance of the data in the window. The variance is the average of the squares of the differences between each value and its mean, which reflects the discreteness or fluctuation of the data. The specific calculation method is shown in the following formula:

[0077]

[0078] Among them, s 2 is the variance, N is the amount of training data, h avg is the average value, h i For each training data.

[0079] The mean and variance of the training data are input into a fully connected layer, and a value is output as the final output result of the pooling layer, that is, the fused feature data. The specific calculation method is shown in the following formula:

[0080] h stat =fc(h avg ,s 2 )

[0081] Among them, h stat is the fusion feature data, h avg is the average value, s 2 is the variance.

[0082] In this embodiment, the statistical pooling layer calculates the mean and variance in the sliding window, which has a certain robustness to small changes or noise in the input data, helping the model to maintain stable performance on complex or changing data sets. By calculating the mean and variance, the training data is reduced and compressed, reducing the amount of calculation in the subsequent network layer, which helps to accelerate the training process of the model and reduce training time and computing resource consumption. The fully connected layer fuses the mean and variance to form a more comprehensive and rich feature representation.

[0083] In a feasible implementation, step S30 may include steps S301 to S304:

[0084] Step S301, obtaining domain data;

[0085] Step S302, predicting the domain data using the trained deep convolutional neural network model to obtain a target label and a probability distribution of the target label;

[0086] Step S303, training a shallow convolutional neural network model using the trained deep convolutional neural network model and the domain data to obtain a target label loss and a probability distribution loss of the target label;

[0087] Step S304, calculating a loss function according to the target label loss and the probability distribution loss of the target label, and optimizing the shallow convolutional neural network model according to the loss function to obtain a trained shallow convolutional neural network model.

[0088] It should be noted that domain data refers to data related to the specific problem or domain that the model is trying to solve. These data may come from different sources and have different formats and characteristics. For example, in the field of baby cry recognition, the domain data can be baby cry detection data and baby cry meaning classification data. The trained deep convolutional neural network model receives domain data as input, and after multiple layers of convolution, pooling, and activation operations, it outputs the predicted value of the target label and the probability distribution of the target label. The predicted value of the target label is the judgment of the trained deep convolutional neural network model on the category to which the domain data belongs, while the probability distribution shows the model's confidence in each category.

[0089] The deep convolutional neural network model is used as the teacher model, and the shallow convolutional neural network model is used as the student model. The domain data is input into both models at the same time. The deep convolutional neural network model outputs the predicted value and probability distribution of the target label, while the shallow convolutional neural network model outputs its own prediction results. By comparing the prediction results of the two models, the target label loss (i.e., the difference between the label output by the deep convolutional neural network model and the predicted label output by the shallow convolutional neural network model) and the probability distribution loss of the target label (i.e., the difference between the probability distribution output by the deep convolutional neural network model and the probability distribution output by the shallow convolutional neural network model) can be calculated. The total loss function is calculated based on the target label loss and the probability distribution loss of the target label. The loss function is used to measure the difference between the model prediction result and the true result. The smaller its value, the more accurate the model prediction. Then, an optimization algorithm, such as backpropagation algorithm, gradient descent method, etc., is used. The parameters of the shallow convolutional neural network model are updated and optimized according to the loss function to reduce the prediction error and improve the performance of the model. After multiple iterations and optimizations, the trained shallow convolutional neural network model is obtained.

[0090] In this embodiment, after the domain data is predicted by the trained deep convolutional neural network model, the target label and the probability distribution of the target label are provided, so that when training the shallow convolutional neural network model, not only can it be optimized according to the loss of the target label, but it can also be adjusted more finely according to the loss of the probability distribution. This multi-level optimization helps to improve the prediction accuracy and generalization ability of the shallow convolutional neural network model. Migrating the rich knowledge and experience learned by the trained deep convolutional neural network model to the shallow convolutional neural network model can improve the training efficiency and performance of the shallow convolutional neural network model.

[0091] In a feasible implementation, step S304 may include steps S3041 to S3042:

[0092] Step S3041, performing a weighted operation according to the target label loss and the probability distribution loss of the target label to obtain the loss function;

[0093] Step S3042: According to the loss function, back-propagation optimizes the shallow convolutional neural network model to obtain a trained shallow convolutional neural network model.

[0094] It should be noted that the target label loss and the target label probability distribution loss measure the performance of the shallow convolutional neural network model in label prediction and probability distribution prediction respectively. In order to comprehensively consider the performance of these two aspects, the target label loss and the target label probability distribution loss are combined into a single loss function through weighted operation. A weight is assigned to each loss term, and different losses are added according to the weight to obtain the final loss function.

[0095] Using the loss function as the optimization target, the back-propagation algorithm is used to update the parameters of the shallow convolutional neural network model. The gradient is calculated according to the loss function, and the gradient is back-propagated from the output layer to the input layer layer by layer. At the same time, the parameters in the model are updated, which can be achieved through the gradient descent algorithm. Iterative optimization is performed according to the above method until the performance of the model on the validation set reaches the preset standard or the maximum number of iterations is reached. In each iteration, the model updates its parameters according to the new loss function to gradually reduce the prediction error. After multiple iterative optimizations, the trained shallow convolutional neural network model is obtained, which can better adapt to the tasks and data processing in the corresponding field.

[0096] In this embodiment, the loss function not only considers the accuracy of the target label through the target label loss, but also considers the model's ability to predict the probability distribution of the target label through the probability distribution loss of the target label, which helps to improve the generalization ability and robustness of the model. By weighting the target label loss and the probability distribution loss of the target label, they are customized for different tasks and data sets, and the relative importance of the two in the loss function is flexibly set to achieve better performance. By constructing a loss function and observing the numerical changes of different loss terms, we can understand the performance of the model during the training process, understand the advantages and disadvantages of the model in label prediction and probability distribution prediction, and then perform targeted optimization. Using the back propagation algorithm, the gradient of the model parameters can be efficiently calculated according to the loss function, and the parameters can be updated through the gradient descent method. The model can accelerate the training process of the model and can automatically adapt to the noise and outliers of the data during the training process, thereby showing stronger robustness and better adapting to different tasks and data sets.

[0097] In a feasible implementation, step S40 may include steps S401 to S403:

[0098] Step S401, performing forward propagation processing on the trained shallow convolutional neural network model to obtain output statistical information and weight statistical information;

[0099] Step S402, calculating a quantization parameter according to the output statistical information and the weight statistical information to obtain a quantization parameter;

[0100] Step S403, quantizing the trained shallow convolutional neural network model according to the quantization parameters and the preset quantization strategy to obtain a quantized shallow convolutional neural network model.

[0101] It should be noted that forward propagation is the standard process for a neural network to calculate a predicted value or output value, in which the input data passes through each layer of the convolutional neural network model and undergoes a series of linear transformations and nonlinear activation functions to finally obtain the output.

[0102] During the forward propagation process, in addition to paying attention to the final output of the shallow convolutional neural network model, it is also necessary to collect statistical information about the output of the shallow convolutional neural network model, such as the distribution, range, mean, standard deviation, etc. of the output value; as well as collect statistical information about the weights of the shallow convolutional neural network model, including the convolution kernel, fully connected layer weights, etc., the range, distribution, sparsity, etc. of the statistical information weights.

[0103] Quantization is a process used to reduce the size of the model and increase the speed of inference. It is achieved by mapping the floating-point weights and activation values ​​in the model to a smaller set of integers. Based on the output statistics and weight statistics, the quantization parameters are calculated. Quantization parameters usually include quantization level (i.e., the number of integers), quantization range (i.e., the upper and lower bounds of the integer set), quantization step size (i.e., the interval between adjacent integers), etc. Quantization parameters ensure that the quantized model can maintain high accuracy and performance. According to the calculated quantization parameters and the preset quantization strategy, the trained shallow convolutional neural network model is quantized. The quantization strategy can select quantization methods such as uniform quantization and non-uniform quantization. After the quantization process is completed, the quantized shallow convolutional neural network model is obtained, which is easier to deploy on resource-constrained devices, such as mobile devices and embedded systems.

[0104] In this embodiment, by forward propagating the trained model, the quantization parameters are calculated according to the output statistical information and the weight statistical information, and the quantization parameters can be flexibly adjusted according to different models and data. The calculated quantization parameters can more accurately reflect the characteristics of the model, minimize the impact of quantization on model performance, and make the quantized model as close as possible to the performance of the model before quantization. Reasonable quantization parameter settings can significantly reduce the storage space and computing resource requirements required for the model, making it easier to deploy on resource-constrained devices, such as mobile devices, embedded systems, etc.

[0105] The present application also provides a language model training device, please refer to Figure 3 , the language model training device comprises:

[0106] An acquisition module 10, used for acquiring training data;

[0107] A first model training module 20, used for training a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer;

[0108] A second model training module 30 is used to perform knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model;

[0109] The quantization module 40 is used to perform parameter quantization processing on the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

[0110] Optionally, the first model training module 20 further includes:

[0111] Performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data;

[0112] Get domain data;

[0113] The deep convolutional neural network model is fine-tuned according to the fused feature data and the domain data to obtain a trained deep convolutional neural network model.

[0114] Optionally, the first model training module 20 further includes:

[0115] Performing sliding window processing on the training data through the statistical pooling layer to obtain the average value of the training data and the variance of the training data;

[0116] The mean value of the training data and the variance of the training data are subjected to feature fusion through the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data.

[0117] Optionally, the second model training module 30 is further used for:

[0118] Get domain data;

[0119] Predicting the domain data using the trained deep convolutional neural network model to obtain a target label and a probability distribution of the target label;

[0120] Training a shallow convolutional neural network model using the trained deep convolutional neural network model and the domain data to obtain a target label loss and a probability distribution loss of the target label;

[0121] A loss function is calculated according to the target label loss and the probability distribution loss of the target label, and the shallow convolutional neural network model is optimized according to the loss function to obtain a trained shallow convolutional neural network model.

[0122] Optionally, the second model training module 30 is further used for:

[0123] Performing a weighted operation according to the target label loss and the probability distribution loss of the target label to obtain the loss function;

[0124] According to the loss function, back propagation is used to optimize the shallow convolutional neural network model to obtain a trained shallow convolutional neural network model.

[0125] Optionally, the quantization module 40 further includes:

[0126] Performing forward propagation processing on the trained shallow convolutional neural network model to obtain output statistical information and weight statistical information;

[0127] Calculate a quantization parameter according to the output statistical information and the weight statistical information to obtain a quantization parameter;

[0128] According to the quantization parameters and the preset quantization strategy, the trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model.

[0129] The language model training device provided in the embodiment of the present application adopts the language model training method in the above embodiment, which can solve the technical problems of language model training. Compared with the prior art, the beneficial effects of the language model training device provided in the embodiment of the present application are the same as the beneficial effects of the language model training method provided in the above embodiment, and other technical features in the language model training device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0130] The present application provides a language model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the language model training method in the above-mentioned embodiment 1.

[0131] Reference below Figure 4 , which shows a schematic diagram of the structure of a language model training device suitable for implementing the embodiment of the present application. The language model training device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The language model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0132] like Figure 4As shown, the language model training device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the language model training device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the language model training device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a language model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0133] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0134] The language model training device provided by the present application adopts the language model training method in the above embodiment to solve the technical problems of language model training. Compared with the prior art, the beneficial effects of the language model training device provided by the present application are the same as the beneficial effects of the language model training method provided by the above embodiment, and the other technical features in the language model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0135] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0136] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0137] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the language model training method in the above-mentioned embodiment.

[0138] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0139] The computer-readable storage medium may be included in the language model training device; or may exist independently without being assembled into the language model training device.

[0140] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the language model training device, the language model training device: obtains training data; trains a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer; performs knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model; and quantizes the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

[0141] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0144] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned language model training method, and can solve the technical problems of language model training. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the language model training method provided in the above-mentioned embodiment, and will not be repeated here.

[0145] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned detection method when executed by a processor.

[0146] The computer program product provided in this application can solve the technical problem of language model training. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the language model training method provided in the above embodiment, which will not be repeated here.

[0147] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A language model training method, characterized in that: The language model training method comprises: Get training data; Training a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer; Performing knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model; The trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model.

2. The method according to claim 1, characterized in that The step of training the deep convolutional neural network model according to the training data to obtain the trained deep convolutional neural network model comprises: Performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data; Get domain data; The deep convolutional neural network model is fine-tuned according to the fused feature data and the domain data to obtain a trained deep convolutional neural network model.

3. The method according to claim 2, characterized in that The step of performing feature extraction and feature fusion on the training data based on the statistical pooling layer and the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data includes: Performing sliding window processing on the training data through the statistical pooling layer to obtain the average value of the training data and the variance of the training data; The mean value of the training data and the variance of the training data are subjected to feature fusion through the fully connected layer of the deep convolutional neural network model to obtain fused feature data of the training data.

4. The method according to claim 1, characterized in that The step of performing knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model comprises: Get domain data; Predicting the domain data using the trained deep convolutional neural network model to obtain a target label and a probability distribution of the target label; Training a shallow convolutional neural network model using the trained deep convolutional neural network model and the domain data to obtain a target label loss and a probability distribution loss of the target label; A loss function is calculated according to the target label loss and the probability distribution loss of the target label, and the shallow convolutional neural network model is optimized according to the loss function to obtain a trained shallow convolutional neural network model.

5. The method according to claim 4, characterized in that The step of calculating the loss function according to the target label loss and the probability distribution loss of the target label, and optimizing the shallow convolutional neural network model according to the loss function to obtain the trained shallow convolutional neural network model includes: Performing a weighted operation according to the target label loss and the probability distribution loss of the target label to obtain the loss function; According to the loss function, back propagation is used to optimize the shallow convolutional neural network model to obtain a trained shallow convolutional neural network model.

6. The method according to claim 1, characterized in that The step of quantizing the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model comprises: Performing forward propagation processing on the trained shallow convolutional neural network model to obtain output statistical information and weight statistical information; Calculate a quantization parameter according to the output statistical information and the weight statistical information to obtain a quantization parameter; According to the quantization parameters and the preset quantization strategy, the trained shallow convolutional neural network model is quantized to obtain a quantized shallow convolutional neural network model.

7. A language model training device, characterized in that: The language model training device comprises: An acquisition module is used to acquire training data; A first model training module is used to train a deep convolutional neural network model according to the training data to obtain a trained deep convolutional neural network model, wherein the deep convolutional neural network model is established according to a preset statistical pooling layer; A second model training module is used to perform knowledge distillation training on a preset shallow convolutional neural network model through the trained deep convolutional neural network model to obtain a trained shallow convolutional neural network model; The quantization module is used to perform parameter quantization processing on the trained shallow convolutional neural network model to obtain a quantized shallow convolutional neural network model.

8. A language model training device, characterized in that: The language model training device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the language model training method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 6 are implemented.

Images