Network model processing method, device, equipment and computer-readable storage medium

Abstract

The present disclosure provides a network model processing method, apparatus, device, and computer-readable storage medium. The method comprises: obtaining a dataset to be trained, wherein the dataset to be trained includes audio training data in a target scenario and text annotation results corresponding to the audio training data; determining a CTC loss function, a knowledge distillation loss function, and an attention loss function based on a preset recognition model and the model to be trained, wherein the recognition model is a network model trained using preset old-scenario training data, and the model to be trained is obtained by copying the recognition model; determining a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function; and training the model to be trained using the dataset to be trained and the target loss function to obtain a trained speech recognition model. This allows a speech recognition model with both target and old-scenario recognition capabilities to be quickly trained using only a small amount of training data in the target scenario.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a network model processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of deep learning technology, automatic speech recognition (ASR) based on end-to-end deep neural networks (DNNs) has become increasingly important in practical applications such as telephone robots, voice assistants, and voice input methods. End-to-end deep neural network ASR models convert input speech signals into output text by learning the hierarchical representations and inherent patterns of sample data. However, due to the large parameter size of end-to-end deep neural network models, the model's speech recognition performance often relies on large amounts of speech training data. Typically, ASR models trained using speech training data for a specific scenario are difficult to apply to new scenarios. Because speech data annotation requires at least one human listening session, annotation costs are often high. Consequently, it is difficult to obtain large amounts of speech data for model training in new business scenarios in a short period of time, seriously hindering the rapid development of new businesses.

[0003] To solve the above technical problems, the existing technology generally uses mixed data for training. Specifically, a large amount of old scene speech training data is mixed with a small amount of new scene speech training data to form a new mixed training set, and the ASR model is retrained.

[0004] However, the aforementioned model training method requires a long time to retrain and fine-tune the model due to the large size of the mixed training set. Each time a new business scenario emerges, the complete ASR model training and fine-tuning process must be repeated, significantly increasing R&D costs. Summary of the Invention

[0005] The present disclosure provides a network model processing method, apparatus, device and computer-readable storage medium, which are used to solve the technical problems that existing model processing methods are time-consuming and have high R&D costs.

[0006] A first aspect of the present disclosure is to provide a network model processing method, comprising:

[0007] Obtaining a training data set from a database, wherein the training data set includes audio training data in a target scene and text annotation results corresponding to the audio training data;

[0008] Determine the CTC loss function, the knowledge distillation loss function, and the attention loss function based on a preset recognition model and a model to be trained, wherein the recognition model is a network model trained using preset old scene training data, and the model to be trained is obtained by copying the recognition model;

[0009] Determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function;

[0010] The model to be trained is trained using the data set to be trained and the target loss function to obtain a trained speech recognition model.

[0011] A second aspect of the present disclosure is to provide a network model processing method, comprising:

[0012] Obtaining a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized;

[0013] According to the speech recognition instruction, the speech to be recognized is input into a preset speech recognition model to obtain text information corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function, and an attention loss function, and a training data set including audio training data in a target scenario and text annotation results corresponding to the audio training data;

[0014] The text message is sent to the terminal device.

[0015] A third aspect of the present disclosure is to provide a network model processing device, comprising:

[0016] An acquisition module is used to acquire a training data set from a database, wherein the training data set includes audio training data in a target scene and text annotation results corresponding to the audio training data;

[0017] A calculation module, configured to determine a CTC loss function, a knowledge distillation loss function, and an attention loss function based on a preset recognition model and a model to be trained, wherein the recognition model is a network model trained using preset old scene training data, and the model to be trained is obtained by copying the recognition model;

[0018] A determination module, configured to determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function;

[0019] The training module is used to train the model to be trained using the training data set and the target loss function to obtain a trained speech recognition model. The network model processing device includes:

[0020] An acquisition module is used to acquire a training data set from a database, wherein the training data set includes audio training data in a target scene and text annotation results corresponding to the audio training data;

[0021] A calculation module, configured to determine a CTC loss function, a knowledge distillation loss function, and an attention loss function based on a preset recognition model and a model to be trained, wherein the recognition model is a network model trained using preset old scene training data, and the model to be trained is obtained by copying the recognition model;

[0022] A determination module, configured to determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function;

[0023] The training module is used to train the model to be trained using the data set to be trained and the target loss function to obtain a trained speech recognition model.

[0024] A fourth aspect of the present disclosure is to provide a network model processing device, comprising:

[0025] An instruction acquisition module is used to acquire a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized;

[0026] A recognition module, configured to input the speech to be recognized into a preset speech recognition model according to the speech recognition instruction, and obtain text information corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function, and an attention loss function, and a training data set including audio training data in a target scenario and text annotation results corresponding to the audio training data;

[0027] A sending module is used to send the text message to the terminal device.

[0028] A fifth aspect of the present disclosure is to provide a network model processing device, comprising: a memory, a processor;

[0029] Memory; a memory for storing instructions executable by the processor;

[0030] The processor is used to call the program instructions in the memory to execute the network model processing method as described in the first aspect or the second aspect.

[0031] The sixth aspect of the present disclosure is to provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the network model processing method as described in the first aspect or the second aspect.

[0032] The network model processing method, apparatus, device, and computer-readable storage medium provided by the present disclosure obtain a training data set including training data in a target scenario and construct a target loss function based on a CTC loss function, a knowledge distillation loss function, and an attention loss function. Thus, based on the training data set and the target loss function, an attention loss-based training operation can be performed on a preset training model that has been trained using preset old scene training data. This allows for rapid training to obtain a speech recognition model that has both target and old scene recognition capabilities using only a small amount of training data in the target scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0034] Figure 1 This is a system architecture diagram based on which the present disclosure is based;

[0035] Figure 2 A flowchart of a network model processing method provided in the first embodiment of the present disclosure;

[0036] Figure 3 A schematic diagram of annotated audio training data provided by an embodiment of the present disclosure;

[0037] Figure 4 A flowchart of a network model processing method provided in the second embodiment of the present disclosure;

[0038] Figure 5 A flowchart of a network model processing method provided in the third embodiment of the present disclosure;

[0039] Figure 6 A flowchart of a network model processing method provided in the fourth embodiment of the present disclosure;

[0040] Figure 7 A flowchart of a network model processing method provided in Embodiment 5 of the present disclosure;

[0041] Figure 8 This is another system architecture diagram on which the present disclosure is based;

[0042] Figure 9 A schematic diagram of speech recognition provided in an embodiment of the present disclosure;

[0043] Figure 10 A schematic diagram of the structure of a network model processing device provided in Example 6 of the present disclosure;

[0044] Figure 11 A schematic diagram of the structure of a network model processing device provided in Embodiment 7 of the present disclosure;

[0045] Figure 12 A schematic diagram of the structure of a network model processing device provided in Embodiment 8 of the present disclosure;

[0046] Figure 13 A schematic diagram of the structure of a network model processing device provided in Embodiment 9 of the present disclosure;

[0047] Figure 14 A schematic diagram of the structure of a network model processing device provided in the tenth embodiment of the present disclosure;

[0048] Figure 15 This is a structural diagram of the network model processing device provided in Example 11 of the present disclosure. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained based on the embodiments in the present disclosure are within the scope of protection of the present disclosure.

[0050] In response to the technical problems mentioned above that the existing model processing methods are time-consuming and have high R&D costs, the present disclosure provides a network model processing method, device, equipment and computer-readable storage medium.

[0051] It should be noted that the network model processing method, device, equipment and computer-readable storage medium provided by the present disclosure can be used in any speech recognition model processing scenario.

[0052] In actual applications, when the speech recognition model needs to be applied to other application scenarios, the speech data of the application scenario needs to be used to train the model. Specifically, the amount of data required to train the model is large. The speech data used to train the model first needs to be manually listened to at least once, and the speech data is annotated according to the results of the manual listening, and the model is trained based on the annotated speech data. When the above method is used to train the model for a new application scenario, the annotation process consumes a lot of human resources, and it is impossible to quickly obtain a large amount of speech data for new business scenarios for model training. In order to solve the above technical problems, the existing technology generally mixes large-scale old scenario speech training data and a small amount of new scenario speech training data together to form a new mixed training set, and uses the mixed training set to train the model. However, due to the large scale of the mixed training set data, the time cost of retraining the model and tuning is very high.

[0053] In the process of solving the above-mentioned technical problems, the inventors discovered through research that, in order to effectively utilize small-scale training data for new scenarios, avoid model retraining, and quickly and cost-effectively obtain speech recognition models for new business scenarios, a method can be used to combine knowledge distillation and attention loss to construct a target loss function, which is then used to train the model. This allows the model to retain both the recognition results of the old model and the reasons why the old model achieved these results. Consequently, a speech recognition model suitable for new business scenarios can be obtained using only a small amount of audio data from the new business scenario.

[0054] Figure 1 This is the system architecture diagram based on which the present disclosure is based, such as Figure 1 As shown, the network architecture underlying the present disclosure includes at least: a server 1 and a database 2. Server 1 is equipped with a network model processing device. The network model processing device is written in a language such as C / C++, Java, Shell, or Python. Database 2 can be a cloud server or server cluster, storing a large amount of data. Server 1 and database 2 are in communication with each other, enabling information exchange between them.

[0055] Figure 2 This is a flow chart of the network model processing method provided in the first embodiment of the present disclosure, as shown in FIG. Figure 2 As shown, the method includes:

[0056] Step 101: Acquire a dataset to be trained from a database, wherein the dataset to be trained includes audio training data in a target scenario and text annotation results corresponding to the audio training data.

[0057] The execution subject of this embodiment is a network model processing device, which can be coupled to a server. The server can be connected to a database for communication and information exchange. The database stores a large amount of data to be trained.

[0058] In this embodiment, in order to train the network model, it is first necessary to obtain a training data set. Since the trained network model needs to have the ability to recognize speech data in the target scenario, the training data set can include audio training data in the target scenario and the text annotation results corresponding to the audio training data.

[0059] Figure 3 A schematic diagram of the annotated audio training data provided by the embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the left side is the audio training data, and the right side is the text annotation result corresponding to the audio training data. The text annotation result corresponding to the audio training data is Good morning, everyone.

[0060] Step 102: Determine the CTC loss function, the knowledge distillation loss function, and the attention loss function based on the preset recognition model and the model to be trained, wherein the recognition model is a network model trained using the preset old scene training data, and the model to be trained is obtained by copying the recognition model.

[0061] In this embodiment, in order to improve the training efficiency of the network model so that the trained network model can recognize speech data in the old scene as well as speech data in the target scene, a recognition model that has been trained using preset old scene training data can be selected as the to-be-trained model. The target scene is different from the old scene.

[0062] In existing technologies, transfer learning is generally used to train the target model. However, since the training process focuses solely on the model's adaptability to the target scenario, it can lead to poor model versatility. Furthermore, when there is little training data for the target scenario, overfitting may occur.

[0063] To avoid the above technical problems, an attention-based loss function can be used to train the model. Specifically, the recognition model can be copied to obtain a model to be trained. The CTC loss function, knowledge distillation loss function, and attention loss function are determined based on the output data and parameter information of the recognition model and the model to be trained during the training process.

[0064] The CTC loss model is specifically used to ensure that the trained speech recognition model has speech recognition capabilities. The knowledge distillation loss function is specifically used to ensure that the output results of the recognition model and the trained speech recognition model are the same. The attention loss function is specifically used to ensure that the parameters of the recognition model and the trained speech recognition model are consistent during the recognition process.

[0065] Step 103: Determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function.

[0066] In this embodiment, since the requirements for each loss function may be different in different application scenarios, in order to improve the versatility of the trained speech recognition model, the target loss function can be determined based on the CTC loss function, the knowledge distillation loss function and the attention loss function.

[0067] Step 104: Use the to-be-trained data set and the target loss function to train the to-be-trained model to obtain a trained speech recognition model.

[0068] In this embodiment, after determining the target loss function based on the CTC loss function, the knowledge distillation loss function and the attention loss function, the model to be trained can be trained based on the target loss function and the training data set to obtain a trained speech recognition model.

[0069] Since the target loss function is constructed by the CTC loss function, the knowledge distillation loss function and the attention loss function, only the training data in the target scenario is needed to train the model.

[0070] Furthermore, based on the first embodiment, before step 104, the following steps are further included:

[0071] For each audio training data in the to-be-trained data set, a speech spectrum feature extraction operation is performed on the audio training data to obtain input feature information corresponding to the audio training data.

[0072] In this embodiment, in order to improve the training efficiency of the model to be trained, before the training data set is used to train the model to be trained, a preprocessing operation may be performed on the training data in the training data set.

[0073] Specifically, for each audio training data in the to-be-trained data set, a speech spectrum feature extraction operation may be performed on the audio training data to obtain input feature information corresponding to the audio training data.

[0074] Furthermore, based on the first embodiment, the speech spectrum feature extraction operation on the audio training data includes:

[0075] A speech spectrum feature extraction operation is performed on the audio training data through short-time Fourier transform.

[0076] Specifically, a conventional short-time Fourier transform (STFT) can be used to extract speech spectrum features from the audio training data. In practice, a window size of 20ms and a window sliding step of 10ms are used to obtain an 81-dimensional spectrum information sequence for each segment of audio training data.

[0077] The network model processing method provided in this embodiment obtains a training dataset including training data for a target scenario and constructs a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function. Thus, based on the training dataset and the target loss function, an attention loss-based training operation can be performed on a preset training model that has been trained using preset old scene training data. This allows for rapid training to obtain a speech recognition model that has both target and old scene recognition capabilities using only a small amount of training data for the target scenario.

[0078] Figure 4 This is a flow chart of the network model processing method provided in the second embodiment of the present disclosure, based on the first embodiment. Figure 4 As shown, step 102 specifically includes:

[0079] Step 201: input the to-be-trained data in the to-be-trained data set into the to-be-trained model, and calculate the CTC loss function corresponding to the to-be-trained model so that the output result of the to-be-trained model is consistent with the text annotation result corresponding to the audio training data.

[0080] In this embodiment, in order to calculate the target loss function, the CTC loss function corresponding to the model to be trained can be calculated first. Specifically, the training data in the training data set can be input into the model to be trained to obtain the output result of the model. The CTC loss function is calculated so that the output result of the training model is consistent with the text annotation result corresponding to the audio training data. Among them, the CTC loss function L1 is shown in Formula 1:

[0081]

[0082] where l c (·) is the CTC loss function, N b is the number of each batch in training. In practical applications, N b =32. f(h 0,i ,θ n) is the nonlinear mapping function corresponding to the old scene to be trained model, h 0,i is the input feature of the model to be trained, θ n The model parameters obtained by training the training data of the model to be trained. i This is the Chinese character labeling result of the sample.

[0083] Step 202: input the training data in the training data set into the recognition model and the training model respectively, and calculate the knowledge distillation loss function corresponding to the training model so that the output result of the training model is consistent with the output result of the recognition model.

[0084] In this embodiment, the knowledge distillation loss function is specifically used to make the recognition model f(h 0,i ,θ n ) and the trained speech recognition model output f(h 0,i ,θ o Specifically, the training data in the training data set can be input into the recognition model and the training model respectively, and the knowledge distillation loss function corresponding to the training model can be calculated.

[0085] Specifically, the output result of the model to be trained can be controlled to be consistent with the output result of the recognition model according to the KL divergence, so as to obtain the knowledge distillation loss function corresponding to the model to be trained.

[0086] The knowledge distillation loss function L2 can be specifically expressed as Formula 2:

[0087]

[0088] Among them, f(h 0,i ,θ n ) is the output result of the recognition model, f(h 0,i ,θ o ) is the output result of the trained speech recognition model, l KL (·) is the KL divergence.

[0089] Step 203: Determine at least one influencing factor corresponding to the model to be trained and the recognition model respectively, and calculate the attention loss function corresponding to the model to be trained so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

[0090] In this embodiment, the attention loss function is specifically used to keep the parameters of the recognition model and the trained speech recognition model consistent during the recognition process. Specifically, at least one influencing factor corresponding to the model to be trained and the recognition model can be determined respectively, and the attention loss function corresponding to the model to be trained can be calculated.

[0091] Among them, the attention loss function L3 can be shown as formula 3:

[0092]

[0093] Among them, l A (·) is the attention loss function.

[0094] Specifically, based on the first embodiment, step 203 specifically includes:

[0095] Calculating the gradient information corresponding to the model to be trained and the recognition model respectively;

[0096] Determine the attention vectors corresponding to the to-be-trained model and the recognition model respectively according to the gradient information;

[0097] Normalize the attention vectors corresponding to the model to be trained and the recognition model to obtain the attention loss function corresponding to the model to be trained.

[0098] Specifically, in order to realize the calculation of attention loss function, we can first calculate

[0099] The network model processing method provided in this embodiment uses the gradient information corresponding to the model to be trained and the recognition model. Specifically, the output result f(h 0,i ,θ o ) Each layer output is The trained speech recognition model outputs the result f(h 0,i ,θ n ) Each layer output is Calculate the gradient corresponding to the recognition model. The gradient corresponding to the recognition model can be specifically shown as Formula 4:

[0100]

[0101] Where max(h) is the maximum value of each column of h. Let argmax(h) be the index of the maximum value of each column of h. The gradient corresponding to the model to be trained is calculated. The gradient corresponding to the model to be trained can be specifically shown in Formula 5:

[0102]

[0103] in for The index value in each column corresponds to The vector composed of.

[0104] The attention vector v of the recognition model is obtained by formula (4-5) o and the attention vector v of the model to be trained n, as shown in Formula 6-7:

[0105]

[0106]

[0107] in, Represents the multiplication of corresponding elements of vectors.

[0108] The attention loss function L3 of the normalized vector can be calculated by formula (6-7), as shown in formula 8:

[0109]

[0110] where ||·|| and |·| represent the 2-norm and 1-norm, respectively.

[0111] The network model processing method provided in this embodiment calculates the CTC loss function, the knowledge distillation loss function, and the attention loss function respectively, and then constructs a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function. This target loss function can then be used to train the model to be trained. This allows for rapid training of a speech recognition model capable of recognizing both the target scene and previous scenes using only a small amount of training data in the target scene.

[0112] Figure 5 This is a flow chart of the network model processing method provided in the third embodiment of the present disclosure. Based on any of the above embodiments, Figure 5 As shown, step 103 specifically includes:

[0113] Step 301: Determine parameter information corresponding to the application scenario according to the application scenario of the speech recognition model.

[0114] Step 302: Determine a target loss function based on the parameter information and the CTC loss function, the knowledge distillation loss function, and the attention loss function.

[0115] In this embodiment, after respectively calculating the CTC loss function, the knowledge distillation loss function, and the attention loss function, a target loss function can be constructed based on the CTC loss function, the knowledge distillation loss function, and the attention loss function. The target loss function can be specifically shown in Formula 9:

[0116] L=L1+αL2+βL3 (9)

[0117] α and β are parameter information for the application scenario in which the speech recognition model is used. α and β are used to balance the model's ability to retain recognition model knowledge and learn new business information.

[0118] Due to different application scenarios, the requirements for each loss function may be different. Taking practical applications as an example, the larger α or β is, the stronger the model's retention of the recognition model's knowledge distillation and attention features. Conversely, the stronger the learning adaptability to new business information. When α = β = 0, model training degenerates into the existing deep transfer learning method. At this time, the model training only focuses on the model's adaptability to new business scenarios. When α≠0, β = 0, model training degenerates into the existing incremental learning method, that is, it only focuses on the consistency of the final output results of the new model and the recognition model. When α≠0, β≠0, while paying attention to new scenarios and retaining the consistency of the model output results, the model also retains the reasons why the recognition model outputs the result, thereby further improving the new model's retention of recognition model knowledge.

[0119] Therefore, according to the application scenario of the speech recognition model, parameter information corresponding to the application scenario can be determined, and the target loss function can be determined based on the parameter information and the CTC loss function, the knowledge distillation loss function, and the attention loss function.

[0120] The network model processing method provided in this embodiment determines parameter information corresponding to the application scenario of the speech recognition model based on the application scenario of the model. The target loss function is determined based on this parameter information and the CTC loss function, the knowledge distillation loss function, and the attention loss function. This allows the trained speech recognition model to be applied to more application scenarios, thereby improving the versatility of the speech recognition model.

[0121] Figure 6 This is a flow chart of the network model processing method provided in the fourth embodiment of the present disclosure. Based on any of the above embodiments, Figure 6 As shown, step 104 specifically includes:

[0122] Step 401: Input the training data in the training data set into the training model to obtain the loss value corresponding to the target loss function.

[0123] Step 402: Adjust the parameters of the model to be trained according to the loss value until the loss value is less than a preset threshold.

[0124] In this embodiment, the training data set can be used to perform backpropagation training on the training model. Specifically, the training data in the training data set can be input into the training model to obtain a loss value corresponding to the target loss function. Based on this loss value, the parameters of the training model are adjusted until the loss value is less than a preset threshold, thereby obtaining a trained speech recognition model.

[0125] The network model processing method provided in this embodiment performs backpropagation training on the training model using the training data set, and continuously adjusts the parameters of the training model according to the loss value, thereby effectively improving the recognition accuracy of the trained speech recognition model.

[0126] Figure 7 This is a flow chart of the network model processing method provided in the fifth embodiment of the present disclosure, as shown in FIG. Figure 7 As shown, the method includes:

[0127] Step 501: Acquire a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized.

[0128] Step 502: According to the speech recognition instruction, the speech to be recognized is input into a preset speech recognition model to obtain text information corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function, and an attention loss function, and a training data set including audio training data in a target scenario and text annotation results corresponding to the audio training data.

[0129] Step 503: Send the text message to the terminal device.

[0130] The execution subject of this embodiment is a network model processing device, which can be coupled to a server. The server can communicate with a terminal device and can further exchange information with the terminal device. The terminal device can be, for example, a desktop computer, a tablet computer, a mobile phone, etc.

[0131] Figure 8 This is another system architecture diagram based on the present disclosure, such as Figure 8 As shown, the network architecture underlying the present disclosure specifically includes a server 1 and a terminal device 3, wherein the server 1 is provided with a network model processing device. The terminal device can be a mobile phone, tablet computer, desktop computer, etc. The server 1 can be in communication with the terminal device 3, so that the two can exchange information.

[0132] In this embodiment, the network model processing device can obtain a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized, input the voice to be recognized into a preset voice recognition model according to the voice recognition instruction, and obtain text information corresponding to the voice to be recognized.

[0133] The speech recognition model is trained using a training dataset that includes training data in the target scenario and a target loss function constructed using a CTC loss function, a knowledge distillation loss function, and an attention loss function. Based on the training dataset and the target loss function, an attention loss-based training operation can be performed on a preset training model that has been trained using preset old-scenario training data. This allows rapid training to obtain a speech recognition model capable of recognizing both the target scenario and the old-scenario using only a small amount of training data in the target scenario.

[0134] In order to enable the user to understand the recognized text information in a timely manner, after the speech recognition model outputs the text information, the speech information can be sent to the terminal device for the user to view.

[0135] Figure 9 The speech recognition diagram provided by the embodiment of the present disclosure is as follows: Figure 9 As shown, the speech to be recognized in the speech recognition instruction sent by the terminal device can be input into the speech recognition model to obtain the recognition result output by the speech recognition model.

[0136] The network model processing method provided in this embodiment recognizes the speech to be recognized by adopting a speech recognition model obtained by training a training data set including the data to be trained in the target scenario and a target loss function constructed by the CTC loss function, the knowledge distillation loss function and the attention loss function, thereby improving the recognition accuracy of the speech data.

[0137] Figure 10 This is a structural diagram of a network model processing device provided in Example 6 of the present disclosure, as shown in FIG. Figure 10 As shown, the device includes: an acquisition module 61, a calculation module 62, a determination module 63 and a training module 64. The acquisition module 61 is used to obtain the data set to be trained from the database, wherein the data set to be trained includes audio training data in the target scene and text annotation results corresponding to the audio training data. The calculation module 62 is used to determine the CTC loss function, the knowledge distillation loss function and the attention loss function according to the preset recognition model and the model to be trained, wherein the recognition model is a network model trained with preset old scene training data, and the model to be trained is obtained by copying the recognition model. The determination module 63 is used to determine the target loss function based on the CTC loss function, the knowledge distillation loss function and the attention loss function. The training module 64 is used to train the model to be trained using the data set to be trained and the target loss function to obtain a trained speech recognition model.

[0138] Furthermore, based on the sixth embodiment, the device further includes:

[0139] The feature extraction module is used to perform a speech spectrum feature extraction operation on each audio training data in the to-be-trained data set to obtain input feature information corresponding to the audio training data.

[0140] Furthermore, based on the sixth embodiment, the feature extraction module is used to:

[0141] A speech spectrum feature extraction operation is performed on the audio training data through short-time Fourier transform.

[0142] The network model processing device provided in this embodiment obtains a training data set including training data in a target scenario and constructs a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function. Thus, based on the training data set and the target loss function, an attention loss-based training operation can be performed on a preset training model that has been trained using preset old scene training data. This allows for rapid training to obtain a speech recognition model that has both target and old scene recognition capabilities using only a small amount of training data in the target scenario.

[0143] Figure 11 This is a structural diagram of a network model processing device provided in the seventh embodiment of the present disclosure, based on the sixth embodiment. Figure 11 As shown, the calculation module includes: a first calculation unit 71, a second calculation unit 72 and a third calculation unit 73. The first calculation unit 71 is used to input the data to be trained in the data set to be trained into the model to be trained, and calculate the CTC loss function corresponding to the model to be trained, so that the output result of the model to be trained is consistent with the text annotation result corresponding to the audio training data. The second calculation unit 72 is used to input the data to be trained in the data set to be trained into the recognition model and the model to be trained respectively, and calculate the knowledge distillation loss function corresponding to the model to be trained, so that the output result of the model to be trained is consistent with the output result of the recognition model. The third calculation unit 73 is used to respectively determine at least one influencing factor corresponding to the model to be trained and the recognition model, and calculate the attention loss function corresponding to the model to be trained, so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

[0144] Furthermore, based on Example 6, the second calculation unit is used to: control the output result of the model to be trained to be consistent with the output result of the recognition model according to the KL divergence, and obtain the knowledge distillation loss function corresponding to the model to be trained.

[0145] Furthermore, based on Example 6, the third calculation unit is used to: respectively calculate the gradient information corresponding to the model to be trained and the recognition model; respectively determine the attention vectors corresponding to the model to be trained and the recognition model based on the gradient information; perform normalization operations on the attention vectors corresponding to the model to be trained and the recognition model to obtain the attention loss function corresponding to the model to be trained.

[0146] Figure 12 This is a structural diagram of a network model processing device provided in the eighth embodiment of the present disclosure. Based on any of the above embodiments, Figure 12 As shown, the determination module includes: a determination unit 81 and an adjustment unit 82. The determination unit 81 is used to determine parameter information corresponding to the application scenario of the speech recognition model according to the application scenario of the speech recognition model; the adjustment unit 82 is used to determine the target loss function based on the parameter information and the CTC loss function, the knowledge distillation loss function, and the attention loss function.

[0147] Figure 13 This is a structural diagram of a network model processing device provided in the ninth embodiment of the present disclosure. Based on any of the above embodiments, Figure 13 As shown, the training module includes an input unit 91 and a training unit 92. The input unit 91 is used to input the training data in the training data set into the training model to obtain a loss value corresponding to the target loss function. The training unit 92 is used to adjust the parameters of the training model according to the loss value until the loss value is less than a preset threshold.

[0148] Figure 14 This is a structural diagram of a network model processing device provided in the tenth embodiment of the present disclosure, as shown in FIG. Figure 14 As shown, the device includes: an instruction acquisition module 111, a recognition module 112 and a sending module 113, wherein the instruction acquisition module 111 is used to obtain a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized. The recognition module 112 is used to input the voice to be recognized into a preset voice recognition model according to the voice recognition instruction, and obtain text information corresponding to the voice to be recognized, wherein the voice recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function and an attention loss function, and a data set to be trained including audio training data in a target scenario and text annotation results corresponding to the audio training data. The sending module 113 is used to send the text information to the terminal device.

[0149] Figure 15 This is a structural diagram of a network model processing device provided in the eleventh embodiment of the present disclosure, as shown in FIG. Figure 15 As shown, the device includes: a memory 121, a processor 122;

[0150] Memory 121; Memory 121 for storing instructions executable by the processor 122;

[0151] The processor 122 is configured to call the program instructions in the memory 121 to execute the network model processing method as described in any of the above embodiments.

[0152] Memory 121 is used to store programs. Specifically, the programs may include program code, which includes computer operating instructions. Memory 121 may include high-speed RAM memory or non-volatile memory, such as at least one disk storage device.

[0153] The processor 122 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure.

[0154] Optionally, in a specific implementation, if the memory 121 and the processor 122 are implemented independently, the memory 121 and the processor 122 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0155] Optionally, in a specific implementation, if the memory 121 and the processor 122 are integrated on a chip, the memory 121 and the processor 122 can communicate with each other through an internal interface.

[0156] Another embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the network model processing method as described in any of the above embodiments.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0158] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A network model processing method, characterized in that: include: Obtaining a training data set from a database, wherein the training data set includes audio training data in a target scene and text annotation results corresponding to the audio training data; Determine the CTC loss function, the knowledge distillation loss function, and the attention loss function based on a preset recognition model and a model to be trained, wherein the recognition model is a network model trained using preset old scene training data, and the model to be trained is obtained by copying the recognition model; Determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function; Training the model to be trained using the data set to be trained and the target loss function to obtain a trained speech recognition model; The method of determining the CTC loss function, the knowledge distillation loss function, and the attention loss function based on the preset recognition model and the model to be trained includes: Inputting the training data in the training data set into the training model, and calculating the CTC loss function corresponding to the training model so that the output result of the training model is consistent with the text annotation result corresponding to the audio training data; Input the training data in the training data set into the recognition model and the model to be trained respectively, and calculate the knowledge distillation loss function corresponding to the model to be trained so that the output result of the model to be trained is consistent with the output result of the recognition model; Determine at least one influencing factor corresponding to the model to be trained and the recognition model respectively, and calculate the attention loss function corresponding to the model to be trained so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

2. The method according to claim 1, characterized in that The calculating of the knowledge distillation loss function corresponding to the model to be trained includes: According to the KL divergence, the output result of the model to be trained is controlled to be consistent with the output result of the recognition model, and the knowledge distillation loss function corresponding to the model to be trained is obtained.

3. The method according to claim 1, characterized in that The determining of at least one influencing factor corresponding to the model to be trained and the recognition model respectively, and calculating the attention loss function corresponding to the model to be trained so as to make the influencing factors corresponding to the model to be trained and the recognition model consistent, includes: Calculating the gradient information corresponding to the model to be trained and the recognition model respectively; Determine the attention vectors corresponding to the to-be-trained model and the recognition model respectively according to the gradient information; Normalize the attention vectors corresponding to the model to be trained and the recognition model to obtain an attention loss function corresponding to the model to be trained.

4. The method according to any one of claims 1 to 3, characterized in that Determining the target loss function according to the CTC loss function, the knowledge distillation loss function, and the attention loss function includes: Determining parameter information corresponding to the application scenario according to the application scenario of the speech recognition model; The target loss function is determined according to the parameter information and the CTC loss function, the knowledge distillation loss function and the attention loss function.

5. The method according to any one of claims 1 to 3, characterized in that Before training the model to be trained using the dataset to be trained and the target loss function, the method further includes: For each audio training data in the to-be-trained data set, a speech spectrum feature extraction operation is performed on the audio training data to obtain input feature information corresponding to the audio training data.

6. The method according to claim 5, characterized in that The performing speech spectrum feature extraction operation on the audio training data includes: A speech spectrum feature extraction operation is performed on the audio training data through short-time Fourier transform.

7. The method according to any one of claims 1 to 3, characterized in that The step of training the model to be trained using the dataset to be trained and the target loss function includes: Inputting the to-be-trained data in the to-be-trained data set into the to-be-trained model to obtain a loss value corresponding to the target loss function; The parameters of the model to be trained are adjusted according to the loss value until the loss value is less than a preset threshold.

8. A network model processing method, characterized in that: include: Obtaining a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized; According to the speech recognition instruction, the speech to be recognized is input into a preset speech recognition model to obtain text information corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function, and an attention loss function, and a training data set including audio training data in a target scenario and text annotation results corresponding to the audio training data; Sending the text message to the terminal device; The CTC loss function, knowledge distillation loss function, and attention loss function are determined in the following way: Inputting the training data in the training data set into the training model, and calculating the CTC loss function corresponding to the training model so that the output result of the training model is consistent with the text annotation result corresponding to the audio training data; Input the training data in the training data set into the recognition model and the model to be trained respectively, and calculate the knowledge distillation loss function corresponding to the model to be trained so that the output result of the model to be trained is consistent with the output result of the recognition model; Determine at least one influencing factor corresponding to the model to be trained and the recognition model respectively, and calculate the attention loss function corresponding to the model to be trained so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

9. A network model processing device, characterized in that: include: An acquisition module is used to acquire a training data set from a database, wherein the training data set includes audio training data in a target scene and text annotation results corresponding to the audio training data; A calculation module, configured to determine a CTC loss function, a knowledge distillation loss function, and an attention loss function based on a preset recognition model and a model to be trained, wherein the recognition model is a network model trained using preset old scene training data, and the model to be trained is obtained by copying the recognition model; A determination module, configured to determine a target loss function based on the CTC loss function, the knowledge distillation loss function, and the attention loss function; A training module, configured to train the model to be trained using the data set to be trained and the target loss function to obtain a trained speech recognition model; The calculation module includes: A first computing unit is configured to input the to-be-trained data in the to-be-trained data set into the to-be-trained model, and calculate a CTC loss function corresponding to the to-be-trained model, so that an output result output by the to-be-trained model is consistent with a text annotation result corresponding to the audio training data; A second calculation unit is used to input the training data in the training data set into the recognition model and the model to be trained, respectively, and calculate the knowledge distillation loss function corresponding to the model to be trained, so that the output result of the training model is consistent with the output result of the recognition model; The third calculation unit is used to respectively determine at least one influencing factor corresponding to the model to be trained and the recognition model, and calculate the attention loss function corresponding to the model to be trained so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

10. The device according to claim 9, characterized in that The second computing unit is configured to: According to the KL divergence, the output result of the model to be trained is controlled to be consistent with the output result of the recognition model, and the knowledge distillation loss function corresponding to the model to be trained is obtained.

11. The device according to claim 9, characterized in that The third computing unit is configured to: Calculating the gradient information corresponding to the model to be trained and the recognition model respectively; Determine the attention vectors corresponding to the to-be-trained model and the recognition model respectively according to the gradient information; Normalize the attention vectors corresponding to the model to be trained and the recognition model to obtain an attention loss function corresponding to the model to be trained.

12. The device according to any one of claims 9 to 11, characterized in that The determination module includes: a determining unit, configured to determine parameter information corresponding to an application scenario according to the application scenario of the speech recognition model; An adjustment unit is used to determine a target loss function based on the parameter information and the CTC loss function, the knowledge distillation loss function, and the attention loss function.

13. The device according to any one of claims 9 to 11, characterized in that The device further comprises: The feature extraction module is used to perform a speech spectrum feature extraction operation on each audio training data in the to-be-trained data set to obtain input feature information corresponding to the audio training data.

14. The device according to claim 13, characterized in that The feature extraction module is used to: A speech spectrum feature extraction operation is performed on the audio training data through short-time Fourier transform.

15. The device according to any one of claims 9 to 11, characterized in that The training module includes: An input unit, configured to input the to-be-trained data in the to-be-trained data set into the to-be-trained model to obtain a loss value corresponding to the target loss function; A training unit is used to adjust the parameters of the model to be trained according to the loss value until the loss value is less than a preset threshold.

16. A network model processing device, characterized in that: include: An instruction acquisition module is used to acquire a voice recognition instruction sent by a terminal device, wherein the voice recognition instruction includes a voice to be recognized; A recognition module, configured to input the speech to be recognized into a preset speech recognition model according to the speech recognition instruction, and obtain text information corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training a target loss function determined by a CTC loss function, a knowledge distillation loss function, and an attention loss function, and a training data set including audio training data in a target scenario and text annotation results corresponding to the audio training data; A sending module, configured to send the text message to the terminal device; The CTC loss function, knowledge distillation loss function, and attention loss function are determined in the following way: Inputting the training data in the training data set into the training model, and calculating the CTC loss function corresponding to the training model so that the output result of the training model is consistent with the text annotation result corresponding to the audio training data; Input the training data in the training data set into the recognition model and the model to be trained respectively, and calculate the knowledge distillation loss function corresponding to the model to be trained so that the output result of the model to be trained is consistent with the output result of the recognition model; Determine at least one influencing factor corresponding to the model to be trained and the recognition model respectively, and calculate the attention loss function corresponding to the model to be trained so that the influencing factors corresponding to the model to be trained and the recognition model are consistent.

17. A network model processing device, characterized in that: include: Memory, processor; Memory; a memory for storing instructions executable by the processor; The processor is used to call the program instructions in the memory to execute the network model processing method as described in any one of claims 1-7 or 8.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the network model processing method according to any one of claims 1-7 or 8.

Images