Model training method, speech processing method, and corresponding apparatuses

Abstract

Disclosed in embodiments of the present application are a model training method, a speech processing method, and corresponding apparatuses. In the present application, after a first speech processing model trained by using first training data is acquired, on the basis of the first speech processing model, only second training data needs to be used to connect to an adaptation network for further training. Compared to the manner of reusing the second training data and the first training data to retrain the first speech processing model, the training efficiency is improved, and computing resources occupied by training and training costs are reduced. In addition, the present application uses an "orthogonal gradient descent" manner to update parameters of the adaptation network, that is, the parameter space of the adaptation network is orthogonal to the updated parameter space used in first training, thereby maintaining the performance of the model on historical training data, and reducing a forgetting phenomenon.

Description

Model training method, speech processing method and corresponding device

[0001] The present application claims priority to the Chinese patent application No. 202410875834.0, filed on July 1, 2024, and entitled "Model training method, speech processing method and corresponding device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a model training method, a speech processing method and a corresponding device. BACKGROUND

[0003] A large number of pre-trained basic models have appeared in the field of speech, and these models have shown good results in speech processing tasks such as speech recognition, translation, etc. However, the pre-trained speech processing model will face the problem of optimization, which currently mainly includes the following two ways:

[0004] The first way is to retrain the speech processing model with new training data and original training data, which requires a huge amount of calculation, low training efficiency and high training cost.

[0005] The second way is to fine-tune the speech processing model using new training data, for example, fine-tuning the speech processing model for training data corresponding to a new language, so that the speech processing model can also perform speech processing for the new language. This way can significantly improve the training efficiency and reduce the training cost, but the traditional fine-tuning method will cause a serious forgetting phenomenon, for example, after fine-tuning the speech processing model for training data corresponding to a new language, the performance of the speech processing model for the language used in the previous training will decrease significantly. SUMMARY

[0006] Therefore, the present application provides a model training method, a speech processing method and a corresponding device to reduce the forgetting phenomenon of the fine-tuned speech processing model.

[0007] The present application provides the following solutions:

[0008] In a first aspect, a model training method is provided, comprising:

[0009] obtaining a speech processing model to be optimized, wherein the speech processing model is obtained by pre-training a first training data comprising a plurality of first training samples, and the first training sample comprises a speech sample and its corresponding labeled text;

[0010] obtaining a second training data comprising a plurality of second training samples, wherein the second training sample comprises a speech sample and its corresponding labeled text;

[0011] The adaptation network is set for the speech processing model, and the adaptation network is used to correct a parameter matrix of the speech processing model; the speech processing model with the set adaptation network is second trained by using the second training data, to obtain an optimized speech processing model, wherein a speech sample of a second training sample is input into the speech processing model with the set adaptation network, and a text result output by the speech processing model with the set adaptation network is acquired; a loss function is determined according to a difference between the text result corresponding to the input speech sample and a labeled text, and the parameters of the adaptation network are updated by using the gradient descent method according to the loss function, wherein a parameter space of the adaptation network is orthogonal to an updated parameter space used in the first training.

[0012] According to an implementable manner in the embodiments of the present application, updating the parameters of the adaptation network by using the gradient descent method according to the loss function includes:

[0013] performing low-rank decomposition on the parameter matrix of the adaptation network to obtain two low-rank matrices;

[0014] updating the two low-rank matrices by using the gradient descent method according to the loss function.

[0015] According to an implementable manner in the embodiments of the present application, updating the parameters of the adaptation network by using the gradient descent method according to the loss function includes:

[0016] performing singular value decomposition on the parameter matrix of the adaptation network to obtain a left singular vector, a diagonal matrix and a right singular vector;

[0017] updating the left singular vector, the diagonal matrix and the right singular vector by using the gradient descent method according to the loss function.

[0018] According to an implementable manner in the embodiments of the present application, updating the diagonal matrix by using the gradient descent method according to the loss function includes:

[0019] gradient calculation is performed according to the loss function;

[0020] the importance of each parameter in the parameter matrix corresponding to the adaptation network is determined by using the calculated gradient;

[0021] the diagonal matrix is updated according to the importance of each parameter.

[0022] According to an implementable manner in the embodiments of the present application, the importance of the parameter is determined according to the gradient of the loss function at the parameter and the value of the parameter;

[0023] The updating the diagonal matrix according to the importance of each parameter comprises: if the importance of the parameter meets a preset requirement, determining the singular value corresponding to the parameter in the diagonal matrix obtained in the current iteration by using the gradient of the loss function at the parameter and the singular value corresponding to the parameter in the diagonal matrix obtained in the last iteration; otherwise, determining the singular value corresponding to the parameter in the diagonal matrix obtained in the current iteration as 0.

[0024] According to an implementable manner in embodiments of the present application, the parameters of the speech processing model are kept unchanged in the second training, and only the parameters of the adaptation network are updated.

[0025] The number of parameters of the adaptation network is less than the number of parameters of the feature extraction network.

[0026] According to an implementable manner in embodiments of the present application, the speech processing model is a speech recognition model, the labeled text is a recognition result corresponding to a speech sample, and the speech sample in the first training sample and the speech sample in the second training sample correspond to different languages.

[0027] The speech processing model is a translation model, the labeled text is a translation result corresponding to a speech sample, and the speech sample in the first training sample and the speech sample in the second training sample correspond to different languages.

[0028] In a second aspect, a speech processing method is provided, and the method comprises:

[0029] obtaining input speech;

[0030] performing speech processing on the input speech by using the optimized speech processing model to obtain text corresponding to the input speech, wherein the optimized speech processing model is trained by using the method in any one of the first aspect above, the input speech and the speech sample in the first training sample are in the same language, or the input speech and the speech sample in the second training sample are in the same language.

[0031] According to an implementable manner in embodiments of the present application, the performing speech processing on the input speech by using the optimized speech processing model to obtain text corresponding to the input speech comprises:

[0032] performing speech recognition on the input speech by using the optimized speech processing model to obtain recognition text corresponding to the input speech, and the recognition text and the input speech correspond to the same language; or

[0033] performing translation on the input speech by using the optimized speech processing model to obtain translation text corresponding to the input speech, and the translation text and the input speech correspond to different languages.

[0034] In a third aspect, a model training apparatus is provided, and the apparatus comprises:

[0035] a model obtaining unit configured to obtain a speech processing model to be optimized, the speech processing model being obtained by pre-training using first training data comprising a plurality of first training samples, the first training samples comprising speech samples and corresponding annotated texts;

[0036] a sample obtaining unit configured to obtain second training data comprising a plurality of second training samples, the second training samples comprising speech samples and corresponding annotated texts;

[0037] a model optimizing unit configured to set an adaptation network for the speech processing model, the adaptation network being used to correct a parameter matrix of the speech processing model, and perform second training on the speech processing model with the set adaptation network using the second training data to obtain an optimized speech processing model, wherein the speech sample of the second training sample is input into the speech processing model with the set adaptation network, a text result output by the speech processing model with the set adaptation network is obtained, a difference between the corresponding text result of the input speech sample and the annotated text is used to determine a loss function, and the parameters of the adaptation network are updated using the gradient descent method according to the loss function, wherein the parameter space of the adaptation network is orthogonal to the updated parameter space used in the first training.

[0038] In a fourth aspect, a speech processing apparatus is provided, and the apparatus comprises:

[0039] a speech obtaining unit configured to obtain input speech;

[0040] a speech processing unit configured to perform speech processing on the input speech using the optimized speech processing model to obtain a text corresponding to the input speech, wherein the optimized speech processing model is obtained by training using the apparatus of the third aspect, the input speech and the speech sample in the first training sample are in the same language, or the input speech and the speech sample in the second training sample are in the same language.

[0041] In a fifth aspect, a computer readable storage medium is provided, and the medium stores a computer program, which, when executed by a processor, implements the steps of the method of any one of the first aspect and the second aspect.

[0042] In a sixth aspect, an electronic device is provided, and the device comprises:

[0043] one or more processors; and

[0044] memory associated with the one or more processors, the memory for storing program instructions that, when read and executed by the one or more processors, perform the steps of the method of any one of the first aspect and the second aspect.

[0045] In a seventh aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method of any one of the first aspect and the second aspect.

[0046] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0047] 1) On the one hand, after obtaining the first speech processing model trained by the first training data, the present application only needs to further train the adaptive network based on the first speech processing model by using the second training data, that is, in the form of fine-tuning. Compared with the way of retraining the first speech processing model by using the second training data and the first training data, the training efficiency is improved, and the computing resources and training cost occupied by the training are reduced. On the other hand, the present application adopts a "orthogonal gradient descent" method to update the parameters of the adaptive network, that is, the parameter space of the adaptive network is orthogonal to the update parameter space used in the first training, so as to maintain the performance of the model on the first training data and reduce the forgetting phenomenon.

[0048] 2) The present application performs singular value decomposition on the parameter matrix of the adaptive network, and updates the left singular vector, the diagonal matrix and the right singular vector obtained by the singular value decomposition by using the loss function, so as to reduce the amount of parameters that need to be calculated in the training process, further save the computing resources occupied by the training, and improve the training efficiency.

[0049] 3) When updating the diagonal matrix, the present application considers the importance of each parameter in the parameter matrix, which is determined by the gradient of the loss function. That is, the importance of the parameter reflects the influence on the gradient of the loss function. Based on the importance of the parameter, the diagonal matrix is updated. In essence, this way is to adaptively allocate the amount of training parameters by controlling the rank of the parameter matrix, which speeds up the model convergence speed and improves the model training effect.

[0050] 4) When fine-tuning, the present application only updates the parameters of the adaptive network in the whole training process, that is, the parameters of the first speech processing model are frozen, and the amount of parameters of the adaptive network is less than that of the first speech processing model. This way needs to update a small amount of parameters, saves computing resources, and improves the training efficiency.

[0051] 5) The application can be widely applied to voice recognition, translation and other scenarios, so that the voice recognition model and the translation model can guarantee the performance in multiple languages, and this method is particularly suitable for the optimization learning of small languages and guarantees the support of the model for small languages.

[0052] Of course, implementing any product of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0054] Fig. 1 is a system architecture diagram to which the embodiments of the present application are applicable;

[0055] Fig. 2 is a flowchart of the model training method provided by the embodiments of the present application;

[0056] Fig. 3 is a schematic diagram of the principle of using LoRA to fine-tune the model provided by the embodiments of the present application;

[0057] Fig. 4 is a schematic diagram of the principle of using LoRA combined with the orthogonal gradient update method to fine-tune the model provided by the embodiments of the present application;

[0058] Fig. 5 is a schematic diagram of the principle of using AdaLoRA combined with the orthogonal gradient update method to fine-tune the model provided by the embodiments of the present application;

[0059] Fig. 6 is a schematic diagram of the multi-stage fine-tuning of the voice processing model provided by the embodiments of the present application;

[0060] Fig. 7 is a schematic block diagram of the model training device provided by the embodiments of the present application;

[0061] Fig. 8 is a schematic block diagram of the model training device provided by the embodiments of the present application;

[0062] Fig. 9 is a schematic block diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0064] The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0065] It should be understood that the term "and / or" as used herein merely describes associated objects, which can exist in three relationships, for example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0066] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0067] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, first, the system architecture based on the present application is described. Figure 1 shows an exemplary system architecture to which the embodiments of the present application can be applied, as shown in Figure 1, the system architecture can include: a user terminal, and a model training device and a speech processing device located at the server side.

[0068] Among them, the model training device of the server side adopts the method provided by the embodiments of the present application to train the model in the offline stage, and obtain the speech processing model. The speech processing model can include multiple stages of optimization tasks, and in each stage of optimization task, the method provided by the embodiments of the present application can be used to further train the speech processing model obtained in the previous optimization task, and obtain the speech processing model optimized in the current stage.

[0069] The user terminal can interact with the speech processing device of the server side through the network.

[0070] As one of the implementable ways, the user can input speech through the user terminal, and the user terminal sends the input speech to the speech processing device of the server side through the network. The speech processing device of the server side processes the input speech by using the optimized speech processing model to obtain the text corresponding to the input speech. The speech processing model can include a speech recognition model, a translation model, etc., which will be described in subsequent embodiments.

[0071] The user terminal can include, but is not limited to, a smart mobile terminal, a smart home device, a wearable device, a smart medical device, a PC (Personal Computer), and the like. The smart mobile device can include a mobile phone, a tablet computer, a notebook computer, a PDA (Personal Digital Assistant), an Internet car, and the like. The smart home device can include a smart home appliance device, such as a smart television, a smart sound box, and the like. The wearable device can include a smart watch, smart glasses, a smart bracelet, a VR (Virtual Reality) device, an AR (Augmented Reality) device, a mixed reality device (i.e., a device capable of supporting virtual reality and augmented reality), and the like.

[0072] The model training apparatus and the speech processing apparatus can be respectively provided as independent servers, or can be provided in a same server or a server group, or can be provided in independent or same cloud servers. The cloud server, also referred to as a cloud computing server or a cloud host, is a host product in a cloud computing service system, and is used to solve the defects of large management difficulty and weak service scalability in traditional physical hosts and virtual private server (VPS) services. The model training apparatus and the speech processing apparatus can also be provided in a computer terminal with strong computing capability.

[0073] It should be noted that, in addition to online speech processing, the speech processing apparatus can also perform speech processing in an offline manner.

[0074] It should be understood that the number of user terminals, model training apparatuses, and speech processing apparatuses in FIG. 1 is merely illustrative. According to the needs of implementation, there can be any number of user terminals, model training apparatuses, and speech processing apparatuses.

[0075] FIG. 2 is a flowchart of a model training method provided by an embodiment of the present application. The method can be performed by the model training apparatus in the system shown in FIG. 1. As shown in FIG. 2, the method can include the following steps:

[0076] Step 201: obtaining a speech processing model to be optimized, the speech processing model being obtained by pre-training a first training data including a plurality of first training samples, the first training sample including a speech sample and corresponding labeled text.

[0077] Step 203: obtaining second training data including a plurality of second training samples, the second training sample including a speech sample and corresponding labeled text.

[0078] Step 205: setting an adaptation network for the speech processing model, the adaptation network being used to correct the parameter matrix of the speech processing model; performing second training on the speech processing model with the set adaptation network by using the second training data, to obtain an optimized speech processing model, wherein the speech sample of the second training sample is input into the speech processing model with the set adaptation network, and a text result output by the speech processing model with the set adaptation network is obtained; a loss function is determined according to the difference between the text result corresponding to the input speech sample and the labeled text, and the parameters of the adaptation network are updated by using the gradient descent method according to the loss function, wherein the parameter space of the adaptation network is orthogonal to the update parameter space used in the first training.

[0079] As can be seen from the above process, on the one hand, after obtaining the first speech processing model trained by using the first training data, the present application only needs to set an adaptation network on the basis of the first speech processing model by using the second training data for further training, that is, in a fine-tuning manner, which improves the training efficiency and reduces the calculation resources and training cost occupied by the training, as compared with the way of retraining the first speech processing model by using the second training data and the first training data. On the other hand, the present application adopts an orthogonal gradient descent method to update the parameters of the adaptation network, that is, the parameter space of the adaptation network is orthogonal to the update parameter space used in the first training, so as to maintain the performance of the model on the first training data and reduce the forgetting phenomenon.

[0080] The steps in the above process and the effects that can be further produced will be described in detail below in combination with embodiments. It should be noted that the "first", "second", and the like involved in the present disclosure do not have the limitations of size, order, and quantity, and are only used to distinguish in name. For example, "first training sample" and "second training sample" are used to distinguish two training samples in name. For example, "first training" and "second training" are used to distinguish two training stages in name, and the like.

[0081] First, the above step 201, that is, "obtaining a speech processing model to be optimized", and step 203, that is, "obtaining second training data including a plurality of second training samples, the second training sample including a speech sample and its corresponding labeled text", will be described in detail in combination with embodiments.

[0082] In the embodiments of the present application, the obtained speech processing model to be optimized refers to a speech processing model that has been trained for at least one stage.

[0083] Suppose that the initial model M0 is trained on the basis of the training set D0, and can be further trained based on M0, suppose For a data set composed of N groups of training data, each group of training data includes a plurality of training samples. Each group of training data is used for model fine-tuning training of one stage. Then, M0 is fine-tuned by D1 to obtain an optimized model M1. M1 is fine-tuned by D1 to obtain an optimized model M2. M2 is fine-tuned by D2 to obtain an optimized model M3, and so on.

[0084] Then in this step, the speech processing model to be optimized can be the model M1, M2, M3, and the like that has been trained.

[0085] If the speech processing model to be optimized obtained is M1, the first training data is D1 and the second training data is D2; if the speech processing model to be optimized obtained is M2, the first training data is D2 and the second training data is D3, and so on.

[0086] That is, in each stage of training task after M1 is obtained, the speech processing model can be optimized by using the training method provided in the embodiments of the present application. The process of optimizing the model M1 based on the initial model M0 can use the LoRA (Low-Rank Adaptation) method, which is described below.

[0087] For the initial model M0, the parameter matrix is a pre-trained parameter matrix, and an adaptation network can be set for the initial model M0, which is used to correct the parameter matrix of M0. As shown in FIG. 3, in the training process, the parameter matrix W i and b i of M0 is frozen (indicated by a snowflake icon in the figure), and only the parameters A i and B i of the adaptation network are updated (indicated by a fire icon in the figure). Wherein, the adaptation network can be set for one or more neural network layers in the initial model, and the neural network layer can be a linear layer.

[0088] The i-th forward process of the initial model M0 can be represented as: f i (x)=xW i T +b i (1)

[0089] Wherein, x represents the input of the neural network layer, represents the weight matrix and the bias matrix, i.e., the parameter matrix belonging to M0, and d1 and d2 are dimensions.

[0090] The parameter matrix corresponding to the adaptation network can be expressed as ΔW i The adaptation network corrects the parameter matrix of M0,

[0091] In the i-th forward process, it can be expressed as: f i (x)=x(W i +ΔW i ) T +b i ΔW i (2)

[0092] The parameter matrix ΔW i of the adaptation network is decomposed into low rank, which can be expressed as: ΔW i =A i B i (3)

[0093] wherein, rank r << min{d1, d2}

[0094] The above A i and B i can be randomly initialized or initialized to a preset value. For example, B i can be initialized to 0, and A i is randomly initialized according to a normal distribution. In the training process, a supervised manner is adopted, and the training samples include speech samples x and their corresponding labeled texts. After inputting each sample x into M0, the neural network layer with the adaptation network performs forward propagation according to formula (2), and finally predicts the text. The training target is to minimize the difference between the predicted text and the labeled text corresponding to the sample x. A loss function can be constructed according to the above training target, and in each iteration, the value of the loss function is used to update the parameters of the adaptation network, i.e. A i and B i , keeping the parameters of M0 unchanged, until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0095] After training, the parameters of the adaptation network, i.e. A i and B i , are used to correct the parameter matrix W i and b i of M0, to obtain the updated W i and b i , i.e. the optimized model M1.

[0096] In addition to using the LoRA method, other fine-tuning methods can also be used to fine-tune M0 to obtain the optimized model M1.

[0097] The following describes the step 205 in detail in combination with an embodiment: "Setting an adaptation network for the speech processing model, the adaptation network being used to correct the parameter matrix of the speech processing model; using the second training data to perform second training on the speech processing model with the set adaptation network to obtain an optimized speech processing model."

[0098] The processing of optimizing the model M1 and any subsequent training stage can use the method provided in this step. Assuming that the speech processing model to be optimized is represented as M j , the second training data used is represented as D j . Similar to the LoRA method described above, an adaptation network can be set for the model M j , the adaptation network being used to correct the parameter matrix of M j , and the parameter matrix of M j is frozen during training, and only the parameters of the adaptation network are updated. Among them, an adaptation network can be set for one or more neural network layers in the model M j , and the neural network layers can be linear layers.

[0099] In order to avoid reducing the forgetting of past training tasks by the model during the process of optimizing M j using D j , in this application, a new training task can be learned in a direction orthogonal to the update parameter space used in the past training task. Assuming that the speech processing model to be optimized is represented as M j , and the further optimized M t is obtained, the parameter matrix of the adaptation network obtained during the training of M j is represented as A j and B j , and the further optimization of M j to obtain M t is performed in the training task t, and the parameters of the adaptation network set for M j are represented as and r<<min{d1,d2}. As shown in FIG. 4, the parameters of M j obtained in the previous training process are kept unchanged, including the pre-trained parameter matrix and A j and B j , and only A t and B t are updated.

[0100] The following orthogonality needs to be ensured during training:

[0101] Ensuring the above orthogonality is actually in the direction of B j and B t The above formula is also similar to the above formula.

[0102] Then in the training process, A t and B t need to be updated in the direction of the above orthogonality. This can be ensured by a second sub-function in a loss function. For example, the second sub-function can be expressed as:

[0103] Where O j,t [k,q] represents the element in the qth row and the kth column of O j,t It can be seen that A j and B j are fixed throughout the process, and A t and B t are optimized along the space direction orthogonal to them, thereby reducing forgetting of past knowledge.

[0104] In this case, the loss function L can be expressed as: L=L whisper +λL orth (A j ,A t ) (6)

[0105] Where the first sub-function L whisper is used to minimize the difference between the text result corresponding to the speech sample M t input into the training process and the labeled text. λ is a weight coefficient, which can be set in advance, for example, it can be set to an empirical value or an experimental value, etc.

[0106] In the embodiments of the present application, a more preferred implementation is provided, which can adopt a more optimal way compared to the LoRA method, namely AdaLoRA (Adaptive Low-Rank Adaptation). Although the LoRA method reduces the number of parameters that need to be updated by low-rank decomposition, it assigns a uniform rank to the parameters of the adaptation network. AdaLoRA, however, introduces two key improvements:

[0107] First, singular value decomposition is performed on the parameter matrix of the adaptation network to obtain left singular vectors, a diagonal matrix, and right singular vectors. That is: ΔW j =A t Λ t B t ;A t T At = B t B t T = I (7)

[0108] In the above formula (7), A t , Λ t and B t are left singular vector, diagonal matrix and right singular vector respectively, and I is unit matrix. In this case, A t and B t are orthogonal. Λ t ∈ R r×r contains r singular values, where r << min{d1, d2}. The above A t , Λ t and B t can be randomly initialized or initialized as a preset value at initialization. For example, Λ t can be initialized as 0, and A t and B t are randomly initialized according to normal distribution.

[0109] The training schematic diagram in this case is shown in FIG. 5. In the training construction, only A t , Λ t and B t are updated, and the parameters of the model M j are kept unchanged, including the parameter matrices A j , Λ j and B j of the adaptation network used in the last training task.

[0110] Secondly, the rank of the parameters of the adaptation network is dynamically assigned in the updating process, which is essentially realized by updating the diagonal matrix.

[0111] Specifically, the importance of each parameter in the parameter matrix corresponding to the adaptation network can be determined according to the gradient calculated by the loss function, and the diagonal matrix Λ t is updated according to the importance of each parameter.

[0112] As one of the implementable ways, the importance of the parameter can be determined according to the gradient of the loss function at the parameter and the value of the parameter, for example, the product of the gradient of the loss function at the parameter and the value of the parameter can be used to determine the importance.

[0113] When updating the diagonal matrix Λ t according to the importance of each parameter, the diagonal matrix Λ tThe pruning manner is performed. As one of the implementable manners, if the importance of the parameter meets preset requirements, for example, ranks in the first few positions, or is greater than or equal to a preset importance threshold, etc., the singular value of the parameter in the diagonal matrix obtained in the current iteration is determined by using the gradient of the loss function at the parameter and the singular value of the parameter in the diagonal matrix obtained in the last iteration, for example, first, the gradient of the loss function at the parameter is multiplied by the learning rate to obtain a first value, then the singular value of the parameter in the diagonal matrix obtained in the last iteration is subtracted by the first value to obtain a second value, and the second value is taken as the singular value of the parameter in the diagonal matrix for updating. Otherwise, the singular value of the parameter in the diagonal matrix obtained in the current iteration is determined as 0.

[0114] The above process is essentially to adaptively allocate the amount of trained parameters by controlling the rank of the parameter matrix, which accelerates the model convergence speed and improves the effect of model training.

[0115] In the above training process, the parameter amount of the adaptation network ΔW j is less than the parameter amount of the speech processing model M j to be optimized, and only the parameters of the adaptation network are updated in the entire training process, that is, the parameters of the first speech processing model are frozen, which greatly reduces the parameter amount to be updated, saves computing resources, and improves the training efficiency.

[0116] In this implementation manner, the loss function L can be represented as: L=L whisper +λ1L orth (A j ,A t )+λ2L adalora (A t ,B t ) (8)

[0117] Wherein, L adalora (A t ,B t ) is a third sub-function, which can be represented as:

[0118] Wherein, ||| 2 represents the 2-norm, and λ1 and λ2 are weight coefficients, which can adopt empirical values or experimental values, etc.

[0119] After the training is completed, the parameters of the adaptation network obtained by training can be merged into M j to obtain M t , and M t can be further optimized by using the above manner subsequently.

[0120] As shown in FIG. 6, if the speech processing model provided with the adaptation network 1 (denoted as AdaLoRA1 in the figure) is further trained by using the training data a, an optimized model 1 is obtained. Then the speech processing model provided with the adaptation network 2 (denoted as AdaLoRA2 in the figure) can be further trained by using the training data b, and the orthogonal gradient descent is used in the training process to ensure that the parameter spaces of the adaptation network 2 and the adaptation network 1 are orthogonal, and an optimized model 2 is obtained. Further, the speech processing model provided with the adaptation network 3 (denoted as AdaLoRA3 in the figure) can be further trained by using the training data c, and the orthogonal gradient descent is used in the training process to ensure that the parameter spaces of the adaptation network 3 and the adaptation network 2 are orthogonal, and an optimized model 3 is obtained. In this way, the optimized model 4 is obtained.

[0121] After the optimized speech processing model is obtained, the speech processing model can be used for speech processing. For example, after an input speech is obtained, the input speech is processed by using the optimized speech processing model to obtain the text corresponding to the input speech. However, the input speech is at least one of the languages of the speech samples used in the historical training tasks for training the speech processing model, for example, the same language as the speech sample in the first training sample or the speech sample in the second training sample.

[0122] The above method provided by the embodiments of the present application can be applied to various application scenarios, including but not limited to:

[0123] 1) Speech recognition scenario

[0124] In this scenario, the speech recognition model can be optimized in the above manner to adapt to more languages. For example, the speech recognition model is pre-trained in the first language for which the training data is most easily obtained or the data is most abundant. If the speech recognition model is required to support other languages, the above method provided by the embodiments of the present application can be used to construct training tasks for other languages respectively, and the speech recognition model that has been trained is further optimized.

[0125] For example, whisper is a speech recognition model that has good recognition effect for English, Chinese and the like, but has poor recognition ability on some small languages such as Tibetan. The speech of Tibetan and its corresponding recognized text can be obtained as training samples to construct training data, and the training data is used as the second training data in the above process, so as to fine-tune the whisper model using the second training data. In the fine-tuning process, the parameters of the adaptive network are updated in the direction orthogonal to the update parameter space used in the historical training by using the gradient descent method, and after the training is completed, the parameters of the adaptive network are used to correct the parameters of the whisper model, for example, the parameters of the adaptive network are merged into the whisper model, so as to obtain an optimized whisper model. The optimized whisper model can also have good recognition effect on Tibetan, and reduce the forgetting of other supported languages.

[0126] The optimized whisper model can perform speech recognition on the input speech to obtain the recognized text corresponding to the input speech. In this case, the recognized text and the input speech usually correspond to the same language. For example, input English speech into the whisper model, and the whisper model outputs English text. Input Chinese speech into the whisper model, and the whisper model outputs Chinese text. Input Tibetan speech into the whisper model, and the whisper model outputs Tibetan text, and so on.

[0127] 2) Translation scenario

[0128] In this scenario, the above method can be used to optimize the translation model to adapt to more languages. For example, the translation model is pre-trained in the first language for which training data is most easily obtained or for which data is most abundant. If you want the translation model to support other languages, you can use the above method provided by the embodiments of the present application to construct training tasks for other languages respectively, and further optimize the translation model that has been trained.

[0129] For example, the translation model has good recognition effect on English, Chinese and the like, but the translation capability on some small languages such as Albanian and Persian is poor. The speech of Persian and its corresponding translation result (in text form) can be obtained as training samples to construct training data, and the training data is used as the second training data in the above process, so as to fine-tune the translation model by using the second training data. In the fine-tuning process, the parameters of the adaptive network are updated in the direction orthogonal to the update parameter space used in the historical training by using the gradient descent method. After the training is completed, the parameters of the adaptive network are used to correct the parameters of the translation model, for example, the parameters of the adaptive network are merged into the translation model, so as to obtain an optimized translation model. The optimized translation model can also have good translation effect on Persian, and reduce the forgetting of other supported languages.

[0130] The optimized translation model can translate the input speech to obtain the translation result corresponding to the input speech. In this case, the translation result and the input speech usually correspond to different languages. For example, Chinese speech is input to the translation model, and the translation model outputs English text. English speech is input to the translation model, and the translation model outputs Chinese text. Persian speech is input to the translation model, and the translation model outputs Chinese text, and the like.

[0131] In order to verify the effect of the method provided in the embodiments of the present application, first, based on the initial whisper v2 model, 20000 times of iterative training are performed using a Chinese data set, and then at least two small language data sets are respectively fine-tuned for 5000 times of iteration based on the method provided in the embodiments of the present application. Aishell-1 (a kind of speech data set) is used to test the performance decline of Chinese. In order to reflect the effect of the method provided in the embodiments of the present application, full-ft (representing full-ft), LoRA method and the method provided in the embodiments of the present application (represented as O-AdaLoRA) are used, and the experimental results are shown in Table 1:

[0132] Table 1

[0133] It can be seen that the parameter amount trained by the manner provided in the embodiment of the present application is the least, only accounting for 1.36% of the total parameter amount, and the degree of performance decline of Chinese is also the lowest. That is, the forgetting of the previous learning of Chinese knowledge can be effectively reduced, and the parameter amount of training is reduced, thereby improving the efficiency and saving the computing resources and cost. Further verification of the WER (World Error Rate, word error rate) of Tibetan language shows that the WER using the full amount fine-tuning manner is about 60%, the WER using the LoRA manner is about 53%, and the WER using the manner provided in the embodiment of the present application is reduced to about 50%. It can be seen that the recognition accuracy of small languages is also improved.

[0134] The above describes the present application based on the embodiments, but the present application is not limited to only these embodiments. In the above detailed description of the present application, some specific details are described in detail. The present application can also be fully understood without the description of these details by those skilled in the art. In order to avoid confusion of the essence of the present application, the well-known methods, processes, flows, elements and circuits are not described in detail. Moreover, the above actions or steps can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or can be advantageous.

[0135] According to another aspect, embodiments of the present application provide a model training apparatus. FIG. 7 is a schematic block diagram of a model training apparatus provided by an embodiment of the present application. As shown in FIG. 7, the apparatus 700 includes a model obtaining unit 701, a sample obtaining unit 702 and a model optimizing unit 703, wherein the main functions of each component unit are as follows:

[0136] The model obtaining unit 701 is configured to obtain a speech processing model to be optimized, wherein the speech processing model is obtained by pre-training using first training data including a plurality of first training samples, and the first training sample includes a speech sample and its corresponding labeled text.

[0137] The sample obtaining unit 702 is configured to obtain second training data including a plurality of second training samples, and the second training sample includes a speech sample and its corresponding labeled text.

[0138] The model optimization unit 703 is configured to set an adaptation network for the speech processing model, the adaptation network being used to correct a parameter matrix of the speech processing model; perform second training on the speech processing model with the set adaptation network by using second training data, to obtain an optimized speech processing model, wherein a speech sample of a second training sample is input into the speech processing model with the set adaptation network, and a text result output by the speech processing model with the set adaptation network is obtained; determine a loss function according to a difference between the text result corresponding to the input speech sample and a labeled text, and update parameters of the adaptation network by using the gradient descent method and the loss function, wherein a parameter space of the adaptation network is orthogonal to an updated parameter space used in the first training.

[0139] As one of the implementable manners, when the parameters of the adaptation network are updated by using the gradient descent method and the loss function, the model optimization unit 703 can perform low-rank decomposition on the parameter matrix of the adaptation network, to obtain two low-rank matrices; and the two low-rank matrices are updated by using the gradient descent method and the loss function.

[0140] As another implementable manner, when the parameters of the adaptation network are updated by using the gradient descent method and the loss function, the model optimization unit 703 can perform singular value decomposition on the parameter matrix of the adaptation network, to obtain a left singular vector, a diagonal matrix, and a right singular vector; and the left singular vector, the diagonal matrix, and the right singular vector are updated by using the gradient descent method and the loss function.

[0141] When the diagonal matrix is updated by using the gradient descent method and the loss function, the model optimization unit 703 can perform gradient calculation according to the loss function; determine the importance of each parameter in the parameter matrix corresponding to the adaptation network by using the calculated gradient; and update the diagonal matrix according to the importance of each parameter.

[0142] As one of the implementable manners, the importance of the parameter can be determined according to the gradient of the loss function at the parameter and the value of the parameter.

[0143] Correspondingly, when the diagonal matrix is updated according to the importance of each parameter, if the importance of the parameter meets a preset requirement, the singular value of the parameter in the diagonal matrix obtained in the current iteration is determined by using the gradient of the loss function at the parameter and the singular value of the parameter in the diagonal matrix obtained in the last iteration; otherwise, the singular value of the parameter in the diagonal matrix obtained in the current iteration is determined to be 0.

[0144] Preferably, during the second training, the parameters of the speech processing model are kept unchanged, and only the parameters of the adaptation network are updated; and the number of parameters of the adaptation network is less than the number of parameters of the feature extraction network.

[0145] FIG. 8 is a schematic block diagram of a model training apparatus according to an embodiment of the present application. As shown in FIG. 8, the apparatus 800 includes a speech obtaining unit 801 and a speech processing unit 802. The main functions of the constituent units are as follows.

[0146] The speech obtaining unit 801 is configured to obtain input speech.

[0147] The speech processing unit 802 is configured to perform speech processing on the input speech by using the optimized speech processing model to obtain text corresponding to the input speech, wherein the optimized speech processing model is trained by using the apparatus shown in FIG. 6, the input speech and the speech samples in the first training samples are in the same language, or the input speech and the speech samples in the second training samples are in the same language.

[0148] As one of the implementable manners, the speech processing unit 802 can perform speech recognition on the input speech by using the optimized speech processing model to obtain recognition text corresponding to the input speech, and the recognition text and the input speech are in the same language.

[0149] As another implementable manner, the speech processing unit 802 can perform translation on the input speech by using the optimized speech processing model to obtain translation text corresponding to the input speech, and the translation text and the input speech are in different languages.

[0150] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the apparatus embodiment, since it is basically similar to the method embodiment, it is described more simply, and the relevant parts can be referred to the part of the method embodiment. The apparatus embodiment described above is only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0151] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0152] In addition, the embodiments of the present application further provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method according to any one of the preceding method embodiments.

[0153] An electronic device comprising:

[0154] one or more processors; and

[0155] a memory associated with the one or more processors, the memory for storing program instructions, the program instructions, when read and executed by the one or more processors, performing the steps of the method according to any one of the preceding method embodiments.

[0156] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of the preceding method embodiments.

[0157] In some embodiments, the electronic device can include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920 can be connected by a communication bus 930.

[0158] The processor 910 can be implemented by a general-purpose CPU, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the present application.

[0159] The memory 920 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, a basic input / output system (BIOS) 922 for controlling the low-level operation of the electronic device 900. In addition, a web browser 923, a data storage management system 924, and a model training apparatus / voice processing apparatus 925, etc. can also be stored. The above-mentioned model training apparatus / voice processing apparatus 925 can be an application program for specifically implementing the above-mentioned steps in the embodiments of the present application. In summary, when the technical solutions provided by the present application are implemented by software or firmware, the relevant program codes are stored in the memory 920 and executed by the processor 910.

[0160] The input / output interface 913 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0161] The network interface 914 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0162] The bus 930 includes a path for transmitting information between various components (such as the processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920) of the device.

[0163] It should be noted that although the above-mentioned device only shows the processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, the memory 920, the bus 930, etc., in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above-mentioned device can also only contain the components necessary for implementing the solutions of the present application, and does not necessarily contain all the components shown in the figure.

[0164] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions of the application can be embodied in the form of a computer program product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the application.

[0165] The technical solutions provided by the application are described in detail above, and the principles and implementation manners of the application are described by using specific examples. The above description of the embodiments is only used to help understand the methods and core ideas of the application. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the ideas of the application. In conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a speech processing model to be optimized, wherein the speech processing model is obtained by pre-training a first training data comprising a plurality of first training samples, and the first training samples comprise speech samples and corresponding annotated texts; obtaining second training data comprising a plurality of second training samples, wherein the second training samples comprise speech samples and corresponding annotated texts; setting an adaptive network for the speech processing model, wherein the adaptive network is used to correct a parameter matrix of the speech processing model; performing second training on the speech processing model with the set adaptive network by using the second training data to obtain an optimized speech processing model, wherein the speech samples of the second training samples are input into the speech processing model with the set adaptive network, and a text result output by the speech processing model with the set adaptive network is obtained; determining a loss function according to a difference between the text result corresponding to the input speech sample and the annotated text, and updating parameters of the adaptive network by using the gradient descent method according to the loss function, wherein a parameter space of the adaptive network is orthogonal to an updated parameter space used in the first training.

2. The method of claim 1, wherein, The updating of the parameters of the adaptive network by using the gradient descent method according to the loss function comprises: performing low-rank decomposition on the parameter matrix of the adaptive network to obtain two low-rank matrices; updating the two low-rank matrices by using the gradient descent method according to the loss function.

3. The method of claim 1, wherein, The updating of the parameters of the adaptive network by using the gradient descent method according to the loss function comprises: performing singular value decomposition on the parameter matrix of the adaptive network to obtain left singular vectors, a diagonal matrix and right singular vectors; updating the left singular vectors, the diagonal matrix and the right singular vectors by using the gradient descent method according to the loss function.

4. The method of claim 3, wherein, The updating of the diagonal matrix by using the gradient descent method according to the loss function comprises: performing gradient calculation according to the loss function; determining importance of each parameter in the parameter matrix corresponding to the adaptive network according to the calculated gradient; updating the diagonal matrix according to the importance of each parameter.

5. The method of claim 4, wherein, The importance of the parameter is determined according to the gradient of the loss function at the parameter and a value of the parameter; The updating of the diagonal matrix according to the importance of each parameter comprises: if the importance of the parameter meets a preset requirement, determining a singular value of the parameter in the diagonal matrix obtained in the current iteration by using the gradient of the loss function at the parameter and a singular value of the parameter in the diagonal matrix obtained in the last iteration; otherwise, determining the singular value of the parameter in the diagonal matrix obtained in the current iteration as 0.

6. The method according to any one of claims 1 to 5, characterized in that, The parameters of the speech processing model are kept unchanged during the second training, and only the parameters of the adaptive network are updated; The amount of parameters of the adaptive network is less than the amount of parameters of the feature extraction network.

7. The method according to any one of claims 1 to 5, characterized in that, The speech processing model is a speech recognition model, the annotated texts are recognition results corresponding to the speech samples, the speech samples in the first training samples and the speech samples in the second training samples correspond to different languages; or The speech processing model is a translation model, the labeled text is a translation result corresponding to the speech sample, and the speech sample in the first training sample and the speech sample in the second training sample correspond to different languages.

8. A voice processing method characterized by, The method comprises: obtaining input speech; performing speech processing on the input speech by using the optimized speech processing model to obtain text corresponding to the input speech, wherein the optimized speech processing model is obtained by training by using the method in any one of claims 1 to 6, the input speech and the speech sample in the first training sample are in the same language, or the input speech and the speech sample in the second training sample are in the same language.

9. The method of claim 8, wherein, The speech processing on the input speech by using the optimized speech processing model to obtain the text corresponding to the input speech comprises: performing speech recognition on the input speech by using the optimized speech processing model to obtain recognized text corresponding to the input speech, the recognized text and the input speech correspond to the same language; or performing translation on the input speech by using the optimized speech processing model to obtain translated text corresponding to the input speech, the translated text and the input speech correspond to different languages.

10. A model training apparatus, comprising: The device comprises: a model obtaining unit configured to obtain a speech processing model to be optimized, the speech processing model being obtained by first training by using first training data comprising a plurality of first training samples, the first training sample comprising a speech sample and labeled text corresponding to the speech sample; a sample obtaining unit configured to obtain second training data comprising a plurality of second training samples, the second training sample comprising a speech sample and labeled text corresponding to the speech sample; a model optimization unit configured to set an adaptation network for the speech processing model, the adaptation network being used to correct a parameter matrix of the speech processing model, and to perform second training on the speech processing model with the set adaptation network by using the second training data to obtain an optimized speech processing model, wherein the speech sample of the second training sample is input into the speech processing model with the set adaptation network, a text result output by the speech processing model with the set adaptation network is obtained, a loss function is determined by using a difference between the text result corresponding to the input speech sample and the labeled text, and a gradient descent method is used to update parameters of the adaptation network by using the loss function, wherein a parameter space of the adaptation network is orthogonal to an updated parameter space used in the first training.

11. A speech processing device, characterized by The device comprises: a speech obtaining unit configured to obtain input speech; a speech processing unit configured to perform speech processing on the input speech by using an optimized speech processing model to obtain text corresponding to the input speech, wherein the optimized speech processing model is obtained by training by using the device in claim 10, the input speech and the speech sample in the first training sample are in the same language, or the input speech and the speech sample in the second training sample are in the same language.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps of the method in any one of claims 1 to 9.

13. An electronic device, comprising: comprises: one or more processors; and a memory associated with the one or more processors and used to store program instructions, which program instructions, when read and executed by the one or more processors, cause the performance of the steps of the method of any one of claims 1 to 9.

14. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method of any one of claims 1 to 9.

Images