Multi-task voice big language model training method and device, equipment and medium
By using a loss function and a meta-learner to update the model in a multi-task speech large language model, the problem of supporting new speech tasks after training is solved, and the model is able to adapt efficiently and maintain performance on new tasks.
Patent Information
- Application Number
- CN202510105797.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-23
- Publication Date
- 2025-12-26
AI Technical Summary
Multi-task speech large language models have difficulty supporting new speech tasks after training.
A multi-task speech large language model is trained using a first loss function and existing multi-task data, and a similarity index is trained using a meta-learner and a second loss function to update the model to support new speech tasks.
A multi-task speech large language model has been implemented that can be trained on both existing and new speech tasks simultaneously, maintaining the performance of existing tasks without degrading and supporting the efficient addition of new tasks.
Smart Images
Figure CN121214918A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a training method and device of a multi-task speech large language model, equipment and medium. BACKGROUND
[0002] A multi-task speech large language model (LLM) is a new intelligent speech technology, which refers to a deep neural network large language model framework that widely covers various speech tasks, which can include speech recognition, speech synthesis, speech enhancement, etc. Unlike traditional small-scale single-task speech models, multi-task speech large language models usually have larger neural network parameter quantities, which can fully utilize a large amount of training data of multiple tasks for learning. The multi-task speech large language model performs joint learning of multiple tasks in the training stage, and can mutually encourage and improve each task by virtue of the inherent correlation between tasks. Moreover, since it can widely cover multiple tasks in the field of intelligent speech, it has the potential to realize each link in human-computer voice interaction by relying on a single model. Therefore, the multi-task speech large language model is widely considered to be the future of intelligent speech technology.
[0003] The multi-task speech large language model is trained by data of multiple speech tasks, but it is still difficult to cover all possible speech tasks in the future, which means that after the multi-task speech large language model is trained with existing task data, there will still be new speech tasks, i.e., the need for speech task expansion. Therefore, how the multi-task speech large language model supports new speech tasks is a problem to be solved. SUMMARY
[0004] At least one embodiment of the present application provides a training method, device, equipment and medium of a multi-task speech large language model, which is used to solve the problem of how to support new speech tasks after the training of the multi-task speech large language model in the prior art ends.
[0005] To solve the above technical problems, the present application is implemented as follows:
[0006] In a first aspect, the embodiments of the present application provide a training method of a multi-task speech large language model, comprising:
[0007] In a case that a first speech task is added to the multi-task speech large model, the multi-task speech large language model is trained according to a first loss function and existing multi-task data, a trained multi-task speech large language model is obtained, and a meta-learner is trained according to a second loss function and existing second speech tasks, and a trained meta-learner is obtained; wherein the first loss function and the second loss function are obtained according to a difference measurement between each two second speech tasks;
[0008] The trained meta-learner is used to obtain a similarity index between the first speech task and each second speech task;
[0009] The trained multi-task speech large language model is updated according to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data.
[0010] Optionally, the method for training the multi-task speech large language model further comprises:
[0011] According to a target element, the properties of each second speech task are determined, and the target element includes at least one of the following: input language category, output language category, audio feature category, whether semantic information of speech is required, whether speaker recognition information of speech is required, whether emotional information of speech is required, whether environmental noise information of speech is required, and whether audio event information of speech is required.
[0012] According to the properties of each second speech task, a difference measurement between each two second speech tasks is obtained.
[0013] Optionally, the method for training the multi-task speech large language model further comprises:
[0014] According to a cross-entropy loss of the multi-task speech large language model on a task target sequence and the difference measurement between each two second speech tasks, the first loss function is obtained.
[0015] Optionally, the method for training the multi-task speech large language model further comprises:
[0016] According to the difference measurement between each two second speech tasks, a similarity index output by the meta-learner is obtained.
[0017] According to the similarity index and a task embedding vector difference value between each two second speech tasks, the second loss function is obtained.
[0018] Optionally, the method for training the multi-task speech large language model, wherein the meta-learner is configured to output a similarity index between two speech tasks according to a difference metric between the two speech tasks.
[0019] Optionally, the method for training the multi-task speech large language model, wherein the updating of the trained multi-task speech large language model according to the similarity index between the first speech task and each of the second speech tasks, the first speech task, and the existing multi-task data comprises:
[0020] According to the similarity index between the first speech task and each of the second speech tasks, a target second speech task is obtained, which has a similarity index less than a similarity index threshold with the first speech task.
[0021] According to the target second speech task, a task embedding vector estimation value corresponding to the first speech task is obtained.
[0022] According to the task embedding vector estimation value, a parameter of a task embedding module in the trained multi-task speech large language model is obtained.
[0023] According to the parameter of the task embedding module in the trained multi-task speech large language model, the first speech task, and the existing multi-task data, the trained multi-task speech large language model is updated.
[0024] In a second aspect, an embodiment of the present application provides a training device for a multi-task speech large language model, comprising:
[0025] The training module is configured to, in a case where a first speech task is added to a multi-task speech large model, train the multi-task speech large language model according to a first loss function and existing multi-task data, obtain a trained multi-task speech large language model, and train a meta-learner according to a second loss function and existing second speech tasks, and obtain a trained meta-learner; wherein the first loss function and the second loss function are both obtained according to a difference metric between each two of the second speech tasks.
[0026] The first obtaining module is configured to use the trained meta-learner to obtain a similarity index between the first speech task and each of the second speech tasks.
[0027] The updating module is configured to update the trained multi-task speech large language model according to the similarity index between the first speech task and each of the second speech tasks, the first speech task, and the existing multi-task data.
[0028] In a third aspect, an embodiment of the present application provides a training device of a multi-task speech large language model, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the processor implements the steps of the training method of the multi-task speech large language model according to the first aspect.
[0029] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the training method of the multi-task speech large language model according to the first aspect.
[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, and the computer instructions are executed by a processor to implement the steps of the training method of the multi-task speech large language model according to the first aspect.
[0031] Compared with the prior art, the training method, device and storage medium of the multi-task speech large language model provided by the embodiment of the present application, in the case of adding a first speech task to the multi-task speech large model, the multi-task speech large language model is trained according to a first loss function and existing multi-task data, a trained multi-task speech large language model is obtained, and a meta-learner is trained according to a second loss function and existing multiple second speech tasks, and a trained meta-learner is obtained; wherein the first loss function and the second loss function are obtained according to the difference between each two second speech tasks; the trained meta-learner is used to obtain a similarity index between the first speech task and each second speech task; and the trained multi-task speech large language model is updated according to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data. In this way, the multi-task speech large language model can support the addition of a speech task, can train on existing speech tasks and newly added speech tasks at the same time, and solves the problem of how to support the addition of a speech task after the training of the multi-task speech large language model in the prior art is completed. BRIEF DESCRIPTION OF DRAWINGS
[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the present application. Moreover, the same reference numerals are used throughout the same components in the entire drawings. In the drawings:
[0033] Figure 1 A flowchart of the training method of the multi-task speech large language model according to an embodiment of the present application is shown in FIG. 1.
[0034] Figure 2 A framework schematic diagram of the multi-task voice large language model according to the embodiments of the present application;
[0035] Figure 3 A flow schematic diagram of an implementation of the training method of the multi-task voice large language model according to the embodiments of the present application;
[0036] Figure 4 A structure schematic diagram of the training device of the multi-task voice large language model according to the embodiments of the present application;
[0037] Figure 5 A hardware block diagram of the training device of the multi-task voice large language model according to the embodiments of the present application. DETAILED DESCRIPTION
[0038] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second" are usually a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, scenario one: including A and not including B; scenario two: including B and not including A; scenario three: including A and B. The character " / " generally represents that the objects before and after are in an "or" relationship.
[0039] Please refer to Figure 1 The embodiments of the present application provide a training method of a multi-task voice large language model, comprising the following steps:
[0040] 101, in the case of adding a first voice task to the multi-task voice large language model, training the multi-task voice large language model according to a first loss function and existing multi-task data to obtain a trained multi-task voice large language model, and training a meta-learner according to a second loss function and existing multiple second voice tasks to obtain a trained meta-learner; wherein the first loss function and the second loss function are obtained according to the difference between each two second voice tasks.
[0041] The first voice task is a new voice task of the multi-task voice large language model compared to the second voice task. The second voice task is an existing voice task of the multi-task voice large language model compared to the first voice task.
[0042] Figure 2 A framework schematic diagram of the multi-task voice large language model according to the embodiments of the present application. As Figure 2As shown, the multi-task speech large language model described in the embodiments of the present application includes at least one of the following:
[0043] The speech encoder, the text encoder, the position embedding module, the task embedding module, the time sequence splicing module, the large language model backbone network, the speech token classifier, the text token classifier, the category label classifier, the speech decoder and the text decoder.
[0044] Since the modalities involved in the multi-task speech large language model include both speech and text, some modules are divided into two types to process data of different modalities. Table 1 lists some common speech tasks and their task source sequences, task target sequences, and the modalities to which the multi-task speech large language model proposed in the embodiments of the present application is related. The task source sequence is in the form of a sequence of discrete tokens or a sequence of continuous features, and the task target sequence is in the form of a sequence of discrete tokens or a sequence of continuous features. Figure 2 As shown in the process shown in the framework of the multi-task speech large language model, each component module processes the process, and finally outputs the task target sequence.
[0045]
[0046] Table 1: Some common speech tasks and their task source sequences, task target sequences
[0047] Next, the various component modules of the multi-task speech large language model described in the embodiments of the present application will be described: Figure 2
[0048] Speech / text encoder: used to encode the task source sequence to be processed from its original form into a sequence of discrete tokens or continuous features, and map it to the input space of the multi-task speech large language model. For task source sequences of two different modalities of speech and text, the corresponding encoder of the corresponding modality is used for processing. Among them, the speech encoder includes: HuBERT, a speech pre-training model based on self-supervised technology, used to transform the original speech waveform into a high-order speech feature sequence; also includes a linear layer for mapping the feature sequence to the input space of the multi-task speech large language model. The text encoder includes: a byte pair encoding (BPE) module for splitting the original text into a plurality of text tokens; also includes an embedding layer for mapping discrete tokens to the input space of the multi-task speech large language model.
[0049] Position embedding module: in order to solve the problem that the widely used self-attention based neural network cannot capture the position order information of each time step in the time sequence, a position embedding module is used to assign a unique vector representation to each position in the input sequence. Here, the multi-task speech large language model uses a learnable linear embedding layer as the position embedding module.
[0050] Task embedding module: Each speech task has a unique task label in the multi-task speech large language model, which enables the multi-task speech large language model to distinguish which speech task will be performed next when the task source sequence is the same. The task embedding module is used to map different task labels to different embedding vectors in the input space of the multi-task speech large language model.
[0051] Temporal concatenation module: Temporally concatenates the task source sequence mapped into the input space of the multi-task speech large language model with the task label embedding vector as the input of the large language model backbone network.
[0052] Large language model backbone network: Processes features of various modalities, and performs understanding, reasoning, and decision-making. It is stacked by multiple neural network layers with the same structure, and generally adopts a network structure of Transformer self-attention mechanism.
[0053] Speech token / text token / category label classifier: Used to convert the vector sequence output by the large language model backbone network into a probability distribution representing the posterior probability of the next time step token or classification label, which consists of a linear layer and a softmax function.
[0054] Speech / text decoder: Through autoregressive decoding, the speech / text decoder infers the target token sequence with maximum posterior probability from the target token probability at each time step output by the speech / text token classifier, and converts it into the original form of the corresponding modality. Among them, the speech decoder converts the speech token sequence into a speech waveform using a vocoder. The text decoder uses a mapping table or a BPE decoding tool to convert the text token sequence into a text unit sequence and splice it to obtain readable text.
[0055] In one embodiment, the method further comprises:
[0056] According to the target element, the properties of each second speech task are determined, and the target element includes at least one of the following: input language category, output language category, audio feature category, whether semantic information of speech is required, whether speaker recognition information of speech is required, whether emotional information of speech is required, whether environmental noise information of speech is required, and whether audio event information of speech is required.
[0057] According to the properties of each second speech task, the difference measure between each two second speech tasks is obtained.
[0058] It should be noted that the application system of the training method of the multi-task voice large language model described in the embodiments of the present application, that is, the training system of the multi-task voice large language model, comprises a task property analysis module, which is used to perform the above-mentioned step of obtaining the difference measure between each two second voice tasks.
[0059] It should also be noted that in addition to the above at least one, the target element can also include at least one of the following:
[0060] Input modalities (audio, text, or others);
[0061] If the input is audio, the type of audio features used;
[0062] Output modalities (audio, text, or others);
[0063] If the output is audio, the type of audio features used.
[0064] Of course, in addition to the above at least one, the target element can also be increased, deleted or modified according to the actual voice tasks involved in the training.
[0065] In the embodiments of the present application, the difference measure d(t a , t b ) between each two second voice tasks t a , t b is an N-dimensional vector, which is calculated using the following formula (1):
[0066]
[0067] Where N represents the number of target elements; m i (t a ), m i (t b ) represent the value of the i-th target element of the two second voice tasks, respectively; d i represents the difference measure function of the i-th target element; and a i represents the weight coefficient of the i-th target element.
[0068] Different target elements can use appropriate difference measure functions d i according to their characteristics, for example, for target elements with only a fixed number of discrete values (such as "type of audio features used", "whether semantic information of voice is needed"), d iFor the 01 function, it is 0 when the two values of the second speech task are the same, and 1 otherwise; and for some target elements with multiple possible values, a special difference measure function needs to be used, for example, the distance between two languages on the language phylogenetic tree can be selected as the difference measure function for the target element of "input language category".
[0069] Therefore, here, first, the nature of each second speech task is determined according to the target element, that is, the value of the target element corresponding to each second speech task and the difference measure function are determined; and then the difference measure between each two second speech tasks is obtained according to the nature of each second speech task.
[0070] In an embodiment, optionally, the method further comprises:
[0071] According to the cross-entropy loss of the multi-task speech large language model on the task target sequence and the difference measure between each two second speech tasks, the first loss function is obtained.
[0072] In the embodiments of the present application, the first loss function is obtained according to the cross-entropy loss of the multi-task speech large language model on the task target sequence, the difference between the task embedding vectors of each two second speech tasks, and the difference measure between each two second speech tasks
[0073] Specifically, the following formula (2) is used for calculation:
[0074]
[0075] Wherein, α, β are weighting weights; represents the cross-entropy loss of the multi-task speech large language model on the task target sequence, which can use the cross-entropy loss of the autoregressive language model; is obtained according to the difference between the task embedding vectors of each two second speech tasks and the difference measure, and is used to minimize the difference measure between each two second speech tasks, which is calculated by the following formula (3):
[0076]
[0077] Wherein, the emb function represents the task embedding module in the multi-task speech large language model; C is a constant.
[0078] In an embodiment, optionally, the meta-learner is used to output a similarity index between two speech tasks according to the difference measure between the two speech tasks.
[0079] It should be noted that the meta-learner is essentially a Multilayer Perceptron (MLP) that takes as input the difference measure d(t a ,t b ) between two speech tasks output by the task property analysis module in the training system of the multi-task speech large language model and outputs a similarity indicator s(t a ,t b ) between the two speech tasks, which is expressed by the following formula (4):
[0080] s(t a ,t b )=MLP(d(t a ,t b ))(4)
[0081] Here, the smaller the similarity indicator, the more similar the two speech tasks are.
[0082] In an embodiment, the method further comprises:
[0083] obtaining a similarity indicator output by the meta-learner according to the difference measure between each two of the second speech tasks;
[0084] obtaining the second loss function according to the similarity indicator and the difference between the task embedding vectors of each two of the second speech tasks.
[0085] In the embodiments of the present application, first, a similarity indicator s(t a ,t b ) between each two of the second speech tasks output by the meta-learner is obtained according to the difference measure d(t a ,t b ) between each two of the second speech tasks output by the task property analysis module in the training system of the multi-task speech large language model, and then a second loss function is obtained according to the similarity indicator s(t a ,t b ) between each two of the second speech tasks and the difference between the task embedding vectors of each two of the second speech tasks ‖emb(t a )-emb(t b )‖.
[0086] Specifically, the following formula (5) is used for calculation:
[0087]
[0088] It should be noted that, since the task property analysis module in the training system of the multi-task speech large language model is used to perform the step of obtaining the difference measure between each two second speech tasks, in step 101, the difference measure between each two second speech tasks is obtained by the task property analysis module, that is, with the aid of the task property analysis module, so as to obtain the first loss function and the second loss function; the multi-task speech large language model is trained to convergence according to the first loss function and the existing multi-task data; and the meta-learner is trained to minimize the second loss function according to the existing plurality of second speech tasks.
[0089] In the process of training the meta-learner, the parameters of the task embedding module in the multi-task speech large language model need to be kept fixed, and only the parameters of the meta-learner are optimized.
[0090] 102, using the trained meta-learner, obtaining the similarity index between the first speech task and each second speech task.
[0091] In the embodiments of the present application, the meta-learner is trained according to the second loss function and the plurality of existing second speech tasks, and the second loss function is minimized as the goal, to obtain a trained meta-learner. The similarity index between the newly added first speech task of the multi-task speech large language model and each existing second speech task is calculated using the trained meta-learner.
[0092] 103, according to the similarity index between the first speech task and each second speech task, the first speech task and the existing multi-task data, updating the trained multi-task speech large language model.
[0093] In one embodiment, optionally, according to the similarity index between the first speech task and each second speech task, the first speech task and the existing multi-task data, updating the trained multi-task speech large language model, comprising:
[0094] According to the similarity index between the first speech task and each second speech task, obtaining a target second speech task whose similarity index with the first speech task is less than a similarity index threshold;
[0095] According to the target second speech task, obtaining a task embedding vector estimate value corresponding to the first speech task;
[0096] According to the task embedding vector estimate value, obtaining the parameters of the task embedding module in the trained multi-task speech large language model;
[0097] According to the parameters of the task embedding module in the trained multi-task speech large language model, the first speech task, and the existing multi-task data, the trained multi-task speech large language model is updated.
[0098] In the embodiments of the present application, since the target second speech task is a second speech task with a similarity index less than a similarity index threshold with respect to the first speech task, the target second speech task can be understood as the second speech task most similar to the first speech task, and the number of target second speech tasks can be multiple.
[0099] Specifically, the task embedding vector estimate value corresponding to the first speech task is calculated using the following formula (6):
[0100]
[0101] wherein, denotes a set composed of K target second speech tasks, t new denotes a first speech task.
[0102] The task embedding vector estimate value corresponding to the first speech task is added to the parameters of the task embedding module in the trained multi-task speech large language model as a task embedding vector initial value.
[0103] The trained multi-task speech large language model is updated and trained using the task embedding vector initial value, the first speech task, and the randomly sampled existing multi-task data until the first speech task converges.
[0104] Updating and training the multi-task speech large language model can simultaneously optimize the existing second speech tasks and the newly added first speech task, ensuring that the performance of the existing second speech tasks does not decrease significantly.
[0105] It should be noted that all parameters of the task embedding module participate in the update and training.
[0106] Figure 3 A flowchart of an embodiment of the training method of the multi-task speech large language model according to the present application is shown. Using this embodiment, the method includes the following steps:
[0107] 301, randomly initialize the parameter values of the multi-task speech large language model and the meta-learner.
[0108] 302, with the assistance of the task property analysis module, using the existing multi-task data to train the multi-task speech large language model to convergence, using the first loss function, obtaining the trained multi-task speech large language model.
[0109] 303, with the assistance of the task property analysis module, using the existing multi-task data to train the multi-task speech large language model to convergence, using the first loss function, obtaining the trained multi-task speech large language model.
[0110] 304, using the trained meta-learner to calculate the similarity index between the newly added first speech task and each existing second speech task.
[0111] 305, obtaining the target second speech task most similar to the first speech task, and obtaining the task embedding vector estimate value corresponding to the first speech task according to the target second speech task.
[0112] 306, adding the task embedding vector estimate value corresponding to the first speech task into the parameters of the task embedding module as the initial value of the task embedding vector.
[0113] 307, using the first speech task and randomly sampled existing multi-task data to train the multi-task speech large language model until the first speech task converges.
[0114] In summary, the training method of the multi-task speech large language model described in the embodiments of the present application analyzes the properties of speech tasks from the perspectives of input modalities, output modalities, feature types, information types extracted from speech, and target elements such as languages involved, and gives quantitative difference measures and similarity indexes between speech tasks, so that the meta-learner can model the correlation between tasks, so that some existing speech tasks can help the newly added speech tasks, and the multi-task speech large language model is updated and trained, combined with the idea of continuous learning, so that the multi-task speech large language model can be trained on both existing speech tasks and newly added speech tasks, solving the problem of how to support newly added speech tasks after the multi-task speech large language model is trained, and maintaining the performance of existing speech tasks while efficiently adding new speech tasks.
[0115] The above introduces various methods of the embodiments of the present application. The following will further provide a device for implementing the above method.
[0116] Please refer to Figure 4 The embodiments of the present application also provide a multi-task speech large language model training device, comprising:
[0117] The training module 401 is configured to train the multi-task speech large language model according to a first loss function and existing multi-task data in a case where a first speech task is added to the multi-task speech large model, to obtain a trained multi-task speech large language model, and to train a meta-learner according to a second loss function and existing second speech tasks, to obtain a trained meta-learner; wherein the first loss function and the second loss function are obtained according to a difference measurement between each two second speech tasks.
[0118] The first obtaining module 402 is configured to use the trained meta-learner to obtain a similarity index between the first speech task and each second speech task.
[0119] The updating module 403 is configured to update the trained multi-task speech large language model according to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data.
[0120] Optionally, the device for training a multi-task speech large language model further comprises:
[0121] The determining module is configured to determine the properties of each second speech task according to target elements, the target elements including at least one of the following: an input language type, an output language type, an audio feature type, whether semantic information of speech is needed, whether speaker recognition information of speech is needed, whether emotional information of speech is needed, whether environmental noise information of speech is needed, and whether audio event information of speech is needed.
[0122] The second obtaining module is configured to obtain a difference measurement between each two second speech tasks according to the properties of each second speech task.
[0123] Optionally, the device for training a multi-task speech large language model further comprises:
[0124] The first obtaining module is configured to obtain the first loss function according to a cross-entropy loss of the multi-task speech large language model on a task target sequence and the difference measurement between each two second speech tasks.
[0125] Optionally, the device for training a multi-task speech large language model further comprises:
[0126] The second obtaining module is configured to obtain a similarity index output by the meta-learner according to the difference measurement between each two second speech tasks.
[0127] The third obtaining module is configured to obtain the second loss function according to the similarity index and a difference value of task embedding vectors between each two of the second speech tasks.
[0128] Optionally, the training device of the multi-task speech large language model, wherein the meta-learner is configured to output a similarity index between two speech tasks according to a difference measurement between the two speech tasks.
[0129] Optionally, the training device of the multi-task speech large language model, wherein the updating module 403 is specifically configured to:
[0130] obtain a target second speech task between the first speech task and each of the second speech tasks according to the similarity index;
[0131] obtain an estimated value of a task embedding vector corresponding to the first speech task according to the target second speech task;
[0132] obtain a parameter of a task embedding module in the trained multi-task speech large language model according to the estimated value of the task embedding vector;
[0133] update the trained multi-task speech large language model according to the parameter of the task embedding module in the trained multi-task speech large language model, the first speech task, and the existing multi-task data.
[0134] It should be noted that the device in this embodiment is a device corresponding to the above-mentioned training method of the multi-task speech large language model, and the implementation modes in the above-mentioned embodiments are applicable to the embodiments of the device and can achieve the same technical effects. The above-mentioned device provided in the embodiments of the present application can realize all the method steps realized by the method embodiments and can achieve the same technical effects. Here, the same parts and beneficial effects in the method embodiments will not be described in detail.
[0135] The embodiments of the present application also provide a training device of a multi-task speech large language model, as shown in Figure 5 The training device comprises:
[0136] The processor 501, the memory 502, the transceiver 503, and a program or instructions stored on the memory 502 and executable on the processor 501; the processor 501 executes the program or instructions to realize each process of the above-mentioned training method of the multi-task speech large language model and can achieve the same technical effects. To avoid repetition, this will not be described in detail.
[0137] The transceiver 503 is configured to receive and send data under the control of the processor 501.
[0138] wherein, in Figure 5 The bus architecture can include any number of interconnected buses and bridges, specifically, various circuitry linking the one or more processors represented by the processor 501 and the memory represented by the memory 502. The bus architecture can also link various other circuitry such as peripheral devices, voltage regulators, and power management circuitry, which are well known in the art and thus, not further described herein. The bus interface provides an interface. The transceiver 503 can be a plurality of elements, i.e., including a transmitter and a receiver, providing a means for communicating with various other apparatus over a transmission medium. The user interface 504 can also be an interface capable of externally connecting the required devices for different user equipment, including but not limited to a keypad, a display, a speaker, a microphone, a joystick, etc.
[0139] The processor 501 is responsible for managing the bus architecture and general processing, and the memory 502 can store data used by the processor 501 in performing operations.
[0140] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the training method of the multi-task speech large language model, and can achieve the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0141] The embodiment of the present application further provides a computer program product, which includes computer instructions. The computer instructions are executed by a processor to implement each process of the training method of the multi-task speech large language model, and can achieve the same technical effects. To avoid repetition, details are not described herein.
[0142] It should be noted that, in this document, the term "comprising" or "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.
[0143] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software product, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc), including a plurality of instructions to make a terminal (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.
[0144] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not limited, those skilled in the art can make many forms without departing from the purpose of the present application and the scope of the claims, which are all within the protection of the present application.
Claims
1. A method for training a multi-task speech large language model, characterized in that, The method comprises: In the case of adding a first speech task to a multi-task speech large model, training the multi-task speech large language model according to a first loss function and existing multi-task data to obtain a trained multi-task speech large language model, and training a meta-learner according to a second loss function and existing second speech tasks to obtain a trained meta-learner; wherein the first loss function and the second loss function are obtained according to the difference between each two second speech tasks; Using the trained meta-learner, obtaining a similarity index between the first speech task and each second speech task; According to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data, updating the trained multi-task speech large language model. 2.The method of Claim 1, wherein, The method further comprises: According to the target element, determining the properties of each second speech task, the target element comprising at least one of the following: input language category, output language category, audio feature category, whether semantic information of speech is required, whether speaker recognition information of speech is required, whether emotional information of speech is required, whether environmental noise information of speech is required, and whether audio event information of speech is required; According to the properties of each second speech task, obtaining a difference measure between each two second speech tasks.
3. The method of Claim 1 or 2, wherein, The method further comprises: According to the cross-entropy loss of the multi-task speech large language model on the task target sequence and the difference measure between each two second speech tasks, obtaining the first loss function.
4. The method of Claim 1 or 2, wherein, The method further comprises: According to the difference measure between each two second speech tasks, obtaining a similarity index output by the meta-learner; According to the similarity index and the difference between the task embedding vectors of each two second speech tasks, obtaining the second loss function. 5.The method of Claim 1, wherein, The meta-learner is used to output a similarity index between two speech tasks according to a difference measure between the two speech tasks.
6. The method of Claim 1, wherein, According to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data, updating the trained multi-task speech large language model, comprises: According to the similarity index between the first speech task and each second speech task, obtaining a target second speech task whose similarity index with the first speech task is less than a similarity index threshold; According to the target second speech task, obtaining a task embedding vector estimation value corresponding to the first speech task; According to the task embedding vector estimation value, obtaining parameters of a task embedding module in the trained multi-task speech large language model; According to the parameters of the task embedding module in the trained multi-task speech large language model, the first speech task, and the existing multi-task data, updating the trained multi-task speech large language model.
7. An apparatus for training a multitask speech large language model, comprising: The method comprises: The training module is configured to train the multi-task speech large language model according to a first loss function and existing multi-task data when a first speech task is added to the multi-task speech large model, to obtain a trained multi-task speech large language model, and to train a meta-learner according to a second loss function and existing second speech tasks, to obtain a trained meta-learner; wherein the first loss function and the second loss function are obtained according to a difference between each two second speech tasks; The first obtaining module is configured to use the trained meta-learner to obtain a similarity index between the first speech task and each second speech task; The updating module is configured to update the trained multi-task speech large language model according to the similarity index between the first speech task and each second speech task, the first speech task, and the existing multi-task data.
8. A training device of a multitask voice large language model, comprising: The processor, the memory, and the program or the instructions stored on the memory and executable on the processor, wherein the processor implements the steps of the training method of the multi-task speech large language model according to any one of claims 1 to 6. The computer program is stored on the computer-readable storage medium, and the computer program is executed by the processor to implement the steps of the training method of the multi-task speech large language model according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer instructions are executed by the processor to implement the steps of the training method of the multi-task speech large language model according to any one of claims 1 to 6.
10. A computer program product, characterised in that,