Training method of audio understanding model, audio understanding method, device, storage medium and program product

Abstract

The present disclosure relates to a method for training an audio understanding model, an audio understanding method, an apparatus, a storage medium and a program product. The method comprises: obtaining a pre-trained speech recognition model and a large language model; the speech recognition model comprises an encoding module, a prediction module and a first fusion module; constructing an audio understanding model according to the speech recognition model and the large language model, wherein the audio understanding model comprises an encoding module, a large language model main body and a second fusion module; extracting an acoustic feature sequence corresponding to an audio sample through the encoding module, and extracting a semantic feature sequence corresponding to the audio sample through the large language model main body; determining, through the second fusion module, all effective alignment paths capable of generating a target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating a total probability of all the effective alignment paths; and updating parameters of the audio understanding model according to the total probability. The present disclosure can improve the accuracy of speech recognition.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for an audio understanding model, an audio understanding method, a training device for an audio understanding model, an audio understanding device, a non-volatile computer-readable storage medium, and a computer program product. Background Technology

[0002] Audio understanding technology refers to the technique of semantic parsing and content recognition of audio signals using artificial intelligence models, capable of converting speech content into understandable text information. Audio understanding can be divided into non-streaming audio understanding and streaming audio understanding. Non-streaming processing performs overall analysis of complete audio and outputs results, suitable for scenarios such as meeting transcript analysis. Streaming processing, on the other hand, parses audio in real time and provides dynamic feedback, widely used in fields requiring instant interaction such as intelligent voice assistants, real-time caption generation, and smart homes. How to improve the training efficiency and processing performance of streaming audio understanding models is an important research topic. Summary of the Invention

[0003] In view of this, this disclosure provides an audio understanding technology solution.

[0004] According to one aspect of this disclosure, a method for training an audio understanding model is provided, comprising:

[0005] A pre-trained speech recognition model and a pre-trained large language model are obtained; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are interconnected;

[0006] An audio understanding model is constructed based on the speech recognition model and the large language model, wherein the audio understanding model includes the encoding module, the large language model body and the second fusion module, and the second fusion module is connected to the encoding module and the large language model body respectively;

[0007] For any audio sample in the audio training set, the acoustic feature sequence corresponding to the audio sample is extracted through the encoding module in the audio understanding model, and the semantic feature sequence corresponding to the audio sample is extracted through the large language model body in the audio understanding model;

[0008] The second fusion module determines all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculates the total probability of all valid alignment paths.

[0009] The parameters of the audio understanding model are updated based on the total probability.

[0010] In one possible implementation, the second fusion module includes the first fusion module and the first header, wherein the first header is initialized based on the large language model header, and the first header adds the dimension corresponding to the blank symbol to the large language model header.

[0011] In one possible implementation, the method further includes:

[0012] During pre-training, the output dimension of the first fusion module is controlled to be the same as the input dimension of the large language model head.

[0013] In one possible implementation, updating the parameters of the audio understanding model based on the total probability includes:

[0014] Taking the negative logarithm of the total probability yields the value of the first loss function corresponding to the audio understanding model;

[0015] The parameters of the audio understanding model are updated based on the value of the first loss function.

[0016] In one possible implementation, the audio understanding model further includes a second head, which is connected to the large language model body, and the second head is initialized according to the large language model head.

[0017] The method further includes:

[0018] The second head outputs the text prediction result corresponding to the audio sample;

[0019] Based on the text prediction results corresponding to the audio samples and the target text label sequence, determine the value of the second loss function corresponding to the audio understanding model;

[0020] The parameters of the audio understanding model are updated based on the value of the second loss function.

[0021] In one possible implementation, the parameters of the second head remain fixed during the training of the audio understanding model.

[0022] In one possible implementation, the audio understanding model performs a first-stage training and a second-stage training based on the audio training set;

[0023] In the first training phase, the parameters of the encoding module remain fixed; in the second training phase, the parameters of the encoding module are adjusted.

[0024] In one possible implementation, the method further includes:

[0025] In the second stage of training, the audio samples are divided into multiple audio segments according to a preset duration, and the multiple audio segments are sequentially input into the encoding module of the audio understanding model;

[0026] By using an attention mask, the audio understanding model is restricted to only acquiring information about the current audio segment and historical audio segments preceding it when processing the current audio segment.

[0027] In one possible implementation, during the training of the audio understanding model, the first head undergoes full-parameter training, while the parameters of the large language model body are updated using fine-tuning.

[0028] In one possible implementation, the speech recognition model employs a recurrent neural network transcriber.

[0029] According to another aspect of this disclosure, an audio understanding method is provided, comprising:

[0030] The audio understanding model is obtained by training the audio understanding model using the training method described above;

[0031] The audio to be processed is input into the audio understanding model, and the audio understanding model outputs the text prediction result corresponding to the audio to be processed.

[0032] In one possible implementation, before inputting the audio to be processed into the audio understanding model, the method further includes:

[0033] Remove the second head from the audio understanding model.

[0034] In one possible implementation, the audio to be processed is either streaming audio or non-streaming audio.

[0035] According to another aspect of this disclosure, a training apparatus for an audio understanding model is provided, comprising:

[0036] An acquisition module is used to acquire a pre-trained speech recognition model and a pre-trained large language model; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are interconnected.

[0037] A construction module is used to construct an audio understanding model based on the speech recognition model and the large language model, wherein the audio understanding model includes the encoding module, the large language model body and the second fusion module, and the second fusion module is connected to the encoding module and the large language model body respectively;

[0038] The extraction module is used to extract the acoustic feature sequence corresponding to any audio sample in the audio training set through the encoding module in the audio understanding model, and to extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model.

[0039] The calculation module is used to determine, through the second fusion module, all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and to calculate the total probability of all valid alignment paths;

[0040] The first update module is used to update the parameters of the audio understanding model based on the total probability.

[0041] In one possible implementation, the second fusion module includes the first fusion module and the first header, wherein the first header is initialized based on the large language model header, and the first header adds the dimension corresponding to the blank symbol to the large language model header.

[0042] In one possible implementation, the device further includes:

[0043] The control module is used to control the output dimension of the first fusion module to be the same as the input dimension of the large language model head during the pre-training process.

[0044] In one possible implementation, the first update module is used to:

[0045] Taking the negative logarithm of the total probability yields the value of the first loss function corresponding to the audio understanding model;

[0046] The parameters of the audio understanding model are updated based on the value of the first loss function.

[0047] In one possible implementation, the audio understanding model further includes a second head, which is connected to the large language model body, and the second head is initialized according to the large language model head.

[0048] The device further includes:

[0049] The second output module is used to output the text prediction result corresponding to the audio sample through the second head;

[0050] The determination module is used to determine the value of the second loss function corresponding to the audio understanding model based on the text prediction result corresponding to the audio sample and the target text label sequence;

[0051] The second update module is used to update the parameters of the audio understanding model based on the value of the second loss function.

[0052] In one possible implementation, the parameters of the second head remain fixed during the training of the audio understanding model.

[0053] In one possible implementation, the audio understanding model performs a first-stage training and a second-stage training based on the audio training set;

[0054] In the first training phase, the parameters of the encoding module remain fixed; in the second training phase, the parameters of the encoding module are adjusted.

[0055] In one possible implementation, the device further includes:

[0056] The segmentation module is used to segment the audio samples into multiple audio segments according to a preset duration during the second stage of training, and to input the multiple audio segments into the encoding module in the audio understanding model in sequence.

[0057] The restriction module is used to restrict the audio understanding model to only acquire information about the current audio segment and historical audio segments preceding the current audio segment when processing the current audio segment, by means of an attention mask.

[0058] In one possible implementation, during the training of the audio understanding model, the first head undergoes full-parameter training, while the parameters of the large language model body are updated using fine-tuning.

[0059] In one possible implementation, the speech recognition model employs a recurrent neural network transcriber.

[0060] According to another aspect of this disclosure, an audio understanding apparatus is provided, comprising:

[0061] The acquisition module is used to acquire the audio understanding model trained by the training method of the audio understanding model;

[0062] The first output module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

[0063] In one possible implementation, the device further includes:

[0064] The deletion module is used to delete the second head in the audio understanding model.

[0065] In one possible implementation, the audio to be processed is either streaming audio or non-streaming audio.

[0066] According to another aspect of this disclosure, a training apparatus for an audio understanding model is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0067] According to another aspect of this disclosure, an audio understanding apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0068] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0069] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0070] In this embodiment, a pre-trained speech recognition model and a pre-trained large-scale language model are obtained. The speech recognition model includes an encoding module, a prediction module, and a first fusion module, which is connected to the encoding module and the prediction module respectively. The large-scale language model includes a large-scale language model body and a large-scale language model head connected to each other. Based on the speech recognition model and the large-scale language model, an audio understanding model is constructed. The audio understanding model includes the encoding module, the large-scale language model body, and a second fusion module, which is connected to the encoding module and the large-scale language model body respectively. For any audio sample in the audio training set, the audio is extracted through the encoding module in the audio understanding model. The audio understanding model extracts the acoustic feature sequence corresponding to the audio sample and the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model. The second fusion module determines all effective alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, calculates the total probability of all effective alignment paths, and updates the parameters of the audio understanding model according to the total probability. In this way, by utilizing the powerful language understanding and generation capabilities of the pre-trained large language model and its ability to learn quickly with a small amount of audio data, combined with the audio-text alignment mechanism, efficient streaming speech recognition can be achieved, and the accuracy of speech recognition can be significantly improved even when there is little audio training data.

[0071] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0072] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0073] Figure 1 A flowchart illustrating the training method of the audio understanding model provided in the embodiments of this disclosure is shown.

[0074] Figure 2 A schematic diagram of a recurrent neural network transcriptograph is shown.

[0075] Figure 3 A schematic diagram of the audio understanding model provided in an embodiment of this disclosure is shown.

[0076] Figure 4 This diagram illustrates the dynamic alignment process of the audio understanding model provided in the embodiments of this disclosure during the decoding stage.

[0077] Figure 5A block diagram of a training apparatus for an audio understanding model provided in an embodiment of this disclosure is shown.

[0078] Figure 6 This is a block diagram illustrating a training device or audio understanding device 1900 for an audio understanding model according to an exemplary embodiment. Detailed Implementation

[0079] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0080] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0081] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0082] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0083] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0084] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0085] This disclosure provides a method for training an audio understanding model. It involves obtaining a pre-trained speech recognition model and a pre-trained large-scale language model. The speech recognition model includes an encoding module, a prediction module, and a first fusion module, which is connected to both the encoding and prediction modules. The large-scale language model includes a main body and a head, both interconnected. An audio understanding model is constructed based on the speech recognition model and the large-scale language model. This audio understanding model includes the encoding module, the main body of the large-scale language model, and a second fusion module, which is connected to both the encoding module and the main body of the large-scale language model. For any audio sample in the audio training set, the encoding module in the audio understanding model... The module extracts the acoustic feature sequence corresponding to the audio sample and extracts the semantic feature sequence corresponding to the audio sample through the large language model in the audio understanding model. The second fusion module determines all effective alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, calculates the total probability of all effective alignment paths, and updates the parameters of the audio understanding model according to the total probability. Thus, by utilizing the powerful language understanding and generation capabilities of the pre-trained large language model and its ability to learn quickly with a small amount of audio data, combined with the audio-text alignment mechanism, efficient streaming speech recognition can be achieved, and the accuracy of speech recognition can be significantly improved even when there is little audio training data.

[0086] The training method of the audio understanding model provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0087] Figure 1 A flowchart illustrating a training method for an audio understanding model provided in an embodiment of this disclosure is shown. In one possible implementation, the main body executing the audio understanding model training method can be a training device for the audio understanding model. For example, the audio understanding model training method can be executed by a terminal device, a server, or other electronic devices. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the audio understanding model training method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method of the audio understanding model includes steps S11 to S15.

[0088] In step S11, a pre-trained speech recognition model and a pre-trained large language model are obtained; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are connected to each other.

[0089] In step S12, an audio understanding model is constructed based on the speech recognition model and the large language model. The audio understanding model includes the encoding module, the large language model body, and the second fusion module. The second fusion module is connected to the encoding module and the large language model body, respectively.

[0090] In step S13, for any audio sample in the audio training set, the acoustic feature sequence corresponding to the audio sample is extracted by the encoding module in the audio understanding model, and the semantic feature sequence corresponding to the audio sample is extracted by the large language model body in the audio understanding model.

[0091] In step S14, the second fusion module determines all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculates the total probability of all valid alignment paths.

[0092] In step S15, the parameters of the audio understanding model are updated according to the total probability.

[0093] In this embodiment of the disclosure, the pre-trained speech recognition model can refer to a speech recognition model that has been trained on a speech dataset beforehand. By learning and extracting features from speech samples, the speech recognition model can capture the acoustic features, prosodic patterns, and correspondence between speech and text, and convert the input speech into corresponding text content.

[0094] In this embodiment, the pre-trained speech recognition model may include an encoder, a predictor, and a first jointer. In some applications, the encoder may also be called an encoder, the predictor may be called a predictor network, and the first jointer may be called a joint network or joint module; no limitation is made here. In one possible implementation, the encoder may be used to extract acoustic features, the predictor may be used to extract semantic features, and the first jointer may be used to fuse the acoustic and semantic features.

[0095] In one possible implementation, the speech recognition model employs a Recurrent Neural Network Transducer (RNNT).

[0096] Figure 2 A schematic diagram of a recurrent neural network transcriptome is shown. Figure 2 As shown, a recurrent neural network transcriptome may include an encoding module, a prediction module, a first fusion module, and an activation layer.

[0097] The encoding module can be used to model acoustic features. For example, it can convert an input audio frame sequence into a high-dimensional acoustic feature representation. For instance, the encoding module can take the audio frame x at the current time step t as input. t Output the acoustic features corresponding to the current time step t. Here, the time step refers to the smallest discrete time unit in audio processing, corresponding to a single frame of data processed or output by the model at a certain moment. For an audio frame sequence x = (x1, x2, ..., x...), ... T The encoding module can output acoustic feature sequences. The encoding module can be implemented using a Conformer or LSTM (Long Short-Term Memory) network, and no specific restriction is imposed here.

[0098] The prediction module can be used to model semantic features. During the training phase, the prediction module can be inputted with historical text labels (such as y) before the current text step u. u-1 Or y ＜u =(y1,y2,...,y u-1 Output the semantic features of the current text step u. During the prediction phase, the prediction module can take the generated (i.e., predicted) text labels as input and output the semantic features of the current text step. Here, a text step refers to a discrete unit in the text generation process, corresponding to a single word processed or output by the model. The prediction module can use either an LSTM or a Transformer structure; no specific limitation is made here.

[0099] The training samples in the training set can consist of audio-text pairs {x, y}, where x represents an audio sample and y represents the corresponding target text label sequence (i.e., labeled text). For example, if the target text label sequence y is "How's the weather today?", then the text format input to the prediction module during training can be: <sos>How's the weather today? <sos>This is the start of sentence marker, used to initialize the hidden state of the prediction module and indicate the beginning of text generation. The target text label sequence used to calculate the loss can be in the format: "What's the weather like today?" <eos>.in, <eos>This is the sentence end marker. <eos>As part of the training objective, it helps the model learn when to stop outputting.

[0100] The first fusion module can fuse the outputs of the encoding module and the prediction module. For example, the output of the first fusion module can be: The first fusion module may include linear layers and activation layers (e.g., softmax).

[0101] like Figure 2 As shown, after the first fusion module, an activation layer (e.g., softmax) can be connected to generate the probability distribution P(y|t,u) of all possible tokens (e.g., characters, words, symbols).

[0102] In this implementation, the pre-trained speech recognition model, employing a recurrent neural network transcriber structure, effectively achieves real-time dynamic alignment of audio and text sequences, demonstrating significant advantages in streaming speech recognition tasks. The recurrent neural network transcriber structure extracts acoustic features through an encoding module, models text context dependencies through a prediction module, and fuses multimodal information through a first fusion module. This results in powerful feature representation capabilities, significantly improving the model's recognition accuracy and real-time performance in complex scenarios.

[0103] In this embodiment, a pre-trained Large Language Model (LLM) can refer to a model pre-trained on a large-scale text dataset. In some applications, a large language model may also be called a large language model, a large text model, etc., without limitation. In one possible implementation, the pre-trained large language model can be an open-source pre-trained large language model, such as Llama or Qwen. These open-source models have already been learned and trained on massive amounts of text, accumulating rich linguistic knowledge and semantic understanding capabilities. In another possible implementation, the pre-trained large language model can be a model trained by the developers themselves.

[0104] Large-scale language models, pre-trained on massive amounts of text data, demonstrate powerful language understanding and generation capabilities. Compared to traditional models that rely on large amounts of data, large-scale language models are more advantageous in data-scarce languages ​​and complex speech tasks, and their large number of parameters makes them more robust and adaptable to noisy environments. Furthermore, multilingual pre-trained large-scale language models can reduce the need for language-specific data and lower the complexity of model management.

[0105] In this embodiment, the large language model may include an interconnected large language model body (LLM-body) and a large language model head (LLM-head). The large language model body may be composed of multi-layer neural networks (such as Transformer modules), and is responsible for deep feature extraction and semantic understanding of the input text, obtaining general language representation capabilities through pre-training on massive amounts of text data. The large language model head, as the output layer of the large language model, may consist of linear layers and activation layers, and is responsible for mapping the high-dimensional features extracted by the large language model body to the target output space (such as the probability distribution of the lexicon). In some application scenarios, the large language model head may also be referred to as the output layer of the large language model.

[0106] In this embodiment of the disclosure, an audio understanding model can be constructed based on a pre-trained speech recognition model and a pre-trained large-scale language model. The audio understanding model includes an encoding module from the pre-trained speech recognition model, a large-scale language model body from the pre-trained large-scale language model, and a second fusion module. In the audio understanding model, the second fusion module is connected to both the encoding module and the large-scale language model body, and can be used to fuse the outputs of the encoding module and the large-scale language model body.

[0107] In one possible implementation, an audio understanding model can be built upon a pre-trained speech recognition model. Specifically, the encoding module in the pre-trained speech recognition model can be retained, the prediction module in the pre-trained speech recognition model can be replaced with the main body of a large language model, and the first fusion module in the pre-trained speech recognition model can be reconstructed into a second fusion module.

[0108] In the audio understanding model, the encoding module can be used to process audio input and output acoustic features h. enc In audio understanding models, large language models can take text as input and output semantic features h. LLM In audio understanding models, the ability to model text context can be enhanced by replacing the prediction module in a speech recognition model with a large language model body. The input to the second fusion module in the audio understanding model can include the acoustic features h output by the encoding module. enc and the semantic features h output by the main body of the large language model LLM The second fusion module can dynamically align acoustic and semantic features.

[0109] In one possible implementation, the second fusion module includes the first fusion module and the first header, wherein the first header is initialized based on the large language model header, and the first header adds the dimension corresponding to the blank symbol to the large language model header.

[0110] In this implementation, the second fusion module can be composed of the first fusion module and the first header.

[0111] The first fusion module retains the structure of the fusion module in the pre-trained speech recognition model (such as linear layer + activation layer), and can be used to initially fuse acoustic features and semantic features.

[0112] The first head can be initialized based on the head of a large language model. That is, the parameters of the first head can be inherited from the pre-trained large language model head to leverage the existing language modeling capabilities of the pre-trained large language model. Additionally, in this implementation, a dimension corresponding to blank characters is added to the large language model head to obtain the first head. This is to adapt to the streaming alignment requirements of speech recognition models (such as recurrent neural network transcribers), because speech recognition models need to handle blank characters (representing no output or waiting for subsequent input) during decoding. For example, if the output dimension of the large language model head is the vocabulary size V, the output dimension of the modified first head is V+1, with the added dimension dedicated to calculating the probability of blank characters.

[0113] In this implementation, the language generation capabilities of a large language model are preserved by reusing the parameters of the large language model's head. At the same time, the alignment mechanism specific to speech recognition models (such as recurrent neural network transcribers) is supported by expanding the dimensions. The introduction of blank enables the audio understanding model to dynamically align audio and text (e.g., moving the path to the right represents blank space, and moving it upwards represents generating characters), thereby achieving streaming decoding.

[0114] In one possible implementation, the method further includes: during pre-training, controlling the output dimension of the first fusion module to be the same as the input dimension of the large language model head.

[0115] In this implementation, during pre-training, the output dimension of the first fusion module is controlled to be the same as the input dimension of the large language model head. This ensures seamless structural integration between the two modules, enabling effective feature fusion. The first fusion module is responsible for jointly processing the acoustic features extracted by the encoding module and the semantic features output by the prediction module (or the replaced large language model head). Its output needs to match the input dimension of the large language model head to avoid information loss or computational errors caused by dimensionality mismatch. This alignment operation allows the second fusion module to directly utilize the parameters of the pre-trained large language model head without additional adjustments, thus preserving the original language modeling capabilities of the large language model while remaining compatible with feature inputs from audio modalities.

[0116] This implementation enables the multimodal features (acoustic and semantic) to be correctly processed by the large language model head after fusion. For example, if the input dimension of the large language model head is D, the output of the first fusion module also needs to be adjusted to D to ensure that its joint features can be directly input into the large language model head for probability distribution calculation. This consistency not only simplifies the structure of the audio understanding model but also improves training efficiency, allowing the audio understanding model to converge quickly during the fine-tuning phase. Furthermore, this dimensionality control lays the foundation for subsequent streaming processing (such as adding blank dimensions), ensuring that the audio understanding model maintains stability and robustness when dynamically aligning audio and text.

[0117] In this embodiment of the disclosure, during the training process of the audio understanding model, for each audio sample in the audio training set, the encoding module can process the original audio signal frame by frame, outputting an acoustic feature sequence. The large language model body can autoregressively generate the corresponding semantic feature sequence based on historical text tags. The acoustic feature sequence and the semantic feature sequence provide multimodal input for the subsequent second fusion module.

[0118] In this embodiment of the disclosure, the second fusion module can determine all valid alignment paths that can generate the target text tag sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculate the total probability of all valid alignment paths.

[0119] The alignment path can refer to the mapping relationship between the acoustic feature sequence (time step t) and the semantic feature sequence (text step u). For example:

[0120] Horizontal movement (→): Time step t+1, text step u remains unchanged (corresponding output blank, indicating that no text has been generated yet).

[0121] Vertical movement (↑): Time step t remains unchanged, text step u+1 (corresponding to the generation of an actual text label).

[0122] A valid alignment path refers to all legal paths from the starting point (t=0, u=0) to the ending point (t=T, u=U), and the generated text sequence is completely identical to the target text label sequence. Here, T is the total time step of the audio sample, and U is the length of the target text label sequence.

[0123] The second fusion module can receive the acoustic feature sequence h from the encoding module. t and semantic feature sequences from the main body of a large language model u The joint probability distribution p(y) is output through a linear layer and an activation layer (such as softmax). t,u |h t ,s u ), where y t,u It can represent the predicted label for the current step.

[0124] The probability of each aligned path is the product of the probabilities of all steps along the path. For example, a path is Path k :(t1,u1)→(t2,u2)→...→(t N ,u N If ), then its probability is Sum the probabilities of all valid paths:

[0125] For example, if the target text is "AB" and the audio has 3 time steps, the two possible valid paths are:

[0126] Path 1: (t=1)→(t=2)→(t=3↑)→(t=3↑)

[0127] (Output: blank→blank→A→B)

[0128] Path 2: (t=1↑)→(t=2)→(t=3↑)

[0129] (Output: A→blank→B)

[0130] Total probability P total = P(Path1) + P(Path2).

[0131] In one possible implementation, updating the parameters of the audio understanding model based on the total probability includes: taking the negative logarithm of the total probability to obtain the value of a first loss function corresponding to the audio understanding model; and updating the parameters of the audio understanding model based on the value of the first loss function.

[0132] In this implementation, during the training of the audio understanding model, after calculating the total probability of all valid alignment paths, this total probability can be maximized (i.e., minimizing the negative log probability) by optimizing the model parameters. Specifically, the total probability P can be optimized. total Taking the negative logarithm yields the value L of the first loss function. RNNT =-log(P total The probability range is [0,1]. After taking the negative logarithm, the higher the probability (closer to 1), the lower the loss (closer to 0); conversely, the lower the probability (closer to 0), the higher the loss (approaching infinity). This transformation converts the probability maximization problem into the loss minimization problem, which is consistent with the gradient descent optimization framework.

[0133] For example, if an audio sample has two valid alignment paths with probabilities of 0.6 and 0.4 respectively, then the total probability P is... total =0.6 + 0.4 = 1, loss value L RNNT =-log(1)=0; if the path probability distribution is not ideal (e.g., 0.1 and 0.1), the loss value L RNNT = -log(0.2)≈1.61, the audio understanding model will adjust the parameters through gradient descent to increase the likelihood of generating high-probability paths.

[0134] In this implementation, the gradient can be calculated based on the value of the first loss function, and the parameters of the audio understanding model can be updated through backpropagation. For example, the parameters of at least some modules in the second fusion module, the large language model body, and the encoding module can be updated.

[0135] In this implementation, based on the first loss function, the alignment path between acoustic and semantic features can be directly optimized, ensuring dynamic matching between the text sequence generated by the audio understanding model and the audio input. By minimizing the first loss function, the second fusion module can learn to effectively combine the acoustic features of the encoding module and the semantic features of the large language model body.

[0136] In one possible implementation, the audio understanding model further includes a second head connected to the main body of the large language model, and the second head is initialized according to the head of the large language model; the method further includes: outputting the text prediction result corresponding to the audio sample through the second head; determining the value of a second loss function corresponding to the audio understanding model based on the text prediction result corresponding to the audio sample and the target text label sequence; and updating the parameters of the audio understanding model based on the value of the second loss function.

[0137] In this implementation, the second head can directly reuse the original head structure of a pre-trained large language model, for example, it can include linear layers and softmax layers to inherit the powerful language generation capabilities of the large language model. The parameters of the second head can be copied from the head of the large language model, but are independent of the first head, forming a dual-task output branch. Furthermore, the second head can be connected to the main body of the large language model to receive its output semantic features.

[0138] The second head can directly generate text prediction results (such as the probability distribution of characters or words) corresponding to audio samples based on the semantic features output by the main body of the large language model. By calculating the second loss function (such as cross-entropy loss) between the text prediction results and the true labels (i.e., the target text label sequence), additional supervision signals can be provided to enhance the language modeling ability.

[0139] In one possible implementation, the parameters of the second head remain fixed during the training of the audio understanding model.

[0140] In this implementation, the parameters of the second head can be kept fixed during the training of the audio understanding model (i.e., it does not participate in gradient updates). This allows the incorporation of pre-trained knowledge from the large language model while preventing its text generation capabilities from being interfered with by the speech alignment task. The second head directly reuses the parameters of the original head of the large language model, and by freezing its weights, ensures that the audio understanding model retains the powerful language modeling capabilities of the large language model. This design leverages the prior knowledge of the large language model to improve text prediction quality while preventing parameter conflicts during multi-task training, enabling the audio understanding model to converge more stably.

[0141] Figure 3 A schematic diagram of the audio understanding model provided in an embodiment of this disclosure is shown. Figure 3 As shown, the audio understanding model may include an encoding module, an LLM body, a second fusion module, and a second head. The input to the encoding module can be audio, and the input to the LLM body can be text. Based on the output of the second fusion module, the value of a first loss function (e.g., RNNT loss) can be calculated. Based on the output of the second head, the value of a second loss function (e.g., CE loss) can be calculated.

[0142] In one possible implementation, the audio understanding model undergoes a first-stage training and a second-stage training based on the audio training set; wherein, in the first-stage training, the parameters of the encoding module remain fixed; and in the second-stage training, the parameters of the encoding module are adjusted.

[0143] In this implementation, a two-stage training strategy can be used to optimize the audio understanding model.

[0144] In the first phase of training, the parameters of the encoding module can be kept fixed to leverage the acoustic feature extraction capabilities already learned by the pre-trained encoding module, while parameters of other parts of the audio understanding model (such as the large language model body and the second fusion module) are adjusted. The encoding module in the pre-trained speech recognition model already possesses the ability to extract effective acoustic features from audio signals. Acoustic features can include spectral information, prosodic patterns, etc., providing a foundation for subsequent text generation. Keeping the parameters of the encoding module fixed in the first phase avoids excessive adjustments during the initial training stage, thus preserving its general acoustic feature extraction capabilities learned in the pre-training phase. This ensures that the audio understanding model can still effectively extract acoustic features when processing new audio data. In the first phase, the training focus is on combining the pre-trained large language model body with the encoding module, aligning and fusing acoustic and semantic features through the second fusion module. At this stage, the main updates are to the parameters of the large language model body and the second fusion module, enabling the audio understanding model to better understand and generate text.

[0145] In one example, the audio understanding model obtained from the first stage of training can be called a non-streaming audio understanding model.

[0146] In the second phase of training, the parameters of the encoding module can be unfrozen to optimize its ability to extract local acoustic features. That is, in the second phase, the audio understanding model can simultaneously update the parameters of the encoding module, the large language model body, and the second fusion module to achieve more accurate alignment and fusion of audio and text.

[0147] In one example, the audio understanding model obtained from the second stage of training can be called a streaming audio understanding model.

[0148] In this implementation, phased training avoids gradient conflicts caused by simultaneously optimizing all modules. Progressive training allows the encoding module to transition from global feature extraction to local real-time processing, making it suitable for streaming audio understanding scenarios.

[0149] In one possible implementation, the method further includes: in the second stage of training, dividing the audio sample into multiple audio segments according to a preset duration, and sequentially inputting the multiple audio segments into the encoding module in the audio understanding model; and restricting the audio understanding model to only acquire information about the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment by using an attention mask.

[0150] In this implementation, during the second stage of training, to achieve streaming audio processing capabilities, the audio understanding model can employ a block-based training strategy combined with attention mask control. Specifically, complete audio samples can be divided into consecutive audio segments of a preset duration (e.g., 320ms / block), and input sequentially into the encoding module. For example, a 3-second audio segment (3000ms) can be divided into approximately nine 320ms audio segments (padding with zeros at the end), and processed sequentially. Furthermore, a unidirectional mask can be applied to the self-attention layer of the audio understanding model, allowing the model to access only information from the current and historical audio segments, while completely blocking information from future audio segments. This simulates the causality of streaming scenarios, enabling the audio understanding model to output results in real time based solely on the received audio data.

[0151] The encoding module can learn to extract effective features from local parts (i.e., audio segments) of audio samples, rather than relying on the complete context. Through segmented training, the encoding module can progressively optimize its ability to capture short-term acoustic patterns (such as phonemes and syllables).

[0152] In one possible implementation, during the training of the audio understanding model, the first head undergoes full-parameter training, while the parameters of the large language model body are updated using fine-tuning.

[0153] In this implementation, the first head in the audio understanding model is directly responsible for text generation (such as generating word probability distributions) and needs to quickly adapt to the special requirements of speech recognition tasks (such as adding blank dimensions). By training the first head with all parameters, that is, by completely updating the weights of the first head through gradient descent, the output layer of the audio understanding model can flexibly learn the language patterns after audio-text alignment.

[0154] Large-scale language models, having already acquired general semantic understanding capabilities through pre-training on massive amounts of text, are prone to overfitting when directly trained with all parameters (especially when audio data is limited). This approach employs efficient parameter fine-tuning techniques (such as LoRA and Adapter) to train only a small number of newly added low-rank matrices or adaptation layers, freezing the original parameters of the large-scale language model. This allows for minor adjustments to feature representations to adapt to the audio context while preserving pre-trained knowledge.

[0155] In this implementation, full-parameter training of the first head allows for precise optimization of the output distribution, improving recognition accuracy. Fine-tuning the parameters of the large language model body avoids compromising the core capabilities of the pre-trained language model, reducing training risks. Furthermore, compared to full-model training, it significantly reduces computational overhead (especially when the large language model body has a large number of parameters).

[0156] This disclosure also provides an audio understanding method, comprising: obtaining an audio understanding model trained by the training method of the audio understanding model; inputting the audio to be processed into the audio understanding model; and outputting a text prediction result corresponding to the audio to be processed through the audio understanding model.

[0157] In this embodiment, an optimized audio understanding model can be obtained through the training method of the audio understanding model described above. This audio understanding model integrates the acoustic feature extraction capability of the encoding module in a pre-trained speech recognition model with the semantic understanding capability of a large language model. In practical applications, the audio to be processed (such as real-time speech streams or complete recordings) can be input into the audio understanding model. The audio understanding model extracts acoustic features through the encoding module and combines this with the semantic reasoning of the main body of the large language model. Finally, the second fusion module dynamically aligns the audio and text sequences, outputting the corresponding text prediction results. The audio understanding model provided in this embodiment supports streaming processing, can generate word-by-word text (such as subtitles) in real time, and can also process non-streaming audio, making it suitable for scenarios such as intelligent assistants and meeting transcription.

[0158] In one possible implementation, before inputting the audio to be processed into the audio understanding model, the method further includes: deleting a second head from the audio understanding model.

[0159] In this implementation, the second head is used only during the training phase to assist in optimizing the text generation task (e.g., through cross-entropy loss). Removing the second head reduces redundant computation and lowers inference latency, which is especially crucial for real-time streaming processing (such as voice assistants), while ensuring that the audio understanding model retains only the necessary output paths.

[0160] In one possible implementation, the audio to be processed is either streaming audio or non-streaming audio.

[0161] In this implementation, the audio understanding model can be used for inference of streaming or non-streaming audio, depending on the actual business scenario.

[0162] In one possible implementation, the audio understanding model can use beam search for decoding to find the optimal output path. Beam search is a heuristic search algorithm that maintains a set of candidate solutions at each step. These candidate solutions are sorted according to a scoring mechanism, and typically the solutions with the highest scores are selected as candidates for the next step.

[0163] Figure 4 This diagram illustrates the dynamic alignment process of the audio understanding model provided in the embodiments of this disclosure during the decoding stage. Figure 4 It can demonstrate the streaming alignment mechanism between acoustic feature sequences (time step t) and semantic feature sequences (text step u).

[0164] exist Figure 4 In the diagram, the horizontal axis (t) can represent the time step of the audio input (such as an audio frame sequence), and the acoustic feature A output by the encoding module is... t (Black circle). The vertical axis (u) can represent the number of steps in text generation (such as the words or characters already output), and the semantic features T output by the main body of the large language model. u (White circle). Node y(t,u) can represent the joint decoding result at time step t and text step u, which may be the actual text label (such as Chinese characters, words) or blank. Here, blank indicates that no text has been generated at the current time step, and only the time step is advanced.

[0165] Figure 4 The red arrow in the image shows an example of a decoding path. Horizontal movement (→) indicates time step t+1, with no change in text step u (output blank, e.g.) Vertical movement (↑) indicates that the time step t remains unchanged, and the text step u+1 (outputs the actual text, such as y(3,2)).

[0166] In this way, the audio understanding model can align acoustic features with semantic features in real time when processing streaming audio input and generate corresponding text results. This alignment capability not only improves the interactive experience but also significantly enhances the performance of streaming speech recognition systems.

[0167] The training method and audio understanding method for the audio understanding model provided in this disclosure can be applied to technical fields such as artificial intelligence, speech recognition, large-scale audio understanding models, and streaming speech recognition, and are not limited thereto. Furthermore, the audio understanding model trained using the training method provided in this disclosure can be applied to scenarios such as real-time captioning, voice search, and smart homes, and is not limited thereto.

[0168] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0169] In addition, this disclosure also provides a training device for an audio understanding model, a non-volatile computer-readable storage medium for an audio understanding device, and a computer program product. All of the above can be used to implement the training of any audio understanding model or audio understanding method provided in this disclosure. The corresponding technical solutions and technical effects can be found in the relevant descriptions in the method section, and will not be repeated here.

[0170] Figure 5 A block diagram of a training apparatus for an audio understanding model provided in an embodiment of this disclosure is shown. Figure 5 As shown, the training device for the audio understanding model includes:

[0171] The acquisition module 51 is used to acquire a pre-trained speech recognition model and a pre-trained large language model; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are connected to each other.

[0172] Construction module 52 is used to construct an audio understanding model based on the speech recognition model and the large language model, wherein the audio understanding model includes the encoding module, the large language model body and the second fusion module, and the second fusion module is connected to the encoding module and the large language model body respectively;

[0173] Extraction module 53 is used to extract the acoustic feature sequence corresponding to any audio sample in the audio training set through the encoding module in the audio understanding model, and to extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model.

[0174] The calculation module 54 is used to determine, through the second fusion module, all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and to calculate the total probability of all valid alignment paths;

[0175] The first update module 55 is used to update the parameters of the audio understanding model according to the total probability.

[0176] In one possible implementation, the second fusion module includes the first fusion module and the first header, wherein the first header is initialized based on the large language model header, and the first header adds the dimension corresponding to the blank symbol to the large language model header.

[0177] In one possible implementation, the device further includes:

[0178] The control module is used to control the output dimension of the first fusion module to be the same as the input dimension of the large language model head during the pre-training process.

[0179] In one possible implementation, the first update module 55 is used to:

[0180] Taking the negative logarithm of the total probability yields the value of the first loss function corresponding to the audio understanding model;

[0181] The parameters of the audio understanding model are updated based on the value of the first loss function.

[0182] In one possible implementation, the audio understanding model further includes a second head, which is connected to the large language model body, and the second head is initialized according to the large language model head.

[0183] The device further includes:

[0184] The second output module is used to output the text prediction result corresponding to the audio sample through the second head;

[0185] The determination module is used to determine the value of the second loss function corresponding to the audio understanding model based on the text prediction result corresponding to the audio sample and the target text label sequence;

[0186] The second update module is used to update the parameters of the audio understanding model based on the value of the second loss function.

[0187] In one possible implementation, the parameters of the second head remain fixed during the training of the audio understanding model.

[0188] In one possible implementation, the audio understanding model performs a first-stage training and a second-stage training based on the audio training set;

[0189] In the first training phase, the parameters of the encoding module remain fixed; in the second training phase, the parameters of the encoding module are adjusted.

[0190] In one possible implementation, the device further includes:

[0191] The segmentation module is used to segment the audio samples into multiple audio segments according to a preset duration during the second stage of training, and to input the multiple audio segments into the encoding module in the audio understanding model in sequence.

[0192] The restriction module is used to restrict the audio understanding model to only acquire information about the current audio segment and historical audio segments preceding the current audio segment when processing the current audio segment, by means of an attention mask.

[0193] In one possible implementation, during the training of the audio understanding model, the first head undergoes full-parameter training, while the parameters of the large language model body are updated using fine-tuning.

[0194] In one possible implementation, the speech recognition model employs a recurrent neural network transcriber.

[0195] According to another aspect of this disclosure, an audio understanding apparatus is provided, comprising:

[0196] The acquisition module is used to acquire the audio understanding model trained by the training method of the audio understanding model;

[0197] The first output module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

[0198] In one possible implementation, the device further includes:

[0199] The deletion module is used to delete the second head in the audio understanding model.

[0200] In one possible implementation, the audio to be processed is either streaming audio or non-streaming audio.

[0201] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and technical effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0202] According to another aspect of this disclosure, a training apparatus for an audio understanding model is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0203] According to another aspect of this disclosure, an audio understanding apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0204] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0205] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0206] Figure 6 This is a block diagram illustrating a training apparatus or audio understanding apparatus 1900 for an audio understanding model according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 6 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0207] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0208] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0209] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0210] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0211] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0212] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0213] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0214] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0215] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0216] Computer program products can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0217] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0218] If the technical solution of this disclosure involves personal information, the product applying the technical solution of this disclosure has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this disclosure involves sensitive personal information, the product applying the technical solution of this disclosure has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to indicate that the user has entered the scope of personal information collection and that personal information will be collected. If the user voluntarily enters the collection scope, it is deemed to have consented to the collection of their personal information; or on the personal information processing device, with clear signs / information informing the user of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0219] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.< / eos> < / eos> < / eos> < / sos> < / sos>

Claims

1. A training method for an audio understanding model, characterized in that, include: A pre-trained speech recognition model and a pre-trained large language model are obtained; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are interconnected; An audio understanding model is constructed based on the speech recognition model and the large language model, wherein the audio understanding model includes the encoding module, the large language model body and the second fusion module, and the second fusion module is connected to the encoding module and the large language model body respectively; For any audio sample in the audio training set, the acoustic feature sequence corresponding to the audio sample is extracted through the encoding module in the audio understanding model, and the semantic feature sequence corresponding to the audio sample is extracted through the large language model body in the audio understanding model; The second fusion module determines all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculates the total probability of all valid alignment paths. The parameters of the audio understanding model are updated based on the total probability.

2. The method according to claim 1, characterized in that, The second fusion module includes the first fusion module and the first header, wherein the first header is initialized according to the large language model header, and the first header adds the dimension corresponding to the blank symbol to the large language model header.

3. The method according to claim 2, characterized in that, The method further includes: During pre-training, the output dimension of the first fusion module is controlled to be the same as the input dimension of the large language model head.

4. The method according to any one of claims 1 to 3, characterized in that, Updating the parameters of the audio understanding model based on the total probability includes: Taking the negative logarithm of the total probability yields the value of the first loss function corresponding to the audio understanding model; The parameters of the audio understanding model are updated based on the value of the first loss function.

5. The method according to any one of claims 1 to 3, characterized in that, The audio understanding model also includes a second head, which is connected to the main body of the large language model, and the second head is initialized according to the head of the large language model. The method further includes: The second head outputs the text prediction result corresponding to the audio sample; Based on the text prediction results corresponding to the audio samples and the target text label sequence, determine the value of the second loss function corresponding to the audio understanding model; The parameters of the audio understanding model are updated based on the value of the second loss function.

6. The method according to claim 5, characterized in that, During the training of the audio understanding model, the parameters of the second head remain fixed.

7. The method according to any one of claims 1 to 3, characterized in that, The audio understanding model is trained in the first and second stages based on the audio training set. In the first training phase, the parameters of the encoding module remain fixed; in the second training phase, the parameters of the encoding module are adjusted.

8. The method according to claim 7, characterized in that, The method further includes: In the second stage of training, the audio samples are divided into multiple audio segments according to a preset duration, and the multiple audio segments are sequentially input into the encoding module of the audio understanding model; By using an attention mask, the audio understanding model is restricted to only acquiring information about the current audio segment and historical audio segments preceding it when processing the current audio segment.

9. The method according to claim 2 or 3, characterized in that, During the training process of the audio understanding model, the first head undergoes full-parameter training, while the parameters of the large language model body are updated using a fine-tuning method.

10. The method according to any one of claims 1 to 3, characterized in that, The speech recognition model uses a recurrent neural network transcriber.

11. An audio understanding method, characterized in that, include: Obtain the audio understanding model trained by the training method of the audio understanding model as described in any one of claims 1 to 10; The audio to be processed is input into the audio understanding model, and the audio understanding model outputs the text prediction result corresponding to the audio to be processed.

12. The method according to claim 11, characterized in that, Before inputting the audio to be processed into the audio understanding model, the method further includes: Remove the second head from the audio understanding model.

13. The method according to claim 11 or 12, characterized in that, The audio to be processed is either streaming audio or non-streaming audio.

14. A training device for an audio understanding model, characterized in that, include: An acquisition module is used to acquire a pre-trained speech recognition model and a pre-trained large language model; wherein, the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are interconnected. A construction module is used to construct an audio understanding model based on the speech recognition model and the large language model, wherein the audio understanding model includes the encoding module, the large language model body and the second fusion module, and the second fusion module is connected to the encoding module and the large language model body respectively; The extraction module is used to extract the acoustic feature sequence corresponding to any audio sample in the audio training set through the encoding module in the audio understanding model, and to extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model. The calculation module is used to determine, through the second fusion module, all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and to calculate the total probability of all valid alignment paths; The first update module is used to update the parameters of the audio understanding model based on the total probability.

15. An audio understanding device, characterized in that, include: The acquisition module is used to acquire the audio understanding model trained by the training device of the audio understanding model as described in claim 14; The first output module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

16. A training apparatus for an audio understanding model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

17. An audio understanding device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 11 to 13.

18. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

19. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images