Audio understanding model training method, audio understanding method, device, storage medium and program product

Abstract

The present disclosure relates to the field of speech recognition technology, and in particular to a training method for an audio understanding model, an audio understanding method, an apparatus, a storage medium, and a program product. The method comprises: obtaining a speech recognition model and a large language model; the speech recognition model comprises a coding module, a prediction module, and a first fusion module; constructing an audio text classification model according to the coding module, the large language model body, the first fusion module, and the binary classification layer; the first fusion module and the binary classification layer obtain modal prediction results according to the acoustic features output by the coding module or the semantic features output by the large language model body; determining the inverted gradient value of the first fusion module according to the true modality and updating the parameters of the first fusion module; in response to meeting a preset condition, constructing and training an audio understanding model according to the coding module, the large language model body, and the second fusion module, wherein the second fusion module comprises the first fusion module. The present disclosure can improve the accuracy of audio understanding.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a training method for an audio understanding model, an audio understanding method, a training device for an audio understanding model, an audio understanding device, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] The application of speech recognition technology has expanded across numerous industries, but its effectiveness depends heavily on the quality and diversity of the training data used. Because modern speech recognition systems must handle complex acoustic variables, including accents, speaking speeds, and background noise, obtaining accurate, diverse, and sufficient training data is crucial for building effective speech recognition systems. Furthermore, to enhance model performance in specific vertical industries, detailed annotation of speech data from those scenarios is essential.

[0003] In this context, the use of large language models (LLMs) for audio understanding model training is particularly important. Large language models, pre-trained on massive amounts of text data, have demonstrated their powerful capabilities in language understanding and generation. In contrast, traditional models often rely on large amounts of training data to achieve acceptable performance. This reliance on big data increases resource acquisition challenges, particularly when working with data-scarce languages. However, due to their powerful pre-training capabilities, large language models can achieve good results in specific audio understanding tasks even with limited data. Furthermore, traditional models typically have a small number of parameters, which limits their learning capabilities and ability to handle complex problems, resulting in poor robustness. In contrast, large language models have a larger number of parameters, enabling them to demonstrate greater robustness and adaptability to speech data in complex, unknown, or noisy environments. Traditionally, separate models have been trained and deployed for different source and target languages. This not only increases the complexity of model management but also makes developing high-quality models for every language combination unrealistic given limited resources. Large language models, especially those pre-trained in multiple languages, can more easily adapt to different languages, reducing the need for large amounts of language-specific data. Finally, traditional models are often limited in their effectiveness, especially when dealing with complex linguistic phenomena. Large language models, with their powerful language understanding and generation capabilities, can achieve better results across a wider range of speech tasks.

[0004] Speech recognition systems are generally categorized into two scenarios: non-streaming and streaming, each suited to different business needs. Non-streaming speech recognition involves the system performing offline speech recognition on a complete audio clip uploaded by the user, ultimately outputting the recognition result. Streaming speech recognition, on the other hand, involves the system recognizing the audio simultaneously with the user's speech input, and displaying the results to the user in real time. Because users receive immediate feedback, streaming speech recognition offers advantages in interactive experience. It is therefore commonly used in scenarios such as real-time captioning, voice search, and voice interaction (e.g., in smart homes, in-vehicle devices, and digital humans). Improving the training efficiency and speech recognition performance of large streaming audio understanding models is a key research topic. Summary of the Invention

[0005] The present disclosure provides a technical solution for audio understanding.

[0006] According to one aspect of the present disclosure, a method for training an audio understanding model is provided, comprising:

[0007] Obtaining a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module, and a first fusion module, the first fusion module being connected to the encoding module and the prediction module respectively; and the large language model includes a large language model body and a large language model head that are connected to each other;

[0008] Constructing an audio-text classification model according to the encoding module, the large language model body, the first fusion module, and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body, respectively, and the binary classification layer is connected to the first fusion module;

[0009] Processing the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module and the binary classification layer to obtain a modality prediction result;

[0010] Determining a reverse gradient value of the first fusion module according to the modality prediction result and a true modality corresponding to the modality prediction result, and updating parameters of the first fusion module according to the reverse gradient value of the first fusion module to train the audio text classification model;

[0011] In response to the audio-text classification model being trained to meet preset conditions, an audio understanding model is constructed and trained based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

[0012] In a possible implementation, determining the reverse gradient value of the first fusion module according to the modality prediction result and the true modality corresponding to the modality prediction result includes:

[0013] Determining a value of a classification loss function corresponding to the audio text classification model according to the modality prediction result and the true modality corresponding to the modality prediction result;

[0014] Calculating a gradient value of the first fusion module according to a value of the classification loss function;

[0015] The gradient value of the first fusion module is reversed to obtain a reversed gradient value of the first fusion module.

[0016] In one possible implementation, processing the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module and the binary classification layer to obtain a modality prediction result includes:

[0017] For any audio-text pair in the training set, the encoding module in the audio-text classification model is used to extract the acoustic features corresponding to the audio sample in the audio-text pair, and the large language model body in the audio-text classification model is used to extract the semantic features corresponding to the target text label sequence in the audio-text pair;

[0018] Inputting the acoustic feature or the semantic feature into the first fusion module, and processing the acoustic feature or the semantic feature by the first fusion module to obtain a fusion feature;

[0019] The fusion features are processed by the binary classification layer to obtain a modality prediction result.

[0020] In one possible implementation, during the training of the audio-text classification model, the parameters of the encoding module and the large language model body remain fixed.

[0021] In a possible implementation, the training of the audio understanding model includes:

[0022] For any audio sample in the training set, extract the acoustic feature sequence corresponding to the audio sample through the encoding module in the audio understanding model, and extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model;

[0023] Determining, by the second fusion module, all valid alignment paths that can generate a target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating a total probability of all valid alignment paths;

[0024] Taking the negative logarithm of the total probability to obtain a value of a streaming sequence alignment loss function corresponding to the audio understanding model;

[0025] The parameters of the audio understanding model are updated according to the value of the streaming sequence alignment loss function.

[0026] In a possible implementation, the second fusion module further includes a first head;

[0027] The method further comprises:

[0028] Obtaining a target word table corresponding to the audio understanding model, wherein the word quantity of the target word table is smaller than the word quantity of the original word table corresponding to the large language model;

[0029] Extracting the initial weight corresponding to each word in the target word table from the weight matrix of the head of the large language model according to the position index of each word in the target word table in the original word table;

[0030] The weight matrix of the first head is initialized according to the initial weight corresponding to each word in the target word table.

[0031] In a possible implementation, the first header further includes a dimension corresponding to a blank symbol.

[0032] In one possible implementation, obtaining a target vocabulary corresponding to the audio understanding model includes:

[0033] Perform word frequency statistics on the words in the training set;

[0034] A target word table corresponding to the audio comprehension model is constructed based on a preset number of word units with the highest word frequency in the training set.

[0035] In one possible implementation, the audio understanding model further includes a second head, the second head is connected to the large language model body, and the second head is initialized according to the large language model head;

[0036] The method further comprises:

[0037] Determining a value of a knowledge distillation loss function according to a first text prediction result output by the first head and a second text prediction result output by the second head;

[0038] The parameters of the audio understanding model are updated according to the value of the knowledge distillation loss function.

[0039] In a possible implementation, the method further includes:

[0040] Determining a value of a word unit distribution loss function corresponding to the audio understanding model according to the second text prediction result and a target text label sequence corresponding to the second text prediction result;

[0041] The parameters of the audio understanding model are updated according to the value of the word unit distribution loss function.

[0042] In a possible implementation, during the training of the audio understanding model, parameters of the second head remain fixed.

[0043] In a possible implementation, the audio understanding model performs first-stage training and second-stage training based on a training set;

[0044] In the first stage of training, the parameters of the encoding module remain fixed; in the second stage of training, the parameters of the encoding module are adjusted.

[0045] In one possible implementation, the speech recognition model uses a recurrent neural network transcriber.

[0046] According to one aspect of the present disclosure, there is provided an audio understanding method, comprising:

[0047] Obtain an audio understanding model trained using the audio understanding model training method;

[0048] The audio to be processed is input into the audio understanding model, and the audio understanding model outputs the text prediction result corresponding to the audio to be processed.

[0049] According to one aspect of the present disclosure, a training device for an audio understanding model is provided, comprising:

[0050] A first acquisition module is configured to obtain a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module, and a first fusion module, the first fusion module being connected to the encoding module and the prediction module, respectively; and the large language model includes a large language model body and a large language model head that are connected to each other;

[0051] A construction module, configured to construct an audio-text classification model based on the encoding module, the large language model body, the first fusion module, and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body, respectively, and the binary classification layer is connected to the first fusion module;

[0052] a modality prediction module, configured to process the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module and the binary classification layer to obtain a modality prediction result;

[0053] a first training module, configured to determine a reverse gradient value of the first fusion module according to the modality prediction result and a true modality corresponding to the modality prediction result, and update parameters of the first fusion module according to the reverse gradient value of the first fusion module to train the audio text classification model;

[0054] A second training module is configured to construct an audio understanding model and train the audio understanding model in response to the audio-text classification model being trained to meet preset conditions based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

[0055] In a possible implementation, the first training module is used to:

[0056] Determining a value of a classification loss function corresponding to the audio text classification model according to the modality prediction result and the true modality corresponding to the modality prediction result;

[0057] Calculating a gradient value of the first fusion module according to a value of the classification loss function;

[0058] The gradient value of the first fusion module is reversed to obtain a reversed gradient value of the first fusion module.

[0059] In one possible implementation, the modality prediction module is used to:

[0060] For any audio-text pair in the training set, the encoding module in the audio-text classification model is used to extract the acoustic features corresponding to the audio sample in the audio-text pair, and the large language model body in the audio-text classification model is used to extract the semantic features corresponding to the target text label sequence in the audio-text pair;

[0061] Inputting the acoustic feature or the semantic feature into the first fusion module, and processing the acoustic feature or the semantic feature by the first fusion module to obtain a fusion feature;

[0062] The fusion features are processed by the binary classification layer to obtain a modality prediction result.

[0063] In one possible implementation, during the training of the audio-text classification model, the parameters of the encoding module and the large language model body remain fixed.

[0064] In a possible implementation, the second training module is used to:

[0065] For any audio sample in the training set, extract the acoustic feature sequence corresponding to the audio sample through the encoding module in the audio understanding model, and extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model;

[0066] Determining, by the second fusion module, all valid alignment paths that can generate a target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating a total probability of all valid alignment paths;

[0067] Taking the negative logarithm of the total probability to obtain a value of a streaming sequence alignment loss function corresponding to the audio understanding model;

[0068] The parameters of the audio understanding model are updated according to the value of the streaming sequence alignment loss function.

[0069] In a possible implementation, the second fusion module further includes a first head;

[0070] The device further comprises:

[0071] A second obtaining module is configured to obtain a target word table corresponding to the audio understanding model, wherein the word quantity of the target word table is smaller than the word quantity of the original word table corresponding to the large language model;

[0072] an extraction module, configured to extract, from the weight matrix of the head of the large language model, the initial weight corresponding to each word in the target word table according to the position index of each word in the target word table in the original word table;

[0073] An initialization module is used to initialize the weight matrix of the first head according to the initial weight corresponding to each word in the target word table.

[0074] In a possible implementation, the first header further includes a dimension corresponding to a blank symbol.

[0075] In a possible implementation, the second obtaining module is configured to:

[0076] Perform word frequency statistics on the words in the training set;

[0077] A target word table corresponding to the audio comprehension model is constructed based on a preset number of word units with the highest word frequency in the training set.

[0078] In one possible implementation, the audio understanding model further includes a second head, the second head is connected to the large language model body, and the second head is initialized according to the large language model head;

[0079] The device further comprises:

[0080] a first determining module, configured to determine a value of a knowledge distillation loss function according to a first text prediction result output by the first head and a second text prediction result output by the second head;

[0081] The first updating module is used to update the parameters of the audio understanding model according to the value of the knowledge distillation loss function.

[0082] In a possible implementation, the apparatus further includes:

[0083] A second determination module is configured to determine a value of a word unit distribution loss function corresponding to the audio understanding model based on the second text prediction result and a target text label sequence corresponding to the second text prediction result;

[0084] The second updating module is used to update the parameters of the audio understanding model according to the value of the word unit distribution loss function.

[0085] In a possible implementation, during the training of the audio understanding model, parameters of the second head remain fixed.

[0086] In a possible implementation, the audio understanding model performs first-stage training and second-stage training based on a training set;

[0087] In the first stage of training, the parameters of the encoding module remain fixed; in the second stage of training, the parameters of the encoding module are adjusted.

[0088] In one possible implementation, the speech recognition model uses a recurrent neural network transcriber.

[0089] According to one aspect of the present disclosure, there is provided an audio understanding apparatus, comprising:

[0090] An acquisition module, configured to acquire an audio understanding model trained by the audio understanding model training device;

[0091] The audio understanding module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

[0092] According to one aspect of the present disclosure, a training device for an audio understanding model is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned audio understanding model training method.

[0093] According to one aspect of the present disclosure, an audio understanding device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned audio understanding method.

[0094] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0095] According to one aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0096] In an embodiment of the present disclosure, a pre-trained speech recognition model and a pre-trained large language model are obtained; wherein the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are connected to each other, and an audio text classification model is constructed according to the encoding module, the large language model body, the first fusion module and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body respectively, and the binary classification layer is connected to the first fusion module, and the acoustic features output by the encoding module or the semantic features output by the large language model body are processed by the first fusion module and the binary classification layer to obtain a modal prediction result, and the modal prediction result and the modal prediction result are used to classify the audio text. According to the real mode of the audio text classification model, the reverse gradient value of the first fusion module is determined, and the parameters of the first fusion module are updated according to the reverse gradient value of the first fusion module to train the audio text classification model. In response to the audio text classification model being trained to meet the preset conditions, an audio understanding model is constructed according to the encoding module, the large language model body and the second fusion module, and the audio understanding model is trained, wherein the second fusion module includes the first fusion module. Thus, before training the audio understanding model, the first fusion module for fusing the audio modality and the text modality is pre-trained, and the gradient reversal mechanism is used to enable the first fusion module to learn modality-independent representations, so that the first fusion module can better learn how to balance and fuse the acoustic features from the encoding module with the semantic features of the large language model body, avoiding the problem of oscillation or bias towards a certain modality due to modality differences during training. In addition, the disclosed embodiment utilizes the powerful language understanding and generation capabilities of the pre-trained large language model and its ability to quickly learn through a small amount of audio data to achieve efficient streaming speech recognition, and significantly improve the accuracy of speech recognition even when there is less audio training data.

[0097] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0098] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0100] Figure 1A flowchart of a method for training an audio understanding model provided by an embodiment of the present disclosure is shown.

[0101] Figure 2 Schematic diagram showing a recurrent neural network transcriber.

[0102] Figure 3 A schematic diagram illustrating an audio-text classification model in the audio understanding model training method provided by an embodiment of the present disclosure.

[0103] Figure 4 A schematic diagram of an audio understanding model provided by an embodiment of the present disclosure is shown.

[0104] Figure 5 A schematic diagram illustrating the dynamic alignment process of the audio understanding model provided by an embodiment of the present disclosure during the decoding phase.

[0105] Figure 6 A block diagram of a training device for an audio understanding model provided by an embodiment of the present disclosure is shown.

[0106] Figure 7 It is a block diagram of a training device for an audio understanding model or an audio understanding device according to an exemplary embodiment. DETAILED DESCRIPTION

[0107] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0108] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0109] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0110] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0111] The present disclosure provides a method for training an audio understanding model by obtaining a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are connected to each other, and an audio text classification model is constructed according to the encoding module, the large language model body, the first fusion module and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body respectively, and the binary classification layer is connected to the first fusion module, and the acoustic features output by the encoding module or the semantic features output by the large language model body are processed by the first fusion module and the binary classification layer to obtain a modal prediction result, and the modal prediction result and the large language model head are processed according to the modal prediction result and the The real mode corresponding to the modal prediction result is determined, and the inverted gradient value of the first fusion module is determined, and the parameters of the first fusion module are updated according to the inverted gradient value of the first fusion module to train the audio text classification model. In response to the audio text classification model being trained to meet the preset conditions, an audio understanding model is constructed according to the encoding module, the large language model body and the second fusion module, and the audio understanding model is trained, wherein the second fusion module includes the first fusion module. Thus, before training the audio understanding model, the first fusion module for fusing the audio modality and the text modality is pre-trained, and the gradient reversal mechanism is used to enable the first fusion module to learn a modality-independent representation, so that the first fusion module can better learn how to balance and fuse the acoustic features from the encoding module with the semantic features of the large language model body, avoiding the problem of oscillation or bias towards a certain mode due to modal differences during training. In addition, the embodiment of the present disclosure utilizes the powerful language understanding and generation capabilities of the pre-trained large language model and its ability to quickly learn through a small amount of audio data to achieve efficient streaming speech recognition, and significantly improve the accuracy of speech recognition even when there is less audio training data.

[0112] The following describes in detail the training method of the audio understanding model provided by the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0113] Figure 1A flow chart of the audio understanding model training method provided by an embodiment of the present disclosure is shown. In one possible implementation, the large language model body that executes the audio understanding model training method may be a training device for the audio understanding model. For example, the audio understanding model training method may be executed by a terminal device or a server or other electronic device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the audio understanding model training method may be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 1 As shown, the training method of the audio understanding model includes steps S11 to S15.

[0114] In step S11, a pre-trained speech recognition model and a pre-trained large language model are obtained; wherein the speech recognition model includes an encoding module, a prediction module and a first fusion module, the first fusion module is connected to the encoding module and the prediction module respectively, and the large language model includes a large language model body and a large language model head that are connected to each other.

[0115] In step S12, an audio-text classification model is constructed based on the encoding module, the large language model body, the first fusion module and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body respectively, and the binary classification layer is connected to the first fusion module.

[0116] In step S13, the acoustic features output by the encoding module or the semantic features output by the large language model body are processed by the first fusion module and the binary classification layer to obtain a modality prediction result.

[0117] In step S14, the inverted gradient value of the first fusion module is determined according to the modal prediction result and the true modality corresponding to the modal prediction result, and the parameters of the first fusion module are updated according to the inverted gradient value of the first fusion module to train the audio text classification model.

[0118] In step S15, in response to the audio-text classification model being trained to meet preset conditions, an audio understanding model is constructed and trained based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

[0119] In the embodiments of the present disclosure, a pre-trained speech recognition model may refer to a speech recognition model that has been pre-trained on a speech dataset. By learning and extracting features from speech samples, the speech recognition model can capture the acoustic characteristics, prosodic patterns, and correspondence between speech and text, and convert the input speech into corresponding text content.

[0120] In an embodiment of the present disclosure, the pre-trained speech recognition model may include an encoding module (encoder), a prediction module (predictor), and a first fusion module (jointer). In some application scenarios, the encoding module may also be referred to as an encoder, the prediction module may also be referred to as a predictor, a prediction network, etc., and the first fusion module may also be referred to as a joint network, a joint module, etc., without limitation herein. In one possible implementation, the encoding module may be used to extract acoustic features, the prediction module may be used to extract semantic features, and the first fusion module may be used to fuse acoustic features and semantic features.

[0121] In one possible implementation, the speech recognition model uses a recurrent neural network transducer (RNNT).

[0122] Figure 2 Figure 2 shows a schematic diagram of a recurrent neural network transcriber. Figure 2 As shown, the recurrent neural network transcriber may include an encoding module, a prediction module, a first fusion module, and an activation layer.

[0123] The encoding module can be used to model acoustic features. For example, the encoding module can convert the input audio frame sequence into a high-dimensional acoustic feature representation. For example, the encoding module can input the audio frame at the current time step t , output the acoustic features corresponding to the current time step t Among them, the time step can refer to the smallest discrete time unit of audio processing, corresponding to a single frame of data processed or output by the model at a certain moment. , the encoding module can output the acoustic feature sequence The encoding module can be implemented using a Conformer or LSTM (Long Short-Term Memory) network, which is not limited here.

[0124] The prediction module can be used to model semantic features. During the training phase, the prediction module can input the historical text labels before the current text step u (such as or ), output the semantic features of the current text step u During the prediction phase, the prediction module takes as input the generated (i.e., predicted) text labels and outputs the semantic features of the current text step. A text step refers to a discrete unit in the text generation process, corresponding to a single token processed or output by the model. The prediction module can employ either an LSTM or Transformer architecture, without limitation here.

[0125] The training samples in the training set can consist of audio-text pairs {x, y}, where x represents the audio sample and y represents the target text label sequence (i.e., the annotated text) corresponding to the audio sample. For example, if the target text label sequence y is "What's the weather like today?", the text format input to the prediction module during training can be: <sos>How is the weather today? <sos>The Start Of Sentence is used to initialize the hidden state of the prediction module, indicating the beginning of text generation. The target text label sequence used to calculate the loss can be in the format of: How is the weather today? <eos>.in, <eos>It is the end of sentence symbol. <eos>As part of the training objective, this helps the model learn when to terminate its output.

[0126] The first fusion module can fuse the outputs of the encoding module and the prediction module. For example, the output of the first fusion module can be: The first fusion module may include a linear layer and an activation layer (eg, softmax).

[0127] like Figure 2 As shown, after the first fusion module, an activation layer (such as softmax) can be connected to generate the probability distribution of all possible word units (such as characters, words, and symbols) through this activation layer. .

[0128] In this implementation, the pre-trained speech recognition model, through its recurrent neural network transcriber architecture, effectively achieves real-time dynamic alignment of audio and text sequences, demonstrating significant advantages in streaming speech recognition tasks. This architecture, through an encoding module to extract acoustic features, a prediction module to model text context dependencies, and a first fusion module to fuse multimodal information, offers powerful feature representation capabilities, significantly improving the model's recognition accuracy and real-time performance in complex scenarios.

[0129] In the embodiments of the present disclosure, a pre-trained large language model (LLM) may refer to a model pre-trained on large-scale text data. In some application scenarios, a large language model may also be referred to as a large language model, a large text model, etc., without limitation herein. In one possible implementation, the pre-trained large language model may be an open-source pre-trained large language model, such as Llama or Qwen. These open-source models have been learned and trained on massive amounts of text and have accumulated rich language knowledge and semantic understanding capabilities. In another possible implementation, the pre-trained large language model may be a model trained by the developer themselves.

[0130] Large language models, pre-trained on massive amounts of text data, demonstrate powerful language understanding and generation capabilities. Compared to traditional models that rely on large amounts of data, large language models offer advantages in data-scarce languages ​​and complex speech tasks. Their large number of parameters also makes them more robust and adaptable in noisy environments. Furthermore, large language models pre-trained in multiple languages ​​can reduce the need for language-specific data and ease model management complexity.

[0131] In an embodiment of the present disclosure, the large language model may include a large language model body (LLM-body) and a large language model head (LLM-head) that are interconnected. The large language model body may be composed of a multi-layer neural network (such as a Transformer module), which may be responsible for deep feature extraction and semantic understanding of the input text, and obtain general language representation capabilities through pre-training on massive text data; the large language model head, as the output layer of the large language model, may be composed of a linear layer and an activation layer, and may be responsible for mapping the high-dimensional features extracted by the large language model body to the target output space (such as the probability distribution of the word table). In some application scenarios, the large language model head may also be called the output layer of the large language model.

[0132] In the embodiment of the present disclosure, an audio-text classification model can be constructed based on a pre-trained speech recognition model and a pre-trained large language model. Figure 3 Schematic diagram of the audio text classification model in the training method of the audio understanding model provided by the embodiment of the present disclosure. Figure 3 As shown, the audio-text classification model can include an encoding module from a pre-trained speech recognition model, a large language model (LLM) body from a pre-trained large language model, a first fusion module from the pre-trained speech recognition model, and a binary classification layer. The encoding module can input audio, the large language model body can input text, and the binary classification layer can output modality prediction results.

[0133] In one possible implementation, an audio-text classification model can be built on top of a pre-trained speech recognition model. Specifically, the encoding module and the first fusion module in the pre-trained speech recognition model can be retained, the prediction module in the pre-trained speech recognition model can be replaced with the main body of the large language model, and a binary classification layer can be added after the first fusion module.

[0134] In an embodiment of the present disclosure, during the training process of the audio text classification model, the acoustic features output by the encoding module or the semantic features output by the large language model body can be processed by the first fusion module and the binary classification layer to obtain a modal prediction result. Based on the modal prediction result and the corresponding true modality, the inverted gradient value of the first fusion module is determined, and based on the inverted gradient value of the first fusion module, the parameters of the first fusion module are updated.

[0135] In a possible implementation, the acoustic features output by the encoding module or the semantic features output by the large language model body are processed by the first fusion module and the binary classification layer to obtain a modal prediction result, including: for any audio-text pair in the training set, extracting the acoustic features corresponding to the audio samples in the audio-text pair by the encoding module in the audio-text classification model, and extracting the semantic features corresponding to the target text label sequence in the audio-text pair by the large language model body in the audio-text classification model; inputting the acoustic features or the semantic features into the first fusion module, processing the acoustic features or the semantic features by the first fusion module to obtain a fusion feature; processing the fusion feature by the binary classification layer to obtain a modal prediction result.

[0136] In this implementation, the training set can include multiple audio-text pairs. For any audio-text pair {x, y} in the training set, x can represent an audio sample (e.g., a speech signal) and y can represent the target text label sequence corresponding to the audio sample (e.g., the annotated text content).

[0137] The encoding module in the audio-to-text classification model processes an audio sample x and outputs acoustic features. Acoustic features capture acoustic information such as the audio spectrum and prosody. The large language model in the audio-to-text classification model processes the target text label sequence y and outputs semantic features. Semantic features can include information about the text's context and language structure.

[0138] Either the acoustic feature or the semantic feature may be input into the first fusion module, that is, one of the acoustic feature and the semantic feature may be input into the first fusion module, rather than inputting both the acoustic feature and the semantic feature into the first fusion module at the same time.

[0139] The first fusion module (e.g., a linear layer + activation layer) performs a nonlinear transformation on the input features to generate fused features. Its purpose is to map features from different modalities into the same space. The fused features are then processed by a binary classification layer (e.g., a fully connected layer + softmax), outputting a modality prediction result. The modality prediction result represents the probability of the current feature being classified as originating from the audio or text modality.

[0140] In a possible implementation, determining the reverse gradient value of the first fusion module based on the modal prediction result and the true modality corresponding to the modal prediction result includes: determining the value of the classification loss function corresponding to the audio text classification model based on the modal prediction result and the true modality corresponding to the modal prediction result; calculating the gradient value of the first fusion module based on the value of the classification loss function; and inverting the gradient value of the first fusion module to obtain the reverse gradient value of the first fusion module.

[0141] In one example, if the input of the first fusion module is acoustic features (i.e., the input of the first fusion module comes from the encoding module), the label of the true modality can be 0; if the input of the first fusion module is semantic features (i.e., the input of the first fusion module comes from a large language model body), the label of the true modality can be 1.

[0142] In an example, the cross-entropy loss function can be used to calculate the difference between the modality prediction result and the true modality label to obtain the value of the classification loss function:

[0143]

[0144] Among them, y represents the label of the true mode (0 or 1), and p represents the mode prediction result (i.e., the predicted probability).

[0145] In this implementation, the chain rule can be used to backpropagate the gradient from the classification loss function to the parameters of the first fusion module. Assuming that the parameters of the first fusion module are θ, its gradient can be:

[0146]

[0147] This is the regular gradient descent update direction, which aims to optimize the accuracy of the classification task.

[0148] In order to make the first fusion module generate modality-independent features (i.e., unable to distinguish between audio or text modalities), in this implementation, the gradient direction can be reversed through the Gradient Reversal Layer (GRL):

[0149]

[0150] Here, λ represents the inversion weight (a hyperparameter, which can be set to 1, for example).

[0151] Normal gradient descent minimizes the classification loss (i.e., improves the ability to distinguish between modalities), while gradient reversal is equivalent to maximizing the classification loss, making it impossible for the feature output of the first fusion module to be distinguished by the binary classification layer.

[0152] In this implementation, the inverted gradients can be used to update the parameters of the first fusion module:

[0153]

[0154] where η is the learning rate. Since the gradient is inverted, the actual effect is to encourage the first fusion module to confuse modality features rather than distinguish them.

[0155] In this implementation, through gradient inversion, the first fusion module is forced to generate features that are "indistinguishable" between audio and text modalities, thereby more fairly fusing the two modalities in the subsequent audio understanding model and avoiding training bias towards one modality (such as over-reliance on text semantics while ignoring acoustic features).

[0156] In one possible implementation, during the training of the audio-text classification model, the parameters of the encoding module and the large language model body remain fixed.

[0157] The encoding module of pre-trained speech recognition models (such as RNNT) already has the ability to extract effective acoustic features (such as spectrum and intonation) from audio. Fixing the encoding module's parameters preserves its pre-trained acoustic representation capabilities, preventing degradation of these features during training for new tasks. Pre-trained large-scale language models have already learned universal language representations from massive amounts of text data. Fixing their parameters maintains their semantic understanding capabilities and prevents loss of original language knowledge during fine-tuning for audio tasks.

[0158] In this implementation, only the parameters of the first fusion module and the binary classification layer are trained during the training of the audio-text classification model. These two components are responsible for mapping acoustic and semantic features into the same space and learning a modality-independent fusion representation through a gradient reversal mechanism.

[0159] This approach ensures that the pre-training capabilities of the encoding module and the main body of the large language model are not disrupted by new tasks. Furthermore, it significantly reduces the number of training parameters and computational overhead (especially when the number of LLM parameters is large).

[0160] In an embodiment of the present disclosure, an audio understanding model can be constructed based on a pre-trained speech recognition model, a pre-trained large language model, and an audio text classification model. The audio understanding model may include an encoding module in the pre-trained speech recognition model, a large language model body in the pre-trained large language model, and a second fusion module. The second fusion module may include a first fusion module in the pre-trained audio text classification model. In the audio understanding model, the second fusion module is connected to the encoding module and the large language model body respectively. The second fusion module can be used to perform fusion processing on the output of the encoding module and the output of the large language model body.

[0161] Among them, the encoding module in the audio understanding model can be used to process audio input and output acoustic features The large language model body in the audio understanding model can input text and output semantic features. In the audio understanding model, the text context modeling capability can be enhanced by replacing the prediction module in the pre-trained speech recognition model with a large language model body. The input of the second fusion module in the audio understanding model can include the acoustic features output by the encoding module. and semantic features output by large language model bodies The second fusion module can dynamically align acoustic features and semantic features.

[0162] In one possible implementation, the training of the audio understanding model includes: for any audio sample in the training set, extracting the acoustic feature sequence corresponding to the audio sample through the encoding module in the audio understanding model, and extracting the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model; determining, through the second fusion module, all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating the total probability of all valid alignment paths; taking the negative logarithm of the total probability to obtain the value of the streaming sequence alignment loss function corresponding to the audio understanding model; and updating the parameters of the audio understanding model according to the value of the streaming sequence alignment loss function.

[0163] In this implementation, during the audio understanding model training process, for each audio sample in the training set, the encoding module processes the raw audio signal frame by frame, outputting a sequence of acoustic features. The large language model then autoregressively generates a corresponding sequence of semantic features based on historical text labels. These and other features provide multimodal input for the subsequent second fusion module.

[0164] In this implementation, the second fusion module can determine all valid alignment paths that can generate the target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculate the total probability of all valid alignment paths.

[0165] The alignment path can refer to the mapping relationship between the acoustic feature sequence (time step t) and the semantic feature sequence (text step u). For example:

[0166] Horizontal shift (→): Time step t+1, text step u remains unchanged (corresponding to outputting a blank symbol (blank), indicating that no text has been generated yet).

[0167] Vertical movement (↑): time step t remains unchanged, text step u+1 (corresponding to generating an actual text label).

[0168] A valid alignment path refers to all legal paths from the starting point (t=0, u=0) to the end point (t=T, u=U), where the generated text sequence is completely consistent with the target text label sequence. Here, T is the total time step of the audio sample, and U is the length of the target text label sequence.

[0169] The second fusion module can receive the acoustic feature sequence from the encoding module and a sequence of semantic features from a large corpus of language models , output joint probability distribution through linear layer and activation layer (such as softmax) .in, It can represent the predicted label of the current step.

[0170] The probability of each aligned path is the product of the probabilities of all steps on the path. For example, a path is , then its probability is . Sum the probabilities of all valid paths: .

[0171] For example, if the target text is "AB" and the audio has 3 time steps, the two possible valid paths are:

[0172] Path 1: (t=1)→(t=2)→(t=3↑)→(t=3↑)

[0173] (Output: blank → blank → A → B)

[0174] Path 2: (t=1↑)→(t=2)→(t=3↑)

[0175] (Output: A → blank → B)

[0176] Total probability .

[0177] In this implementation, during the training of the audio understanding model, after calculating the total probability of all valid alignment paths, the total probability can be maximized (i.e., the negative log probability can be minimized) by optimizing the model parameters. Take the negative logarithm to get the value of the streaming sequence alignment loss function The probability range is [0, 1]. After taking the negative logarithm, higher probabilities (closer to 1) correspond to lower loss values ​​(closer to 0); conversely, lower probabilities (closer to 0) correspond to higher loss values ​​(approaching infinity). This transformation transforms the probability maximization problem into the loss minimization problem, which conforms to the gradient descent optimization framework.

[0178] For example, if an audio sample has two valid alignment paths with probabilities of 0.6 and 0.4 respectively, the total probability is , loss value ; If the path probability distribution is not ideal (such as 0.1 and 0.1), the loss value , the audio understanding model will adjust parameters through gradient descent to improve the generation possibility of high-probability paths.

[0179] In this implementation, gradients can be calculated based on the value of the streaming sequence alignment loss function, and backpropagation can be used to update the parameters of the audio understanding model. For example, the parameters of at least some modules in the second fusion module, the large language model body, and the encoding module can be updated.

[0180] In this implementation, based on a streaming sequence alignment loss function, we can directly optimize the alignment path between acoustic and semantic features, ensuring that the text sequences generated by the audio understanding model dynamically match the audio input. By minimizing the streaming sequence alignment loss function, the second fusion module can learn to effectively combine the acoustic features of the encoding module with the semantic features of the large language model body.

[0181] In one possible implementation, the second fusion module also includes a first head; the method also includes: obtaining a target word table corresponding to the audio understanding model, wherein the word quantity of the target word table is smaller than the word quantity of the original word table corresponding to the large language model; extracting the initial weight corresponding to each word in the target word table from the weight matrix of the head of the large language model according to the position index of each word in the target word table in the original word table; initializing the weight matrix of the first head according to the initial weight corresponding to each word in the target word table.

[0182] In this implementation, the second fusion module can be composed of the first fusion module and the first header. The first header can be initialized based on the large language model header. That is, the parameters of the first header can be inherited from the pre-trained large language model header to leverage the existing language modeling capabilities of the pre-trained large language model.

[0183] Large language models typically have massive output layers. For example, their lemmas may contain hundreds of thousands or even more lemmas (e.g., 150,000 lemmas). Such large lemmas are inefficient for speech recognition tasks, as speech recognition typically only focuses on a limited set of lemmas (e.g., Chinese, English, punctuation, and high-frequency words, which may only be around 10,000). Therefore, directly using the complete lemma of a large language model can lead to low training and inference efficiency, as well as wasted resources.

[0184] In this implementation, a target token table (tiny vocab) is constructed to reduce the dimension of the output layer, thereby optimizing the training and inference efficiency of the audio understanding model. Among them, the number of tokens in the target token table is less than that in the original token table corresponding to the large language model. For example, the number of tokens in the original token table corresponding to the large language model is 150,000, and the number of tokens in the target token table is 10,000.

[0185] In this implementation, the position indices of the tokens in the target token table in the original token table corresponding to the large language model can be recorded to form a mapping relationship (map), for example: {v0: V5000, v1: V9000,..., v9999: V142900}. Among them, "v0: V5000" means that index 0 (v0) in the target token table corresponds to index 5000 (V5000) in the original token table, and both represent the same token, such as "Hello". Based on this mapping relationship, the weight matrix corresponding to the target token table can be initialized. For example, the dimension of the weight matrix of the softmax layer in the large language model head is (2048, 150000), then the dimension of the weight matrix of the softmax layer in the first head will be (2048, 10000). Among them, each row in the weight matrix corresponding to the target token table corresponds to the row with the corresponding index in the weight matrix corresponding to the original token table. For example, the first row weight of the target token table is initialized using the 5000th row of the weight matrix of the original token table, and so on. In this way, the target token table inherits the pre-trained weights of the large language model, while significantly reducing the output dimension and optimizing the training and inference efficiency of the audio understanding model.

[0186] In a possible implementation, an audio understanding model can be constructed based on the audio text classification model. Specifically, the encoding module, the large language model body, and the first fusion module in the pre-trained audio text classification model can be retained, and a first head can be connected after the first fusion module to obtain a second fusion module.

[0187] In a possible implementation, the first head further includes a dimension corresponding to the blank symbol.

[0188] In this implementation, a dimension corresponding to the blank symbol (blank) is added to the large language model head to obtain the first head. This is to adapt to the streaming alignment requirements of the speech recognition model (such as the recurrent neural network transducer), because the speech recognition model needs to process the blank symbol (indicating no output or waiting for subsequent input) during decoding.

[0189] In this implementation, by reusing the parameters of the large language model's head, the language generation ability of the large language model is retained, and at the same time, the alignment mechanism unique to the speech recognition model (such as the recurrent neural network transducer) is supported by expanding the dimension. The introduction of blank symbols enables the audio understanding model to dynamically align audio with text (for example, moving the path to the right represents a blank, and moving up represents generating a character), thereby achieving streaming decoding.

[0190] In one possible implementation, obtaining the target vocabulary corresponding to the audio understanding model includes: performing word frequency statistics on the tokens in the training set; constructing the target token table corresponding to the audio understanding model according to a preset number of tokens with the highest word frequencies in the training set.

[0191] In this implementation, all text data in the training set (such as all target text label sequences) can be tokenized, and the occurrence frequency of each token is counted.

[0192] As an example of this implementation, the tokenizer of the large language model can be used to tokenize the text data in the training set. For example, the Chinese text "你好,今天天气如何” can be decomposed by the tokenizer into "你好|,|今天天气|如何|”, and the English text "good morning” can be decomposed into "good| |mor@|ning|”. Here, each part separated by a vertical bar "|” represents a single token. In this way, the text data can be converted into discrete units that the audio understanding model can process, providing a basis for subsequent word frequency statistics and the construction of the target token table.

[0193] In this implementation, according to the word frequency statistics results, a preset number of tokens with the highest frequencies (such as 10,000) can be selected to construct the target token table. These tokens are usually the most commonly used tokens in the speech recognition task.

[0194] In one possible implementation, the audio understanding model further includes a second head, the second head is connected to the large language model body, and the second head is initialized according to the large language model head; the method further includes: determining the value of the knowledge distillation loss function according to the first text prediction result output by the first head and the second text prediction result output by the second head; updating the parameters of the audio understanding model according to the value of the knowledge distillation loss function.

[0195] In this implementation, the second head can directly reuse the original head structure of a pre-trained large language model, for example, including linear layers and softmax layers, to inherit the powerful language generation capabilities of the large language model. The parameters of the second head are copied from the large language model head, but are independent of the first head, forming a dual-task output branch during the training of the audio understanding model. Furthermore, the second head can be connected to the main body of the large language model and can directly generate a second text prediction result (such as the probability distribution of word units) corresponding to the audio sample based on the semantic features output by the large language model body.

[0196] In this implementation, the output layer of the second head has a larger dimension (e.g., 150,000), which provides strong text generation capabilities. The output layer of the first head has a smaller dimension (e.g., 10,000), which is suitable for speech recognition tasks. Through knowledge distillation, the first head can learn the output distribution of the second head, thereby inheriting the knowledge of the large language model.

[0197] Based on the mapping relationship between the target word table and the original word table, the predicted probability distribution of each word in the target word table can be extracted from the second text prediction result output by the second head. Based on the predicted probability distribution of each word in the target word table in the second text prediction result, and the predicted probability distribution of each word in the target word table in the first text prediction result, the value of the knowledge distillation loss function can be determined. In one example, the value of the knowledge distillation loss function can be determined by calculating the KL divergence (Kullback-Leibler Divergence). The KL divergence measures the difference between the two probability distributions. By minimizing this difference, the output of the first head can be made closer to the output of the second head. In this way, the audio understanding model can inherit the powerful language generation capabilities of a large language model while maintaining efficient reasoning.

[0198] In one possible implementation, the method further includes: determining the value of the word unit distribution loss function corresponding to the audio understanding model based on the second text prediction result and the target text label sequence corresponding to the second text prediction result; and updating the parameters of the audio understanding model based on the value of the word unit distribution loss function.

[0199] In this implementation, by calculating the word distribution loss function (such as cross entropy loss) between the second text prediction result and the true label (i.e., the target text label sequence), additional supervision signals can be provided to enhance language modeling capabilities.

[0200] In a possible implementation, during the training of the audio understanding model, parameters of the second head remain fixed.

[0201] In this implementation, the parameters of the second head can be kept fixed during audio understanding model training (i.e., it does not participate in gradient updates). This allows the incorporation of pre-trained knowledge from the large language model while preventing its text generation capabilities from being disrupted by the speech alignment task. The second head directly reuses the parameters of the original head of the large language model, freezing its weights to ensure that the audio understanding model retains the powerful language modeling capabilities of the large language model. This design not only leverages the prior knowledge of the large language model to improve text prediction quality, but also prevents parameter conflicts during multi-task training, resulting in more stable convergence of the audio understanding model.

[0202] Figure 4 Schematic diagram of the audio understanding model provided by the embodiment of the present disclosure is shown. Figure 4 As shown, the audio understanding model may include an encoding module, an LLM body, a first fusion module, a first header, and a second header. The encoding module may receive audio input, and the LLM body may receive text input. Based on the output of the first header, the value of the streaming sequence alignment loss function may be calculated. Based on the output of the first and second headers, the value of the knowledge distillation loss function may be calculated. Based on the output of the second header, the value of the word unit distribution loss function may be calculated.

[0203] In one possible implementation, the audio understanding model performs first-stage training and second-stage training based on a training set; wherein, in the first-stage training, the parameters of the encoding module remain fixed; and in the second-stage training, the parameters of the encoding module are adjusted.

[0204] In this implementation, a two-stage training strategy can be used to optimize the audio understanding model.

[0205] During the first phase of training, the parameters of the encoding module can remain fixed to leverage the acoustic feature extraction capabilities already learned by the pre-trained encoding module, while parameters of other components of the audio understanding model (such as the main large language model and the second fusion module) are adjusted. The encoding module in the pre-trained speech recognition model already has the ability to extract effective acoustic features from audio signals. Acoustic features, such as spectral information and prosodic patterns, can provide a foundation for subsequent text generation. By keeping the encoding module parameters fixed during the first phase, excessive adjustments to the encoding module during initial training are avoided, preserving the general acoustic feature extraction capabilities learned during pre-training. This ensures that the audio understanding model can still effectively extract acoustic features when processing new audio data. In the first phase, training focuses on combining the main pre-trained large language model with the encoding module, aligning and fusing acoustic and semantic features through the second fusion module. At this stage, the parameters of the main large language model and the second fusion module are primarily updated to enable the audio understanding model to better understand and generate text.

[0206] In one example, the audio understanding model trained in the first stage can be called a non-streaming audio understanding model.

[0207] In the second phase of training, the encoding module parameters can be unfrozen to optimize its ability to extract local acoustic features. In other words, in the second phase, the audio understanding model can simultaneously update the parameters of the encoding module, the main large language model, and the second fusion module to achieve more precise alignment and fusion of audio and text.

[0208] In one example, the audio understanding model trained in the second stage can be called a streaming audio understanding model.

[0209] This implementation avoids gradient conflicts caused by optimizing all modules simultaneously through phased training. Through progressive training, the encoding module transitions from global feature extraction to local real-time processing, making it suitable for streaming audio understanding scenarios.

[0210] In one possible implementation, the method further includes: in the second stage training, dividing the audio sample into multiple audio segments according to a preset duration, and inputting the multiple audio segments into the encoding module in the audio understanding model in sequence; and limiting the audio understanding model through an attention mask so that when processing the current audio segment, it can only obtain information about the current audio segment and historical audio segments before the current audio segment.

[0211] In this implementation, in the second stage of training, in order to achieve streaming audio processing capabilities, the audio understanding model can adopt a block training strategy combined with attention mask control. Specifically, the complete audio sample can be divided into continuous audio segments according to a preset duration (such as 320ms / block), and input into the encoding module in sequence. For example, a 3-second audio segment (3000ms) can be divided into approximately 9 320ms audio segments (the missing part at the end is padded with zeros) and processed in sequence. In addition, in the self-attention layer of the audio understanding model, a unidirectional mask can be applied so that the audio understanding model can only access the information of the current audio segment and historical audio segments, while the information of future audio segments is completely blocked. In this way, the causality of the streaming scenario can be simulated, so that the audio understanding model can only rely on the received audio data to output results in real time.

[0212] The encoding module can learn to extract effective features from localized audio samples (i.e., audio segments), rather than relying on the full context. Through segmented training, the encoding module can gradually optimize its ability to capture short-term acoustic patterns (such as phonemes and syllables).

[0213] In a possible implementation, during the training of the audio understanding model, the first head performs full parameter training, and the parameters of the large language model body are updated by fine-tuning.

[0214] In this implementation, the first head in the audio understanding model is directly responsible for text generation (e.g., generating word-unit probability distributions) and must quickly adapt to the specific requirements of speech recognition tasks (e.g., increasing the dimension of whitespace symbols). By fully training the first head—that is, completely updating its weights via gradient descent—the output layer of the audio understanding model can flexibly learn the language patterns after audio-text alignment.

[0215] Large language models have been pre-trained on massive amounts of text to develop general semantic understanding capabilities. Direct full-parameter training can easily lead to overfitting (especially when audio data is limited). In this implementation, efficient parameter fine-tuning techniques (such as LoRA and Adapter) can be used to train only a small number of newly added low-rank matrices or adaptation layers, freezing the original parameters of the large language model. This allows for subtle adjustments to feature representations to adapt to the audio context while preserving pre-trained knowledge.

[0216] In this implementation, full parameter training of the first head allows for precise optimization of the output distribution and improved recognition accuracy. Fine-tuning the parameters of the large language model core avoids damaging the core capabilities of the pre-trained language model and reduces training risk. Furthermore, compared to full model training, this significantly reduces computational overhead (especially when the large language model core has a large number of parameters).

[0217] The embodiment of the present disclosure also provides an audio understanding method, including: obtaining an audio understanding model trained by the audio understanding model training method; inputting the audio to be processed into the audio understanding model, and outputting a text prediction result corresponding to the audio to be processed through the audio understanding model.

[0218] In the embodiment of the present disclosure, an optimized audio understanding model can be obtained by the training method of the audio understanding model described above. The audio understanding model integrates the acoustic feature extraction capability of the encoding module in the pre-trained speech recognition model, the semantic understanding capability of the large language model, and the ability of the first fusion module to fuse audio modalities and text modalities. In actual application, the audio to be processed (such as a real-time voice stream or a complete recording) can be input into the audio understanding model. The audio understanding model extracts acoustic features through the encoding module and combines it with the semantic reasoning of the large language model body. Finally, the second fusion module dynamically aligns the audio and text sequences and outputs the corresponding text prediction results. The audio understanding model provided by the embodiment of the present disclosure supports streaming processing, can generate verbatim text (such as subtitles) in real time, and can also process non-streaming audio. It is suitable for scenarios such as smart assistants and conference transcription.

[0219] In one possible implementation, before inputting the processed audio into the audio understanding model, the method further includes deleting the second header in the audio understanding model. In this implementation, the second header is used only during the training phase to assist in optimizing the text generation task. Removing the second header reduces redundant computation and lowers inference latency, which is particularly critical for real-time streaming processing (such as voice assistants). It also ensures that the audio understanding model retains only necessary output paths.

[0220] In one possible implementation, the audio to be processed is streaming audio or non-streaming audio. In this implementation, the audio understanding model can be used for reasoning about streaming audio or non-streaming audio according to actual business scenarios.

[0221] In one possible implementation, the audio understanding model can use beam search for decoding to find the optimal output path. Beam search is a heuristic search algorithm that maintains a set of candidate solutions at each step. These candidate solutions are ranked according to a scoring mechanism, and the top-scoring solutions are typically selected as candidates for the next step.

[0222] Figure 5 A schematic diagram illustrating the dynamic alignment process of the audio understanding model provided by an embodiment of the present disclosure during the decoding phase. Figure 5 It can demonstrate a streaming alignment mechanism between a sequence of acoustic features (time step t) and a sequence of semantic features (text step u).

[0223] exist Figure 5 In the example, the horizontal axis (t) can represent the time step of the audio input (such as the audio frame sequence), and the acoustic feature A output by the encoding module is t (black circle). The vertical axis (u) can represent the number of steps of text generation (such as words or characters that have been output), and the semantic features T output by the large language model body u (white circle). The node y(t,u) can represent the joint decoding result at time step t and text step u. It can be an actual text label (such as a Chinese character or word) or a blank symbol (∅). The blank symbol indicates that no text has been generated at the current time step and only the time step is advanced.

[0224] Figure 5 The red arrow in the figure shows an example of a decoding path. A horizontal movement (→) indicates that time step t+1 remains unchanged, with the text step u unchanged (outputting a blank symbol, such as ∅(1,2)). A vertical movement (↑) indicates that time step t remains unchanged, with the text step u+1 unchanged (outputting the actual text, such as y(3,2)).

[0225] In this way, the audio understanding model can align acoustic features with semantic features in real time when processing streaming audio input and generate corresponding text results. This alignment capability not only improves the interactive experience but also significantly enhances the performance of the streaming speech recognition system.

[0226] The audio understanding model training method and audio understanding method provided in the embodiments of the present disclosure can be applied to technical fields such as artificial intelligence, speech recognition, large audio understanding models, and streaming speech recognition, without limitation herein. Furthermore, the audio understanding model trained using the audio understanding model training method provided in the embodiments of the present disclosure can be applied to scenarios such as real-time subtitles, voice search, and smart homes, without limitation herein.

[0227] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0228] In addition, the present disclosure also provides a training device for an audio understanding model, a non-volatile computer-readable storage medium for an audio understanding device, and a computer program product. The above can all be used to implement the training of any audio understanding model or audio understanding method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method section and will not be repeated here.

[0229] Figure 6 FIG. 1 is a block diagram of a training device for an audio understanding model provided by an embodiment of the present disclosure. Figure 6 As shown, the training device of the audio understanding model includes:

[0230] A first acquisition module 61 is configured to obtain a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module, and a first fusion module, the first fusion module being connected to the encoding module and the prediction module, respectively; and the large language model includes a large language model body and a large language model head that are connected to each other.

[0231] A construction module 62 is configured to construct an audio-text classification model based on the encoding module, the large language model body, the first fusion module, and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body, respectively, and the binary classification layer is connected to the first fusion module;

[0232] a modality prediction module 63, configured to process the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module and the binary classification layer to obtain a modality prediction result;

[0233] a first training module 64, configured to determine a reverse gradient value of the first fusion module according to the modality prediction result and a true modality corresponding to the modality prediction result, and update parameters of the first fusion module according to the reverse gradient value of the first fusion module to train the audio text classification model;

[0234] The second training module 65 is used to construct an audio understanding model and train the audio understanding model in response to the audio-text classification model being trained to meet preset conditions based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

[0235] In a possible implementation, the first training module 64 is configured to:

[0236] Determining a value of a classification loss function corresponding to the audio text classification model according to the modality prediction result and the true modality corresponding to the modality prediction result;

[0237] Calculating a gradient value of the first fusion module according to a value of the classification loss function;

[0238] The gradient value of the first fusion module is reversed to obtain a reversed gradient value of the first fusion module.

[0239] In a possible implementation, the modality prediction module 63 is configured to:

[0240] For any audio-text pair in the training set, the encoding module in the audio-text classification model is used to extract the acoustic features corresponding to the audio sample in the audio-text pair, and the large language model body in the audio-text classification model is used to extract the semantic features corresponding to the target text label sequence in the audio-text pair;

[0241] Inputting the acoustic feature or the semantic feature into the first fusion module, and processing the acoustic feature or the semantic feature by the first fusion module to obtain a fusion feature;

[0242] The fusion features are processed by the binary classification layer to obtain a modality prediction result.

[0243] In one possible implementation, during the training of the audio-text classification model, the parameters of the encoding module and the large language model body remain fixed.

[0244] In a possible implementation, the second training module 65 is configured to:

[0245] For any audio sample in the training set, extract the acoustic feature sequence corresponding to the audio sample through the encoding module in the audio understanding model, and extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model;

[0246] Determining, by the second fusion module, all valid alignment paths that can generate a target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating a total probability of all valid alignment paths;

[0247] Taking the negative logarithm of the total probability to obtain a value of a streaming sequence alignment loss function corresponding to the audio understanding model;

[0248] The parameters of the audio understanding model are updated according to the value of the streaming sequence alignment loss function.

[0249] In a possible implementation, the second fusion module further includes a first head;

[0250] The device further comprises:

[0251] A second obtaining module is configured to obtain a target word table corresponding to the audio understanding model, wherein the word quantity of the target word table is smaller than the word quantity of the original word table corresponding to the large language model;

[0252] an extraction module, configured to extract, from the weight matrix of the head of the large language model, the initial weight corresponding to each word in the target word table according to the position index of each word in the target word table in the original word table;

[0253] An initialization module is used to initialize the weight matrix of the first head according to the initial weight corresponding to each word in the target word table.

[0254] In a possible implementation, the first header further includes a dimension corresponding to a blank symbol.

[0255] In a possible implementation, the second obtaining module is configured to:

[0256] Perform word frequency statistics on the words in the training set;

[0257] A target word table corresponding to the audio comprehension model is constructed based on a preset number of word units with the highest word frequency in the training set.

[0258] In one possible implementation, the audio understanding model further includes a second head, the second head is connected to the large language model body, and the second head is initialized according to the large language model head;

[0259] The device further comprises:

[0260] a first determining module, configured to determine a value of a knowledge distillation loss function according to a first text prediction result output by the first head and a second text prediction result output by the second head;

[0261] The first updating module is used to update the parameters of the audio understanding model according to the value of the knowledge distillation loss function.

[0262] In a possible implementation, the apparatus further includes:

[0263] A second determination module is configured to determine a value of a word unit distribution loss function corresponding to the audio understanding model based on the second text prediction result and a target text label sequence corresponding to the second text prediction result;

[0264] The second updating module is used to update the parameters of the audio understanding model according to the value of the word unit distribution loss function.

[0265] In a possible implementation, during the training of the audio understanding model, parameters of the second head remain fixed.

[0266] In a possible implementation, the audio understanding model performs first-stage training and second-stage training based on a training set;

[0267] In the first stage of training, the parameters of the encoding module remain fixed; in the second stage of training, the parameters of the encoding module are adjusted.

[0268] In one possible implementation, the speech recognition model uses a recurrent neural network transcriber.

[0269] According to one aspect of the present disclosure, there is provided an audio understanding apparatus, comprising:

[0270] An acquisition module, configured to acquire an audio understanding model trained by the audio understanding model training device;

[0271] The audio understanding module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

[0272] An embodiment of the present disclosure also provides a training device for an audio understanding model, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned audio understanding model training method.

[0273] An embodiment of the present disclosure further provides an audio understanding device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned audio understanding method.

[0274] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0275] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the above method. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

[0276] The embodiment of the present disclosure further provides a computer program, comprising a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0277] An embodiment of the present disclosure further provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0278] Figure 7 1 is a block diagram of an audio understanding model training device or an audio understanding device 1 according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 7 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0279] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0280] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0281] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted via wires.

[0282] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device for storage.

[0283] The computer program (or computer program instructions) used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0284] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0285] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0286] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0287] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0288] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0289] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0290] If the technical solutions of the embodiments of this disclosure involve personal information, the products that apply the technical solutions of the embodiments of this disclosure have clearly informed the individual of the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solutions of the embodiments of this disclosure involve sensitive personal information, the products that apply the technical solutions of the embodiments of this disclosure have obtained the individual's separate consent before processing the sensitive personal information and simultaneously meet the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform the individual that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, while using obvious signs / information to inform the individual of the personal information processing rules, the individual's authorization is obtained through pop-up messages or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0291] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.< / eos> < / eos> < / eos> < / sos> < / sos>

Claims

1. A method for training an audio comprehension model, characterized in that: include: Obtaining a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module, and a first fusion module, the first fusion module being connected to the encoding module and the prediction module respectively; and the large language model includes a large language model body and a large language model head that are connected to each other; Constructing an audio-text classification model according to the encoding module, the large language model body, the first fusion module, and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body, respectively, and the binary classification layer is connected to the first fusion module; Processing the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module to obtain fused features; Processing the fusion features through the binary classification layer to obtain a modality prediction result; Determining a reverse gradient value of the first fusion module according to the modality prediction result and a true modality corresponding to the modality prediction result, and updating parameters of the first fusion module according to the reverse gradient value of the first fusion module to train the audio text classification model; In response to the audio-text classification model being trained to meet preset conditions, an audio understanding model is constructed and trained based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

2. The method according to claim 1, characterized in that The determining, according to the modal prediction result and the true modality corresponding to the modal prediction result, a reverse gradient value of the first fusion module includes: Determining a value of a classification loss function corresponding to the audio text classification model according to the modality prediction result and the true modality corresponding to the modality prediction result; Calculating a gradient value of the first fusion module according to a value of the classification loss function; The gradient value of the first fusion module is reversed to obtain a reversed gradient value of the first fusion module.

3. The method according to claim 1, characterized in that The processing of the acoustic features output by the encoding module or the semantic features output by the large language model body by the first fusion module to obtain fused features includes: For any audio-text pair in the training set, the encoding module in the audio-text classification model is used to extract the acoustic features corresponding to the audio sample in the audio-text pair, and the large language model body in the audio-text classification model is used to extract the semantic features corresponding to the target text label sequence in the audio-text pair; The acoustic features or the semantic features are input into the first fusion module, and the acoustic features or the semantic features are processed by the first fusion module to obtain the fusion features.

4. The method according to claim 1, wherein During the training of the audio-text classification model, the parameters of the encoding module and the large language model body remain fixed.

5. The method according to claim 1, wherein The training of the audio understanding model comprises: For any audio sample in the training set, extract the acoustic feature sequence corresponding to the audio sample through the encoding module in the audio understanding model, and extract the semantic feature sequence corresponding to the audio sample through the large language model body in the audio understanding model; Determining, by the second fusion module, all valid alignment paths that can generate a target text label sequence corresponding to the audio sample based on the acoustic feature sequence and the semantic feature sequence, and calculating a total probability of all valid alignment paths; Taking the negative logarithm of the total probability to obtain a value of a streaming sequence alignment loss function corresponding to the audio understanding model; The parameters of the audio understanding model are updated according to the value of the streaming sequence alignment loss function.

6. The method according to claim 1, characterized in that The second fusion module further includes a first head; The method further comprises: Obtaining a target word table corresponding to the audio understanding model, wherein the word quantity of the target word table is smaller than the word quantity of the original word table corresponding to the large language model; Extracting the initial weight corresponding to each word in the target word table from the weight matrix of the head of the large language model according to the position index of each word in the target word table in the original word table; The weight matrix of the first head is initialized according to the initial weight corresponding to each word in the target word table.

7. The method according to claim 6, characterized in that The first header also includes a dimension corresponding to a blank symbol.

8. The method according to claim 6, characterized in that The obtaining of a target vocabulary corresponding to the audio understanding model includes: Perform word frequency statistics on the words in the training set; A target word table corresponding to the audio comprehension model is constructed based on a preset number of word units with the highest word frequency in the training set.

9. The method according to claim 6, characterized in that The audio understanding model further includes a second head, the second head being connected to the large language model body and the second head being initialized based on the large language model head; The method further comprises: Determining a value of a knowledge distillation loss function according to a first text prediction result output by the first head and a second text prediction result output by the second head; The parameters of the audio understanding model are updated according to the value of the knowledge distillation loss function.

10. The method according to claim 9, characterized in that The method further comprises: Determining a value of a word unit distribution loss function corresponding to the audio understanding model according to the second text prediction result and a target text label sequence corresponding to the second text prediction result; The parameters of the audio understanding model are updated according to the value of the word unit distribution loss function.

11. The method according to claim 9, characterized in that During the training of the audio understanding model, the parameters of the second head remain fixed.

12. The method according to claim 1, characterized in that The audio understanding model performs first-stage training and second-stage training based on the training set; In the first stage of training, the parameters of the encoding module remain fixed; in the second stage of training, the parameters of the encoding module are adjusted.

13. The method according to claim 1, wherein The speech recognition model uses a recurrent neural network transcriber.

14. An audio understanding method, characterized in that: include: Obtaining an audio understanding model trained by the audio understanding model training method according to any one of claims 1 to 13; The audio to be processed is input into the audio understanding model, and the audio understanding model outputs the text prediction result corresponding to the audio to be processed.

15. A training device for an audio comprehension model, characterized in that: include: A first acquisition module is configured to obtain a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoding module, a prediction module, and a first fusion module, the first fusion module being connected to the encoding module and the prediction module, respectively; and the large language model includes a large language model body and a large language model head that are connected to each other; A construction module, configured to construct an audio-text classification model based on the encoding module, the large language model body, the first fusion module, and the binary classification layer, wherein the first fusion module is connected to the encoding module and the large language model body, respectively, and the binary classification layer is connected to the first fusion module; a modality prediction module, configured to process the acoustic features output by the encoding module or the semantic features output by the large language model body through the first fusion module to obtain fused features; and to process the fused features through the binary classification layer to obtain a modality prediction result; a first training module, configured to determine a reverse gradient value of the first fusion module according to the modality prediction result and a true modality corresponding to the modality prediction result, and update parameters of the first fusion module according to the reverse gradient value of the first fusion module to train the audio text classification model; A second training module is configured to construct an audio understanding model and train the audio understanding model in response to the audio-text classification model being trained to meet preset conditions based on the encoding module, the large language model body and the second fusion module, wherein the second fusion module includes the first fusion module.

16. An audio understanding device, characterized in that: include: an acquisition module, configured to acquire an audio understanding model trained by the audio understanding model training device according to claim 15; The audio understanding module is used to input the audio to be processed into the audio understanding model, and output the text prediction result corresponding to the audio to be processed through the audio understanding model.

17. A training device for an audio comprehension model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 13.

18. An audio understanding device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 14.

19. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

20. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that: When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 1 to 14.

Images