Audio understanding method, training method, apparatus, and related products

Abstract

The present disclosure relates to the technical field of computers, and provides an audio understanding method, a training method, an apparatus and related products. The method comprises: inputting obtained audio to be understood into a target audio understanding model, and outputting a text prediction result of the audio to be understood; the target audio understanding model is generated by performing first stage training and second stage training on an initial audio understanding model, and the initial audio understanding model comprises an encoding module and a large language model main body. The embodiment of the present disclosure can obtain a trained target audio understanding model through a two-stage training method. The target audio understanding model integrates the acoustic feature extraction capability of the encoding module and the semantic understanding capability of the large language model, can perform speech recognition in a streaming manner, fully utilizes the text capability of the large language model, and greatly improves the speech recognition effect of the target audio understanding model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio understanding method, training method, apparatus and related products. Background Technology

[0002] Audio understanding technology refers to the technique of semantic parsing and content recognition of audio signals using artificial intelligence models, capable of converting speech content into understandable text information. Audio understanding can be divided into non-streaming audio understanding and streaming audio understanding. Non-streaming processing performs overall analysis of complete audio and outputs results, suitable for scenarios such as meeting transcript analysis. Streaming processing, on the other hand, parses audio in real time and provides dynamic feedback, widely used in fields requiring instant interaction such as intelligent voice assistants, real-time caption generation, and smart homes. How to improve the training efficiency and processing performance of streaming audio understanding models is an important research topic.

[0003] In related technologies, a pre-trained large text model can be used as a basis to fine-tune the audio modality of a speech recognition model, resulting in a large audio understanding model with speech recognition capabilities. While related technologies have designed alignment training mechanisms—that is, using the loss function of the speech recognition model to align audio and text, enabling the large audio understanding model to perform streaming speech recognition—these alignment training mechanisms primarily replace the original semantic modeling module in the speech recognition model with a large text model, without fully utilizing the powerful semantic modeling capabilities of the large text model itself. Summary of the Invention

[0004] In view of this, this disclosure presents an audio understanding method, training method, device, and related products.

[0005] According to one aspect of this disclosure, an audio understanding method is provided, the method comprising: acquiring an audio to be understood; inputting the audio to be understood into a target audio understanding model to obtain a text prediction result of the audio to be understood output by the target audio understanding model; wherein the target audio understanding model is generated by performing a first-stage training and a second-stage training on an initial audio understanding model, the initial audio understanding model comprising an encoding module and a large language model body; in the first-stage training, the parameters of the encoding module are fixed, and the large language model body is used to predict the second half of the first text based on the first half of the first text; in the second-stage training, the parameters of the large language model body are fixed, and the large language model body is used to predict the remaining text segment sequence in the first text based on a random first K text segment sequence after the first text is cropped, where K is an integer greater than 1.

[0006] In one possible implementation, the first stage of training includes: training the initial audio understanding model based on the first audio, the first text, and the first half of the first text, while fixing the parameters of the encoding module in the initial audio understanding model, to obtain the first audio understanding model; the second stage of training includes: training the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, while fixing the parameters of the large language model body in the first audio understanding model, to obtain the target audio understanding model.

[0007] In one possible implementation, the text prediction result contains multiple characters. The audio to be understood is input into a target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model. This includes: inputting the audio to be understood into the target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is greater than or equal to a preset value N, initializing a copy of the large language model body, where N is a positive integer; the copy determines multiple first prediction probabilities for decoding the (j+1)th character based on the first N decoded characters; the target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character; and the target audio understanding model decodes the (j+1)th character according to the multiple first prediction probabilities and the multiple second prediction probabilities.

[0008] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the large language model body, respectively. The target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character, including: the encoding module obtaining the acoustic segment features corresponding to the (j+1)-th audio segment in the audio to be understood; the large language model body obtaining the semantic features corresponding to the (j+1)-th audio segment based on the first j decoded characters; and the first fusion module determining the second prediction probabilities of multiple candidate paths for decoding the (j+1)-th character based on the acoustic segment features corresponding to the (j+1)-th audio segment, the semantic features corresponding to the (j+1)-th audio segment, and the M candidate paths with the highest probability for the j-th character, where M is a positive integer.

[0009] In one possible implementation, decoding the (j+1)th character based on the plurality of first prediction probabilities and the plurality of second prediction probabilities includes: performing a weighted sum of the plurality of first prediction probabilities and the plurality of second prediction probabilities to obtain a plurality of third prediction probabilities; selecting the M third prediction probabilities with the highest probabilities from the plurality of third prediction probabilities as the M groups of candidate paths with the highest probabilities for the (j+1)th character; and decoding the (j+1)th character based on the M groups of candidate paths with the highest probabilities for the (j+1)th character.

[0010] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the large language model body, respectively. The text prediction result contains multiple characters. The audio to be understood is input into a target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model. This includes: inputting the audio to be understood into the target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is less than a preset value N, the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood. The large language model obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters. The first fusion module determines the second prediction probabilities of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M candidate paths with the highest probability for the j-th character. The first fusion module selects the M second prediction probabilities with the highest probability from the second prediction probabilities of the multiple candidate paths as the M candidate paths with the highest probability for the (j+1)th character. The first fusion module decodes the (j+1)th character based on the M candidate paths with the highest probability for the (j+1)th character.

[0011] According to another aspect of this disclosure, a training method for an audio understanding model is provided. The method includes: acquiring a training set and an initial audio understanding model, wherein training samples in the training set include a first audio and a first text, and the initial audio understanding model includes an encoding module, a large language model body, and a first fusion module, wherein the first fusion module is respectively connected to the encoding module and the large language model body; in a first stage of training, while fixing the parameters of the encoding module in the initial audio understanding model, the initial audio understanding model is trained based on the first audio, the first text, and the second text to obtain the first audio understanding model, wherein the second text is the upper half of the first text; in a second stage of training... During the training phase, with the parameters of the large language model body in the first audio understanding model fixed, the first audio understanding model is trained based on the audio segment sequence after the first audio is cropped and the random first K text segment sequence after the first text is cropped, to obtain the target audio understanding model. The target audio understanding model is used to implement the steps of the above-mentioned audio understanding method. The large language model body is used to predict the lower half of the first text based on the second text in the first phase of training, and to predict the remaining text segment sequence in the first text other than the K text segment sequence based on the random first K text segment sequence in the second phase of training, where K is an integer greater than 1.

[0012] In one possible implementation, training the initial audio understanding model based on a first audio, a first text, and a second text to obtain the first audio understanding model includes: extracting a first acoustic feature sequence corresponding to the first audio through the encoding module, and extracting a first semantic feature sequence corresponding to the first text and a second semantic feature sequence corresponding to the second text through the large language model body, wherein the second semantic feature sequence is used to predict the lower half of the first text; determining, through the first fusion module, all first effective alignment paths that can generate the first text corresponding to the first audio, and a first total probability of all first effective alignment paths, based on the first acoustic feature sequence and the first semantic feature sequence; and training the initial audio understanding model according to the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain the first audio understanding model.

[0013] In one possible implementation, training the initial audio understanding model based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain a first audio understanding model includes: determining a first loss based on the first total probability; determining a second loss based on the first semantic feature; determining a third loss based on the second semantic feature; and training the initial audio understanding model based on a weighted sum of the first loss, the second loss, and the third loss to obtain the first audio understanding model.

[0014] In one possible implementation, the first audio understanding model further includes a first head and a second head, the first head and the second head being connected to the main body of the large language model, and both the first head and the second head being initialized based on the large language model head; determining a first loss based on the first total probability includes: determining the first loss based on the negative logarithm of the first total probability; determining a second loss based on the first semantic feature includes: outputting a first text prediction result corresponding to the first audio through the first head based on the first semantic feature; determining the second loss based on the first text prediction result and the first text; determining a third loss based on the second semantic feature includes: outputting a second text prediction result corresponding to the lower half of the first text through the second head based on the second semantic feature; determining the third loss based on the second text prediction result and the lower half of the first text.

[0015] In one possible implementation, during the second stage of training, with the parameters of the large language model body in the first audio understanding model fixed, the first audio understanding model is trained based on the audio segment sequence after the first audio is cropped and the first K text segment sequence after the first text is cropped, to obtain the target audio understanding model. This includes: sequentially inputting the audio segment sequence of the first audio into the encoding module in the first audio understanding model; restricting the first audio understanding model to only acquire information about the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment using an attention mask, so that the acoustic segment features corresponding to the current audio segment extracted by the encoding module are sent to the first fusion module in real time; extracting the first semantic feature sequence corresponding to the first text and the third semantic feature sequence corresponding to the first K text segment sequence through the large language model body, the third semantic feature sequence being used to predict the remaining text in the first text; determining all second effective alignment paths that can generate the first text corresponding to the first audio through the first fusion module based on the first semantic feature sequence and the acoustic segment features corresponding to each audio segment, and calculating the second total probability of all second effective alignment paths; and calculating the second total probability based on the second total probability. The first semantic feature sequence and the third semantic feature sequence are used to train the first audio understanding model to obtain the target audio understanding model.

[0016] In one possible implementation, the method for determining the value of K includes: for any training sample in the training set, obtaining the duration of the first audio and the number of characters in the first text; and determining the value of K based on the mean and variance of the duration occupied by each character.

[0017] According to another aspect of this disclosure, an audio understanding device is provided, comprising: a first acquisition module for acquiring audio to be understood; and an output module for inputting the audio to be understood into a target audio understanding model to obtain a text prediction result of the audio to be understood output by the target audio understanding model; wherein the target audio understanding model is generated by performing a first-stage training and a second-stage training on an initial audio understanding model, the initial audio understanding model including an encoding module and a large language model body; in the first-stage training, the parameters of the encoding module are fixed, and the large language model body is used to predict the second half of the first text based on the first half of the first text; in the second-stage training, the parameters of the large language model body are fixed, and the large language model body is used to predict the remaining text segment sequence in the first text based on the random first K text segment sequence after the first text is cropped, where K is an integer greater than 1.

[0018] In one possible implementation, the first stage of training includes: training the initial audio understanding model based on the first audio, the first text, and the first half of the first text, while fixing the parameters of the encoding module in the initial audio understanding model, to obtain the first audio understanding model; the second stage of training includes: training the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, while fixing the parameters of the large language model body in the first audio understanding model, to obtain the target audio understanding model.

[0019] In one possible implementation, the text prediction result contains multiple characters, and the output module is configured to: input the audio to be understood into a target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is greater than or equal to a preset value N, initialize a copy of the main body of the large language model, where N is a positive integer; the copy determines multiple first prediction probabilities for decoding the (j+1)th character based on the first N decoded characters; the target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the jth character; and the target audio understanding model decodes the (j+1)th character according to the multiple first prediction probabilities and the multiple second prediction probabilities.

[0020] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the large language model body, respectively. The target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character, including: the encoding module obtaining the acoustic segment features corresponding to the (j+1)-th audio segment in the audio to be understood; the large language model body obtaining the semantic features corresponding to the (j+1)-th audio segment based on the first j decoded characters; and the first fusion module determining the second prediction probabilities of multiple candidate paths for decoding the (j+1)-th character based on the acoustic segment features corresponding to the (j+1)-th audio segment, the semantic features corresponding to the (j+1)-th audio segment, and the M candidate paths with the highest probability for the j-th character, where M is a positive integer.

[0021] In one possible implementation, decoding the (j+1)th character based on the plurality of first prediction probabilities and the plurality of second prediction probabilities includes: performing a weighted sum of the plurality of first prediction probabilities and the plurality of second prediction probabilities to obtain a plurality of third prediction probabilities; selecting the M third prediction probabilities with the highest probabilities from the plurality of third prediction probabilities as the M groups of candidate paths with the highest probabilities for the (j+1)th character; and decoding the (j+1)th character based on the M groups of candidate paths with the highest probabilities for the (j+1)th character.

[0022] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the large language model body, respectively. The text prediction result contains multiple characters. The output module is used to: input the audio to be understood into the target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is less than a preset value N, the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment based on the (j+1)th audio segment in the audio to be understood; the large language model body, based on the first j decoded characters... The semantic features corresponding to the (j+1)th audio segment are obtained; the first fusion module determines the second prediction probabilities of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M candidate paths with the highest probability for the j-th character; the first fusion module selects the M second prediction probabilities with the highest probability from the second prediction probabilities of the multiple candidate paths as the M candidate paths with the highest probability for the (j+1)th character; the first fusion module decodes the (j+1)th character based on the M candidate paths with the highest probability for the (j+1)th character.

[0023] According to another aspect of this disclosure, a training apparatus for an audio understanding model is provided. The training apparatus includes: a second acquisition module for acquiring a training set and an initial audio understanding model, wherein the training samples in the training set include a first audio and a first text, and the initial audio understanding model includes an encoding module, a large language model body, and a first fusion module, wherein the first fusion module is respectively connected to the encoding module and the large language model body; and a first training module for training the initial audio understanding model based on the first audio, the first text, and the second text in a first-stage training phase, while fixing the parameters of the encoding module in the initial audio understanding model, to obtain the first audio understanding model, wherein the second text is the upper half of the first text. The second training module is used in the second stage of training to train the first audio understanding model based on the audio segment sequence after the first audio is cropped and the random first K text segment sequence after the first text is cropped, while fixing the parameters of the large language model body in the first audio understanding model, to obtain a target audio understanding model. The target audio understanding model is used to implement the steps of the above-mentioned audio understanding method. The large language model body is used to predict the lower half of the first text based on the second text in the first stage of training, and to predict the remaining text segment sequence in the first text other than the K text segment sequence based on the random first K text segment sequence in the second stage of training, where K is an integer greater than 1.

[0024] In one possible implementation, the first training module is configured to: extract a first acoustic feature sequence corresponding to the first audio through the encoding module, and extract a first semantic feature sequence corresponding to the first text and a second semantic feature sequence corresponding to the second text through the large language model body, wherein the second semantic feature sequence is used to predict the lower half of the first text; determine, based on the first acoustic feature sequence and the first semantic feature sequence, all first effective alignment paths capable of generating the first text corresponding to the first audio through the first fusion module, and a first total probability of all first effective alignment paths; and train the initial audio understanding model according to the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain the first audio understanding model.

[0025] In one possible implementation, training the initial audio understanding model based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain a first audio understanding model includes: determining a first loss based on the first total probability; determining a second loss based on the first semantic feature; determining a third loss based on the second semantic feature; and training the initial audio understanding model based on a weighted sum of the first loss, the second loss, and the third loss to obtain the first audio understanding model.

[0026] In one possible implementation, the first audio understanding model further includes a first head and a second head, the first head and the second head being connected to the main body of the large language model, and both the first head and the second head being initialized based on the large language model head; determining a first loss based on the first total probability includes: determining the first loss based on the negative logarithm of the first total probability; determining a second loss based on the first semantic feature includes: outputting a first text prediction result corresponding to the first audio through the first head based on the first semantic feature; determining the second loss based on the first text prediction result and the first text; determining a third loss based on the second semantic feature includes: outputting a second text prediction result corresponding to the lower half of the first text through the second head based on the second semantic feature; determining the third loss based on the second text prediction result and the lower half of the first text.

[0027] In one possible implementation, the second training module is used to: sequentially input the audio segment sequence of the first audio into the encoding module of the first audio understanding model; restrict the first audio understanding model to only acquire information of the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment using an attention mask, so that the acoustic segment features corresponding to the current audio segment extracted by the encoding module are sent to the first fusion module in real time; extract the first semantic feature sequence corresponding to the first text and the third semantic feature sequence corresponding to the first K text segment sequences through the large language model body, the third semantic feature sequence being used to predict the remaining text in the first text; determine all second effective alignment paths that can generate the first text corresponding to the first audio through the first fusion module based on the first semantic feature sequence and the acoustic segment features corresponding to each audio segment, and calculate the second total probability of all second effective alignment paths; train the first audio understanding model according to the second total probability, the first semantic feature sequence, and the third semantic feature sequence to obtain the target audio understanding model.

[0028] In one possible implementation, the method for determining the value of K includes: for any training sample in the training set, obtaining the duration of the first audio and the number of characters in the first text; and determining the value of K based on the mean and variance of the duration occupied by each character.

[0029] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0030] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0031] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0032] In the audio understanding method provided in this disclosure, a trained target audio understanding model can be obtained through a two-stage training method. This target audio understanding model integrates the acoustic feature extraction capability of the encoding module in a pre-trained speech recognition model with the semantic understanding capability of a large language model. In practical applications, the audio to be understood (such as a real-time speech stream or a complete recording) can be input into the target audio understanding model. The target audio understanding model extracts acoustic features through the encoding module and combines this with the semantic reasoning of the large language model to dynamically align the audio and text sequences, outputting the corresponding text prediction results. Furthermore, since the large language model can predict the second half of the first text based on the first half of the first text in the first stage of training, and predict the remaining text segment sequence in the first text based on the first K text segment sequences in the second stage of training, the trained target audio understanding model can perform streaming speech recognition while fully utilizing the text capabilities of the large language model, thereby further improving the speech recognition performance of the target audio understanding model.

[0033] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0035] Figure 1 A flowchart illustrating the audio understanding method provided in an embodiment of this disclosure is shown.

[0036] Figure 2 A flowchart illustrating the training method of the audio understanding model provided in the embodiments of this disclosure is shown.

[0037] Figure 3 A schematic diagram of a recurrent neural network transcriptograph is shown.

[0038] Figure 4 A schematic diagram of the initial audio understanding model provided in an embodiment of this disclosure is shown.

[0039] Figure 5 This diagram illustrates the dynamic alignment process of the target audio understanding model provided in the present disclosure during the decoding stage.

[0040] Figure 6 This diagram illustrates the decoding stage of the target audio understanding model provided in an embodiment of the present disclosure.

[0041] Figure 7 A block diagram of an audio understanding apparatus according to an embodiment of the present disclosure is shown.

[0042] Figure 8 A block diagram of a training apparatus for an audio understanding model according to an embodiment of the present disclosure is shown.

[0043] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0044] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0045] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0046] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0047] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0048] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0049] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0050] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0051] Figure 1 A flowchart illustrating the audio understanding method provided in this disclosure is shown. In one possible implementation, the entity executing the audio understanding method can be an audio understanding device. For example, the audio understanding method can be executed by a terminal device, a server, or other electronic equipment. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the audio understanding method can be implemented by a processor calling computer-readable instructions stored in memory, wherein the processor may include a graphics processing unit (GPU).

[0052] like Figure 1 As shown, this audio understanding method includes:

[0053] In step S11, the audio to be understood is obtained;

[0054] In step S12, the audio to be understood is input into the target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model;

[0055] The target audio understanding model is generated by performing a first-stage training and a second-stage training on the initial audio understanding model. The initial audio understanding model includes an encoding module and a large language model body. In the first-stage training, the parameters of the encoding module are fixed, and the large language model body is used to predict the second half of the first text based on the first half of the first text. In the second-stage training, the parameters of the large language model body are fixed, and the large language model body is used to predict the remaining text segment sequence in the first text based on the random first K text segment sequence after the first text is cropped, where K is an integer greater than 1.

[0056] In the audio understanding method provided in this disclosure, a trained target audio understanding model can be obtained through a two-stage training method. This target audio understanding model integrates the acoustic feature extraction capability of the encoding module in a pre-trained speech recognition model with the semantic understanding capability of a large language model. In practical applications, the audio to be understood (such as a real-time speech stream or a complete recording) can be input into the target audio understanding model. The target audio understanding model extracts acoustic features through the encoding module and combines this with the semantic reasoning of the large language model to dynamically align the audio and text sequences, outputting the corresponding text prediction results. Furthermore, since the large language model can predict the second half of the first text based on the first half of the first text in the first stage of training, and predict the remaining text segment sequence in the first text based on the first K text segment sequences in the second stage of training, the trained target audio understanding model can perform streaming speech recognition while fully utilizing the text capabilities of the large language model, thereby further improving the speech recognition performance of the target audio understanding model.

[0057] In one possible implementation, in step S11, the audio to be understood can be audio collected by devices such as a recorder, mobile phone, or microphone; audio downloaded from various audio platforms or audio libraries; audio extracted from videos, movies, TV series, or live broadcast files; or artificially customized synthesized audio. This disclosure does not limit the source or specific type of the audio to be understood.

[0058] In one possible implementation, in step S12, the acquired audio to be understood can be input into a target audio understanding model. The target audio understanding model can perform semantic parsing and content recognition on the input audio to be understood, convert the audio to be understood into understandable text information, and use the understandable text information as the text prediction result output by the target audio understanding model.

[0059] In one possible implementation, a trained target audio understanding model can be obtained through a two-stage training method. The first stage of training includes: with the parameters of the encoding module in the initial audio understanding model fixed, training the initial audio understanding model based on the first audio, the first text, and the first half of the first text to obtain the first audio understanding model; the second stage of training includes: with the parameters of the large language model body in the first audio understanding model fixed, training the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut to obtain the target audio understanding model.

[0060] The training method of the audio understanding model provided in the embodiments of this disclosure will be described in detail below.

[0061] Figure 2 A flowchart illustrating a training method for an audio understanding model provided in an embodiment of this disclosure is shown. In one possible implementation, the entity executing the audio understanding model training method can be an audio understanding model training device. For example, the audio understanding model training method can be executed by a terminal device, a server, or other electronic devices. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device, etc. In some possible implementations, the audio understanding model training method can be implemented by a processor calling computer-readable instructions stored in memory, wherein the processor may include a graphics processing unit (GPU).

[0062] like Figure 2 As shown, the training method for this audio understanding model includes:

[0063] In step S21, a training set and an initial audio understanding model are obtained. The training samples in the training set include the first audio and the first text. The initial audio understanding model includes an encoding module, a large language model body, and a first fusion module. The first fusion module is connected to the encoding module and the large language model body, respectively.

[0064] In step S22, during the first stage of training, with the parameters of the encoding module in the initial audio understanding model fixed, the initial audio understanding model is trained based on the first audio, the first text, and the second text to obtain the first audio understanding model, wherein the second text is the upper half of the first text;

[0065] In step S23, during the second stage of training, with the parameters of the large language model body in the first audio understanding model fixed, the first audio understanding model is trained based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, to obtain the target audio understanding model. The target audio understanding model is used to implement the above audio understanding method.

[0066] The large language model is used in the first stage to train the prediction of the second half of the first text based on the second text, and in the second stage to train the prediction of the remaining text fragments in the first text excluding the K text fragment sequences based on the random first K text fragment sequences, where K is an integer greater than 1.

[0067] Compared to related technologies that merely replace the original prediction module in a speech recognition model with a larger model (e.g., replacing the original prediction module with a large language model), although the replaced audio understanding model still consists of an encoding module, a large language model, and a fusion module, it does not fully utilize the powerful semantic modeling capabilities of the large language model itself. The audio understanding model training method provided in this disclosure allows the large language model to predict the second half of a first text based on the first half of the first text in the first training stage, and predict the remaining text segment sequence in the first text based on the first K text segment sequences in the second training stage. Therefore, the trained target audio understanding model can perform streaming speech recognition while fully utilizing the textual capabilities of the large language model, thereby improving the speech recognition performance.

[0068] In one possible implementation, step S21 may involve obtaining an initial audio understanding model, including: obtaining a pre-trained speech recognition model and a pre-trained large language model; wherein the speech recognition model includes an encoder, a predictor, and a jointer, the jointer being connected to the encoder and the predictor respectively, and the large language model including a large language model body and a large language model head that are interconnected; and constructing an initial audio understanding model based on the speech recognition model and the large language model; wherein the initial audio understanding model includes an encoder, a large language model body, and a first jointer, the first jointer being connected to the encoder and the large language model body respectively.

[0069] In this embodiment, the pre-trained speech recognition model may include an encoding module, a prediction module, and a raw fusion module. The encoding module may also be called an encoder, the prediction module may be called a predictor or prediction network, and the raw fusion module may be called a joint network or joint module; no limitation is made herein. In one possible implementation, the encoding module may be used to extract acoustic features, the prediction module may be used to extract semantic features, and the raw fusion module may be used to fuse the acoustic and semantic features.

[0070] In one possible implementation, the speech recognition model employs a recurrent neural network transducer (RNNT).

[0071] Figure 3 A schematic diagram of a recurrent neural network transcriptome is shown. Figure 3 As shown, a recurrent neural network transcriptome may include an encoding module, a prediction module, a raw fusion module, and an activation layer.

[0072] The encoding module can be used to model acoustic features. For example, it can convert an input audio frame sequence into a high-dimensional acoustic feature representation. For instance, the encoding module can take the audio frame x at the current time step t as input. t Output the acoustic features corresponding to the current time step t. Here, a time step refers to the smallest discrete unit of time in audio processing, corresponding to a single frame of data processed or output by the model at a certain moment. For an audio frame sequence... The encoding module can output acoustic feature sequences. The encoding module can be implemented using either a Conformer (Convolution-augmented Transformer end-to-end model architecture) or an LSTM (Long Short-Term Memory) network; no specific restriction is imposed here.

[0073] The prediction module can be used to model semantic features. During the training phase, the prediction module can be inputted with historical text labels from before the current text step u (e.g., ...). or Output the semantic features of the current text step u. In the prediction phase, the prediction module can take the generated (i.e., predicted) text labels as input and output the semantic features of the current text step. Here, a text step can refer to a discrete unit in the text generation process, corresponding to a single word processed or output by the model. The prediction module can use either an LSTM or a Transformer structure; no specific limitation is made here.

[0074] The training samples in the training set used to train the recurrent neural network transcriber can consist of audio-text pairs {x, y}, where x represents an audio sample and y represents the corresponding target text label sequence (i.e., labeled text). For example, if the target text label sequence y is "How's the weather today?", then the text format input to the prediction module during training can be: <sos>How's the weather today? <sos>This is the start of sentence marker, used to initialize the hidden state of the prediction module and indicate the beginning of text generation. The target text label sequence used to calculate the loss can be in the format: "What's the weather like today?" <eos>.in, <eos>This is the sentence end marker. <eos>As part of the training objective, this helps the recurrent neural network transcript learn when to terminate the output.

[0075] The original fusion module can fuse the outputs of the encoding and prediction modules. For example, the output of the original fusion module can be: Where W represents the weight matrix, b represents the bias vector, and z t,u This represents the output of the original fusion module, Joint. This represents the acoustic feature corresponding to time step t. The semantic features of text step u are represented. The original fusion module may include linear layers and activation layers (e.g., softmax).

[0076] like Figure 3 As shown, after the original fusion module, an activation layer (e.g., softmax) can be connected to generate the probability distribution of all possible tokens (e.g., characters, words, symbols). .

[0077] In this implementation, the pre-trained speech recognition model, employing a recurrent neural network transcriber structure, effectively achieves real-time dynamic alignment of audio and text sequences, demonstrating significant advantages in streaming speech recognition tasks. The recurrent neural network transcriber structure extracts acoustic features through an encoding module, models text context dependencies through a prediction module, and fuses multimodal information through a raw fusion module, possessing powerful feature representation capabilities that significantly improve the model's recognition accuracy and real-time performance in complex scenarios.

[0078] In this embodiment of the disclosure, a pre-trained large language model (LMM) can refer to a model that has been pre-trained on a large-scale text dataset. In some application scenarios, a large language model may also be called a large language model, a large text model, etc., and this is not limited thereto. In one possible implementation, the pre-trained large language model can be an open-source, pre-trained large language model. These open-source models have already been learned and trained on massive amounts of text, accumulating rich linguistic knowledge and semantic understanding capabilities. In another possible implementation, the pre-trained large language model can be a model trained by the developers themselves.

[0079] Large-scale language models, pre-trained on massive amounts of text data, demonstrate powerful language understanding and generation capabilities. Compared to traditional models that rely on large amounts of data, large-scale language models are more advantageous in data-scarce languages ​​and complex speech tasks, and their large number of parameters makes them more robust and adaptable to noisy environments. Furthermore, multilingual pre-trained large-scale language models can reduce the need for language-specific data and lower the complexity of model management.

[0080] In this embodiment, the large language model may include an interconnected large language model body (LLM-body) and a large language model head (LLM-head). The large language model body may be composed of multi-layer neural networks (such as Transformer modules), and is responsible for deep feature extraction and semantic understanding of the input text, obtaining general language representation capabilities through pre-training on massive amounts of text data. The large language model head, as the output layer of the large language model, may consist of linear layers and activation layers, and is responsible for mapping the high-dimensional features extracted by the large language model body to the target output space (such as the probability distribution of the lexicon). In some application scenarios, the large language model head may also be referred to as the output layer of the large language model.

[0081] In this embodiment of the disclosure, an initial audio understanding model can be constructed based on a pre-trained speech recognition model and a pre-trained large-scale language model. The initial audio understanding model includes an encoding module from the pre-trained speech recognition model, a large-scale language model body from the pre-trained large-scale language model, and a first fusion module. In the initial audio understanding model, the first fusion module is connected to both the encoding module and the large-scale language model body, and can be used to fuse the outputs of the encoding module and the large-scale language model body.

[0082] For example, the encoding module in the pre-trained speech recognition model can be retained, the prediction module in the pre-trained speech recognition model can be replaced with the main body of a large language model, and the original fusion module in the pre-trained speech recognition model can be reconstructed into the first fusion module.

[0083] The encoding module in the initial audio understanding model can be used to process audio input and output acoustic features. The large language model body in the initial audio understanding model can be used to process the input text and output semantic features. In the initial audio understanding model, the ability to model text context is enhanced by replacing the prediction module in the speech recognition model with a large language model body. The input to the first fusion module in the initial audio understanding model can include acoustic features output by the encoding module. Semantic features output by the main body of a large language model The first fusion module can dynamically align acoustic and semantic features.

[0084] In one possible implementation, the first fusion module includes an original fusion module and an output header, wherein the output header is initialized based on the large language model header, and the output header is based on the large language model header with the dimension corresponding to the blank symbol added.

[0085] In this implementation, the first fusion module can consist of the original fusion module and the output header.

[0086] The original fusion module retains the structure of the fusion module in the pre-trained speech recognition model (such as linear layer + activation layer), and can be used to initially fuse acoustic features and semantic features.

[0087] The output header can be initialized based on the large language model header. That is, the parameters of the output header can be inherited from the pre-trained large language model header to leverage the existing language modeling capabilities of the pre-trained large language model. Additionally, in this implementation, a dimension corresponding to blank characters is added to the large language model header to obtain the output header. This is to adapt to the streaming alignment requirements of speech recognition models (such as recurrent neural network transcribers), because speech recognition models need to handle blank characters (representing no output or waiting for subsequent input) during decoding. For example, if the output dimension of the large language model header is the vocabulary size V, the modified output header has an output dimension of V+1, with the added dimension dedicated to calculating the probability of blank characters.

[0088] In this implementation, the language generation capabilities of a large language model are preserved by reusing the parameters of the large language model's head. At the same time, the alignment mechanism specific to speech recognition models (such as recurrent neural network transcribers) is supported by expanding the dimensions. The introduction of blank enables the initial audio understanding model to dynamically align audio and text (e.g., moving the path to the right represents blanking, and moving it upwards represents generating characters), thereby achieving streaming decoding.

[0089] In one possible implementation, the method further includes controlling the output dimension of the original fusion module to be the same as the input dimension of the large language model head during pre-training.

[0090] In this implementation, during pre-training, the output dimension of the original fusion module is controlled to be the same as the input dimension of the large language model head. This ensures seamless structural integration between the two modules, enabling effective feature fusion. The original fusion module is responsible for jointly processing the acoustic features extracted by the encoding module and the semantic features output by the prediction module (or the replaced large language model head). Its output needs to match the input dimension of the large language model head to avoid information loss or computational errors caused by dimensionality mismatch. This alignment operation allows the first fusion module to directly utilize the parameters of the pre-trained large language model head without additional adjustments, thus preserving the original language modeling capabilities of the large language model while remaining compatible with feature inputs from audio modalities.

[0091] This implementation enables the fused multimodal features (acoustic and semantic) to be correctly processed by the large language model head. For example, if the input dimension of the large language model head is D, the output of the original fusion module also needs to be adjusted to D to ensure that its joint features can be directly input into the large language model head for probability distribution calculation. This consistency not only simplifies the structure of the initial audio understanding module but also improves training efficiency, allowing the initial audio understanding module to converge quickly during the fine-tuning phase. Furthermore, this dimensionality control lays the foundation for subsequent streaming processing (such as adding blank dimensions), ensuring that the initial audio understanding module maintains stability and robustness when dynamically aligning audio and text.

[0092] In this embodiment, during the training of the initial audio understanding module, for each audio sample in the audio training set, the encoding module can process the original audio signal frame by frame, outputting an acoustic feature sequence. The large language model body can autoregressively generate the corresponding semantic feature sequence based on historical text tags. The acoustic feature sequence and the semantic feature sequence provide multimodal input for the subsequent first fusion module.

[0093] Once the initial audio understanding module is obtained, it can be trained using the training set. Similar to the training set used to train the recurrent neural network transcriber, each training sample in the training set of the initial audio understanding module can consist of a first audio and a first text pair, and so on.

[0094] In step S21, the training set and the initial audio understanding model are obtained. The first stage of training can be carried out in step S22, and the second stage of training can be carried out in step S23. In the first stage of training, the parameters of the encoding module remain fixed; in the second stage of training, the parameters of the main body of the large language model remain fixed.

[0095] In this implementation, a two-stage training strategy can be used to optimize the initial audio understanding model.

[0096] In the first stage of training, the parameters of the encoding module can be kept fixed. With the parameters of the encoding module in the initial audio understanding model fixed, the acoustic feature extraction capabilities already learned by the pre-trained encoding module are utilized, while the parameters of other parts of the initial audio understanding model (such as the large language model body and the first fusion module) are adjusted. The encoding module in the pre-trained speech recognition model already possesses the ability to extract effective acoustic features from audio signals. Acoustic features can include audio spectral information, prosodic patterns, etc., providing a foundation for subsequent text generation. In the first stage, keeping the parameters of the encoding module fixed avoids excessive adjustments to the encoding module during the initial training phase, thus preserving its general acoustic feature extraction capabilities learned in the pre-training stage. This ensures that the initial audio understanding model can still effectively extract acoustic features when processing new audio data (such as the first audio). In the first stage, the training focus is on combining the pre-trained large language model body with the encoding module, and aligning and fusing the acoustic features of the first audio and the semantic features of the first text through the first fusion module. At this point, the main updates are to the parameters of the large language model body and the first fusion module, so that the first audio understanding model after the first stage of training can better understand and generate text.

[0097] In this process, to ensure the textual capabilities of the large language model, a second piece of text (the first half of the first text) can be simultaneously input into the large language model. The large language model will only perform predictions and loss calculations based on the second half of the first text. This allows the large language model to be trained more stably, and the more second pieces of text (e.g., the first half of the first text) are input, the more accurate the predictions of future texts will be.

[0098] In one example, the first audio understanding model obtained from the first stage of training can be called a non-streaming audio understanding model.

[0099] In one possible implementation, during the first stage of training of the initial audio understanding model, the output head of the first fusion module is trained with full parameters, while the parameters of the main body of the large language model are updated by fine-tuning.

[0100] In this implementation, the output head of the initial audio understanding model is directly responsible for text generation (such as generating word probability distributions) and needs to quickly adapt to the special requirements of speech recognition tasks (such as adding blank dimensions). By training the output head with all parameters, that is, by completely updating the weights of the output head through gradient descent, the output layer of the first audio understanding model can flexibly learn the language patterns after audio-text alignment.

[0101] Large-scale language models, having already acquired general semantic understanding capabilities through pre-training on massive amounts of text, are prone to overfitting when directly trained with all parameters (especially when audio data is limited). This approach employs efficient parameter fine-tuning techniques, training only a small number of low-rank matrices or adaptation layers. This allows for the small-scale adjustment of feature representations to adapt to the audio context while preserving pre-trained knowledge.

[0102] In this implementation, full-parameter training of the output header allows for precise optimization of the output distribution, improving recognition accuracy. Fine-tuning the parameters of the large language model body avoids compromising the core capabilities of the pre-trained language model, reducing training risks. Furthermore, compared to full-model training, it significantly reduces computational overhead (especially when the large language model body has a large number of parameters).

[0103] In the second phase of training, the main body of the large language model can be fixed. With the parameters of the main body of the large language model in the first audio understanding model fixed, the parameters of the encoding module can be unfrozen, optimizing the encoding module's ability to extract local acoustic features. That is, in the second phase, the first audio understanding model can simultaneously update the parameters of the encoding module and the first fusion module to achieve more accurate alignment and fusion of audio and text.

[0104] In one example, the target audio understanding model obtained from the second stage of training can be called a streaming audio understanding model.

[0105] In this implementation, phased training avoids gradient conflicts caused by simultaneously optimizing all modules. Progressive training allows the encoding module to transition from global feature extraction to local real-time processing, making it suitable for streaming audio understanding scenarios.

[0106] In one possible implementation, during the second stage of training, the first audio is divided into multiple audio segment sequences according to a preset duration, and these sequences are sequentially input into the encoding module of the first audio understanding model. An attention mask is used to restrict the first audio understanding model to only access information about the current audio segment sequence and the preceding historical audio segment sequences when processing the current sequence. Simultaneously, the main body of the large language model begins to randomly use the first K segments of the truncated first text to predict and calculate the loss for the remaining text excluding these K segments.

[0107] This allows the causality of streaming scenarios to be simulated, enabling the first audio understanding model to output results in real time based solely on the received audio data, while fully utilizing the textual capabilities of the large language model to collaboratively decode and improve speech recognition performance. Specifically, the encoding module can learn to extract effective features from local portions (i.e., audio segment sequences) of the first audio, rather than relying on the complete context. Through segmented training, the encoding module can progressively optimize its ability to capture short-term acoustic patterns (such as phonemes and syllables).

[0108] In one possible implementation, the method for determining the value of K includes: for any training sample in the training set, obtaining the duration of the first audio and the number of words in the first text; and determining the value of K based on the mean and variance of the duration occupied by each word.

[0109] For example, for any training sample {x, y} in the training set, where x represents the first audio and y represents the first text, the duration of the first audio x and the number of words in the first text y can be obtained. The duration of the first audio x can be divided by the number of words in the first text y to obtain the mean of each word's duration. Then, based on the duration of the first audio x and the number of words in the first text y, the duration of each word in the first text y can be determined, and the average of the squared differences between the duration of each word and its mean can be used as the difference. A random number K can be generated based on the normal distribution (or other probability distribution) generated from this mean and variance.

[0110] In this way, by adjusting the mean and variance, the distribution characteristics of the generated random number K can be flexibly controlled to match different reading styles or audio characteristics. This is beneficial for determining the value of K according to the speed of pronunciation, so as to adapt to more application scenarios.

[0111] Compared to related technologies that merely replace the original prediction module in a speech recognition model with a larger model (e.g., replacing the original prediction module with a large language model), although the replaced audio understanding model still consists of an encoding module, a large language model, and a fusion module, it does not fully utilize the powerful semantic modeling capabilities of the large language model itself. The audio understanding model training method provided in this disclosure allows the large language model to predict the second half of a first text based on the first half of the first text in the first training stage, and predict the remaining text in the first text based on the first K text segment sequences in the second training stage. Therefore, the trained target audio understanding model can perform streaming speech recognition while fully utilizing the textual capabilities of the large language model, thereby improving the speech recognition performance.

[0112] The training method of the audio understanding model according to the present disclosure will be described by way of example below.

[0113] Figure 4 A schematic diagram of the initial audio understanding model provided in an embodiment of this disclosure is shown. Figure 4 As shown, the initial audio understanding model may include an encoding module, an LLM body, a first fusion module, a first head, and a second head. Both the first and second heads are initialized based on the large language model head. The input to the encoding module can be audio (e.g., the first audio), and the input to the LLM body can be text (e.g., the first text and the second text). Based on the output of the first fusion module, a first loss (e.g., RNNT loss) can be calculated. Based on the output of the first head, a second loss (e.g., the first cross-entropy loss, or first CE loss) can be calculated. Based on the output of the second head, a third loss (e.g., the second cross-entropy loss, or second CE loss) can be calculated.

[0114] The initial audio understanding model can be trained in the first stage to obtain the first audio understanding model. The first audio understanding model also includes a first head and a second head, which are respectively connected to the main body of the large language model.

[0115] In one possible implementation, such as Figure 4 As shown, in the first stage of training, step S22 may include: extracting the first acoustic feature sequence corresponding to the first audio through the encoding module, and extracting the first semantic feature sequence corresponding to the first text and the second semantic feature sequence corresponding to the second text through the large language model body (LLM body), the second semantic feature sequence being used to predict the lower half of the first text; determining all first effective alignment paths that can generate the first text corresponding to the first audio through the first fusion module based on the first acoustic feature sequence and the first semantic feature sequence, and the first total probability of all first effective alignment paths; training the initial audio understanding model based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain the first audio understanding model.

[0116] The encoding module in the initial audio understanding model can be used to process audio input, extract acoustic features from the first audio input, and output a first acoustic feature sequence. The large language model body in the initial audio understanding model can be used to process input text, extract semantic features from the first text, and output a first semantic feature sequence; and extract semantic features from the second text (the first half of the first text), and output a second semantic feature sequence, which can be used to predict the second half of the first text.

[0117] The first fusion module can determine all first effective alignment paths that can generate the first text (target text label sequence, i.e. labeled text) corresponding to the first audio by using the first acoustic feature sequence and the first semantic feature sequence, and calculate the first total probability of all first effective alignment paths.

[0118] The alignment path can refer to the mapping relationship between the first acoustic feature sequence (time step t) and the first semantic feature sequence (text step u). For example:

[0119] Horizontal movement (→): Time step t+1, text step u remains unchanged (corresponding output blank, indicating that no text has been generated yet).

[0120] Vertical movement (↑): Time step t remains unchanged, text step u+1 (corresponding to the generation of an actual text label).

[0121] The first valid alignment path can refer to all legal paths from the starting point (t=0, u=0) to the ending point (t=T, u=U), and the generated text sequence is completely identical to the first text. Here, T is the total time step of the first audio, and U is the length of the first text.

[0122] The first fusion module can receive the first acoustic feature sequence from the encoding module. and the first semantic feature sequence from the main body of a large language model The joint probability distribution is output through linear layers and activation layers (such as softmax). .in, It can represent the predicted label for the current step.

[0123] The probability of each aligned path is the product of the probabilities of all steps along the path. For example, a path is... Then its probability is Sum the probabilities of all valid paths: .

[0124] For example, if the target text is "AB" and the audio has 3 time steps, the two possible valid paths are:

[0125] Path 1: (t=1)→(t=2)→(t=3↑)→(t=3↑)

[0126] (Output: blank → blank → A → B)

[0127] Path 2: (t=1↑)→(t=2)→(t=3↑)

[0128] (Output: A → blank → B)

[0129] First total probability Where P(Path1) represents the probability of path 1 and P(Path2) represents the probability of path 2.

[0130] In one possible implementation, an initial audio understanding model can be trained based on a first total probability, a first semantic feature sequence, and a second semantic feature sequence to obtain a first audio understanding model, including: determining a first loss based on the first total probability; determining a second loss based on the first semantic feature; determining a third loss based on the second semantic feature; and training the initial audio understanding model based on the weighted sum of the first loss, the second loss, and the third loss to obtain the first audio understanding model.

[0131] In one possible implementation, determining the first loss based on the first total probability includes: determining the first loss based on the negative logarithm of the first total probability;

[0132] For example, the negative logarithm of the first total probability can be taken to obtain the first loss function corresponding to the audio understanding model, and the value of the first loss function can be used as the first loss.

[0133] Specifically, we can consider the first total probability. Taking the negative logarithm yields the value of the first loss function. As the first loss, the probability range is [0,1]. After taking the negative logarithm, the higher the probability (closer to 1), the lower the first loss (closer to 0); conversely, the lower the probability (closer to 0), the higher the first loss (approaching infinity). This transformation converts the probability maximization problem into the loss minimization problem, which is consistent with the gradient descent optimization framework.

[0134] For example, if a first audio file has two first valid alignment paths with probabilities of 0.6 and 0.4 respectively, then the first total probability is... First loss value If the path probability distribution is not ideal (e.g., 0.1 and 0.1), the first loss value... This is beneficial for audio understanding models to adjust parameters through gradient descent to increase the likelihood of generating high-probability paths.

[0135] The first loss can be used to calculate the gradient and backpropagate to update the parameters of the audio understanding model. For example, it can update the parameters of at least some modules in the first fusion module, the large language model body, and the encoding module. The first loss can directly optimize the alignment path between acoustic features and semantic features, ensuring dynamic matching between the text sequence generated by the audio understanding model and the audio input. The first fusion module can learn to effectively combine the acoustic features of the encoding module and the semantic features of the large language model body.

[0136] In one possible implementation, determining the second loss based on the first semantic features includes: outputting a first text prediction result corresponding to the first audio based on the first semantic features using the first head; and determining the second loss based on the first text prediction result and the first text.

[0137] like Figure 4 As shown, the audio understanding model also includes a first head, which is connected to the main body of the large language model and initialized based on the large language model's head. The first head can directly reuse the original head structure of the pre-trained large language model, for example, it can include linear layers and softmax layers to inherit the powerful language generation capabilities of the large language model. The parameters of the first head can be copied from the large language model's head, but are independent of the output head, forming output branches for different tasks. Furthermore, the first head can be connected to the main body of the large language model to receive its output first semantic features.

[0138] The first head can directly generate the first text prediction result (such as the probability distribution of characters or words) corresponding to the first audio based on the first semantic features output by the main body of the large language model. By calculating the second loss function (such as cross-entropy loss) between the first text prediction result and the true label (i.e., the first text), a second loss can be obtained. The second loss can be used to provide additional supervision signals and enhance the language modeling ability.

[0139] In one possible implementation, the parameters of the first head are kept fixed during the first phase of training the audio understanding model. In this approach, the parameters of the first head are kept fixed during the first phase of training (i.e., they do not participate in gradient updates), thus preventing the text generation capability from being interfered with by the speech alignment task while incorporating the pre-trained knowledge of the large language model. The first head directly reuses the parameters of the original head of the large language model; by freezing its weights, it ensures that the audio understanding model retains the powerful language modeling capabilities of the large language model. This design leverages the prior knowledge of the large language model to improve text prediction quality while preventing parameter conflicts during multi-task training, leading to more stable convergence of the audio understanding model.

[0140] In one possible implementation, determining the third loss based on the second semantic features includes: outputting the second text prediction result corresponding to the lower half of the first text based on the second semantic features through the second head; and determining the third loss based on the second text prediction result and the lower half of the first text.

[0141] like Figure 4 As shown, the audio understanding model also includes a second head, which is connected to the main body of the large language model and initialized based on the large language model's head. The second head can directly reuse the original head structure of the pre-trained large language model, for example, it can include linear layers and softmax layers to inherit the powerful language generation capabilities of the large language model. The parameters of the second head can be copied from the large language model's head, but are independent of the output head and the first head, forming output branches for different tasks. Furthermore, the second head can be connected to the main body of the large language model to receive its output second semantic features.

[0142] The second head can directly generate the second text prediction result (such as the probability distribution of characters or words) corresponding to the second half of the first text based on the second semantic features output by the large language model body. By calculating the third loss function (such as cross-entropy loss) between the second text prediction result and the lower half of the true label (i.e., the lower half of the first text), the third loss can be obtained. The third loss can be used to provide another additional supervision signal to further enhance the language prediction ability, introducing the ability to predict text to the large speech model body in the first training stage.

[0143] Once the first loss, second loss, and third loss are obtained, the initial audio understanding model can be trained based on the weighted sum of the first loss, second loss, and third loss to obtain the first audio understanding model.

[0144] In this way, large language model subjects can simultaneously use text data to perform joint training that automatically predicts the second half of a sentence based on the first half.

[0145] In one possible implementation, during the second stage of training, step S23 may include: sequentially inputting the audio segment sequence of the first audio into the encoding module of the first audio understanding model; restricting the first audio understanding model to only acquire information about the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment using an attention mask, so that the acoustic segment features corresponding to the current audio segment extracted by the encoding module are sent to the first fusion module in real time; extracting the first semantic feature sequence corresponding to the first text and the third semantic feature sequence corresponding to the first K text segment sequences through a large language model body, the third semantic feature sequence being used to predict the remaining text in the first text; determining all second effective alignment paths that can generate the first text corresponding to the first audio through the first fusion module based on the first semantic feature sequence and the acoustic segment features corresponding to each audio segment, and calculating the second total probability of all second effective alignment paths; training the first audio understanding model based on the second total probability, the first semantic feature sequence, and the third semantic feature sequence to obtain the target audio understanding model.

[0146] To achieve streaming audio processing capabilities, the first audio understanding model can employ a block-based training strategy combined with attention mask control. Specifically, the complete first audio segment can be divided into consecutive audio segment sequences according to a preset duration (e.g., 320ms / block or 640ms / block), and then input into the encoding module sequentially. For example, a 3-second audio segment (3000ms) can be divided into approximately nine 320ms audio segment sequences (padding the ends with zeros), and processed sequentially. Furthermore, a unidirectional mask can be applied in the self-attention layer of the first audio understanding model, allowing it to access only information from the current and historical audio segment sequences, while completely blocking information from future audio segment sequences. This simulates the causality of streaming scenarios, enabling the audio understanding model to output results in real time based solely on the received audio data.

[0147] The large language model in the first audio understanding model can be used to process the input text. By extracting semantic features from the first text, it outputs a first semantic feature sequence. By extracting semantic features from the first K text fragment sequences, it outputs a third semantic feature sequence. This third semantic feature sequence is used to predict the remaining text in the first text excluding the first K text fragment sequences.

[0148] The first fusion module can determine all second effective alignment paths that can generate characters (or words) in the first text corresponding to each audio segment in the first audio by using the first semantic feature sequence and the acoustic segment features corresponding to each audio segment. It can also calculate the second total probability of all second effective alignment paths. The calculation method for the second total probability can refer to the calculation method for the first total probability, and will not be repeated here.

[0149] Then, the fourth loss can be determined based on the second total probability; the fifth loss can be determined based on the first semantic feature; the sixth loss can be determined based on the third semantic feature; and the first audio understanding model can be trained based on the weighted sum of the fourth, fifth, and sixth losses to obtain the target audio understanding model.

[0150] For example, the negative logarithm of the second total probability can be taken to obtain the fourth loss function corresponding to the audio understanding model, and the value of the fourth loss function can be used as the fourth loss. The method for determining the fourth loss can be referred to the method for determining the first loss mentioned above, and will not be repeated here.

[0151] For example, based on the first semantic features, the first header can output the first text prediction result corresponding to the first audio; based on the first text prediction result and the first text, the fifth loss can be determined. The method for determining the fifth loss can refer to the method for determining the second loss mentioned above, and will not be repeated here.

[0152] For example, based on the third semantic features, the second head outputs the prediction results of the remaining text corresponding to the first K text segment sequences in the first text; based on the remaining text prediction results and the remaining text in the first text excluding the first K text segment sequences, the sixth loss is determined. The method for determining the sixth loss can refer to the method for determining the third loss above, and will not be repeated here.

[0153] In this way, the second stage of training can be achieved, simulating the causality of streaming scenarios, allowing the first audio understanding model to output results in real time based solely on the received audio data. Furthermore, the large language model body also begins to possess the ability to randomly use the first few characters of the text as pre-filled lexical units to predict the remaining text. Through multi-stage training, the text decoding capability of the large language model body is further improved.

[0154] Through the first and second stages of training, a trained target audio understanding model can be obtained; the audio to be understood is input into the target audio understanding model, and the target audio understanding model outputs the text prediction result corresponding to the audio to be understood.

[0155] In this embodiment, an optimized target audio understanding model can be obtained through the training method of the audio understanding model described above. This target audio understanding model integrates the acoustic feature extraction capability of the encoding module in a pre-trained speech recognition model with the semantic understanding capability of a large language model. In practical applications, the audio to be understood (such as a real-time speech stream or a complete recording) can be input into the target audio understanding model. The target audio understanding model extracts acoustic features through the encoding module and combines this with the semantic reasoning of the main body of the large language model. Finally, the first fusion module dynamically aligns the audio and text sequences, outputting the corresponding text prediction results. The target audio understanding model provided in this embodiment supports streaming processing, can generate word-by-word text (such as subtitles) in real time, and can also process non-streaming audio, making it suitable for scenarios such as intelligent assistants and meeting transcription.

[0156] In one possible implementation, the method further includes deleting a first head and a second head from the target audio understanding model before inputting the audio to be understood into the target audio understanding model.

[0157] In this implementation, the first and second heads are used only during the training phase to assist in optimizing the text generation task (e.g., through cross-entropy loss). Removing the first and second heads reduces redundant computation and lowers inference latency, which is especially important for real-time streaming processing (such as voice assistants), while ensuring that the target audio understanding model retains only the necessary output paths.

[0158] In one possible implementation, the audio to be understood is either streaming audio or non-streaming audio.

[0159] In this implementation, the target audio understanding model can be used for inference of streaming or non-streaming audio, depending on the actual business scenario.

[0160] In actual deployment, the trained target audio understanding model can be decoded using beam search, and streaming or non-streaming inference can be performed according to the actual business scenario.

[0161] Figure 5 This diagram illustrates the dynamic alignment process of the target audio understanding model provided in the present disclosure during the decoding stage. Figure 5 It can demonstrate the streaming alignment mechanism between acoustic feature sequences (time step t) and semantic feature sequences (text step u).

[0162] exist Figure 5 In the diagram, the horizontal axis (t) can represent the time step of the audio input (such as an audio frame sequence), and the acoustic feature A output by the encoding module is... t (Black circle). The vertical axis (u) can represent the number of steps in text generation (such as the words or characters already output), and the semantic features T output by the main body of the large language model. u (White circle). Node y(t,u) can represent the joint decoding result at time step t and text step u, which may be the actual text label (such as Chinese characters, words) or blank (…). Here, blank indicates that no text has been generated at the current time step, and the time step is only advanced.

[0163] Figure 5 The red arrow in the image shows an example of a decoding path. Horizontal movement (→) indicates time step t+1, text step u) remains unchanged (output blank, e.g.) (1,2)). Vertical movement (↑) means that the time step t remains unchanged, and the text step u+1 (output the actual text, such as y(3,2)).

[0164] In this way, the target audio understanding model can align acoustic features with semantic features in real time when processing streaming audio input and generate corresponding text results. This alignment capability not only improves the interactive experience but also significantly enhances the performance of streaming speech recognition systems.

[0165] In one possible implementation, the text prediction result contains multiple characters. The audio to be understood is input into a target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model. This includes: inputting the audio to be understood into the target audio understanding model; when the number of characters j in the partial text prediction result decoded by the target audio understanding model is less than a preset value N, the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood; the large language model body obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters; the first fusion module determines the second prediction probabilities of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M groups of candidate paths with the highest probability for the jth character; the first fusion module selects the M second prediction probabilities with the highest probability from the second prediction probabilities of multiple candidate paths as the M groups of candidate paths with the highest probability for the (j+1)th character; and the first fusion module decodes the (j+1)th character based on the M groups of candidate paths with the highest probability for the (j+1)th character.

[0166] It should be understood that how the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood; and how the large language model body obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters, can be referred to the encoding module and the large language model body in the training process above, which will not be repeated here.

[0167] Figure 6 This diagram illustrates a decoding method for a target audio understanding model provided in an embodiment of this disclosure. Figure 6 As shown, when the number of characters j decoded by the target audio understanding model is less than N, the target audio understanding model can use beam search to find the optimal output path. Beam search is a heuristic search algorithm that maintains a set of candidate paths at each step. These candidate paths are sorted according to a scoring mechanism, and typically the top few (e.g., M) target paths with the highest scores (highest second-highest prediction probability) are selected to generate the next set of candidate paths. For example, each step corresponds to a character to be decoded. The (j+1)th character to be decoded (j < N) has Vocab candidate paths p1[j+1], p2[j+1], ..., pVocab[j+1]. These paths are determined based on the M groups of paths topM(p1[j], p2[j], ..., pVocab[j]) with the highest second predicted probability for the j-th character, the acoustic segment features corresponding to the (j+1)-th audio segment determined by the encoding module, and the semantic features corresponding to the (j+1)-th audio segment determined by the large language model. Here, Vocab represents the number of possible paths for each character to be decoded, where Vocab > M. Then, the first fusion module selects the M candidate paths with the highest second predicted probability from the Vocab candidate paths and decodes the (j+1)-th character based on these M groups of candidate paths.

[0168] In this way, large language models can be used in real time for collaborative decoding to improve the speech recognition performance of the target audio understanding model.

[0169] In one possible implementation, the text prediction result contains multiple characters. The audio to be understood is input into the target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model. This includes: inputting the audio to be understood into the target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is greater than or equal to a preset value N, initializing a copy of the main body of the large language model, where N is a positive integer; the copy determines multiple first prediction probabilities for decoding the (j+1)th character based on the first N decoded characters; the target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the jth character; and the target audio understanding model decodes the (j+1)th character according to the multiple first prediction probabilities and the multiple second prediction probabilities.

[0170] For example, if the number of characters j decoded by the target audio understanding model is greater than or equal to the preset value N, a copy of the large language model body is initialized based on the large language model body; and multiple first prediction probabilities for decoding the (j+1)th character are determined based on the copy body using the first N decoded characters.

[0171] The first fusion module uses a beam search approach to determine multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character. This includes: the encoding module obtaining the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood; the large language model body obtaining the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters; and the first fusion module determining the second prediction probabilities of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M candidate paths with the highest probability for the j-th character, where M is a positive integer.

[0172] It should be understood that how the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood; and how the large language model body obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters, can be referred to the encoding module and the large language model body in the training process above, which will not be repeated here.

[0173] The (j+1)th character is decoded based on multiple first and second prediction probabilities, including: weighting the multiple first and second prediction probabilities to obtain multiple third prediction probabilities; selecting the M third prediction probabilities with the highest probabilities from the multiple third prediction probabilities as the M candidate paths with the highest probabilities for the (j+1)th character, and decoding the (j+1)th character based on the M candidate paths with the highest probabilities for the (j+1)th character.

[0174] Since the large language model head is introduced with text prediction capabilities in the first stage of training, and the prediction confidence of the large language model head is increased in the second stage of training by partially pre-filled lexical units (such as the first K text segments after the first text is cropped), a copy of the large language model body can be added during the real-time decoding process of the target audio understanding model. The two large language model bodies can be used collaboratively for decoding, that is, a method of using the existing large language model (without adding new parameters to the large language model) for collaborative decoding, so as to further improve the speech recognition effect.

[0175] like Figure 6 As shown, a preset value N can be used. When the number of characters j decoded by the target audio understanding model is greater than or equal to the preset value N, a copy of the large language model body (with the same parameters as the large language model body) can be initialized based on the large language model body. A third header is configured for the copy, which is initialized based on the large language model header. The output header dimension of the first fusion module is controlled to be the same as the output dimension of the third header. This ensures that the two modules can be seamlessly connected in structure, avoiding information loss or calculation errors caused by dimension mismatch.

[0176] The semantic feature sequence extracted from the first N decoded characters is fed into the third head by the copy. The third head then determines multiple first prediction probabilities for decoding the (j+1)th character based on the semantic feature sequence extracted from the first N decoded characters, namely LLM1[j+1], LLM2[j+1], ..., LLMVOcab[j+1]. The first N decoded characters are used to pre-fill the copy. In this way, when the copy predicts characters after N characters, it can directly infer based on the first N decoded characters without calculating intermediate results (for example, when predicting N+2 characters, it does not need to be based on the intermediate predicted N+1th character), which can significantly reduce the amount of computation.

[0177] At this point, the first fusion module in the target audio understanding model begins to decode the (j+1)th character. During the beam search process, the first fusion module maintains the M groups of paths with the highest probability each time. Based on the M groups of candidate paths with the highest probability for the jth character, it can determine the second prediction probability of all Vocab candidate paths used to decode the (j+1)th character, namely: p1[j+1], p2[j+1], ..., pVocab[j+1];

[0178] The first prediction probabilities LLM1[j+1], LLM2[j+1], ..., LLMMVocab[j+1] of Vocab are weighted and summed with the second prediction probabilities p1[j+1], p2[j+1], ..., pVocab[j+1] to obtain the third prediction probabilities of Vocab, namely: p1[j+1] + a LLM1[j+1], p2[j+1] + a LLM2[j+1], ..., pVocab[j+1] + a LLMMVocab[j+1], where a represents the weight, and different parameters can be set according to the actual application scenario. The specific value of the weight a is not limited in the embodiments of this disclosure.

[0179] From the three third prediction probabilities p1[j+1]+a LLM1[j+1], p2[j+1]+a LLM2[j+1], ..., pVocab[j+1]+a LLMVocab[j+1], select the M third prediction probabilities with the highest probabilities, i.e., topM(p1[j+1]+aLLM1[j+1], p2[j+1]+a LLM2[j+1], ..., pVocab[j+1]+a LLMVocab[j+1]), as the M candidate paths with the highest probability for the (j+1)th character. The topM() function is used to extract the M largest elements from multiple elements.

[0180] Assume j=N+1, at this time, the predicted probabilities of the optimal M candidate paths for N+1 characters can be changed from topM(p1[N+1], p2[N+1], ..., pVocab[N+1]) to topM(p1[N+1]+aLLM1[N+1], p2[N+1]+aLLM1[N+1], ..., pVocab[N+1]+aLLMVocab[N+1]). For example, the original probability of the character "今" will be weighted with the probability of "今" in the copy vocabulary, and then participate in the topM calculation. In this way, the copy can be introduced in real time for streaming decoding.

[0181] In this way, the existing large language model (no additional large language model parameters are added separately) can be utilized for collaborative decoding in real time. Although this method uses two large language model subjects collaboratively for decoding, it only needs to load one trained large language model subject into the video memory, and performs two paths of inference synchronously through concurrency or engineering optimization.

[0182] The training method for the audio understanding model and the audio understanding method provided by the embodiments of the present disclosure enable the audio understanding model to perform streaming speech recognition while fully utilizing the text capability of the large language model, so as to improve the effect of speech recognition through collaborative decoding, and can be applied to technical fields such as artificial intelligence, speech recognition, audio understanding, streaming speech recognition, which are not limited herein. Moreover, the audio understanding model trained by the training method for the audio understanding model provided by the embodiments of the present disclosure can be applied to scenarios such as real-time subtitles, voice search, smart home, which are not limited herein.

[0183] Figure 7 shows a block diagram of an audio understanding apparatus according to an embodiment of the present disclosure, as Figure 7 shown in the figure, the audio understanding apparatus includes:

[0184] a first obtaining module 71, configured to obtain audio to be understood;

[0185] an output module 72, configured to input the audio to be understood into a target audio understanding model, and obtain a text prediction result of the audio to be understood output by the target audio understanding model;

[0186] wherein the target audio understanding model is generated after performing a first-stage training and a second-stage training on an initial audio understanding model, and the initial audio understanding model includes an encoding module and a large language model subject; in the first-stage training, parameters of the encoding module are fixed, and the large language model subject is configured to predict the lower half of a first text based on the upper half of the first text; in the second-stage training, parameters of the large language model subject are fixed, and the large language model subject is configured to predict the remaining text segment sequence in the first text based on a random first K text segment sequence obtained by cropping the first text, where K is an integer greater than 1.

[0187] In one possible implementation, the first stage of training includes: with the parameters of the encoding module in the initial audio understanding model fixed, training the initial audio understanding model based on the first audio, the first text, and the first half of the first text to obtain the first audio understanding model; the second stage of training includes: with the parameters of the large language model body in the first audio understanding model fixed, training the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut to obtain the target audio understanding model.

[0188] In one possible implementation, the text prediction result contains multiple characters, and the output module 72 is used to: input the audio to be understood into the target audio understanding model; if the number of characters j in the partial text prediction result decoded by the target audio understanding model is greater than or equal to a preset value N, initialize a copy of the main body of the large language model, where N is a positive integer; the copy determines multiple first prediction probabilities for decoding the (j+1)th character based on the first N decoded characters; the target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the jth character; and the target audio understanding model decodes the (j+1)th character according to the multiple first prediction probabilities and the multiple second prediction probabilities.

[0189] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the main body of the large language model, respectively. The target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character, including: the encoding module obtaining the acoustic segment features corresponding to the (j+1)-th audio segment in the audio to be understood; the main body of the large language model obtaining the semantic features corresponding to the (j+1)-th audio segment based on the first j decoded characters; and the first fusion module determining the second prediction probabilities of multiple candidate paths for decoding the (j+1)-th character based on the acoustic segment features corresponding to the (j+1)-th audio segment, the semantic features corresponding to the (j+1)-th audio segment, and the M groups of candidate paths with the highest probability for the j-th character, where M is a positive integer.

[0190] In one possible implementation, decoding the (j+1)th character based on multiple first prediction probabilities and multiple second prediction probabilities includes: weighting the multiple first prediction probabilities and multiple second prediction probabilities to obtain multiple third prediction probabilities; selecting the M third prediction probabilities with the highest probabilities from the multiple third prediction probabilities as the M candidate paths with the highest probabilities for the (j+1)th character; and decoding the (j+1)th character based on the M candidate paths with the highest probabilities for the (j+1)th character.

[0191] In one possible implementation, the initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the main body of the large language model, respectively. The text prediction result contains multiple characters. The output module 72 is used to: input the audio to be understood into the target audio understanding model; when the number of characters j in the partial text prediction result decoded by the target audio understanding model is less than a preset value N, the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment based on the (j+1)th audio segment in the audio to be understood; the main body of the large language model obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters; the first fusion module determines the second prediction probabilities of multiple candidate paths used to decode the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M groups of candidate paths with the highest probability for the jth character; the first fusion module selects the M second prediction probabilities with the highest probability from the second prediction probabilities of multiple candidate paths as the M groups of candidate paths with the highest probability for the (j+1)th character; the first fusion module decodes the (j+1)th character based on the M groups of candidate paths with the highest probability for the (j+1)th character.

[0192] Figure 8 A block diagram of a training apparatus for an audio understanding model according to an embodiment of the present disclosure is shown, such as Figure 8 As shown, the training device includes:

[0193] The second acquisition module 81 is used to acquire the training set and the initial audio understanding model. The training samples in the training set include the first audio and the first text. The initial audio understanding model includes an encoding module, a large language model body and a first fusion module. The first fusion module is connected to the encoding module and the large language model body respectively.

[0194] The first training module 82 is used to train the initial audio understanding model based on the first audio, the first text, and the second text in the first stage of training, with the parameters of the encoding module in the initial audio understanding model fixed, to obtain the first audio understanding model, wherein the second text is the first half of the first text;

[0195] The second training module 83 is used in the second stage of training to train the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, while fixing the parameters of the large language model body in the first audio understanding model. The target audio understanding model is used to implement the steps of the above audio understanding method.

[0196] The large language model is used in the first stage to train the prediction of the second half of the first text based on the second text, and in the second stage to train the prediction of the remaining text fragments in the first text excluding the K text fragment sequences based on the random first K text fragment sequences, where K is an integer greater than 1.

[0197] In one possible implementation, the first training module 82 is used to: extract a first acoustic feature sequence corresponding to the first audio through an encoding module, and extract a first semantic feature sequence corresponding to the first text and a second semantic feature sequence corresponding to the second text through a large language model body, wherein the second semantic feature sequence is used to predict the lower half of the first text; determine, through a first fusion module, all first effective alignment paths that can generate the first text corresponding to the first audio, and a first total probability of all first effective alignment paths based on the first acoustic feature sequence and the first semantic feature sequence; and train an initial audio understanding model based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain the first audio understanding model.

[0198] In one possible implementation, an initial audio understanding model is trained based on a first total probability, a first semantic feature sequence, and a second semantic feature sequence to obtain a first audio understanding model, including: determining a first loss based on the first total probability; determining a second loss based on the first semantic feature; determining a third loss based on the second semantic feature; and training the initial audio understanding model based on the weighted sum of the first loss, the second loss, and the third loss to obtain the first audio understanding model.

[0199] In one possible implementation, the first audio understanding model further includes a first head and a second head, which are respectively connected to the main body of a large language model, and both the first head and the second head are initialized based on the head of the large language model; determining a first loss based on a first total probability includes: determining the first loss based on the negative logarithm of the first total probability; determining a second loss based on a first semantic feature includes: outputting a first text prediction result corresponding to the first audio through the first head based on the first semantic feature; determining the second loss based on the first text prediction result and the first text; determining a third loss based on a second semantic feature includes: outputting a second text prediction result corresponding to the second half of the first text through the second head based on the second semantic feature; determining the third loss based on the second text prediction result and the second half of the first text.

[0200] In one possible implementation, the second training module 83 is used to: sequentially input the audio segment sequence of the first audio into the encoding module of the first audio understanding model; restrict the first audio understanding model to only acquire information of the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment through an attention mask, so that the acoustic segment features corresponding to the current audio segment extracted by the encoding module are sent to the first fusion module in real time; extract the first semantic feature sequence corresponding to the first text and the third semantic feature sequence corresponding to the first K text segment sequences through a large language model body, the third semantic feature sequence being used to predict the remaining text in the first text; determine all second effective alignment paths that can generate the first text corresponding to the first audio through the first fusion module based on the first semantic feature sequence and the acoustic segment features corresponding to each audio segment, and calculate the second total probability of all second effective alignment paths; train the first audio understanding model according to the second total probability, the first semantic feature sequence, and the third semantic feature sequence to obtain the target audio understanding model.

[0201] In one possible implementation, the method for determining the value of K includes: for any training sample in the training set, obtaining the duration of the first audio and the number of words in the first text; and determining the value of K based on the mean and variance of the duration occupied by each word.

[0202] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0203] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0204] This disclosure also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0205] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0206] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server or terminal device. (Refer to...) Figure 9 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0207] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system stored in memory 1932.

[0208] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0209] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0210] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0211] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0212] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0213] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0214] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0215] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0216] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.< / eos> < / eos> < / eos> < / sos> < / sos>

Claims

1. An audio understanding method, characterized in that, The method includes: Obtain the audio to be understood; The audio to be understood is input into the target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model; The target audio understanding model is generated by performing a first-stage training and a second-stage training on an initial audio understanding model. The initial audio understanding model includes an encoding module and a large language model body. In the first-stage training, the parameters of the encoding module are fixed, and the large language model body is used to predict the second half of the first text based on the first half of the first text. In the second-stage training, the parameters of the large language model body are fixed, and the large language model body is used to predict the remaining text segment sequence in the first text based on the random first K text segment sequence after the first text is cropped, where K is an integer greater than 1.

2. The method according to claim 1, characterized in that, The first stage of training includes: training the initial audio understanding model based on the first audio, the first text, and the first half of the first text, while fixing the parameters of the encoding module in the initial audio understanding model, to obtain the first audio understanding model; The second stage of training includes: with the parameters of the large language model body in the first audio understanding model fixed, training the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, to obtain the target audio understanding model.

3. The method according to claim 2, characterized in that, The text prediction result contains multiple characters. The audio to be understood is input into the target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model, including: The audio to be understood is input into the target audio understanding model. If the number of characters j in the partial text prediction result decoded by the target audio understanding model is greater than or equal to a preset value N, a copy of the main body of the large language model is initialized, where N is a positive integer. The copy determines multiple first prediction probabilities for decoding the (j+1)th character based on the first N decoded characters; The target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character; The target audio understanding model decodes the (j+1)th character based on the plurality of first prediction probabilities and the plurality of second prediction probabilities.

4. The method according to claim 3, characterized in that, The initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the main body of the large language model, respectively. The target audio understanding model determines multiple second prediction probabilities for decoding the (j+1)th character based on the j-th character, including: The encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood; The large language model body obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters; The first fusion module determines the second prediction probability of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M groups of candidate paths with the highest probability of the jth character. M is a positive integer.

5. The method according to claim 3, characterized in that, Based on the plurality of first prediction probabilities and the plurality of second prediction probabilities, the (j+1)th character is decoded, including: Multiple third prediction probabilities are obtained by weighting and summing the multiple first prediction probabilities and the multiple second prediction probabilities; Select the M third prediction probabilities with the highest probabilities from multiple third prediction probabilities, and use them as the M candidate paths with the highest probability for the (j+1)th character; Based on the M candidate paths with the highest probability of the (j+1)th character, decode the (j+1)th character.

6. The method according to claim 2, characterized in that, The initial audio understanding model further includes a first fusion module, which is connected to the encoding module and the main body of the large language model, respectively. The text prediction result contains multiple characters. The audio to be understood is input into the target audio understanding model to obtain the text prediction result of the audio to be understood output by the target audio understanding model, including: The audio to be understood is input into the target audio understanding model. If the number of characters j in the partial text prediction result decoded by the target audio understanding model is less than a preset value N, the encoding module obtains the acoustic segment features corresponding to the (j+1)th audio segment in the audio to be understood. The large language model body obtains the semantic features corresponding to the (j+1)th audio segment based on the first j decoded characters; The first fusion module determines the second predicted probability of multiple candidate paths for decoding the (j+1)th character based on the acoustic segment features corresponding to the (j+1)th audio segment, the semantic features corresponding to the (j+1)th audio segment, and the M groups of candidate paths with the highest probability of the jth character. The first fusion module selects the M second prediction probabilities with the highest probability from the second prediction probabilities of multiple candidate paths, and uses them as the M group of candidate paths with the highest probability of the (j+1)th character. The first fusion module decodes the (j+1)th character based on the M candidate paths with the highest probability of the (j+1)th character.

7. A training method for an audio understanding model, characterized in that, The method includes: A training set and an initial audio understanding model are obtained. The training samples in the training set include a first audio and a first text. The initial audio understanding model includes an encoding module, a large language model body, and a first fusion module. The first fusion module is connected to the encoding module and the large language model body, respectively. In the first stage of training, with the parameters of the encoding module in the initial audio understanding model fixed, the initial audio understanding model is trained based on the first audio, the first text, and the second text to obtain the first audio understanding model, wherein the second text is the upper half of the first text; In the second stage of training, with the parameters of the large language model body in the first audio understanding model fixed, the first audio understanding model is trained based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, to obtain the target audio understanding model, which is used to implement the steps of the method according to any one of claims 1 to 6. The large language model is used to train in the first stage to predict the second half of the first text based on the second text, and in the second stage to train to predict the remaining text segment sequence in the first text based on the random first K text segment sequence, where K is an integer greater than 1.

8. The method according to claim 7, characterized in that, The initial audio understanding model is trained based on the first audio, the first text, and the second text to obtain the first audio understanding model, which includes: The encoding module extracts the first acoustic feature sequence corresponding to the first audio, and the large language model extracts the first semantic feature sequence corresponding to the first text and the second semantic feature sequence corresponding to the second text. The second semantic feature sequence is used to predict the second half of the first text. The first fusion module determines, based on the first acoustic feature sequence and the first semantic feature sequence, all first effective alignment paths that can generate the first text corresponding to the first audio, and the first total probability of all first effective alignment paths. The initial audio understanding model is trained based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence to obtain the first audio understanding model.

9. The method according to claim 8, characterized in that, Based on the first total probability, the first semantic feature sequence, and the second semantic feature sequence, the initial audio understanding model is trained to obtain the first audio understanding model, including: The first loss is determined based on the first total probability; The second loss is determined based on the first semantic feature; The third loss is determined based on the second semantic feature; The initial audio understanding model is trained based on the weighted sum of the first loss, the second loss, and the third loss to obtain the first audio understanding model.

10. The method according to claim 9, characterized in that, The first audio understanding model further includes a first head and a second head, the first head and the second head being connected to the main body of the large language model respectively, and the first head and the second head being initialized based on the head of the large language model. The first loss is determined based on the first total probability, including: The first loss is determined based on the negative logarithm of the first total probability; Determining the second loss based on the first semantic feature includes: Based on the first semantic features, the first head outputs the first text prediction result corresponding to the first audio. Based on the prediction result of the first text and the first text, determine the second loss; The third loss is determined based on the second semantic feature, including: Based on the second semantic features, the second header outputs the second text prediction result corresponding to the lower half of the first text; The third loss is determined based on the prediction result of the second text and the second half of the first text.

11. The method according to claim 7, characterized in that, In the second stage of training, with the parameters of the large language model body in the first audio understanding model fixed, the first audio understanding model is trained based on the audio segment sequence after the first audio clipping and the first K text segment sequence after the first text clipping, to obtain the target audio understanding model, including: The audio segment sequence of the first audio is sequentially input into the encoding module of the first audio understanding model; By using an attention mask, the first audio understanding model is restricted to only acquiring information about the current audio segment and the historical audio segments preceding the current audio segment when processing the current audio segment, so that the acoustic segment features corresponding to the current audio segment extracted by the encoding module are sent to the first fusion module in real time; The first semantic feature sequence corresponding to the first text and the third semantic feature sequence corresponding to the first K text fragment sequences are extracted through the large language model body. The third semantic feature sequence is used to predict the remaining text in the first text. The first fusion module determines all second effective alignment paths that can generate the first text corresponding to the first audio based on the first semantic feature sequence and the acoustic segment features corresponding to each audio segment, and calculates the second total probability of all second effective alignment paths. The first audio understanding model is trained based on the second total probability, the first semantic feature sequence, and the third semantic feature sequence to obtain the target audio understanding model.

12. The method according to claim 7, characterized in that, The methods for determining the value of K include: For any training sample in the training set, obtain the duration of the first audio and the number of words in the first text; The K value is determined based on the mean and variance of the time occupied by each character.

13. An audio understanding device, characterized in that, include: The first acquisition module is used to acquire the audio to be understood; The output module is used to input the audio to be understood into the target audio understanding model and obtain the text prediction result of the audio to be understood output by the target audio understanding model. The target audio understanding model is generated by performing a first-stage training and a second-stage training on an initial audio understanding model. The initial audio understanding model includes an encoding module and a large language model body. In the first-stage training, the parameters of the encoding module are fixed, and the large language model body is used to predict the second half of the first text based on the first half of the first text. In the second-stage training, the parameters of the large language model body are fixed, and the large language model body is used to predict the remaining text segment sequence in the first text based on the random first K text segment sequence after the first text is cropped, where K is an integer greater than 1.

14. A training device for an audio understanding model, characterized in that, include: The second acquisition module is used to acquire a training set and an initial audio understanding model. The training samples in the training set include a first audio and a first text. The initial audio understanding model includes an encoding module, a large language model body, and a first fusion module. The first fusion module is connected to the encoding module and the large language model body, respectively. The first training module is used to train the initial audio understanding model based on the first audio, the first text, and the second text in the first stage of training, while keeping the parameters of the encoding module in the initial audio understanding model fixed, to obtain the first audio understanding model, wherein the second text is the upper half of the first text; The second training module is used in the second stage of training to train the first audio understanding model based on the audio segment sequence after the first audio is cut and the random first K text segment sequence after the first text is cut, while fixing the parameters of the large language model body in the first audio understanding model, to obtain the target audio understanding model. The target audio understanding model is used to implement the steps of the method according to any one of claims 1 to 6. The large language model is used to train in the first stage to predict the second half of the first text based on the second text, and in the second stage to train to predict the remaining text fragments in the first text excluding the K text fragment sequences based on the random first K text fragment sequences, where K is an integer greater than 1.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 12.

16. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

17. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images