Speech abstract extraction method and device and computer readable storage medium

Abstract

The embodiment of the invention relates to the technical field of natural language processing, and discloses a voice abstract extraction method and device and a computer readable storage medium, and the method comprises the steps: obtaining a to-be-converted voice; extracting the to-be-converted voice based on an audio encoder to obtain audio lexical elements; wherein the audio encoder is generated based on a knowledge distillation algorithm; generating a voice abstract corresponding to the to-be-converted voice through a preset voice abstract generation model based on the audio lemma; wherein the voice abstract generation model is obtained by training a language processing model according to an audio lexical element sample and a voice abstract sample. According to the technical scheme of the invention, the feature information of the to-be-converted voice is mined through the audio encoder improved by the knowledge distillation technology, so that the extracted audio lexical elements have more audio features, and the accuracy of extracting the abstract corresponding to the to-be-converted voice by the voice abstract model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of natural language processing, and in particular to a method and device for extracting a speech summary, and a computer-readable storage medium. Background Art

[0002] At present, speech summary generation is to give a piece of speech audio and then extract summary text information from the speech. Speech summary is an important scenario where speech and natural language are combined, and it has important applications in sectors such as conference summary and smart marketing. The existing technology uses a series processing method of automatic speech recognition (Automatic Speech Recognition, referred to as ASR) and text summary technology to generate speech summaries. This method first converts speech into text through ASR, and then applies text summary technology to extract key information and generate a summary. However, the errors generated by the ASR technical step will propagate to the text summary technical step, resulting in low accuracy in generating speech summaries. Summary of the invention

[0003] In view of the above problems, the embodiments of the present invention provide a method and device for extracting a speech summary, and a computer-readable storage medium, which are used to solve the technical problem of low accuracy in generating a speech summary in the prior art.

[0004] According to one aspect of an embodiment of the present invention, a method for extracting a speech summary is provided, the method comprising:

[0005] Get the voice to be converted;

[0006] Extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm;

[0007] Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

[0008] According to another aspect of an embodiment of the present invention, there is provided a device for extracting a speech summary, comprising: a data acquisition module, for acquiring speech to be converted;

[0009] A data extraction module, used to extract the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm;

[0010] The result generation module is used to generate a speech summary corresponding to the speech to be converted based on the audio word unit through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word unit samples and speech summary samples.

[0011] In an optional manner, the training of the audio encoder includes:

[0012] Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0013] Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0014] Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model;

[0015] Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

[0016] In an optional manner, inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model to calculate the initial audio encoder loss includes:

[0017] Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model;

[0018] Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model;

[0019] Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated;

[0020] An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

[0021] In an optional manner, the logarithmic distillation loss is generated based on the predicted logarithms generated after the audio word unit sample and the corresponding text word unit sample are respectively input into the speech summary generation model, including:

[0022] Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model;

[0023] Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model;

[0024] The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

[0025] In an optional manner, generating a feature distillation loss based on hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes:

[0026] Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model;

[0027] Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model;

[0028] The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

[0029] In an optional manner, the generating a speech summary corresponding to the speech to be converted based on the audio word by using a preset speech summary generation model includes:

[0030] Determine whether the user's summary prompt text is received;

[0031] If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text;

[0032] If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

[0033] In an optional manner, combining the audio word-grams and the text word-grams to obtain a word-gram sequence includes:

[0034] The audio word units and the text word units are arranged in a manner that the audio word units are arranged first and the text word units are arranged later to generate a word unit sequence.

[0035] According to another aspect of an embodiment of the present invention, there is provided a device for extracting a speech summary, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0036] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the following operations:

[0037] Get the voice to be converted;

[0038] Extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm;

[0039] Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

[0040] In an optional manner, the training of the audio encoder includes:

[0041] Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0042] Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0043] Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model;

[0044] Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

[0045] In an optional manner, inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model to calculate the initial audio encoder loss includes:

[0046] Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model;

[0047] Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model;

[0048] Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated;

[0049] An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

[0050] In an optional manner, the logarithmic distillation loss is generated based on the predicted logarithms generated after the audio word unit sample and the corresponding text word unit sample are respectively input into the speech summary generation model, including:

[0051] Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model;

[0052] Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model;

[0053] The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

[0054] In an optional manner, generating a feature distillation loss based on hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes:

[0055] Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model;

[0056] Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model;

[0057] The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

[0058] In an optional manner, the generating a speech summary corresponding to the speech to be converted based on the audio word by using a preset speech summary generation model includes:

[0059] Determine whether the user's summary prompt text is received;

[0060] If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text;

[0061] If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

[0062] In an optional manner, combining the audio word-grams and the text word-grams to obtain a word-gram sequence includes:

[0063] The audio word units and the text word units are arranged in a manner that the audio word units are arranged first and the text word units are arranged later to generate a word unit sequence.

[0064] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a speech summary extraction device / apparatus to perform the following operations:

[0065] Get the voice to be converted;

[0066] Extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm;

[0067] Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

[0068] In an optional manner, the training of the audio encoder includes:

[0069] Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0070] Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0071] Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model;

[0072] Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

[0073] In an optional manner, inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model to calculate the initial audio encoder loss includes:

[0074] Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model;

[0075] Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model;

[0076] Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated;

[0077] An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

[0078] In an optional manner, the logarithmic distillation loss is generated based on the predicted logarithms generated after the audio word unit sample and the corresponding text word unit sample are respectively input into the speech summary generation model, including:

[0079] Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model;

[0080] Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model;

[0081] The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

[0082] In an optional manner, generating a feature distillation loss based on hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes:

[0083] Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model;

[0084] Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model;

[0085] The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

[0086] In an optional manner, the generating a speech summary corresponding to the speech to be converted based on the audio word by using a preset speech summary generation model includes:

[0087] Determine whether the user's summary prompt text is received;

[0088] If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text;

[0089] If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

[0090] In an optional manner, combining the audio word-grams and the text word-grams to obtain a word-gram sequence includes:

[0091] The audio word units and the text word units are arranged in a manner that the audio word units are arranged first and the text word units are arranged later to generate a word unit sequence.

[0092] The embodiment of the present invention uses an improved audio encoder involving knowledge distillation technology to mine feature information of the speech to be converted, so that the extracted audio words have more audio features, thereby improving the accuracy of the speech summary model in extracting the corresponding summary of the speech to be converted. Compared with the prior art that first converts the speech to be converted into text and generates a summary based on the converted audio text, the present application can directly process the audio, avoid errors caused by audio-to-text conversion, and thus improve the accuracy of text summary extraction.

[0093] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0095] Figure 1 A schematic flow chart of a first embodiment of a method for extracting a speech summary provided by the present invention is shown;

[0096] Figure 2 A schematic flow chart of a second embodiment of the method for extracting a speech summary provided by the present invention is shown;

[0097] Figure 3 A schematic structural diagram of a first embodiment of a device for extracting a speech summary provided by the present invention is shown;

[0098] Figure 4A schematic diagram showing the structure of an embodiment of a device for extracting a speech summary provided by the present invention;

[0099] Figure 5 A schematic diagram showing the structure of an embodiment of the application of LLM and an audio encoder provided by the present invention;

[0100] Figure 6 A schematic structural diagram of an embodiment of LLM and audio encoder training provided by the present invention is shown. DETAILED DESCRIPTION

[0101] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0102] Figure 1 FIG. 1 is a flow chart showing a first embodiment of a method for extracting a speech summary of the present invention, wherein the method is executed by a device for extracting a speech summary. Figure 1 As shown, the method comprises the following steps:

[0103] Step 110: Acquire the speech to be converted.

[0104] The voice to be converted is received by receiving a voice file input by the user or a voice recorded by the user in real time.

[0105] Step 120: extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm.

[0106] Among them, an audio encoder is used, which can convert continuous speech signals into discrete word units. These words can then be provided as input to LLM (the speech summary generation model adopted in this embodiment, Large Language Model, Large Language Model, LLM for short). The key to the audio encoder is that it can capture the key semantic information in the speech and present it in a format suitable for LLM processing.

[0107] In a specific embodiment, the knowledge distillation algorithm includes: feature distillation technology and logarithmic (Logit) distillation technology, using the LLM response obtained from the text input as a teacher model to guide the audio encoder to generate predictions and hidden states that match the text input.

[0108] Step 130: Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

[0109] The method of generating a speech summary corresponding to the speech to be converted by using a preset speech summary generation model includes:

[0110] Determine whether the user's summary prompt text is received;

[0111] If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text;

[0112] If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

[0113] The step of combining the audio word unit and the text word unit to obtain a word unit sequence includes:

[0114] The audio word units and the text word units are arranged in a manner that the audio word units are arranged first and the text word units are arranged later to generate a word unit sequence.

[0115] In a specific embodiment, the speech summary generation model adopts the concept of modality invariance during training, that is, whether the input is speech or text, as long as the semantic information is the same, the speech summary generation model should be able to generate the same response. This training method ensures the robustness of the system to different input modalities.

[0116] In another optional embodiment, the speech summary generation model can use an instruction-tuned LLM, which is specially trained to generate corresponding text output according to the user's instructions or prompts. In this system, the LLM receives the token representation from the audio encoder and generates a summary based on this information.

[0117] It should be noted that by adjusting the prompt strategy for LLM, the system can generate summaries in different styles, thus overcoming the problem of single style of traditional methods.

[0118] In this embodiment, the speech summary extraction device / system is trained as an end-to-end pipeline, and the audio encoder and LLM are jointly optimized to minimize the difference between the speech input and the text input and improve the quality and consistency of the summary.

[0119] Preferably, the speech summary extraction device / system is about how to implement the function of speech summary generation through software components and algorithms. The speech summary extraction device / system consists of two components: a large language model (LLM) and an audio encoder, which converts speech input into audio word embeddings that the LLM can receive as part of the prompt;

[0120] In this example, MiniChat-3B is used as the specific model of LLM, which is an instruction-adjusted version of MiniChat-3B, which is extracted from Llama 2 7B. Although small by LLM standards, it performs well on multiple benchmarks compared to other 3B parameter models and is close to the performance of Llama 2 7B, making it a suitable choice. Like most other decoder-only language models, MiniChat receives a sequence of token embeddings as input and autoregressively generates text responses;

[0121] Meanwhile, the backbone network of the audio encoder is initialized with a fine-tuned HuBERT-Large model. It receives as input mono audio with a sampling rate of 16kHz. After the HuBERT layer that produces a 1024-dimensional representation every 20ms (50Hz), we perform a 4x temporal downsampling by average pooling with a kernel size of 8 and a stride of 4. This is followed by a linear projection layer to match the dimensionality of the pooled representation with that of the LLM (3072 dimensions). Therefore, the encoder produces audio tokens at a rate of 12.5Hz;

[0122] Therefore, the audio encoder is the front-end part of the speech summary extraction device / system, responsible for converting the input speech signal into a series of word-unit representations. It is usually based on a deep learning model, using HuBERT to extract speech features and downsample and project them into a dimensional space suitable for LLM processing; the large language model (LLM) is the core of the speech summary extraction device / system. LLM receives the word-unit sequence output by the audio encoder and generates a text summary based on the input. LLM can generate corresponding text according to the given prompt through instruction adjustment.

[0123] To better illustrate, here are some examples:

[0124] Suppose the user wants to generate a summary from a recording of a scientific lecture on climate change. Input: a 10-minute recording of a lecture on climate change.

[0125] Reasoning process

[0126] 1. Preprocessing:

[0127] The recording file is converted into a single channel (mono) audio (i.e., the audio to be converted as described in the present invention), and if necessary, the sampling rate is adjusted to 16kHz.

[0128] 2. Feature extraction:

[0129] Use the trained audio encoder to process the mono audio, extract features and generate a series of audio token embeddings (i.e., the audio word unit described in the present invention).

[0130] 3. Sequence construction:

[0131] If there are text prompts or instructions (for example, the user specifies the length or style of the summary), these texts are converted into token embeddings (i.e., text words as described in the present invention) and combined with the audio token embeddings.

[0132] 4. Input Merge:

[0133] The audio token embeddings and text token embeddings (if any) are merged into a single input sequence (i.e., the word-gram sequence described in the present invention).

[0134] 5. Position encoding:

[0135] Add positional encodings to the merged input sequence so that the model understands the order of tokens.

[0136] 6. Enter LLM:

[0137] The final input sequence is provided to the trained LLM (ie, the speech summary generation model of the present invention).

[0138] 7. Generate summary:

[0139] The LLM receives an input sequence and autoregressively generates an initial speech summary, predicting the next token based on the tokens in the sequence until a complete summary is generated or a predetermined summary length is reached.

[0140] 8. Post-processing:

[0141] Perform any necessary post-processing on the generated speech summary, such as removing extraneous tokens, correcting grammatical errors, or formatting.

[0142] 9. Output summary:

[0143] Output the final speech summary, which is a brief summary of the original speech content, that is, output: a paragraph of text that summarizes the main content of the lecture on climate change, such as key points, important findings, or conclusions.

[0144] Specific steps analysis:

[0145] The user uploaded a 10-minute recording of a lecture on climate change.

[0146] The audio encoder processes the recording and generates a series of audio token embeddings.

[0147] A user might provide a text prompt: "Please summarize the main points of this lecture."

[0148] The text prompts are converted into token embeddings and merged with the audio token embeddings.

[0149] The merged sequences were input into the LLM.

[0150] LLM generates a summary based on an input sequence, for example: “This lecture discusses three major aspects of climate change: rising global temperatures, increases in extreme weather events, and impacts on ecosystems.”

[0151] The generated summary is shown to the user, who gets a succinct summary of the lecture content.

[0152] In a specific embodiment, the application of the audio processor and the LLM model is shown in Figure 5 .

[0153] Beneficial Effects

[0154] This embodiment develops an end-to-end algorithm that can directly process speech input and generate text summaries without relying on the traditional automatic speech recognition (ASR) and text summarization tandem processing method. The core of this algorithm is to use the powerful processing and reasoning capabilities of the large language model LLM, combined with the audio encoder technology improved by knowledge distillation, to achieve direct understanding of speech content and summary generation.

[0155] Figure 2 FIG. 2 is a flow chart showing another embodiment of a method for extracting a speech summary of the present invention, wherein the method is executed by a device for extracting a speech summary. Figure 2 As shown, the method comprises the following steps:

[0156] Step 210: Acquire the speech to be converted.

[0157] Herein, a segment of speech is received as speech to be converted, and the segment of speech can be spoken content in any field.

[0158] Step 220: extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm; the training of the audio encoder includes:

[0159] Step 221: obtaining a voice sample and a corresponding summary prompt text sample; wherein the voice sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0160] Step 222: input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0161] Step 223: input the audio word-unit sample and the corresponding text word-unit sample into the speech summary generation model, and calculate the initial audio encoder loss based on the feedback of the speech summary generation model;

[0162] Step 224: Based on the initial audio encoder loss, adjust parameters of the initial audio encoder to obtain a final audio encoder.

[0163] The audio encoder uses the Hubert model to process the speech signal and extract key speech features. In an optional embodiment, the training of the audio encoder includes:

[0164] Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0165] Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0166] Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model;

[0167] Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

[0168] In a specific embodiment, a Hubert model is trained using a paired speech-text dataset. The audio encoder learns how to convert speech into a sequence of tokens that the LLM can process. The LLM learns how to generate accurate summaries based on the token sequence. At a high level, it involves aligning the LLM's response to the audio with the responses to the matching text prompts that have the same semantic information. In order to leverage the power of the LLM without affecting its original performance, the weights of the LLM are kept fixed in all experiments, and the audio encoder is the only trainable component in our system.

[0169] In an optional embodiment, the training of the audio encoder includes:

[0170] 1. Data preparation:

[0171] In a dataset containing speech and corresponding text (such as Librispeech), select a speech sample and the corresponding summary prompt text sample.

[0172] 2. Feature extraction:

[0173] The speech sample is input into an audio encoder (e.g. based on the HuBERT model), features are extracted and audio word samples are generated.

[0174] 3. Text processing:

[0175] Convert the summary prompt text sample into a text word sample through a word segmenter (such as SentencePiece);

[0176] Convert audio word samples and text word samples into embedding form.

[0177] 4. Sequence construction:

[0178] Arrange the audio word sample embedding and the text word sample embedding in the order they appear in the original input to form a sequence.

[0179] 5. Position code added:

[0180] Add positional encoding to each token in the sequence to provide token order information.

[0181] 6. Input to LLM:

[0182] The constructed sequence is provided as input to the LLM.

[0183] 7. Freeze LLM weight:

[0184] During training, the weights of the LLM remain frozen and are not updated.

[0185] 8. Training the audio encoder:

[0186] The audio encoder is trained using a prediction (NTP) loss so that its output audio token samples enable the LLM to generate the same response as the summary prompt text sample.

[0187] 9. Knowledge Distillation:

[0188] The audio encoder is trained through knowledge distillation, using the pre-trained LLM response as the teacher model, and guiding the training of the audio encoder through logit distillation and feature distillation. Until the audio encoder can accurately convert speech into audio word sample embeddings that LLM can understand, and LLM can generate high-quality summaries.

[0189] In an optional embodiment, inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model to calculate the initial audio encoder loss includes:

[0190] Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model;

[0191] Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model;

[0192] Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated;

[0193] An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

[0194] In a specific embodiment, the cross entropy loss is calculated, which is the difference between the predicted next word and the true

[0195]

[0196] Cross entropy loss between responses.

[0197] In the formula, x=(x0,…,x t ,x t+1 ,…,x T ) is the complete word sequence input to the model, (x0,…,x t ) is the prompt part, including audio word samples and words in the chat structure of LLM, (x t+1 ,…,x T ) is the token of the LLM’s true response to the summary prompt text sample, p(x i ∣x <i ) is the probability of the model predicting the next word xi given all the previous words, and T is the total length of the sequence.

[0198] In a specific embodiment, Logit Distillation (LD) and Feature Distillation (FD), which are two forms of Knowledge Distillation (KD), are used to train the audio encoder to ensure that it can guide the LLM to generate responses corresponding to the summary prompt text samples.

[0199] By calculating the logarithmic distillation loss, feature distillation loss and cross entropy loss, substituting them into the preset weighted summation formula, the initial audio encoder loss is calculated; the weighted summation formula is specifically:

[0200] L total =λ NTP L NTP +λ LD L LD +λ FD L FD

[0201] Where λNTP, λLD and λFD are the weight factors of cross entropy loss, log distillation loss and feature distillation loss, respectively, which are used to balance the contribution of these three losses to the total loss.

[0202] As a preferred embodiment, according to experimental experience, λNTP=0.5, λ LD =0.5,λFD=1.0.

[0203] In an optional embodiment, the generating the logarithmic distillation loss based on the predicted logarithms generated after the audio word unit sample and the corresponding text word unit sample are respectively input into the speech summary generation model includes:

[0204] Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model;

[0205] Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model;

[0206] The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

[0207] In a specific embodiment, the processing process of logarithmic distillation (LD) is specifically as follows:

[0208] Goal: The goal of log distillation is to make the prediction of LLM for audio input (i.e., prediction of the next token) close to the prediction of LLM for text input.

[0209] operate:

[0210] For a given summary prompt text sample, first obtain the text prediction logits of the next word generated by LLM, denoted as

[0211] Then, using the same summary prompt text sample as a condition, let the audio encoder generate the embedding corresponding to the audio word sample.

[0212] These audio word samples are input into LLM to obtain the audio prediction logarithm of the next token based on the audio word sample input, denoted as

[0213] Logarithmic loss calculation:

[0214] The soft cross entropy loss function is used to calculate the difference between the LLM predictions for text prompts and audio prompts. The soft cross entropy loss function formula is as follows:

[0215]

[0216] Where V is the vocabulary of the LLM and LLM is the number of predictions for the next token.

[0217] optimization:

[0218] Through the back propagation algorithm, according to L LD The parameters of the audio encoder are adjusted so that the LLM predictions for audio input are as close as possible to the predictions for text input.

[0219] In an optional embodiment, generating a feature distillation loss based on hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes:

[0220] Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model;

[0221] Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model;

[0222] The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

[0223] In a specific embodiment, the process of feature distillation (FD) is as follows:

[0224] Goal: The goal of feature distillation is to make the hidden state of the LLM under audio input similar to the hidden state of the LLM under text input.

[0225] operate:

[0226] For the summary prompt text sample, obtain the text hidden state of LLM at a specific connector layer, denoted as

[0227] Using the audio word sample generated by the audio encoder as input, we obtain the audio hidden state of the LLM at the same connector layer as the summary prompt text sample input, denoted as

[0228] Loss calculation:

[0229] The mean squared error (MSE) loss function is used to calculate the difference between the LLM hidden states for textual and audio cues, as follows:

[0230]

[0231] Where L is the index set of the selected connector layer, and They are the hidden states of LLM at the lth layer under the summary prompt text sample input and speech sample input respectively.

[0232] optimization:

[0233] Through the back propagation algorithm, according to L FD The parameters of the audio encoder are adjusted so that the LLM hidden state under audio input is as close as possible to the hidden state under text input.

[0234] Through these two distillation methods, the audio encoder is trained to effectively convert audio input into token embeddings that can guide the LLM to generate responses corresponding to text input. In this way, LLM is able to generate semantically consistent output regardless of whether the input is audio or text.

[0235] To better illustrate audio encoder training, the following example is given:

[0236] Input the 60-second speech sample into the audio encoder to obtain a series of audio word sample embeddings;

[0237] Transcribe the summary prompt text sample, tokenize it, and convert it into text word sample embedding;

[0238] Merge the audio word sample embedding and the text word sample embedding, and add position encoding;

[0239] Input the merged sequence into LLM;

[0240] Freeze the weights of LLM and only train the audio encoder;

[0241] The audio encoder is trained using the NTP loss, with the goal of making the LLM’s response to the merged sequence consistent with the response to the text input;

[0242] Further optimize the audio encoder through knowledge distillation to make it better mimic the behavior of LLM when given text input;

[0243] After many iterations, the audio encoder learns how to effectively convert speech into word embeddings, and the LLM is able to generate accurate summaries based on these embeddings.

[0244] Through this process, the system is able to learn how to generate high-quality text summaries from speech input in any domain.

[0245] Step 230: Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

[0246] Among them, the extracted speech features are converted into word-unit sequences, which can be understood by the speech summary generation model.

[0247] In a specific embodiment, the application of the audio processor and the LLM model is shown in Figure 6 .

[0248] Beneficial Effects

[0249] The technical solution proposed in this embodiment not only improves the accuracy and diversity of the summary, but also improves the efficiency and generalization ability of the system by directly using LLM to process the speech input and generate the summary, while simplifying the user operation and increasing the security and scalability of the system.

[0250] Figure 3 FIG. 2 is a schematic diagram showing a structure of an embodiment of a device for extracting speech summaries of the present invention. Figure 3 As shown, the device 300 includes: a data acquisition module 310, a data extraction module 320 and a result generation module 330.

[0251] A data acquisition module is used to obtain the voice and text prompts to be converted;

[0252] A data extraction module, used to extract the audio word corresponding to the speech to be converted, and to extract the text word corresponding to the text prompt;

[0253] The result generation module is used to extract the speech summary corresponding to the text prompt from the speech to be converted based on the audio word unit and the text word unit through a preset speech summary generation model; wherein the large language model is trained with audio word unit samples and user instructions as input and the speech summary generation model as output.

[0254] In an optional manner, the training of the audio encoder includes:

[0255] Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag;

[0256] Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample;

[0257] Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model;

[0258] Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

[0259] In an optional manner, inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model to calculate the initial audio encoder loss includes:

[0260] Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model;

[0261] Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model;

[0262] Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated;

[0263] An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

[0264] In an optional manner, the logarithmic distillation loss is generated based on the predicted logarithms generated after the audio word unit sample and the corresponding text word unit sample are respectively input into the speech summary generation model, including:

[0265] Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model;

[0266] Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model;

[0267] The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

[0268] In an optional manner, generating a feature distillation loss based on hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes:

[0269] Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model;

[0270] Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model;

[0271] The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

[0272] In an optional manner, the generating a speech summary corresponding to the speech to be converted based on the audio word by using a preset speech summary generation model includes:

[0273] Determine whether the user's summary prompt text is received;

[0274] If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text;

[0275] If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

[0276] In an optional manner, combining the audio word-grams and the text word-grams to obtain a word-gram sequence includes:

[0277] The audio word units and the text word units are arranged in a manner that the audio word units are arranged first and the text word units are arranged later to generate a word unit sequence.

[0278] Beneficial effect: By using an improved audio encoder involving knowledge distillation technology, the characteristic information of the speech to be converted is mined, so that the extracted audio words have more audio characteristics, thereby improving the accuracy of the speech summary model in extracting the corresponding summary of the speech to be converted. Compared with the prior art that first converts the speech to be converted into text and generates a summary based on the converted audio text, this application can directly process the audio, avoid the errors caused by audio-to-text conversion, and thus improve the accuracy of text summary extraction.

[0279] Figure 4The schematic diagram of the structure of an embodiment of the device for extracting speech summaries of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the device for extracting speech summaries.

[0280] like Figure 4 As shown, the speech summary extraction device may include: a processor (processor) 402, a communication interface (Communications Interface) 404, a memory (memory) 406, and a communication bus 408.

[0281] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned embodiment of the method for extracting a speech summary.

[0282] Specifically, the program 410 may include program code including computer executable instructions.

[0283] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the speech summary extraction device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0284] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0285] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a speech summary extraction device / apparatus, the speech summary extraction device / apparatus executes the speech summary extraction method in any of the above method embodiments.

[0286] Beneficial effect: The embodiment of the present invention mines the characteristic information of the speech to be converted through the audio encoder improved by the knowledge distillation technology, so that the extracted audio words have more audio characteristics, thereby improving the accuracy of the speech summary model in extracting the corresponding summary of the speech to be converted. Compared with the prior art that first converts the speech to be converted into text and generates a summary based on the converted audio text, the present application can directly process the audio, avoid the errors caused by audio-to-text conversion, and thus improve the accuracy of text summary extraction.

[0287] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0288] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0289] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

[0290] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A method for extracting a speech summary, characterized in that: The method comprises: Get the voice to be converted; Extracting the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm; Based on the audio word, a speech summary corresponding to the speech to be converted is generated through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word samples and speech summary samples.

2. The method according to claim 1, characterized in that: The training of the audio encoder comprises: Acquire a speech sample and a corresponding summary prompt text sample; wherein the speech sample and the corresponding summary prompt text sample are marked with a preset text summary tag; Input the speech sample into the initial audio encoder to obtain an audio word unit sample corresponding to the speech sample; and input the summary prompt text sample corresponding to the speech sample into the word segmenter to obtain a text word unit sample corresponding to the summary prompt text sample; Inputting the audio word-unit sample and the corresponding text word-unit sample into a speech summary generation model, and calculating an initial audio encoder loss based on feedback from the speech summary generation model; Based on the initial audio encoder loss, parameters of the initial audio encoder are adjusted to obtain a final audio encoder.

3. The method according to claim 2, characterized in that The step of inputting the audio word unit sample and the corresponding text word unit sample into the speech summary generation model and calculating the initial audio encoder loss comprises: Generate a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model; Generate feature distillation loss based on the hidden states of the audio word sample and the corresponding text word sample in the corresponding connector layer of the speech summary generation model; Based on the audio word sample input, the predicted text summary generated by the speech summary generation model is calculated, and the cross entropy loss between the predicted text summary and the corresponding text summary label is calculated; An initial audio encoder loss is determined based on the log distillation loss, the feature distillation loss, and the cross entropy loss.

4. The method according to claim 3, characterized in that: The method generates a logarithmic distillation loss based on the predicted logarithms generated after the audio word sample and the corresponding text word sample are respectively input into the speech summary generation model, including: Inputting the audio word sample into the speech summary generation model to obtain the audio prediction logarithm generated by the speech summary generation model; Inputting the corresponding text word sample into the speech summary generation model to obtain the text prediction logarithm generated by the speech summary generation model; The log distillation loss is calculated based on the audio prediction log, text prediction log, and soft cross entropy loss function.

5. The method according to claim 3, characterized in that: The generating feature distillation loss based on the hidden states of the audio word unit sample and the corresponding text word unit sample in the corresponding connector layer of the speech summary generation model includes: Inputting the audio word unit sample into the speech summary generation model to obtain the audio hidden state of the audio word unit sample at a preset connector layer in the speech summary generation model; Inputting the corresponding text word sample into the speech summary generation model to obtain the text hidden state of the text word sample at the preset connector layer in the speech summary generation model; The feature distillation loss is calculated based on the audio hidden state, text hidden state and mean square error loss function.

6. The method according to claim 1, characterized in that The step of generating a speech summary corresponding to the speech to be converted based on the audio word by using a preset speech summary generation model includes: Determine whether the user's summary prompt text is received; If yes, extract the text word-grams corresponding to the summary prompt text, combine the audio word-grams and the text word-grams to obtain a word-gram sequence, input the word-gram sequence into a preset speech summary generation model, and generate a speech summary corresponding to the speech to be converted and the summary generation prompt text; If not, the audio word is input into a preset speech summary generation model to generate a speech summary corresponding to the speech to be converted.

7. The method according to claim 6, characterized in that The step of combining the audio word unit and the text word unit to obtain a word unit sequence includes: The audio word unit and the text word unit are arranged in a manner that the audio word unit comes first and the text word unit comes later to generate a word unit sequence.

8. A device for extracting speech summaries, characterized in that: The device comprises: A data acquisition module, used to acquire the speech to be converted; A data extraction module, used to extract the speech to be converted based on an audio encoder to obtain audio words; wherein the audio encoder is generated based on a knowledge distillation algorithm; The result generation module is used to generate a speech summary corresponding to the speech to be converted based on the audio word unit through a preset speech summary generation model; wherein the speech summary generation model is obtained by training a language processing model based on audio word unit samples and speech summary samples.

9. A device for extracting speech summaries, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the voice summary extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the device / apparatus for extracting the speech summary, the device / apparatus for extracting the speech summary executes the operation of the method for extracting the speech summary according to any one of claims 1 to 7.

Images