Methods, apparatuses, devices, and media for processing audio data

Abstract

Methods, apparatuses, devices, and media for processing audio data are provided. In one method, audio data is received, the audio data including context data represented in an audio format and question data represented in the audio format, the question data representing a question associated with the context data. Based on the question data in the audio data, the question is determined. An audio feature associated with the audio data and a context feature associated with the context data are respectively determined. Based on the audio feature and the context feature, an answer to the question is provided. With implementations of the present disclosure, context data to be processed and instructions to be performed for the context data can be automatically extracted from mixed audio data. In this way, questions from a user can be specified in a voice manner, audio data including richer information can be processed, thereby being more suitable for processing audio data collected in a variety of environments.

Description

Technical Field

[0001] Implementations of this disclosure generally relate to the field of computers, and in particular to methods, apparatus, devices, and computer-readable storage media for processing audio data. Background Technology

[0002] Machine learning techniques have been widely used to process audio data. For example, machine learning models can be used to perform various tasks on speech data, with specific tasks defined in individual user instructions. For instance, a user instruction might specify, "Please identify the type of sound in the audio data," and so on. However, in real-world applications, user instructions may overlap with ambient audio. For example, a user at the beach might hear waves and ask, "Can you hear the waves?" In this case, the user's question overlaps with the ambient sound of the waves, both serving as input data for the machine learning model. The model struggles to distinguish between the ambient sound and the user's instruction, resulting in inaccurate outputs. Therefore, a more efficient way to process audio data containing more information is desired. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for processing audio data is provided. In this method, audio data is received, the audio data including context data represented in an audio format and question data represented in an audio format, the question data representing a question associated with the context data. Based on the question data in the audio data, a question is determined. Audio features associated with the audio data and context features associated with the context data are determined separately. Based on the audio features and context features, a response to the question is provided.

[0004] In a second aspect of this disclosure, an apparatus for processing audio data is provided. The apparatus includes: a receiving module configured to receive audio data, the audio data including context data represented in an audio format and question data represented in an audio format, the question data representing a question associated with the context data; a question determining module configured to determine a question based on the question data in the audio data; a feature determining module configured to determine audio features associated with the audio data and context features associated with the context data, respectively; and a providing module configured to provide a response to the question based on the audio features and the context features.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] In the following detailed description, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent, taken in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A block diagram showing the structure of audio data according to one implementation of this disclosure is illustrated;

[0011] Figure 2 A block diagram illustrating some implementations of this disclosure for processing audio data is shown;

[0012] Figure 3 A block diagram illustrating the structure of a machine learning model for processing audio data according to some implementations of this disclosure is shown;

[0013] Figure 4 A block diagram of a reference sample for training a machine learning model according to some implementations of this disclosure is shown;

[0014] Figure 5 A block diagram is shown illustrating some implementations of this disclosure for generating reference audio data;

[0015] Figure 6A A block diagram is shown illustrating some implementations of this disclosure for generating reference audio data for overlapping patterns;

[0016] Figure 6BA block diagram is shown illustrating some implementations of this disclosure for generating reference audio data for non-overlapping modes;

[0017] Figure 7 A block diagram of an algorithm for generating reference audio data according to some implementations of this disclosure is shown;

[0018] Figure 8 A block diagram illustrating some implementations of this disclosure for generating datasets is shown;

[0019] Figure 9 A flowchart is shown of a method for processing audio data according to some implementations of this disclosure;

[0020] Figure 10 A block diagram of an apparatus for processing audio data according to some implementations of this disclosure is shown; and

[0021] Figure 11 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0022] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0024] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0027] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0030] Example Environment

[0031] Machine learning models can be used to perform various tasks on speech data. Specific tasks can be defined in individual user instructions, such as "Please identify the type of sound in the audio data," etc. In this case, the audio data to be processed (e.g., audio including wave sounds) and the instruction data represented in text format (e.g., "Are there waves?") can be input into the machine learning model. The machine learning model can then determine whether the audio data includes wave sounds and provide a response. However, in real-world applications, human instructions may overlap with ambient audio. In the field of audio data processing, language models have been applied to process both text and audio data. However, existing language models primarily focus on processing input audio data using text instructions and cannot be used to process mixed audio data that includes both voice instructions and audio events to be processed.

[0032] See Figure 1 This describes the application environment based on some implementations of this disclosure. Figure 1 A block diagram 100 showing the structure of audio data according to one implementation of this disclosure is illustrated. Figure 1 As shown, the audio data 110 to be processed may include context data 120 and question data 130. Here, question data 130 may be in speech format and correspond to a speech instruction to be executed (e.g., a speech corresponding to "Are there waves"); and context data 120 may be in speech format and correspond to an audio event to be processed (e.g., audio including the sound of waves). In this case, audio data 110 may be mixed audio data including context data 120 and question data 130.

[0033] The user's question overlaps with the background sound of ocean waves, both serving as input data for a machine learning model. In this situation, the machine learning model struggles to distinguish between the background sound and the user's question, resulting in inaccurate outputs. Therefore, there is a need for a more efficient way to process audio data containing more information, and for the machine learning model to seamlessly integrate and understand mixed audio data, including voice commands and audio events.

[0034] An overview of audio data processing

[0035] Machine learning models are expected to understand mixed audio data containing contextual data (e.g., audio events, background speech, or background music, etc.) and question data. First, the machine learning model should accurately identify the question from the mixed audio data. Second, it needs to determine the appropriate response based on the information conveyed by the contextual data. Furthermore, real-world scenarios often involve a mixture of speech and background sounds. For example, overlapping audio sources and varying signal-to-noise ratios (SNR) can make it difficult for the model to extract relevant information from the contextual and question data, leading to poor model accuracy.

[0036] To at least partially address the shortcomings of the prior art, a method for processing audio data is proposed according to one implementation of this disclosure. In summary, the technical solution of this disclosure can process audio data including mixed content, wherein the audio data may include context data represented in an audio format and question data represented in an audio format, and the question data represents a question associated with the context data.

[0037] See Figure 2 A summary of one implementation according to this disclosure is described. Figure 2 A block diagram 200 for processing audio data is shown, illustrating some implementations of this disclosure. For example... Figure 2 As shown, audio data 110 can be received. Audio data 110 may include context data 120 represented in audio format and question data 130 represented in audio format, where question data 130 represents a question 210 associated with context data 120. Question 210 can be determined based on question data 130 in audio data 110. For example, automatic speech recognition (ASR) technology can be used to determine question 210 represented in text format. Audio features associated with audio data 110 can be determined (e.g., via an adapter model in a machine learning model). Further, contextual features associated with context data can be determined (e.g., using an audio tagging (AT) model in a machine learning model). A response 220 is then provided for question 210 based on the audio features and contextual features. Here, question 210 and response 220 can be represented in text format; alternatively and / or additionally, question 210 and response 220 can be represented in audio format.

[0038] Based on some implementations of this disclosure, a framework for machine learning models capable of processing mixed audio data is provided. This framework includes an audio tagging model to effectively identify and represent audio events, and a prediction method based on Automatic Speech Recognition (ASR) improves the understanding of speech commands. Experimental results show that the proposed framework outperforms existing technologies on various test sets. In this way, more robust performance in jointly understanding speech commands and the audio data to be processed can be provided.

[0039] Using the implementation method of this disclosure, context data to be processed and instructions to be executed on the context data can be automatically extracted from mixed audio data. In this way, it is possible to support specifying questions from users in a voice manner, process audio data containing richer information, and thus be more suitable for processing audio data collected in a variety of environments.

[0040] Detailed process of audio data processing

[0041] To understand speech commands and acoustic background sounds, this disclosure proposes a novel framework for machine learning models. Specifically, a Speech-Oriented LLM ThatHearsAcoustic Context (SOLLA) is proposed. To help the model better interpret the audio information from the input, an audio tagging model is provided for predicting audio events and generating corresponding event features. Furthermore, an ASR-based auxiliary prediction function is proposed, which allows the machine learning model to first capture the specific content of the speech command and then generate the corresponding response.

[0042] For ease of understanding, the meaning of the various symbols used in some implementations according to this disclosure will be described first below. Mixed audio data s can be given. m :

[0043] s m =s s +s a (1)

[0044] Here, s s This represents voice commands (i.e., questions from the user expressed in voice format), s a This represents the audio events accompanying the voice commands (i.e., contextual data as background), and s m This represents mixed audio data including voice commands and audio events. Here, the machine learning model can generate a text response that includes both types of information. For example, the response may include a question from the user (i.e., a text representation recognized from the voice command) and an answer to the question (i.e., a relevant answer to the question based on the audio events).

[0045] See Figure 3 To describe more details, the Figure 3 A block diagram 300 illustrates the structure of a machine learning model for processing audio data according to some implementations of this disclosure. Figure 3 As shown, a machine learning model 360 can be provided, which may include several key models: an audio encoder model 310, an adapter model 320, an audio tagging model 330, and a language model 340. Alternatively and / or additionally, the machine learning model 360 may include an additional ASR model 350 to assist in understanding speech commands. Audio data 110 can be input to the machine learning model 360 to obtain a response 220.

[0046] According to some implementations of this disclosure, the audio encoder model 310 can be based on an existing encoder architecture. The encoder model 310 can be trained, for example, using multi-task supervised multilingual data, to obtain an audio encoder with accurate speech recognition capabilities. It should be understood that the audio encoder model 310 here can extract multiple aspects of information about the audio data 110, including, for example, information about both contextual data 120 and question data 130, and the intermediate data of the encoder output can include rich data. Therefore, the encoder can be used to extract information from mixed audio data s. m Effectively extracting voice commands s and audio events a The relevant features. During training, the parameters of the audio encoder model can be frozen, and the audio encoder model 310 can be fine-tuned using a low-rank adapter (Lora) 312 with trainable parameters.

[0047] Furthermore, the adapter model 320 is used to transform intermediate features from the audio encoder into a suitable format that can be processed by the language model. According to some implementations of this disclosure, the adapter model 320 may include linear layers, activation function layers, and downsampling layers. Specifically, an activation layer (e.g., for implementing the Gaussian Error Linear Unit (GELU) activation function) may be provided after the first linear layer, and a pooling layer may be provided after the second linear layer to implement a two-dimensional average pooling operation, which downsamples the features, for example, by a factor of 2. The final output sequence is represented as follows: Where n represents the length of the sequence.

[0048] According to some implementations of this disclosure, in the process of determining audio feature 362, an encoder model (e.g., audio encoder model 310) in a machine learning model can be used to determine intermediate audio features; and an adapter model in a machine learning model can be used to convert the intermediate audio features into audio features. Here, the length n of the audio feature is determined based on the length of the audio data. Here, n is proportional to the length of the audio data, and the larger the length of the audio data, the larger the value of n. Here, sequence v a The feature of the i-th element in The dimension can correspond to the dimension of the features in a machine learning model. For example, this dimension can be represented as a vector with a width of 4096 (or other numerical values). Using some implementations of this disclosure, audio encoder models and adapter models can be used to extract multifaceted information about contextual and question data from the raw audio data to be processed, and then a language model can be used to determine the appropriate response.

[0049] According to some implementations of this disclosure, the audio tagging model 330 can capture feature information related to contextual data in audio data. In determining contextual features, an audio tagging model within a machine learning model can be used to convert intermediate features into contextual features, and the length of the contextual features is determined based on the width of the features in the machine learning model. Specifically, the audio tagging model 330 may include a pooling layer, a normalization layer, and a residual attention layer. Here, the pooling layer can apply a two-dimensional average pooling operation to the output of the audio encoder, for example, reducing the temporal resolution by a factor of 20. The normalization layer can perform normalization and transformation on the pooled features through linear projection to produce an internal representation. The residual attention layer can model long-range temporal dependence based on this internal representation. After mean pooling along the time dimension, a compact feature vector 364 (denoted as v) can be obtained. at ).

[0050] By utilizing some implementations of this disclosure, pre-trained speech tagging models can be used to extract more information about contextual data, thereby enabling machine learning models to more accurately identify audio events included in the contextual data, thus improving the accuracy of machine learning models in generating responses.

[0051] According to some implementations of this disclosure, v at The dimension can correspond to the dimension of the features in a machine learning model. For example, this dimension can be represented as a vector with a width of 4096 (or some other value). at and They can have the same dimension. According to some implementations of this disclosure, based on audio features (e.g., denoted as v), a ) and contextual features (e.g., denoted as v) at In the process of providing responses to questions, language models from machine learning models can be used to provide responses based on audio features and contextual features. For example... Figure 3 As shown, a language model 340 (e.g., a chat language model trained with a large amount of data) can be used to process v a and v at v at The data can be normalized and passed through a linear classifier to produce a prediction 366 for the audio event (e.g., represented as a probability). In the example above, the prediction 366 for the audio event could represent the probability that “waves” are present in the audio data. At this point, the language model 340 in the machine learning model 360 can determine that “waves” are included in the audio data 110, thus providing an affirmative response.

[0052] According to some implementations of this disclosure, the parameters of the language model can be kept frozen while fine-tuning is achieved using a trainable LoRA 342. The language model 340 can receive the output of the adapter model. And the output v of the AT model at Furthermore, the language model 340 can output corresponding responses. Using some implementation methods of this disclosure, the fine-tuned LoRA 342 allows the language model 340 to output responses in a more accurate manner.

[0053] According to some implementations of this disclosure, the response can have multiple formats, for example, it can directly output the response "Yes, the sound you are hearing is of waves." According to some implementations of this disclosure, to enhance the understanding of speech instructions, an ASR-assisted prediction strategy can be used. In this case, the question can be further provided as a text representation in natural language. During training, the model first performs an ASR task before generating the final answer. For example, the target output could be a transcription: "Are there waves?" The response could then include "Yes, the sound you are hearing is of waves." According to some implementations of this disclosure, it can be ensured that the model accurately captures the content of the speech instruction and then combines it with audio information to produce the final response. Specifically, both the question and response can be output: "Question: Are there waves? Answer: Yes, the sound you are hearing is of waves." Using some implementations of this disclosure, responses can be generated in a variety of different ways, allowing the user to obtain more information about the mixed audio data in the desired format. It should be understood that although the questions and answers are presented in text format in the example above, alternatively and / or additionally, the questions and answers may be represented in audio format. In this case, the machine learning model can output audio data including the questions and answers.

[0054] According to some implementations of this disclosure, a machine learning model can be trained in the following manner. Specifically, reference samples can be determined, including reference audio data, a reference question, and a reference response to the reference question; a reference prediction of the reference response can be determined using the machine learning model; and the machine learning model can be updated based on the difference between the reference prediction and the reference response. Using some implementations of this disclosure, reference samples can be constructed according to the needs of the machine learning model, and the machine learning model can be updated in a way that makes the predictions more closely match the ground truth data samples.

[0055] Based on some implementations of this disclosure, new datasets can be constructed for training and / or evaluating the framework. These datasets include data related to multiple tasks: audio event classification, audio captioning, and audio question answering. The proposed datasets may include different speech instructions with various speaking styles and can provide sample data with different difficulty levels. See also... Figure 4 Describe more details about the reference samples in the dataset. Figure 4A block diagram 400 shows a reference sample for training a machine learning model according to some implementations of this disclosure.

[0056] like Figure 4 As shown, reference sample 440 may include reference audio data 410, a reference question 420 in text format (e.g., "Can you hear the rain?"), and a reference response 430 in text format to the reference question (e.g., "Yes, the sound you are hearing is of rain"). Reference audio data 410 can be input into the machine learning model described above to determine the response. Furthermore, the difference between the response and the reference response can be determined, and the machine learning model can be updated based on this difference.

[0057] According to some implementations of this disclosure, the reference audio data may include reference context data represented in audio format and reference question data represented in audio format, wherein the reference question data represents a reference question associated with the context data. In determining the prediction of the reference response, a machine learning model can be used to determine reference audio features associated with the reference audio data and reference context features associated with the reference context data; and a language model within the machine learning model can be used to determine the reference prediction of the reference response based on the reference audio features and the reference context features.

[0058] Specifically, it can be directed to, for example Figure 3 The machine learning model 360 shown is input to reference audio data. The adapter model 320 can output reference audio features associated with the reference audio data, and the audio tagging model can output reference context features associated with reference context data. Subsequently, the language model 340 can determine a reference prediction of the reference response based on the reference audio features and the reference context features. Using some implementations of this disclosure, relevant information about speech instructions and audio events in mixed audio data can be fully utilized to determine the reference prediction of the reference response.

[0059] According to some implementations of this disclosure, in the process of determining reference audio data, reference question data can be determined based on a reference question; a reference response to the reference question can be determined based on reference context data; and reference audio data can be generated according to a predetermined generation pattern based on the reference context data and the reference question data. Using some implementations of this disclosure, reference audio data can be generated using a variety of currently available audio datasets. In this way, a large amount of training data containing richer information can be generated, thereby improving the performance of machine learning models.

[0060] See Figure 5To describe more details about generating the reference audio data, the Figure 5 A block diagram 500 for generating reference audio data according to some implementations of this disclosure is shown. For example... Figure 5 As shown, reference context data 510 (e.g., audio including rain sounds) can be obtained from an existing audio dataset. Various types of audio processing tasks 520 can be provided for the reference context data 510, such as classification tasks 522, caption extraction tasks 524, and / or question-and-answer tasks 526, etc. Further, audio processing tasks 520 can be performed to generate text instructions and then convert them into corresponding speech instructions. Here, the speech instructions can be used as reference question data 530. Subsequently, the reference context data 510 in audio format and the reference question data 530 in audio format can be mixed to generate reference audio data 410.

[0061] According to some implementations of this disclosure, audio classification, audio description, and audio question answering can be performed on audio data in an existing audio dataset. This dataset can be mixed with voice commands to generate training data. Specifically, the audio processing tasks shown in Table 1 can be performed on each audio file in the existing audio dataset to determine the corresponding question. Then, ground truth labels for the question can be determined, thereby generating corresponding reference responses. It should be understood that although examples of the tasks are described in English herein, alternatively and / or additionally, examples of the tasks may be described in other languages.

[0062] Table 1 Examples of Tasks

[0063]

[0064]

[0065] Using some implementations of this disclosure, questions spanning different tasks can be generated. Furthermore, to ensure diversity in the question format and sound characteristics of the speech instructions, existing language models can be used to generate multiple question formats for each task, while a text-to-speech model is employed to synthesize different speaking styles. According to some implementations of this disclosure, publicly available datasets involving different tasks can be obtained. Text-formatted instructions (i.e., questions) can be generated as shown in Table 1, and these instructions can be rewritten using a language model with text processing capabilities; for example, a predetermined number (e.g., 100, or other numbers) of rewritten instructions can be generated. Alternatively and / or additionally, rewritten instructions unsuitable for generating speech instructions can be filtered out.

[0066] Subsequently, speech instructions can be generated based on each rewritten instruction. For example, speech instructions with different timbres can be generated. The generated large number of speech instructions can be scored to select those that meet predetermined criteria (e.g., ranking in the top 10% or other proportions) for addition to the dataset. Specifically, speech instructions that are too long, have poor content consistency, and / or contain language errors can be filtered out. Further, problematic records can be filtered out based on content, and the speech dataset can be regenerated using a text-to-speech model. Specifically, the word error rate (WER) for each generated utterance can be determined. For example, speech data with a WER exceeding 40% will be regenerated; alternatively and / or additionally, speech data to be regenerated can be determined in other ways. Alternatively and / or additionally, speech instructions with other timbres can be randomly selected for generation.

[0067] According to some implementations of this disclosure, audio events and voice commands can be mixed to generate mixed audio. Specifically, the voice commands and accompanying audio events can appear in various forms. For example, the voice commands can appear before, after, or overlap with the audio events. In this case, a generation mode for generating reference audio data can be specified. To simulate these different conditions and provide mixed audio with different levels of difficulty, mixed audio can be generated based on overlapping (difficult) modes and non-overlapping (easy) modes.

[0068] According to some implementations of this disclosure, in the process of generating reference audio data, in response to determining that the predetermined generation mode is an overlay mode, reference question data is overlaid in the subsequent portion of the reference context data after a predetermined time point to generate reference audio data. Here, the overlay mode can generate more complex reference audio data and allows machine learning models to process audio data with overlaid voice commands and audio events in a more efficient manner.

[0069] According to some implementations of this disclosure, in response to determining that the predetermined generation mode is a non-overlay mode, reference question data is inserted into the reference context data at predetermined time points to generate reference audio data. Here, the non-overlay mode can generate reference audio data of lower difficulty, and allows the machine learning model to process audio data where voice commands and audio events occur sequentially in a more efficient manner. Using some implementations of this disclosure, training data of different difficulties can be generated separately, thereby improving the machine learning model's ability to process mixed audio data of varying difficulty.

[0070] It should be understood that reference context data and reference problem data can have different time lengths, and problem data can be overlaid at different positions within the reference context data. If the time length after the predetermined time point is less than the time length of the reference problem data (i.e., insufficient to accommodate the reference problem data), then the reference context data and reference problem data may only partially overlap. If the time length after the predetermined time point is greater than or equal to the time length of the reference problem data (i.e., sufficient to accommodate the reference problem data), then the reference context data and reference problem data may overlap.

[0071] According to some implementations of this disclosure, during the process of overlaying reference problem data in a subsequent section, in response to determining that the length of the subsequent section is less than the length of the reference problem data, a portion of the reference problem data is overlaid in the subsequent section. See also Figure 6A To describe more details, the Figure 6A A block diagram 600A is shown, illustrating some implementations of this disclosure for generating reference audio data for overlapping patterns. For example... Figure 6A As shown, reference problem data 530 can be inserted into the reference context data 510 at a predetermined time point T1. At this time, the length of the subsequent data 610 is less than the length of the reference problem data 530, therefore the reference context data 510 and the reference problem data 530 only overlap in the subsequent portion 610. It should be understood that, although... Figure 6A This only illustrates the case where the start time of reference context data 510 precedes the start time of reference problem data 530. Alternatively and / or additionally, the start time of reference context data 510 may be after the start time of reference problem data 530. In this way, reference audio data involving multiple positional relationships can be generated, thereby supporting machine learning models in processing various mixed audio data.

[0072] According to some implementations of this disclosure, in response to determining that the length of the subsequent portion is not less than the length of the reference problem data, the reference problem data is superimposed on the subsequent portion. Using some implementations of this disclosure, fully overlapping or partially overlapping training data can be generated, thereby improving the ability of machine learning models to process mixed audio data of varying difficulty.

[0073] Figure 6B A block diagram 600B is shown, illustrating some implementations of this disclosure for generating reference audio data in a non-overlapping mode. For example... Figure 6BAs shown, reference question data 530 can be concatenated after reference context data 510, with the predetermined time point located at the end of reference context data 510. The resulting reference audio data 410 then includes consecutive reference context data 510 and reference question data 530. This method generates more easily processed, non-overlapping data, thereby improving the ability of machine learning models to handle simple mixed audio data.

[0074] Several scenarios for generating reference audio data have been described below; see below for more details. Figure 7 Describe the algorithm 710 for generating reference audio data. Figure 7 A block diagram 700 of an algorithm for generating reference audio data according to some implementations of this disclosure is shown. It should be understood that... Figure 7 The algorithm's outline is illustrated in pseudocode, and it can be implemented using different programming languages. For example... Figure 7 As shown, the input data is introduced first: s s The input signal representing the voice command, s a The input signal representing the audio event, l s The length of the input signal for the voice command, l a The length of the input signal for the audio event is indicated by m, which represents easy or hard mode.

[0075] Lines 1 through 8 illustrate the processing steps for the hard mode (i.e., overlapping mode). In hard mode, the voice commands and audio events in the generated audio signal overlap. In this case, the start time t of the voice command can be determined separately. s and the start time t of the audio event a Lines 2 through 4 show the case where the length of the audio event is shorter than the length of the voice command. Line 3 sets the start time t based on a random method. s Line 4 adds a voice command to the audio event. Lines 5 through 8 show the cases where the length of the audio event is greater than or equal to the length of the voice command. Line 6 sets the start time t based on a random method. s At line 7, add an audio event to the voice command.

[0076] Lines 9-16 illustrate the processing in easy mode (i.e., non-overlapping mode). In easy mode, the voice commands and audio events in the generated audio signal do not overlap; in other words, they are sequential. The audio at the beginning of the voice command and audio event can be selected based on a random value. Lines 17-22 show the determination of volume for the voice command and audio event respectively. Lines 23-24 show the setting of volume for the voice command and audio event respectively. Line 25 shows the mixing of voice commands and audio events, and the generation of the final mixed audio data s.m .

[0077] According to some implementations of this disclosure, the easy mode can involve non-overlapping audio with a higher SNR, while the hard mode can involve overlapping audio with a lower SNR, thereby mimicking real-world situations, such as high-volume background noise interfering with voice commands. Specifically, by selecting either an overlapping or non-overlapping mode, the volume of the audio event and the voice command can be determined, thereby generating a mixed audio that includes the audio event and the voice command.

[0078] According to some implementations of this disclosure, training datasets can be constructed using publicly available datasets for performing various tasks. See also Figure 8 Describe more details, Figure 8 A block diagram 800 for generating a dataset is shown, according to some implementations of this disclosure. For example... Figure 8 As shown, audio 810 can be selected from a publicly available audio dataset, and text questions 812 can be constructed for each audio. The text questions 812 can be rewritten using a rewriting module 820 (e.g., implemented by a language model) to obtain more text questions. Corresponding speech commands can be generated using a TTS module 822. Unsuitable speech commands can be filtered out using a filtering module 824. The loudness can be adjusted using a loudness module 826, thereby generating a dataset 830.

[0079] Alternatively and / or additionally, parameters for the mixing of voice commands and audio events can be set: the overlap ratio of voice commands and audio events can be 80%. Alternatively and / or additionally, the volume of the voice commands can be randomly set between -38 and 20, while the volume of the audio events can be set between -25 and 25. Finally, training data samples can be selected from the dataset constructed as described above, for example, replacing 20% ​​(or another proportion) of the training samples in the regular training dataset.

[0080] According to some implementations of this disclosure, the performance of the provided machine learning model can be verified across different tasks and on different test data. For example, a newly created dataset can be used to test the performance of the proposed machine learning model. Experimental data shows that the proposed machine learning model can handle mixed audio data, and the accuracy of the results provided is higher than that of tasks executed using text instructions. The proposed solution performs comparably to or better than baseline models. In particular, under difficult modes, the stability and accuracy of the proposed solution are higher than existing solutions in different scenarios. Furthermore, the ASR-assisted prediction method enhances the model's ability to interpret speech instructions, while the AT model plays a key role in understanding the audio content of the input signal, thereby improving the overall performance of the machine learning model.

[0081] It should be understood that although the above description uses English as an example of a natural language to illustrate some implementation details according to this disclosure, alternatively and / or additionally, the methods described above can be implemented in other language environments. For example, the machine learning model described above can be implemented using only a language model that supports multiple natural languages, in which case the methods described above can be implemented in environments such as Chinese, Japanese, French, German, etc.

[0082] Using the implementation method of this disclosure, context data to be processed and instructions to be executed on the context data can be automatically extracted from mixed audio data. In this way, it is possible to support specifying questions from users in a voice manner, process audio data containing richer information, and thus be more suitable for processing audio data collected in a variety of environments.

[0083] Example process

[0084] Figure 9 A flowchart of a method 900 for processing audio data according to some implementations of this disclosure is shown. At block 910, audio data is received, the audio data including context data represented in an audio format and question data represented in an audio format, the question data representing a question associated with the context data; at block 920, a question is determined based on the question data in the audio data; at block 930, audio features associated with the audio data and context features associated with the context data are determined, respectively; and at block 940, a response to the question is provided based on the audio features and the context features.

[0085] According to some implementations of this disclosure, determining audio features includes: using an encoder model in a machine learning model to determine intermediate audio features; and using an adapter model in a machine learning model to convert the intermediate audio features into audio features, wherein the length of the audio features is determined based on the length of the audio data.

[0086] According to some implementations of this disclosure, determining context features includes: using an audio tagging model in a machine learning model to convert intermediate features into context features, and the length of the context features is determined based on the width of the features in the machine learning model.

[0087] According to some implementations of this disclosure, providing responses to questions based on audio features and contextual features includes: utilizing a language model in a machine learning model to provide responses to questions based on audio features and contextual features.

[0088] According to some implementations of this disclosure, the machine learning model is determined based on: determining reference samples, including reference audio data, reference questions, and reference responses to the reference questions; using the machine learning model to determine reference predictions of the reference responses; and updating the machine learning model based on the difference between the reference predictions and the reference responses.

[0089] According to some implementations of this disclosure, the reference audio data includes reference context data represented in audio format and reference question data represented in audio format, wherein the reference question data represents a reference question associated with the context data, and the prediction of the reference response is determined based on: using a machine learning model to determine reference audio features associated with the reference audio data and reference context features associated with the reference context data; and using a language model in the machine learning model to determine the reference prediction of the reference response based on the reference audio features and the reference context features.

[0090] According to some implementations of this disclosure, the reference audio data is determined based on: determining reference question data based on a reference question; determining a reference response to the reference question based on reference context data; and generating reference audio data according to a predetermined generation pattern based on the reference context data and the reference question data.

[0091] According to some implementations of this disclosure, the reference audio data is generated based on: in response to determining that the predetermined generation mode is an overlay mode, overlaying reference question data into the subsequent portion after a predetermined time point in the reference context data to generate reference audio data; or in response to determining that the predetermined generation mode is a non-overlay mode, inserting reference question data into the predetermined time point in the reference context data to generate reference audio data.

[0092] According to some implementations of this disclosure, overlaying reference problem data in a subsequent section includes: in response to determining that the length of the subsequent section is less than the length of the reference problem data, overlaying a portion of the reference problem data in the subsequent section; or in response to determining that the length of the subsequent section is not less than the length of the reference problem data, overlaying the reference problem data in the subsequent section.

[0093] According to some implementations of this disclosure, the adapter model includes a linear layer, an activation function layer, and a downsampling layer, and the audio tagging model includes a pooling layer, a normalization layer, and a residual attention layer. Questions and responses are represented in text or speech format, and the method further includes providing questions represented in natural language.

[0094] Example devices and equipment

[0095] Figure 10A block diagram of an apparatus 1000 for processing audio data according to some implementations of the present disclosure is shown. The apparatus 1000 includes: a receiving module 1010 configured to receive audio data, the audio data including context data represented in an audio format and question data represented in an audio format, the question data representing a question associated with the context data; a question determination module 1020 configured to determine a question based on the question data in the audio data; a feature determination module 1030 configured to determine audio features associated with the audio data and context features associated with the context data, respectively; and a providing module configured to provide a response to the question based on the audio features and the context features.

[0096] According to some implementations of this disclosure, the feature determination module 1030 is further configured to: determine intermediate audio features using an encoder model in a machine learning model; and convert the intermediate audio features into audio features using an adapter model in a machine learning model, wherein the length of the audio features is determined based on the length of the audio data.

[0097] According to some implementations of this disclosure, the feature determination module 1030 is further configured to: convert intermediate features into context features using an audio tagging model in a machine learning model, wherein the length of the context features is determined based on the width of the features in the machine learning model.

[0098] According to some implementations of this disclosure, the providing module 1040 is further configured to: provide a response to a question based on audio features and contextual features using a language model in a machine learning model.

[0099] According to some implementations of this disclosure, the machine learning model is determined based on: determining reference samples, including reference audio data, reference questions, and reference responses to the reference questions; using the machine learning model to determine reference predictions of the reference responses; and updating the machine learning model based on the difference between the reference predictions and the reference responses.

[0100] According to some implementations of this disclosure, the reference audio data includes reference context data represented in audio format and reference question data represented in audio format, wherein the reference question data represents a reference question associated with the context data, and the prediction of the reference response is determined based on: using a machine learning model to determine reference audio features associated with the reference audio data and reference context features associated with the reference context data; and using a language model in the machine learning model to determine the reference prediction of the reference response based on the reference audio features and the reference context features.

[0101] According to some implementations of this disclosure, the reference audio data is determined based on: determining reference question data based on a reference question; determining a reference response to the reference question based on reference context data; and generating reference audio data according to a predetermined generation pattern based on the reference context data and the reference question data.

[0102] According to some implementations of this disclosure, the reference audio data is generated based on: in response to determining that the predetermined generation mode is an overlay mode, overlaying reference question data into the subsequent portion after a predetermined time point in the reference context data to generate reference audio data; or in response to determining that the predetermined generation mode is a non-overlay mode, inserting reference question data into the predetermined time point in the reference context data to generate reference audio data.

[0103] According to some implementations of this disclosure, overlaying reference problem data in a subsequent section includes: in response to determining that the length of the subsequent section is less than the length of the reference problem data, overlaying a portion of the reference problem data in the subsequent section; or in response to determining that the length of the subsequent section is not less than the length of the reference problem data, overlaying the reference problem data in the subsequent section.

[0104] According to some implementations of this disclosure, the adapter model includes a linear layer, an activation function layer, and a downsampling layer, and the audio tagging model includes a pooling layer, a normalization layer, and a residual attention layer. Questions and responses are represented in text or speech format, and the providing module is further configured to provide questions represented in natural language.

[0105] Figure 11 A block diagram of a device 1100 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 11 The computing device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 11 The computing device 1100 shown can be used to implement the method described above.

[0106] like Figure 11 As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors 1110, memory 1120, storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processor 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0107] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1100.

[0108] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 11 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1120 may include computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0109] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0110] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device that enables computing device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0111] According to an implementation of this disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0112] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0114] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing audio data, comprising: Receive audio data, the audio data including context data represented in an audio format and question data represented in the audio format, the question data representing a question associated with the context data; The problem is determined based on the problem data in the audio data; Determine the audio features associated with the audio data and the context features associated with the context data, respectively; as well as Based on the audio features and the context features, a response is provided for the question.

2. The method of claim 1, wherein determining the audio features comprises: Use the encoder model in a machine learning model to determine intermediate audio features; as well as The intermediate audio features are converted into audio features using the adapter model in the machine learning model, and the length of the audio features is determined based on the length of the audio data.

3. The method of claim 2, wherein determining the contextual features comprises: The intermediate features are converted into context features using the audio tagging model in the machine learning model, and the length of the context features is determined based on the width of the features in the machine learning model.

4. The method of claim 3, wherein providing the response to the question based on the audio features and the context features comprises: The language model in the machine learning model is used to provide the response to the question based on the audio features and the context features.

5. The method of claim 4, wherein the machine learning model is determined based on the following: Determine a reference sample, which includes reference audio data, a reference question, and a reference response to the reference question; The machine learning model is used to determine a reference prediction for the reference response; as well as The machine learning model is updated based on the difference between the reference prediction and the reference response.

6. The method of claim 5, wherein the reference audio data includes reference context data represented in the audio format and reference question data represented in the audio format, the reference question data representing a reference question associated with the context data, and the prediction of the reference response is determined based on: Using the machine learning model, reference audio features associated with the reference audio data and reference context features associated with the reference context data are determined; and Using the language model in the machine learning model, the reference prediction of the reference response is determined based on the reference audio features and the reference context features.

7. The method of claim 5, wherein the reference audio data is determined based on the following: The reference problem data is determined based on the reference problem; Based on the reference context data, determine the reference response to the reference question; as well as The reference audio data is generated based on the reference context data and the reference question data, according to a predetermined generation pattern.

8. The method of claim 7, wherein the reference audio data is generated based on: In response to determining that the predetermined generation mode is an overlay mode, the reference question data is overlaid in the subsequent portion of the reference context data after a predetermined time point to generate the reference audio data; or In response to determining that the predetermined generation mode is a non-overlay mode, the reference question data is inserted into the reference context data at a predetermined time point to generate the reference audio data.

9. The method of claim 8, wherein overlaying the reference problem data in the subsequent portion comprises: In response to determining that the length of the subsequent portion is less than the length of the reference problem data, a portion of the reference problem data is superimposed into the subsequent portion; or In response to determining that the length of the subsequent portion is not less than the length of the reference problem data, the reference problem data is superimposed on the subsequent portion.

10. The method of claim 3, wherein the adapter model comprises a linear layer, an activation function layer, and a downsampling layer, and the audio tagging model comprises a pooling layer, a normalization layer, and a residual attention layer, the question and the response are represented in text format or the speech format, and the method further comprises: Provide the question expressed in the natural language.

11. An apparatus for processing audio data, comprising: A receiving module is configured to receive audio data, the audio data including context data represented in an audio format and question data represented in the audio format, the question data representing a question associated with the context data; The problem determination module is configured to determine the problem based on the problem data in the audio data; A feature determination module is configured to determine, respectively, audio features associated with the audio data and context features associated with the context data; as well as A module is provided, configured to provide a response to the question based on the audio features and the context features.

12. An electronic device, comprising: At least one processor; as well as At least one memory, coupled to the at least one processor and storing instructions for execution by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A computer instruction product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images