Audio processing method and device and terminal equipment

Abstract

The invention provides an audio processing method and apparatus, and a terminal device. The method comprises the steps of obtaining a first audio; inputting the first audio into an audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is obtained by training based on a plurality of first sample audios and sample audio events included in each first sample audio, and the sample audio events are obtained based on a plurality of teacher models; the duration of training a second sample audio of the plurality of teacher models is greater than the duration of the first sample audio; and determining an audio event included in the first audio based on the M first probabilities.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of audio processing technology, and in particular to an audio processing method, apparatus, and terminal device. Background Art

[0002] Audio event detection refers to the terminal device detecting the audio content in an audio segment. For example, if an audio segment includes a cat's meow, the terminal device can detect that the audio event in the audio segment is a cat's meow.

[0003] Currently, a terminal device can input any audio segment into an audio event detection model, which will then determine the audio event corresponding to that segment. However, to reduce the cost of manually labeling samples, the audio event detection model is trained on longer audio segments. This trained audio event detection model is unable to accurately detect audio events in shorter audio segments, resulting in lower audio event detection accuracy. Summary of the Invention

[0004] The present disclosure provides an audio processing method, apparatus, and terminal device for solving one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an audio processing method, the method comprising:

[0006] Get the first audio;

[0007] Inputting the first audio into an audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, the sample audio events are obtained based on multiple teacher models, and the duration of second sample audios used to train the multiple teacher models is greater than the duration of the first sample audio;

[0008] An audio event included in the first audio is determined based on the M first probabilities.

[0009] In a second aspect, the present disclosure provides an audio processing device, the audio processing device comprising an acquisition module, an input module, and a determination module, wherein:

[0010] The acquisition module is used to acquire the first audio;

[0011] The input module is used to input the first audio into the audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, the sample audio events are obtained based on multiple teacher models, and the duration of second sample audios used to train the multiple teacher models is greater than the duration of the first sample audio;

[0012] The determination module is configured to determine an audio event included in the first audio based on the M first probabilities.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the audio processing method as described in the first aspect and various possible aspects of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the audio processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0017] The present disclosure provides an audio processing method, apparatus, and terminal device, which obtain a first audio, input the first audio into an audio event detection model, and obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio, wherein M is an integer greater than 1, and the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, and the sample audio events are obtained based on multiple teacher models, and the duration of the second sample audio used to train the multiple teacher models is greater than the duration of the first sample audio, and based on the M first probabilities, the audio events included in the first audio are determined. In the above method, since the multiple teacher models are models trained based on audio events included in audio with longer duration, and the terminal device can guide the audio event detection model to learn the ability to detect audio events included in audio with shorter duration based on the multiple teacher models, this eliminates the need for manual annotation of fine-grained data (i.e., audio events included in audio with shorter duration), reduces the training cost of the audio event detection model, and the audio event detection model can learn the audio event detection ability of the teacher model, thereby improving the accuracy of audio event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of an audio processing method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of determining a first probability provided by an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of a process for determining an audio event provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of a training method for an audio event detection model provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of a training process of a teacher model provided in an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of another method for training an audio event detection model provided in an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of a process for determining M fourth probabilities provided by an embodiment of the present disclosure;

[0027] Figure 9 A schematic diagram of the training process of an audio event detection model provided in an embodiment of the present disclosure;

[0028] Figure 10 A structural diagram of an audio processing device provided by an embodiment of the present disclosure; and

[0029] Figure 11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0031] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0032] Terminal device: is a device with wireless transceiver function. The terminal device can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a vehicle-mounted terminal device, a wireless terminal in self-driving, a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, a wearable terminal device, etc. The terminal device involved in the embodiments of the present disclosure can also be called a terminal, user equipment (UE), an access terminal device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote terminal device, a mobile device, a UE terminal device, a wireless communication device, a UE agent or a UE device, etc. Terminal devices can also be fixed or mobile.

[0033] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0034] Figure 1 This is a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 , including 1 second of audio and a terminal device. The 1 second of audio may include the sound of a cat meowing. After the terminal device receives the 1 second of audio, the terminal device may detect the audio content in the 1 second of audio. After determining that the audio content includes the sound of a cat meowing, the electronic device may determine that the audio event of the 1 second of audio is a cat meowing, and mark the audio segment as an audio event.

[0035] In the related art, the terminal device can detect the content in the audio based on the audio event model and determine the audio event of the audio. At present, due to the high cost of obtaining fine-grained (shorter audio duration) audio event samples, the audio event detection model in the terminal device is trained based on coarse-grained (longer audio duration) audio event samples. In this way, the audio event detection model has a higher detection accuracy for audio events with longer audio durations, and a lower detection accuracy for audio events with shorter audio durations, which in turn leads to lower accuracy in audio event detection. If the audio event detection model is trained based on fine-grained audio events, the training cost of the audio event detection model will be higher. How to improve the accuracy of the audio event detection model for short audio based on coarse-grained audio event samples is a problem that needs to be solved urgently.

[0036] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides an audio processing method, in which a terminal device can obtain a first audio and input the first audio into an audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio. The terminal device can determine a probability greater than a preset threshold among the M first probabilities, and determine the audio event associated with the probability greater than the threshold as the audio event included in the first audio. Since the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, and the sample audio events are obtained based on multiple teacher models, and the duration of the second sample audio used to train the multiple teacher models is greater than the duration of the first sample audio, the terminal device can guide the audio event detection model to learn the ability to detect audio events for short audio based on multiple teacher models trained on coarse-grained sample audio events. In this way, there is no need to perform fine-grained labeling of sample audio events, reducing the training cost of the audio event detection model. In addition, the audio event detection model can accurately perform audio event detection processing on shorter audio, thereby improving the accuracy of audio event detection.

[0037] It should be noted that Figure 1 The above is only an example of an application scenario of the embodiment of the present disclosure, and is not intended to limit the application scenario of the embodiment of the present disclosure.

[0038] The following detailed description of the technical solution of the present disclosure and how the technical solution of the present disclosure solves the above-mentioned technical problems is provided with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.

[0039] Figure 2This is a flowchart of an audio processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0040] S201: Obtain first audio.

[0041] The execution subject of the embodiment of the present disclosure may be a terminal device, or an audio processing device provided in the terminal device. The audio processing device may be implemented based on software, or based on a combination of software and hardware, which is not limited in the embodiment of the present disclosure.

[0042] The first audio may be any audio segment. For example, the first audio may include a cat's meowing sound, a dog's barking sound, rain sounds, and thunder sounds. The first audio may also not include any sound, and this embodiment of the present disclosure does not limit this.

[0043] Optionally, the duration of the first audio can be any duration. For example, the first audio can be 1 second of audio, 10 seconds of audio, or 30 seconds of audio, which is not limited in the present embodiment.

[0044] Optionally, there are two feasible implementations for the terminal device to obtain the first audio:

[0045] A possible implementation:

[0046] In response to a touch operation on an audio acquisition control, an audio page is displayed, wherein the audio page includes a plurality of second audios, and in response to a touch operation on any one of the plurality of second audios, the first audio is obtained.

[0047] For example, the display page of the terminal device may include an audio acquisition control. After the user clicks the audio acquisition control, the display page of the terminal device may jump to the audio page, wherein the audio page may include multiple pre-stored second audios. The user clicks any second audio to obtain the first audio. For example, the audio page may include audio A, audio B, and audio C. If the user clicks audio A, the terminal device determines audio A as the first audio. If the user clicks audio B, the terminal device determines audio B as the first audio. If the user clicks audio C, the terminal device determines audio C as the first audio.

[0048] In this way, the terminal device can flexibly select the first audio, thereby improving the flexibility of determining the first audio.

[0049] Another possible implementation:

[0050] In response to the touch operation on the audio acquisition control, audio recording is performed to obtain the first audio. For example, the audio acquisition control can be a recording control. After the user clicks the recording control, the terminal device can collect surrounding audio to obtain the first audio.

[0051] In this way, the terminal device can obtain any audio segment in real time, thereby improving the flexibility of obtaining the first audio.

[0052] It should be noted that the terminal device may also obtain the first audio based on any feasible implementation method (eg, receiving the first audio sent by other devices), and the embodiments of the present disclosure are not limited to this.

[0053] S202: Input a first audio into an audio event detection model to obtain M first probabilities corresponding to M audio events associated with audio content of the first audio.

[0054] Optionally, the audio event detection model is used to detect audio events in audio. For example, the audio event detection model can be a neural network model of any structure, which is not limited in the embodiments of the present disclosure. For example, if the audio content of the first sample audio includes the sound of a cat meowing, then the audio event of the first sample audio is a cat meowing; if the audio content of the first sample audio includes the sound of rain, then the audio event of the first sample audio is raining.

[0055] Here, M is an integer greater than 1. For example, for any audio segment, the audio event detection model can classify the audio based on multiple audio events, and then determine at least one audio event associated with the audio.

[0056] It should be noted that the number of audio events is related to the audio event detection model. During the audio event detection model training process, the number of audio events is determined. For example, if the audio event detection model has determined that the number of audio events is 10 during training, the audio event detection model can determine the degree of correlation between any audio segment and these 10 audio events during use.

[0057] Optionally, each first probability is associated with an audio event. For example, if the number of audio events is 10, the number of first probabilities output by the audio event detection model is 10. If the number of audio events is 20, the number of first probabilities output by the audio event detection model is 20. For example, if the audio events include a cat meowing, a dog barking, and rain, then after the audio event detection model processes the first sample audio, it can obtain the first probability corresponding to the cat meowing, the first probability corresponding to the dog barking, and the first probability corresponding to rain.

[0058] Optionally, the electronic device may determine M first probabilities based on the following feasible implementation: performing convolution processing on the first audio to obtain audio features associated with the first audio, and determining the M first probabilities based on the audio features. For example, the audio detection model may include multiple convolution layers and fully connected layers. The audio detection model may perform convolution processing on the first audio based on the multiple convolution layers, and then extract the audio features of the first audio, and then process the audio features according to the fully connected layer to obtain M first probabilities. For example, the audio detection model may include a classifier, and the audio features are processed based on the classifier to obtain multiple first probabilities.

[0059] It should be noted that the terminal device may determine the M first probabilities based on the audio features in other implementation ways, which is not limited in the embodiments of the present disclosure.

[0060] Next, combine Figure 3 , a process of determining the first probability by the terminal device is described.

[0061] Figure 3 A schematic diagram of determining a first probability provided by an embodiment of the present disclosure. Figure 3 , including: a first sample audio and an audio event detection model. Terminal device ( Figure 3 (not shown) The first audio can be input into the audio event detection model. The audio event detection model can extract audio features associated with the first audio and determine 13 first probabilities based on the audio features. Among them, the probability that the first audio includes event A is probability 1, the probability that the first audio includes event B is probability 2, ..., and the probability that the first audio includes event M is probability 13. In this way, the terminal device can determine the audio event in the first audio based on the audio event detection model.

[0062] S203: Determine an audio event included in the first audio based on the M first probabilities.

[0063] In which, the terminal device can determine the audio event included in the first audio based on the following feasible implementation method: determine the probability greater than a preset threshold among M first probabilities, and determine the audio event associated with the probability greater than the threshold as the audio event included in the first audio.

[0064] Optionally, the preset threshold value may be any set value. For example, in actual application, the larger the preset threshold value, the more accurate the audio event determination. The preset threshold value may be 0.8, 0.9, 1, etc., which is not limited in the present embodiment.

[0065] For example, after the terminal device inputs the first audio into the audio event detection model, the audio event detection model outputs the first probability 1 as 0.1, the first probability 2 as 0.9, and the first probability 3 as 1. If the preset threshold is 0.8, the terminal device can determine that the probabilities greater than the preset threshold are the first probability 2 and the first probability 3.

[0066] After the terminal device determines that the probability is greater than a preset threshold, the terminal device may determine that the audio event associated with the probability greater than the preset threshold is an audio event included in the first audio. For example, if the audio event detection model outputs a first probability of 0.1 for the audio event of a cat meowing and a first probability of 1 for the audio event of a dog barking, and the preset threshold is 0.9, the terminal device may determine that the first audio may include the sound of a dog barking.

[0067] Next, combine Figure 4 , the process of determining the audio event included in the first audio is described.

[0068] Figure 4 This is a schematic diagram of a process for determining an audio event provided by an embodiment of the present disclosure. Figure 4 , including: a first audio and an audio event detection model. The first audio may be an audio obtained by recording, or may be an audio pre-stored in the terminal device, which is not limited in the embodiment of the present disclosure. The terminal device ( Figure 4 (not shown) can input the first audio into the audio event detection model, and the audio event detection model can output a first probability of a cat's meowing audio event of 0, a first probability of a dog's barking audio event of 0.1, a first probability of a raining audio event of 0.9, and a first probability of a thundering audio event of 1. Because the first probabilities corresponding to rain and thunder are larger, the terminal device can determine that the first audio can include the sound of rain and thunder.

[0069] An audio processing method provided by an embodiment of the present disclosure obtains a first audio, inputs the first audio into an audio event detection model, obtains M first probabilities corresponding to M audio events associated with the audio content of the first audio, determines a probability greater than a preset threshold among the M first probabilities, and determines the audio event associated with the probability greater than the threshold as an audio event included in the first audio. In this way, since the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, and the sample audio events are obtained based on multiple teacher models, and the duration of the second sample audio used to train the multiple teacher models is greater than the duration of the first sample audio, the terminal device can guide the audio event detection model to learn the ability to detect audio events for short audio based on multiple teacher models trained with coarse-grained sample audio events. For any audio segment, the audio event detection model can determine the audio events included in the audio segment, thereby improving the accuracy of audio event detection.

[0070] exist Figure 2 Based on the embodiment shown below, combined with Figure 5 , the training method of the audio event detection model in the above audio processing method is explained.

[0071] Figure 5 A schematic diagram of a training method for an audio event detection model provided in an embodiment of the present disclosure.

[0072] See Figure 5 , the method flow includes:

[0073] S501: Obtain a first audio sample.

[0074] The execution subject of the embodiment of the present disclosure may be a terminal device, or an audio processing device provided in the terminal device. The audio processing device may be implemented based on software, or based on a combination of software and hardware, which is not limited in the embodiment of the present disclosure.

[0075] The first audio sample may be any audio whose duration is less than a threshold. For example, the first audio sample may include speech, music, etc., and may also include white noise, etc., which is not limited in the embodiments of the present disclosure. For example, the first audio sample may be audio less than or equal to 1 second, or less than or equal to 2 seconds. It should be noted that the threshold may be a smaller value such as 1, 2, or 3, which can improve the accuracy of audio event detection.

[0076] Optionally, the terminal device may obtain the first sample audio based on any feasible implementation method. For example, the terminal device may receive the first sample audio sent by other devices, or the terminal device may obtain the first sample audio from a database. This is not limited in the embodiments of the present disclosure.

[0077] S502: Input a first audio sample into an audio event detection model to obtain M second probabilities corresponding to M audio events associated with the audio content of the first audio sample.

[0078] Optionally, the terminal device may input a first sample audio into the audio event detection model, and the audio event detection model may predict a second probability corresponding to each audio event included in the first sample audio.

[0079] It should be noted that the processing process of the first sample audio by the audio event detection model can refer to Figure 2 The processing process of the first audio by the audio event detection model in the illustrated embodiment will not be further described in detail in the embodiment of the present disclosure.

[0080] S503: Input the first sample audio into multiple teacher models associated with the audio event detection model, and obtain M third probabilities corresponding to the M audio events output by each teacher model.

[0081] Optionally, the teacher model is trained based on multiple second sample audios and sample audio events corresponding to each second sample audio. The duration of the second sample audio is greater than that of the first sample audio. For example, the duration of the second sample audio can be 10 seconds, and the duration of the first sample audio can be 1 second. It should be noted that the second sample audio can be weakly annotated data (audio event annotation of a long audio clip). In this way, the training cost of the teacher model is low.

[0082] Optionally, the terminal device may input a first sample audio into the teacher model, and the teacher model may extract audio features of the first sample audio and determine multiple third probabilities based on the audio features. For example, the structure of the teacher model may be the structure of any neural network model, which is not limited in the embodiments of the present disclosure. The teacher model may process the first sample audio based on at least one convolutional layer to obtain audio features, and classify the audio features based on a classifier to obtain multiple third probabilities.

[0083] Optionally, each third probability is associated with an audio event. For example, if the number of audio events is 10, the number of third probabilities output by the teacher model is 10; if the number of audio events is 20, the number of third probabilities output by the teacher model is 20. For example, if the audio events include a cat meowing, a dog barking, and rain, then after the teacher model processes the first sample audio, it can obtain the third probability corresponding to a cat meowing, a third probability corresponding to a dog barking, and a third probability corresponding to rain.

[0084] It should be noted that after the terminal device inputs the first sample audio to each teacher model, M third probabilities can be obtained. For example, the terminal device may include teacher model 1 and teacher model 2. The terminal device can input the first sample audio to teacher model 1, and teacher model 1 can output M third probabilities. The terminal device can input the first sample audio to teacher model 2, and teacher model 2 can output M third probabilities. The terminal device can obtain 2M third probabilities.

[0085] Optionally, the terminal device can obtain the teacher model based on the following feasible implementation method: for any group of training samples, the terminal device can extract audio features from the second sample audio and input the audio features to the teacher model (the terminal device can also directly input the second sample audio to the teacher model, and the convolution layer in the teacher model extracts audio features, which is not limited to this in the embodiments of the present disclosure). The teacher model can output M third probabilities associated with M audio events based on the audio features. The terminal device determines the loss function based on the M third probabilities output by the teacher model and the M probabilities corresponding to the M audio events associated with the second sample audio, and trains the teacher model based on the loss function, wherein the loss function can be any feasible loss function, which is not limited in the embodiments of the present disclosure.

[0086] Next, combine Figure 6 , the training process of the teacher model is explained.

[0087] Figure 6 This is a diagram of the training process of a teacher model provided in an embodiment of the present disclosure. Figure 6 , including a second sample audio, a teacher model, and a true probability. The second sample audio may be a longer audio, the teacher model may be structured as any neural network model, and the true probability may include a probability a of event A, b probability of event B, and c probability of event C in the second sample audio.

[0088] See Figure 6 , terminal equipment ( Figure 6(not shown) A second sample audio can be input into the teacher model, and the teacher model can extract the audio features of the second sample audio and obtain a predicted probability based on the audio features, where the predicted probability can include a probability of 1 for event A, a probability of 2 for event B, and a probability of 3 for event C. The terminal device can determine a loss function based on the predicted probability and the true probability, and update the model parameters in the teacher model based on the loss function. In this way, the trained teacher model can accurately identify the audio events included in the long audio clip.

[0089] The model parameters of multiple teacher models are larger than those of the audio event detection model. For example, the parameter size of each teacher model can be 1GB, while the parameter size of the audio event detection model can be 100MB. This allows the audio event detection model to adapt to a wider range of application scenarios (such as mobile phones), improving the user experience. Moreover, based on the guidance of the teacher model to the audio event detection model, the audio event detection model can learn the knowledge of the teacher model with a smaller parameter size, thereby improving the accuracy of audio event detection.

[0090] Optionally, the model structures of multiple teacher models can be different. For example, to improve the effectiveness of the teacher model in guiding the audio event detection model, the terminal device can train multiple classroom models with different model structures. This can improve the training accuracy of the audio event detection model and, in turn, improve the accuracy of audio event detection.

[0091] S504: Train the audio event detection model based on the M second probabilities and the M third probabilities output by each teacher model.

[0092] Optionally, since the cost of labeling audio events included in audio with a shorter duration is high, a teacher model obtained by training multiple audios with longer durations is used to guide the audio event detection model to learn the ability to detect audio events included in audio with a shorter duration. This can effectively reduce the training cost of the audio event detection model.

[0093] Optionally, the terminal device can use the M third probabilities output by each teacher model as pseudo labels, and then train the model parameters in the audio event detection model based on the M third probabilities output by each teacher model and the M second probabilities output by the audio event detection model. In this way, there is no need to label the audio events corresponding to the first sample audio (audio with shorter duration), thereby reducing training costs.

[0094] The embodiment of the present disclosure provides a training method for an audio event detection model, which obtains a first sample audio, inputs the first sample audio into the audio event detection model, obtains M second probabilities corresponding to M audio events associated with the audio content of the first sample audio, inputs the first sample audio into multiple teacher models associated with the audio event detection model, obtains M third probabilities corresponding to the M audio events output by each teacher model, and trains the audio event detection model based on the M second probabilities and the M third probabilities output by each teacher model. In this way, the terminal device can guide the audio event detection model to learn the ability to detect audio events included in short audio through multiple teacher models trained based on coarse-grained sample audio events. In this way, there is no need to label fine-grained data, which reduces the training cost of the audio event detection model. In addition, the audio event detection model can accurately perform audio event detection processing on shorter audio, thereby improving the accuracy of audio event detection.

[0095] exist Figure 5 Based on the embodiment shown below, combined with Figure 7 , the method for training the audio event detection model based on the M second probabilities and the M third probabilities output by each teacher model in the training method of the above-mentioned audio event detection model is further explained.

[0096] Figure 7 This is a schematic diagram of another method for training an audio event detection model provided by an embodiment of the present disclosure. Figure 7 , the method flow includes:

[0097] S701. Perform weighted averaging on multiple groups of M third probabilities output by multiple teacher models to obtain M fourth probabilities.

[0098] Among them, the terminal device can obtain M fourth probabilities based on the following feasible implementation method: obtain multiple third probabilities corresponding to each audio event, perform weighted averaging on the multiple third probabilities corresponding to each audio event, and obtain M fourth probabilities corresponding to the M audio events. For example, if the number of teacher models is 2, the number of third probabilities corresponding to each audio event is 2; if the number of teacher models is 4, the number of third probabilities corresponding to each audio event is 4. For example, if audio event A corresponds to third probability 1 and third probability 2, the terminal device can perform weighted averaging on third probability 1 and third probability 2 to obtain the fourth probability corresponding to audio event A; if audio event B corresponds to third probability 3 and third probability 4, the terminal device can perform weighted averaging on third probability 3 and third probability 4 to obtain the fourth probability corresponding to audio event B.

[0099] For example, audio events may include audio events of a cat's meowing, audio events of a dog's barking, and audio events of rain. The number of teacher models is 2. When the terminal device inputs the first sample audio to the first teacher model, the first teacher model can output the third probability 1 corresponding to the audio event of a cat's meowing, the third probability 2 corresponding to the audio event of a dog's barking, and the third probability 3 corresponding to the audio event of rain. When the terminal device inputs the first sample audio to the second teacher model, the second teacher model can also output the third probability A corresponding to the audio event of a cat's meowing, the third probability B corresponding to the audio event of a dog's barking, and the third probability 4 corresponding to the audio event of rain. The corresponding third probability C, therefore, the third probability corresponding to the audio event of a cat's meowing can include the third probability 1 and the third probability A, the third probability corresponding to the audio event of a dog's barking can include the third probability 2 and the third probability B, and the third probability corresponding to the audio event of rain can include the third probability 3 and the third probability C. Therefore, the fourth probability corresponding to the audio event of a cat's meowing can be the average of the third probability 1 and the third probability A, the fourth probability corresponding to the audio event of a dog's barking can be the average of the third probability 2 and the third probability B, and the fourth probability corresponding to the audio event of rain can be the average of the third probability 3 and the third probability C.

[0100] Next, combine Figure 8 , the process of determining M fourth probabilities is explained.

[0101] Figure 8 A schematic diagram of a process for determining M fourth probabilities provided by an embodiment of the present disclosure. Figure 8 , including: first sample audio, teacher model 1, teacher model 2 and teacher model 3. Terminal device ( Figure 8 (Not shown) The first sample audio may be input to teacher model 1, teacher model 2, and teacher model 3. Teacher model 1 may output a third probability a, teacher model 2 may output a third probability b, and teacher model 3 may output a third probability c.

[0102] See Figure 8 , the third probability a may include probability 1 of event A and probability 2 of event B, the third probability b may include probability 3 of event A and probability 4 of event B, and the third probability c may include probability 5 of event A and probability 6 of event B. The terminal device may determine the fourth probability of event A as the average of probability 1, probability 3, and probability 5, and the fourth probability of event B as the average of probability 2, probability 4, and probability 6. In this way, through weighted averaging, the audio event detection model can learn the audio event detection capability of each teacher model, thereby improving the training effect of the audio event detection model and improving the accuracy of audio event detection.

[0103] S702: Train an audio event detection model based on the M second probabilities and the M fourth probabilities.

[0104] Optionally, since the teacher model is trained based on multiple second sample audios and sample audio events corresponding to each second sample audio, the terminal device can randomly extract a segment of audio from the second sample audio to obtain the first sample audio. For example, the second sample audio can be 10 seconds of audio, and the terminal device can randomly extract 1 second of audio from the second sample audio to obtain the first sample audio.

[0105] Optionally, the terminal device may train the audio event detection model based on the following feasible implementation method: determine the target loss between each audio event associated with M second probabilities and M fourth probabilities, and update the model parameters of the audio event detection model according to the target loss. For example, for any audio event, the terminal device may determine the target loss based on the second probability corresponding to the audio event and the fourth probability corresponding to the audio event, and then construct a loss function based on the target loss, and update the model parameters of the audio event detection model based on the loss function. It should be noted that the loss function may be any feasible loss function, and the embodiments of the present disclosure are not limited to this.

[0106] For example, the terminal device inputs the first sample audio into the audio event detection model, and the audio event detection model can output the probability of event A as probability 1 (second probability) and the probability of event B as probability 2 (second probability). The terminal device can input the first sample audio into multiple teacher models, and based on the third probability output by multiple teacher models, determine the fourth probability of event A as probability a and the fourth probability of event B as probability b. The terminal device can determine a target loss based on the difference between probability 1 and probability a corresponding to event A, and determine another target loss based on the difference between probability 2 and probability b corresponding to event B, and then construct a loss function based on the two target losses, and update the model parameters of the audio event detection model based on the loss function.

[0107] For example, the terminal device can randomly extract a short audio clip of 1 second from the second sample audio, and convert it into a feature vector i based on a filter, whose feature dimension is t*f, where t is the number of frames and f is the number of frequency points (determined by the number of filters). The terminal device can input the feature vector i into N teacher models to obtain results y1, y2..., yN, and perform weighted averaging on the N results to obtain Yt, which can be a pseudo-label to guide the learning of the audio event detection model, where the dimension of Yt is C*1, where C is the number of preset audio event categories. The terminal device can input the feature vector i into the audio event detection model, and the audio event detection model can output the result Ys, where the dimension of Ys is also C*1, where C is the number of preset audio event categories. The terminal device can calculate the loss function L (which can be any classification loss function, and the embodiment of the present disclosure is not limited to this) between the pseudo-label Yt and the result Ys output by the audio event detection model, and perform gradient backpropagation on the loss function L to train the audio event detection model.

[0108] Next, combine Figure 9 , explains the training process of the audio event detection model.

[0109] Figure 9 This is a diagram of the training process of an audio event detection model provided by an embodiment of the present disclosure. Figure 9 , including: a first sample audio, an audio event detection model, a teacher model 1 and a teacher model 2. Terminal device ( Figure 9 (Not shown) A first sample audio can be input to the audio event detection model, teacher model 1, and teacher model 2. The audio event detection model outputs the second probability of event A as second probability 1 and the second probability of event B as second probability 2. Teacher model 1 outputs the third probability of event A as third probability a and the third probability of event B as third probability b. Teacher model 3 outputs the third probability of event A as third probability c and the third probability of event B as third probability d.

[0110] See Figure 9 , the terminal device can average the two third probabilities output by teacher model 1 and the two third probabilities output by teacher model 2 to obtain the fourth probability of event A as fourth probability 1 and the fourth probability of event B as fourth probability 2. Among them, the fourth probability 1 corresponding to event A can be the average of the third probability a and the third probability c, and the fourth probability 2 corresponding to event B can be the average of the third probability b and the third probability d. The terminal device determines loss 1 based on the difference between the second probability 1 and the fourth probability 1 corresponding to event A, and determines loss 2 based on the difference between the second probability 2 and the fourth probability 2 corresponding to event B, and then updates the model parameters of the audio event detection model according to loss 1 and loss 2.

[0111] Another method for training an audio event detection model provided by an embodiment of the present disclosure performs weighted averaging on multiple groups of M third probabilities output by multiple teacher models to obtain M fourth probabilities, and trains the audio event detection model based on the M second probabilities and the M fourth probabilities. In this way, the terminal device can determine the pseudo-label for training the audio event detection model in combination with the third probabilities output by multiple teacher models, and then train the audio event detection model based on the pseudo-label and the second probability output by the audio event detection model, without the need to label fine-grained data, thereby reducing the cost of model training. Moreover, since the first sample audio is audio of a shorter duration, the audio event detection model can better learn the ability to detect audio events of short and long audio, thereby improving the accuracy of audio event detection.

[0112] Figure 10 This is a structural diagram of an audio processing device provided by an embodiment of the present disclosure. Figure 10 The audio processing device 100 includes an acquisition module 101, an input module 102 and a determination module 103, wherein:

[0113] The acquisition module 101 is used to acquire a first audio;

[0114] The input module 102 is configured to input the first audio into the audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, the sample audio events are obtained based on multiple teacher models, and the duration of second sample audios used to train the multiple teacher models is greater than the duration of the first sample audio;

[0115] The determination module 103 is configured to determine the audio event included in the first audio based on the M first probabilities.

[0116] According to one or more embodiments of the present disclosure, the determining module 103 is specifically configured to:

[0117] Determining a probability greater than a preset threshold among the M first probabilities;

[0118] The audio event associated with the probability greater than the threshold is determined as the audio event included in the first audio.

[0119] According to one or more embodiments of the present disclosure, the acquisition module 101 is specifically configured to:

[0120] In response to a touch operation on the audio acquisition control, displaying an audio page, wherein the audio page includes a plurality of second audios;

[0121] The first audio is obtained in response to a touch operation on any one of the plurality of second audios.

[0122] According to one or more embodiments of the present disclosure, the acquisition module 101 is specifically configured to:

[0123] In response to a touch operation on the audio acquisition control, audio recording is performed to obtain the first voice.

[0124] According to one or more embodiments of the present disclosure, the audio event detection apparatus further includes a training module 104, wherein the training module 104 is specifically configured to:

[0125] Get the first sample audio;

[0126] Inputting the first sample audio into an audio event detection model to obtain M second probabilities corresponding to M audio events associated with audio content of the first sample audio;

[0127] Inputting a first sample audio into multiple teacher models associated with the audio event detection model, and obtaining M third probabilities corresponding to M audio events output by each teacher model;

[0128] The audio event detection model is trained based on the M second probabilities and the M third probabilities output by each teacher model.

[0129] According to one or more embodiments of the present disclosure, the training module 104 is specifically configured to:

[0130] Performing weighted averaging on the multiple groups of M third probabilities output by the multiple teacher models to obtain M fourth probabilities;

[0131] The audio event detection model is trained based on the M second probabilities and the M fourth probabilities.

[0132] According to one or more embodiments of the present disclosure, the training module 104 is specifically configured to:

[0133] Obtaining multiple third probabilities corresponding to each audio event;

[0134] A weighted average is performed on the multiple third probabilities corresponding to each audio event to obtain M fourth probabilities corresponding to the M audio events.

[0135] According to one or more embodiments of the present disclosure, the model parameter amount of the multiple teacher models is greater than the model parameter amount of the audio event detection model, and the model structures of the multiple teacher models are different.

[0136] The audio processing device provided in the embodiment of the present disclosure can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0137] Figure 11 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present disclosure. Figure 11 , which shows a schematic structural diagram of a terminal device 1100 suitable for implementing the embodiments of the present disclosure. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 11 The terminal device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0138] like Figure 11 As shown, terminal device 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of terminal device 1100 are also stored in RAM 1103. Processing device 1101, ROM 1102, and RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to bus 1104.

[0139] Typically, the following devices may be connected to the I / O interface 1105: an input device 1106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1109. The communication device 1109 may allow the terminal device 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 11The terminal device 1100 is shown as having various devices, but it should be understood that it is not required to implement or possess all of the devices shown, and more or fewer devices may be implemented or possessed instead.

[0140] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 1109, or installed from the storage device 1108, or installed from the ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0141] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The computer-readable medium may be included in the terminal device, or may exist independently without being incorporated into the terminal device.

[0143] The computer-readable medium carries one or more programs. When the one or more programs are executed by the terminal device, the terminal device executes the method shown in the above embodiment.

[0144] An embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the audio processing method that may be involved in various embodiments above is implemented.

[0145] An embodiment of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, the audio processing method that may be involved in various embodiments described above is implemented.

[0146] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0149] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0152] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0153] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0154] For example, in response to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thereby, the user can independently choose whether to provide personal information to the software or hardware such as the terminal device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from the user, the method of sending the prompt message to the user can be, for example, a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the terminal device.

[0155] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0156] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws and regulations. Data may include information, parameters and messages, such as flow switching indication information.

[0157] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0158] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0159] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An audio processing method, characterized in that: include: Get the first audio; Input the first audio to an audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is trained based on a plurality of first sample audios and sample audio events included in each of the first sample audios, the sample audio events are obtained based on a plurality of teacher models, and the duration of the second sample audios used to train the plurality of teacher models is greater than the duration of the first sample audio; Based on the M first probabilities, an audio event included in the first audio is determined.

2. The method according to claim 1, characterized in that The step of determining the audio event included in the first audio based on the M first probabilities comprises: Determine a probability greater than a preset threshold among the M first probabilities; The audio event associated with the probability greater than the threshold is determined as the audio event included in the first audio.

3. The method according to claim 1, characterized in that The obtaining of the first audio includes: In response to a touch operation on the audio acquisition control, displaying an audio page, wherein the audio page includes a plurality of second audios; In response to a touch operation on any one of the plurality of second audios, the first audio is obtained.

4. The method according to claim 1, characterized in that: The obtaining of the first audio includes: In response to a touch operation on the audio acquisition control, audio recording is performed to obtain the first voice.

5. The method according to any one of claims 1 to 4, characterized in that: The audio event detection model is trained based on the following steps: Get the first sample audio; Inputting the first sample audio into an audio event detection model to obtain M second probabilities corresponding to M audio events associated with audio content of the first sample audio; Inputting a first sample audio into a plurality of teacher models associated with the audio event detection model, and obtaining M third probabilities corresponding to M audio events output by each teacher model; The audio event detection model is trained based on the M second probabilities and the M third probabilities output by each teacher model.

6. The method according to claim 5, characterized in that The step of training the audio event detection model based on the M second probabilities and the M third probabilities output by each teacher model comprises: Performing weighted average processing on multiple groups of M third probabilities output by the multiple teacher models to obtain M fourth probabilities; The audio event detection model is trained based on the M second probabilities and the M fourth probabilities.

7. The method according to claim 6, characterized in that The step of performing weighted average processing on the multiple groups of M third probabilities output by the multiple teacher models to obtain M fourth probabilities includes: Obtain multiple third probabilities corresponding to each audio event; A weighted average is performed on the multiple third probabilities corresponding to each audio event to obtain M fourth probabilities corresponding to the M audio events.

8. The method according to any one of claims 1 to 4, characterized in that: The model parameter quantity of the multiple teacher models is greater than the model parameter quantity of the audio event detection model, and the model structures of the multiple teacher models are different.

9. An audio processing device, characterized in that: It includes an acquisition module, an input module and a determination module, wherein: The acquisition module is used to acquire the first audio; The input module is used to input the first audio into the audio event detection model to obtain M first probabilities corresponding to M audio events associated with the audio content of the first audio; wherein M is an integer greater than 1, the audio event detection model is trained based on multiple first sample audios and sample audio events included in each first sample audio, the sample audio events are obtained based on multiple teacher models, and the duration of the second sample audios for training the multiple teacher models is greater than the duration of the first sample audio; The determination module is used to determine the audio event included in the first audio based on the M first probabilities.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the audio processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the audio processing method according to any one of claims 1 to 8 is implemented.

Images