Audio processing method and device, computer-readable storage medium, and computing device

Abstract

Embodiments of the present disclosure provide an audio processing method and apparatus, a computer-readable storage medium, and a computing device, wherein the method comprises: extracting audio features of the audio to be processed; determining the audio category of the audio to be processed using an audio tag prediction model based on the audio features; and generating a summary statement for the audio to be processed using a pre-trained audio summary generation model based on the audio features and the audio category. Because the summary statement is generated by combining audio features and audio categories, the audio category can indicate the direction of the summary statement, thereby narrowing the scope of the summary statement. Therefore, compared with the summary statement generated using only audio features in the related art, the accuracy and efficiency are greatly improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of audio processing. More specifically, the embodiments of the present disclosure relate to an audio processing method and apparatus, a computer-readable storage medium, and a computing device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] In recent years, with the continuous development of artificial intelligence technology, automated audio processing has become increasingly common. Among the many types of audio processing, audio summarization is gaining increasing popularity. Summarizing audio involves generating natural language to describe the content of a specific audio clip. For example, given an audio clip of a door opening and closing, the following summary might be generated: "The door squeaks as it slowly swings back." However, the accuracy and efficiency of audio summarization models used in related technologies for summarizing audio are relatively low. Summary of the Invention

[0004] In this context, embodiments of the present disclosure are intended to provide an audio processing method and apparatus, a computer-readable storage medium, and a computing device to improve the accuracy and efficiency of generating audio summaries.

[0005] In a first aspect of the embodiments of the present disclosure, there is provided an audio processing method, comprising:

[0006] Extracting audio features of the audio to be processed;

[0007] Determining the audio category of the audio to be processed using an audio tag prediction model based on the audio features;

[0008] A summary sentence of the audio to be processed is generated using a pre-trained audio summary generation model according to the audio features and the audio category.

[0009] In one embodiment of the present disclosure, extracting audio features of the audio to be processed includes:

[0010] Sampling the audio to be processed using a preset frequency to obtain a sampling result;

[0011] Feature extraction is performed on the sampling result to obtain a feature sequence as the audio feature.

[0012] In another embodiment of the present disclosure, determining the audio category of the audio to be processed by using an audio tag prediction model according to the audio features includes:

[0013] Inputting the audio feature into the audio tag prediction model, wherein the audio tag prediction model outputs a score of at least one audio tag in the tag library;

[0014] selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score;

[0015] Determine, from a label library, a candidate audio category corresponding to each of the reserved labels;

[0016] An audio category of the audio to be processed is determined from a preset number of candidate audio categories.

[0017] In yet another embodiment of the present disclosure, further comprising:

[0018] After outputting the score of at least one audio tag in the tag library, normalizing the score;

[0019] The step of selecting a preset number of audio tags from the at least one audio tag as retained tags according to the score includes:

[0020] A preset number of audio tags are selected from the at least one audio tag according to the normalized score as reserved tags.

[0021] In yet another embodiment of the present disclosure, selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score includes:

[0022] Among the at least one audio tag, retaining an audio tag having a score higher than a preset score threshold;

[0023] In response to the number of the reserved audio tags being less than or equal to the preset number, taking all the reserved audio tags as reserved tags;

[0024] In response to the number of the retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags.

[0025] In another embodiment of the present disclosure, the tag library includes a plurality of audio tags, the plurality of audio tags have a tree-like hierarchical relationship, each audio tag has an upper-level and / or lower-level audio tag, and at least one audio tag in the tag library is pre-marked as a first-category tag;

[0026] Determining, from the tag library, a candidate audio category corresponding to each of the reserved tags includes:

[0027] In response to the reserved label being a first category label, determining the first category label as a candidate audio category corresponding to the reserved label;

[0028] Otherwise, taking the reserved tag as the starting level, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

[0029] In yet another embodiment of the present disclosure, further comprising:

[0030] At least one audio tag in the tag library is marked as a first-category tag.

[0031] In yet another embodiment of the present disclosure, determining one audio category from a preset number of candidate audio categories as the audio category to be processed:

[0032] Determine a comprehensive score for each candidate audio category, wherein the comprehensive score is the sum of scores of all subordinate audio tags corresponding to the candidate audio category;

[0033] The candidate audio category with the highest comprehensive score is determined as the audio category of the audio to be processed.

[0034] In yet another embodiment of the present disclosure, generating a summary sentence of the audio to be processed using a pre-trained audio summary generation model based on the audio features and the audio category includes:

[0035] The encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector;

[0036] The decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes the initial word vector and the label word vector of the audio category.

[0037] In yet another embodiment of the present disclosure, generating a summary sentence of the audio to be processed based on the encoding vector and the initial word sequence includes:

[0038] generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence;

[0039] The i+1th group of word vectors is generated according to the initial word sequence and the word sequence composed of the first i groups of word vectors, and the encoding vector until a terminal word vector is generated, where i≥1 and each group of word vectors includes at least one word vector.

[0040] In yet another embodiment of the present disclosure, generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence includes:

[0041] Embedding the initial word sequence into the encoded vector to form a first vector, and generating a first set of word vectors for the summary sentence by decoding the first vector;

[0042] Generating the i+1th group of word vectors according to the initial word sequence, the first i groups of word vectors, and the encoding vector includes:

[0043] The word sequence consisting of the initial word sequence and the first i groups of word vectors is embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

[0044] In yet another embodiment of the present disclosure, the i+1th group of word vectors is generated in the following manner:

[0045] Determine a target probability of at least one word vector in a preset vocabulary, where the target probability is a probability that the word vector belongs to the i+1th group of word vectors;

[0046] The (i+1)th group of word vectors is generated according to the target probability of the at least one word vector.

[0047] In yet another embodiment of the present disclosure, further comprising:

[0048] Extracting audio features of training audio in a training set, wherein the training audio is labeled with a category label and a summary label;

[0049] Inputting the audio features and category labels of the training audio into the audio summary generation model, and the audio summary generation model outputting a predicted label of the training audio;

[0050] A network loss value is determined according to the summary label and the predicted label, and network parameters of an encoding module and a decoding module of the audio summary generation model are adjusted according to the network loss value.

[0051] In a second aspect of the embodiments of the present disclosure, there is provided an audio processing device, comprising:

[0052] A feature module, used to extract audio features of the audio to be processed;

[0053] A classification module, configured to determine the audio category of the audio to be processed using an audio tag prediction model based on the audio features;

[0054] The summary module is used to generate a summary sentence of the audio to be processed based on the audio features and the audio category using a pre-trained audio summary generation model.

[0055] In one embodiment of the present disclosure, the feature module is specifically configured to:

[0056] Sampling the audio to be processed using a preset frequency to obtain a sampling result;

[0057] Feature extraction is performed on the sampling result to obtain a feature sequence as the audio feature.

[0058] In another embodiment of the present disclosure, the category module is specifically configured to:

[0059] Inputting the audio feature into the audio tag prediction model, wherein the audio tag prediction model outputs a score of at least one audio tag in the tag library;

[0060] selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score;

[0061] Determine, from a label library, a candidate audio category corresponding to each of the reserved labels;

[0062] An audio category of the audio to be processed is determined from a preset number of candidate audio categories.

[0063] In yet another embodiment of the present disclosure, the category module is further configured to:

[0064] After outputting the score of at least one audio tag in the tag library, normalizing the score;

[0065] The category module is configured to select a preset number of audio tags from the at least one audio tag according to the score as reserved tags, specifically for:

[0066] A preset number of audio tags are selected from the at least one audio tag according to the normalized score as reserved tags.

[0067] In yet another embodiment of the present disclosure, the category module is configured to select a preset number of audio tags from the at least one audio tag as retained tags based on the scores, specifically to:

[0068] Among the at least one audio tag, retaining an audio tag having a score higher than a preset score threshold;

[0069] In response to the number of the reserved audio tags being less than or equal to the preset number, taking all the reserved audio tags as reserved tags;

[0070] In response to the number of the retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags.

[0071] In another embodiment of the present disclosure, the tag library includes a plurality of audio tags, the plurality of audio tags have a tree-like hierarchical relationship, each audio tag has an upper-level and / or lower-level audio tag, and at least one audio tag in the tag library is pre-marked as a first-category tag;

[0072] When the category module is used to determine the candidate audio category corresponding to each of the reserved tags from the tag library, it is specifically used to:

[0073] In response to the reserved label being a first category label, determining the first category label as a candidate audio category corresponding to the reserved label;

[0074] Otherwise, taking the reserved tag as the starting level, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

[0075] In yet another embodiment of the present disclosure, a marking module is further included, configured to:

[0076] At least one audio tag in the tag library is marked as a first-category tag.

[0077] In yet another embodiment of the present disclosure, the summary module is specifically configured to:

[0078] Determine a comprehensive score for each candidate audio category, wherein the comprehensive score is the sum of scores of all subordinate audio tags corresponding to the candidate audio category;

[0079] The candidate audio category with the highest comprehensive score is determined as the audio category of the audio to be processed.

[0080] In yet another embodiment of the present disclosure, when the summary module is configured to generate a summary statement of the audio to be processed based on the audio features and the audio category using a pre-trained audio summary generation model, the summary module is specifically configured to:

[0081] The encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector;

[0082] The decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes the initial word vector and the label word vector of the audio category.

[0083] In yet another embodiment of the present disclosure, when the summary module is configured to generate a summary sentence of the audio to be processed based on the encoding vector and the initial word sequence, the summary module is specifically configured to:

[0084] generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence;

[0085] The i+1th group of word vectors is generated according to the initial word sequence and the word sequence composed of the first i groups of word vectors, and the encoding vector until a terminal word vector is generated, where i≥1 and each group of word vectors includes at least one word vector.

[0086] In yet another embodiment of the present disclosure, when the summary module is configured to generate a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence, the summary module is specifically configured to:

[0087] Embedding the initial word sequence into the encoded vector to form a first vector, and generating a first set of word vectors for the summary sentence by decoding the first vector;

[0088] Generating the i+1th group of word vectors according to the initial word sequence, the first i groups of word vectors, and the encoding vector includes:

[0089] The word sequence consisting of the initial word sequence and the first i groups of word vectors is embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

[0090] In yet another embodiment of the present disclosure, the summary module is configured to generate the i+1th group of word vectors in the following manner:

[0091] Determine a target probability of at least one word vector in a preset vocabulary, where the target probability is a probability that the word vector belongs to the i+1th group of word vectors;

[0092] The (i+1)th group of word vectors is generated according to the target probability of the at least one word vector.

[0093] In yet another embodiment of the present disclosure, a training module is further included, for:

[0094] Extracting audio features of training audio in a training set, wherein the training audio is labeled with a category label and a summary label;

[0095] Inputting the audio features and category labels of the training audio into the audio summary generation model, and the audio summary generation model outputting a predicted label of the training audio;

[0096] A network loss value is determined according to the summary label and the predicted label, and network parameters of an encoding module and a decoding module of the audio summary generation model are adjusted according to the network loss value.

[0097] In a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0098] In a fourth aspect of the embodiments of the present disclosure, a computing device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method described in the first aspect when executing the computer instructions.

[0099] According to the audio processing method and apparatus, computer-readable storage medium, and computing device of the embodiments of the present disclosure, audio features of the audio to be processed are extracted, and then, based on the audio features, the audio category of the audio to be processed is determined using an audio tag prediction model. Finally, a summary statement for the audio to be processed is generated using a pre-trained audio summary generation model based on the audio features and the audio category. Because the summary statement is generated by combining audio features and audio categories, the audio category can indicate the direction of summary statement generation, thereby narrowing the range of summary statement selection. Therefore, compared with summary statements generated using only audio features in related technologies, the accuracy and efficiency are greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0101] Figure 1 The following schematically illustrates a flow of an audio processing method according to an embodiment of the present disclosure;

[0102] Figure 2 The following schematically illustrates a process for determining an audio category according to an embodiment of the present disclosure;

[0103] Figure 3 The following schematically shows a partial tag diagram of a tag library according to an embodiment of the present disclosure;

[0104] Figure 4 Schematically shows a partial tag diagram of a tag library according to another embodiment of the present disclosure;

[0105] Figure 5 The following schematically illustrates a process for generating an audio summary according to an embodiment of the present disclosure;

[0106] Figure 6 The following is a schematic diagram schematically showing a process of generating an audio summary according to an embodiment of the present disclosure;

[0107] Figure 7 The structure of an audio processing device according to an embodiment of the present disclosure is schematically shown;

[0108] Figure 8 Schematically illustrates a computer-readable storage medium according to an embodiment of the present disclosure;

[0109] Figure 9 The figure schematically shows a structural diagram of a computing device according to an embodiment of the present disclosure.

[0110] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0111] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0112] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0113] According to embodiments of the present disclosure, an audio processing method and apparatus, a computer-readable storage medium, and a computing device are provided. It should be understood that any number of elements in the accompanying drawings is for illustrative purposes only and not limiting, and any nomenclature is for distinction only and does not have any limiting meaning.

[0114] The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0115] Audio summarization models, such as the AAC (automatic speech recognition) model, must extract audio events, audio scenes, the temporal order of individual sound sources, and logical relationships such as foreground and background during the summary generation process, making this a complex task. Therefore, related technologies primarily employ the following two approaches to improve the accuracy and efficiency of summary generation.

[0116] The first approach is to introduce more training data. Introducing more training data can make the model training more comprehensive, but the cost of labeling training data for audio summarization models is high and expensive, and it is even more difficult to label audio with poor summary results.

[0117] The second approach involves using a pre-trained model as the encoder. This encoder is pre-trained using training data from other tasks, and the resulting parameters are used to initialize the parameters of the audio summarization model. This approach places high demands on the relevance of the transfer task, and the improvement is limited.

[0118] In summary, current methods can only introduce a large amount of resources for data labeling or introduce similar tasks for model pre-training. Moreover, the above two methods do not significantly improve the accuracy and efficiency of audio summary generation.

[0119] Based on this, an embodiment of the present disclosure provides an audio processing method that can be used to generate a summary for the audio to be processed. The summary is a sentence consisting of at least one word that is used to describe the audio content. The method predicts the category of the audio to be processed through an audio tag prediction (AT) model. The category can indicate the direction of generating the summary sentence, thereby narrowing the selection range of the summary sentence. Therefore, the method does not require the introduction of a large amount of resources for data annotation, nor does it require the introduction of related tasks for model training, and can greatly improve the accuracy and efficiency of summary generation.

[0120] The audio processing method provided by the embodiments of the present disclosure can be applied to a variety of audio processing scenarios. For example, an audio processing website has a summary generation function, and a user can input a segment of audio and activate the summary generation function for the segment of audio. The website can then run the method provided by this embodiment for the segment of audio to obtain an audio summary, and the website can display it to the customer. For another example, an audio processing application has a summary generation function, and a user can input a segment of audio and activate the summary generation function for the segment of audio. The website can then run the method provided by this embodiment for the segment of audio to obtain an audio summary, and the website can display it to the customer.

[0121] Please refer to the attached Figure 1 , which shows a process of an audio processing method provided by an embodiment of the present disclosure, including steps S101 to S103.

[0122] The audio processing method can be used to generate a summary for the audio to be processed, and the summary is a sentence consisting of at least one word, which is used to describe the audio content. In addition, the audio processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA) handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server, which can be a local server, a cloud server, etc.

[0123] In step S101, audio features of the audio to be processed are extracted.

[0124] In this step, the audio to be processed may be sampled at a preset frequency to obtain a sampling result; and then feature extraction may be performed on the sampling result to obtain a feature sequence as the audio feature.

[0125] The sampling frequency can be determined based on the audio tag prediction model in step S102 because the sampling frequency can affect the sampling results, which in turn can affect the audio features, which in turn can be matched with the audio tag prediction model. Optionally, a 32k sampling frequency is used to sample the processed audio, i.e., 32,000 points of the processed audio are sampled in 1 second.

[0126] A pre-trained feature extraction network can be used to extract features from the sampling results. For example, the audio feature can be a logarithmic Mel-spectrogram.

[0127] In step S102, the audio category of the audio to be processed is determined using an audio tag prediction model according to the audio features.

[0128] In this step, the audio tag prediction (AT) model can first use the audio features extracted in step S101 to predict the audio tag of the audio to be processed, and then determine the audio category of the audio to be processed based on the predicted audio tags. Each of the above-mentioned predicted audio tags is a possible category of the audio to be processed, and these audio tags can be tags in a tag library, which includes audio tags of various categories under various scenarios.

[0129] Optionally, PANN is used as the AT model. PANN is a model that uses multiple CNN architectures as feature extractors to predict audio tags.

[0130] In addition, the audio tag prediction model can be pre-trained to accurately predict audio tags. During training, the predicted audio tags and the true values ​​of the audio tags can be used to adjust the model parameters until the model converges.

[0131] In step S103, a summary sentence of the audio to be processed is generated using a pre-trained audio summary generation model according to the audio features and the audio category.

[0132] The audio summary generation model can be an AAC (automatic speech recognition) model, and the ACC model can adopt a seq2seq model architecture. The audio summary generation model can include an encoder module and a decoder module. The encoder can be composed of multiple stacked convolutional layers (CNN), and the decoder can adopt a Transformer structure.

[0133] Optionally, the audio summary generation model can generate an audio summary in a word-by-word manner. That is, the audio summary generation model does not generate the entire audio summary at one time, but generates each part of the audio summary one by one to obtain the entire audio summary; and can also use the previously generated parts when generating subsequent parts.

[0134] When generating an audio summary, the audio category can indicate the direction of the audio summary and limit its scope. For example, if the audio category is vocal, the audio summary must be related to vocals; if the audio category is music, the audio summary must be related to music. Without an audio category indicator, the selection range of audio summaries is unlimited, making audio summary generation more difficult, resulting in lower accuracy and efficiency.

[0135] According to the audio processing method and apparatus, computer-readable storage medium, and computing device of the embodiments of the present disclosure, audio features of the audio to be processed are extracted, and then, based on the audio features, the audio category of the audio to be processed is determined using an audio tag prediction model. Finally, a summary statement of the audio to be processed is generated using a pre-trained audio summary generation model based on the audio features and the audio category. Because the audio features and audio category are combined when generating the summary statement, the audio category can indicate the direction of the summary statement, thereby narrowing the scope of the summary statement. Therefore, compared with the summary statements generated using only audio features in the related art, the accuracy and efficiency are greatly improved.

[0136] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0137] In some embodiments of the present disclosure, the following Figure 2 The method shown in FIG. 1 is to determine the audio category of the audio to be processed, including steps S201 to S204 .

[0138] In step S201, the audio feature is input into the audio tag prediction model, and the audio tag prediction model outputs a score of at least one audio tag in a tag library.

[0139] The audio tags in the tag library may include common categories covered by audio. The tag library contains multiple audio tags, and the multiple audio tags have a tree-like hierarchical relationship. Each audio tag has an upper-level and / or lower-level audio tag, that is, the first-level tag only has a lower-level tag, the last-level tag only has an upper-level tag, and the tags of other levels have both upper-level tags and lower-level tags. Optionally, the tag library in this embodiment has a total of 527 tags, please refer to the attached Figure 3 , which shows the first-level tags and second-level tags in the tag library, including 7 first-level tags and 43 second-level tags in total.

[0140] The score of an audio tag represents the probability that the audio tag matches the audio content of the audio being processed. For example, if the audio being processed is audio content from a hunting scene, including the sounds of a hunting rifle and the barking of hunting dogs, and the audio tag is hunting, and the score of the audio tag is 0.85, then the score indicates that the probability that the audio tag matches the audio content of the audio being processed is 0.85.

[0141] Optionally, the audio tag prediction model outputs a score for each audio tag in the tag library. Note that the tag library contains many audio tags, so when outputting scores for a particular audio file, some audio tags may receive a score of 0, indicating that these audio tags are completely irrelevant to the audio file.

[0142] In step S202, a preset number of audio tags are selected from the at least one audio tag according to the score as reserved tags.

[0143] In this step, audio tags with scores higher than a preset score threshold can be retained from the at least one audio tag; then, in response to the number of retained audio tags being less than or equal to the preset number, all the retained audio tags are used as retained tags; in response to the number of retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags. That is, retained tags are selected from at least one audio tag according to the two dimensions of the score threshold and the preset number, and the preset number of tags is retained as much as possible, but at the same time, it is ensured that the scores of all retained tags are greater than the score threshold, so that tags with a higher probability of being consistent with the audio content quality inspection of the audio to be processed can be retained, while interfering tags with a lower probability of being consistent with the audio content quality inspection of the audio to be processed can be removed. Optionally, the preset number is 10.

[0144] It should be noted that to facilitate score comparison, after obtaining the score of at least one audio tag in the tag library, the score can be normalized. A preset score threshold can then be set based on the normalized score. Specifically, a preset number of audio tags are selected from the at least one audio tag based on the normalized score as retained tags. Optionally, the score threshold can be set to 0.01.

[0145] In step S203, a candidate audio category corresponding to each of the reserved tags is determined from the tag library.

[0146] At least one audio tag in the tag library is pre-marked as a first-class tag. The first-class tag represents a tag that can be used as a category of audio, while other tags cannot be used as categories of audio. Figure 4 , which is an abbreviated schematic diagram of all tags under the first-level tag Sounds of things, where the first-level tag Sounds of things, the second-level tag Vehicle, and the third-level tag Motor vehicle (road) are all marked as first-class tags.

[0147] Based on the structure of the above-mentioned tag library, this step can be in response to determining that the first-category tag is the candidate audio category corresponding to the reserved tag; otherwise (that is, the reserved tag is not a first-category tag), starting from the reserved tag, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

[0148] That is to say, when the reserved label is a first-category label, since it can be used as a category of audio, the reserved label itself is determined as its corresponding candidate audio category; and when the reserved label is not a first-category label, since it cannot be used as an audio category, the first-category label with the smallest level difference from the parent label of the reserved label is used as its corresponding candidate audio category. In other words, each first-category label is not only used as its own corresponding candidate audio category, but also as the candidate audio category corresponding to all non-first-category labels of its subordinates. In addition, a label mapping table for each first-category label can be established in advance, and the mapping table can record all non-first-category labels corresponding to each first-category label, that is, record all labels that use the first-category label as candidate audio categories, so as to facilitate the determination of candidate audio categories in this step.

[0149] For example, Figure 4 The sub-tags of Motor vehicle (road) shown include Car, Truck, Bus, etc., as well as sub-tags of these sub-tags, such as Car alarm, etc. If the tag Car is retained, then Car should be mapped to Motor vehicle (road) and Motor vehicle (road) should be used as the corresponding candidate tag category, without further mapping upward to Vehicle and Sounds of things. If the tags Boat and Water vehicle are retained, then they should be mapped to Vehicle and Vehicle should be used as the corresponding candidate audio category. If the tag Engine is retained, then it should be mapped to Sounds of things and Sounds of things should be used as the corresponding candidate audio category.

[0150] It should be noted that at least one audio tag in the tag library can also be marked as a first-class tag in advance. When selecting the first-class tag, the higher-level tag can be directly selected as the first-class tag, for example, all the first-level tags and all the second-level tags can be selected as the first-class tags, because the higher-level tags have a certain degree of generalization and representativeness; the tags of the training audios that appear more frequently in the training set can also be selected as the first-class tags, for example, the tags of the top 13 training audios with the highest frequency of appearance can all be marked as first-class tags. These tags that appear more frequently have a certain degree of popularity, so the probability of these tags appearing is high, and they can be targeted as categories. By pre-marking the first-class tags, the candidate audio categories determined subsequently can be more accurate, and they are both qualified as audio categories, but not too high in level to be targeted.

[0151] In step S204, an audio category is determined from a preset number of candidate audio categories as the audio category to be processed.

[0152] In this step, a comprehensive score can be first determined for each candidate audio category, where the comprehensive score is the sum of the scores of all subordinate audio tags corresponding to the candidate audio category. The candidate audio category with the highest comprehensive score is then determined as the audio category of the audio to be processed. The tag mapping table mentioned in step S203 can be used to determine all subordinate audio tags for each candidate audio category. Determining the category of the audio to be processed from multiple candidate audio categories using comprehensive scores is accurate, convenient, and reliable.

[0153] When all subordinate tags of the candidate audio category have output scores in step S201, the scores of each tag are directly summed up. When only some of the subordinate tags of the candidate audio category have output scores in step S201, the scores of the tags that have not output scores are recorded as 0, and then the scores of each tag are summed up.

[0154] Optionally, the comprehensive score is calculated using the following formula:

[0155] S c =∑ l∈F(c) S l ;

[0156] Among them, F(c) is the set of all subordinate labels corresponding to the candidate audio category c; S l The score of label l.

[0157] For example, in Figure 4 In the simplified version of the sticky note library shown, when the candidate audio category is Sounds of things, the sum of the scores of all its subordinate tags (including first-category tags and non-first-category tags) is its comprehensive score.

[0158] In some embodiments of the present disclosure, the following Figure 5 The method shown generates a summary sentence of the audio to be processed, including steps S501 and S502.

[0159] In step S501, the encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector.

[0160] The coding vector may be an audio coding sequence in vector form.

[0161] In step S502, the decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes an initial word vector (sos) and a label word vector of the audio category.

[0162] Among them, when the decoding module predicts the result sequence, the first position of the result sequence predicts the audio category, and the subsequent positions predict the audio summary normally. This step directly adds the audio category obtained in the above step to the first position of the result sequence to guide the prediction process. In other words, <sos>The decoding module will start decoding from this initial sequence and combine it with the encoding vector obtained by the encoding module. It will use greedy search or beam-search and other search methods to generate the word vectors of the audio summary sentence one by one until it reaches <eos>Mark till the end.

[0163] Optionally, a first group of word vectors for the summary sentence is first generated based on the encoding vector and the initial word sequence. For example, the initial word sequence can be embedded in the encoding vector to form a first vector, and the first group of word vectors for the summary sentence is generated by decoding the first vector; then, the i+1th group of word vectors is generated based on the word sequence consisting of the initial word sequence and the first i groups of word vectors, and the encoding vector, until an end word vector (eos) is generated, where i≥1, and each group of word vectors includes at least one word vector. For example, the word sequence consisting of the initial word sequence and the first i groups of word vectors can be embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

[0164] Among them, the i+1th group of word vectors can be generated in the following manner: first determine the target probability of at least one word vector in the preset vocabulary, where the target probability is the probability that the word vector belongs to the i+1th group of word vectors; then generate the i+1th group of word vectors based on the target probability of the at least one word vector.

[0165] In one example, the audio category of the audio to be processed is TGA_human-voice. The encoding module is first used to obtain the encoding vector, and then the decoding module is required to decode the text sequence according to the encoding vector and the text sequence. <sos>TGA_human-voice" starts predicting the next word vector. The predicted word vector is added to the end of the text sequence and continues to be input into the decoding module for prediction. This cycle continues until the end-of-sentence marker is encountered. <eos>Then stop generating. In this process, greedy search or beam-search can be used to search for paths.

[0166] In some embodiments of the present disclosure, the audio summary generation module can also be pre-trained in the following manner: first, audio features of training audio in a training set are extracted, wherein the training audio is marked with a category label and a summary label; next, the audio features and category label of the training audio are input into the audio summary generation model, and the audio summary generation model outputs a predicted label of the training audio; finally, a network loss value is determined based on the summary label and the predicted label, and the network parameters of the encoding module and the decoding module of the audio summary generation model are adjusted based on the network loss value.

[0167] Through the above training process, the audio summary generation model can generate audio summaries more accurately. The training audio used in the above training process is pre-labeled with category labels. It should be understood that training audio without category labels can also be used for training. Here is an example to illustrate the detailed process of training. Taking an audio "00332lake beach 1.wav" in the Clotho training set as an example, the audio category is confirmed to be TGA_human-voice. The correct audio summary of this audio is "children play and adults converse in a noisy city area." During training, 14 labels including TGA_human-voice will also be regarded as words and added to the preset vocabulary. The training algorithm first requires the decoding module to obtain the encoding vector and the generated text sequence according to the encoding module. <sos>Predict TGA_human-voice; then require the decoding module to calculate the encoding vector and the generated text sequence" <sos>TGA_human-voice" predicts children; the third step is to use the encoding vector and the generated text sequence" <sos>TGA_human-voice children" predicts the next word play, and so on, until the decoding module is required to predict the word "TGA_human-voice children play and adults converse in a noisy city area." based on the encoding vector and the generated text sequence <eos>until.

[0168] The training method provided in this embodiment adds the dimension of audio category, thereby improving the training stability and training effect of the model.

[0169] Please refer to the attached Figure 6 , which exemplarily illustrates the process of generating an audio summary according to an embodiment of the present disclosure. As can be seen from the figure, the pre-extracted audio features are pre-input into the AT task model, and then the output of the AT task model obtains the audio category through the audio category confirmation step; then the pre-extracted audio features are also encoded by the encoding module of the AAC model to obtain an encoding vector, and at the same time, the word vector composed of sos, the above audio category and the generated audio summary is embedded into the above encoding vector by the decoding module, and further the probability distribution of the preset word list is obtained by decoding the above embedding result, and then the next word vector is determined according to the probability distribution.

[0170] Figure 6 The process shown is the generation of a word vector in the middle of the audio summary. That is, each word vector is generated according to the above process. It is important to note that when the first word vector is generated, the input to the decoding module is sos and the audio category, and when the next word vector obtained is eos, the generation of the audio summary stops.

[0171] Please refer to the table below, which shows audio summaries generated by audio summarization methods in related art and audio summaries generated using the audio processing method provided by this disclosure. As can be seen from the table, by using audio category guidance, descriptions of bird calls can be generated; however, without this strategy in related art, the generated descriptions differ significantly from the standard manually written answers.

[0172] Table 1: Comparison of audio summary generation results

[0173]

[0174] The present disclosure also provides an audio processing device. Figure 7 , which shows the structure of the device, including:

[0175] Feature module 701, used to extract audio features of the audio to be processed;

[0176] A classification module 702 is configured to determine the audio category of the audio to be processed using an audio tag prediction model based on the audio features;

[0177] The summary module 703 is configured to generate a summary sentence of the audio to be processed based on the audio features and the audio category using a pre-trained audio summary generation model.

[0178] In one embodiment of the present disclosure, the feature module is specifically configured to:

[0179] Sampling the audio to be processed using a preset frequency to obtain a sampling result;

[0180] Feature extraction is performed on the sampling result to obtain a feature sequence as the audio feature.

[0181] In another embodiment of the present disclosure, the category module is specifically configured to:

[0182] Inputting the audio feature into the audio tag prediction model, wherein the audio tag prediction model outputs a score of at least one audio tag in the tag library;

[0183] selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score;

[0184] Determine, from a label library, a candidate audio category corresponding to each of the reserved labels;

[0185] An audio category of the audio to be processed is determined from a preset number of candidate audio categories.

[0186] In yet another embodiment of the present disclosure, the category module is further configured to:

[0187] After outputting the score of at least one audio tag in the tag library, normalizing the score;

[0188] The category module is configured to select a preset number of audio tags from the at least one audio tag according to the score as reserved tags, specifically for:

[0189] A preset number of audio tags are selected from the at least one audio tag according to the normalized score as reserved tags.

[0190] In yet another embodiment of the present disclosure, the category module is configured to select a preset number of audio tags from the at least one audio tag as retained tags based on the scores, specifically to:

[0191] Among the at least one audio tag, retaining an audio tag having a score higher than a preset score threshold;

[0192] In response to the number of the reserved audio tags being less than or equal to the preset number, taking all the reserved audio tags as reserved tags;

[0193] In response to the number of the retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags.

[0194] In another embodiment of the present disclosure, the tag library includes a plurality of audio tags, the plurality of audio tags have a tree-like hierarchical relationship, each audio tag has an upper-level and / or lower-level audio tag, and at least one audio tag in the tag library is pre-marked as a first-category tag;

[0195] When the category module is used to determine the candidate audio category corresponding to each of the reserved tags from the tag library, it is specifically used to:

[0196] In response to the reserved label being a first category label, determining the first category label as a candidate audio category corresponding to the reserved label;

[0197] Otherwise, taking the reserved tag as the starting level, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

[0198] In yet another embodiment of the present disclosure, a marking module is further included, configured to:

[0199] At least one audio tag in the tag library is marked as a first-category tag.

[0200] In yet another embodiment of the present disclosure, the summary module is specifically configured to:

[0201] Determine a comprehensive score for each candidate audio category, wherein the comprehensive score is the sum of scores of all subordinate audio tags corresponding to the candidate audio category;

[0202] The candidate audio category with the highest comprehensive score is determined as the audio category of the audio to be processed.

[0203] In yet another embodiment of the present disclosure, when the summary module is configured to generate a summary statement of the audio to be processed based on the audio features and the audio category using a pre-trained audio summary generation model, the summary module is specifically configured to:

[0204] The encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector;

[0205] The decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes the initial word vector and the label word vector of the audio category.

[0206] In yet another embodiment of the present disclosure, when the summary module is configured to generate a summary sentence of the audio to be processed based on the encoding vector and the initial word sequence, the summary module is specifically configured to:

[0207] generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence;

[0208] The i+1th group of word vectors is generated according to the initial word sequence and the word sequence composed of the first i groups of word vectors, and the encoding vector until a terminal word vector is generated, where i≥1 and each group of word vectors includes at least one word vector.

[0209] In yet another embodiment of the present disclosure, when the summary module is configured to generate a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence, the summary module is specifically configured to:

[0210] Embedding the initial word sequence into the encoded vector to form a first vector, and generating a first set of word vectors for the summary sentence by decoding the first vector;

[0211] Generating the i+1th group of word vectors according to the initial word sequence, the first i groups of word vectors, and the encoding vector includes:

[0212] The word sequence consisting of the initial word sequence and the first i groups of word vectors is embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

[0213] In yet another embodiment of the present disclosure, the summary module is configured to generate the i+1th group of word vectors in the following manner:

[0214] Determine a target probability of at least one word vector in a preset vocabulary, where the target probability is a probability that the word vector belongs to the i+1th group of word vectors;

[0215] The (i+1)th group of word vectors is generated according to the target probability of the at least one word vector.

[0216] In yet another embodiment of the present disclosure, a training module is further included, for:

[0217] Extracting audio features of training audio in a training set, wherein the training audio is labeled with a category label and a summary label;

[0218] Inputting the audio features and category labels of the training audio into the audio summary generation model, and the audio summary generation model outputting a predicted label of the training audio;

[0219] A network loss value is determined according to the summary label and the predicted label, and network parameters of an encoding module and a decoding module of the audio summary generation model are adjusted according to the network loss value.

[0220] The present disclosure also provides a computer-readable storage medium. Figure 8 As shown, the storage medium stores a computer program 801, and when the computer program 801 is executed by a processor, the advertisement pushing method of any embodiment of the present disclosure can be executed.

[0221] An embodiment of the present disclosure further provides a computing device, which may include a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the advertising push method of any embodiment of the present disclosure when executing the computer instructions.

[0222] Figure 9 An example of a structure of the computing device is shown as follows: Figure 9 As shown, the computing device 90 may include, but is not limited to: a processor 91 , a memory 92 , and a bus 93 connecting different system components (including the memory 92 and the processor 91 ).

[0223] The memory 92 stores computer instructions that can be executed by the processor 91, enabling the processor 91 to perform the advertising push method according to any embodiment of the present disclosure. The memory 92 may include a random access memory unit (RAM) 921, a cache memory unit (922), and / or a read-only memory unit (ROM) 923. The memory 92 may also include a program tool 925 having a set of program modules 924, including but not limited to an operating system, one or more application programs, other program modules, and program data. One or more combinations of these program modules may include an implementation of a network environment.

[0224] The bus 93 may include, for example, a data bus, an address bus, and a control bus. The computing device 90 may also communicate with an external device 95 via an I / O interface 94. The external device 95 may be, for example, a keyboard, a Bluetooth device, etc. The computing device 90 may also communicate with one or more networks via a network adapter 96. For example, the network may be a local area network, a wide area network, a public network, etc. Figure 9 As shown, the network adapter 96 can also communicate with other modules of the computing device 90 via the bus 93 .

[0225] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0226] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.< / eos> < / sos> < / sos> < / sos> < / eos> < / sos> < / eos> < / sos>

Claims

1. An audio processing method, characterized in that: include: Extracting audio features of the audio to be processed; Determining the audio category of the audio to be processed using an audio tag prediction model based on the audio features; Generate a summary sentence of the audio to be processed using a pre-trained audio summary generation model according to the audio features and the audio category; Determining the audio category of the audio to be processed by using an audio tag prediction model according to the audio features includes: Inputting the audio feature into the audio tag prediction model, wherein the audio tag prediction model outputs a score of at least one audio tag in the tag library; selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score; Determine, from a label library, a candidate audio category corresponding to each of the reserved labels; Determining an audio category as the audio to be processed from a preset number of candidate audio categories; Determining an audio category from a preset number of candidate audio categories as the audio category to be processed: Determine a comprehensive score for each candidate audio category, wherein the comprehensive score is the sum of scores of all subordinate audio tags corresponding to the candidate audio category; The candidate audio category with the highest comprehensive score is determined as the audio category of the audio to be processed.

2. The audio processing method according to claim 1, wherein: The step of extracting audio features of the audio to be processed includes: Sampling the audio to be processed using a preset frequency to obtain a sampling result; Feature extraction is performed on the sampling result to obtain a feature sequence as the audio feature.

3. The audio processing method according to claim 1, wherein: Also includes: After outputting the score of at least one audio tag in the tag library, normalizing the score; The step of selecting a preset number of audio tags from the at least one audio tag as retained tags according to the score includes: A preset number of audio tags are selected from the at least one audio tag according to the normalized score as reserved tags.

4. The audio processing method according to claim 1, wherein: The step of selecting a preset number of audio tags from the at least one audio tag as retained tags according to the score includes: Among the at least one audio tag, retaining an audio tag having a score higher than a preset score threshold; In response to the number of the reserved audio tags being less than or equal to the preset number, taking all the reserved audio tags as reserved tags; In response to the number of the retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags.

5. The audio processing method according to claim 1, wherein: The tag library includes a plurality of audio tags, wherein the plurality of audio tags have a tree-like hierarchical relationship, each audio tag has an upper-level and / or lower-level audio tag, and at least one audio tag in the tag library is pre-marked as a first-category tag; Determining, from the tag library, a candidate audio category corresponding to each of the reserved tags includes: In response to the reserved label being a first category label, determining the first category label as a candidate audio category corresponding to the reserved label; Otherwise, taking the reserved tag as the starting level, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

6. The audio processing method according to claim 1 or 5, characterized in that: Also includes: At least one audio tag in the tag library is marked as a first-category tag.

7. The audio processing method according to claim 1, characterized in that: Generating a summary sentence of the audio to be processed using a pre-trained audio summary generation model according to the audio features and the audio category includes: The encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector; The decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes the initial word vector and the label word vector of the audio category.

8. The audio processing method according to claim 7, characterized in that: Generating a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence includes: generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence; The i+1th group of word vectors is generated according to the initial word sequence and the word sequence composed of the first i groups of word vectors, and the encoding vector until a terminal word vector is generated, where i≥1 and each group of word vectors includes at least one word vector.

9. The audio processing method according to claim 8, characterized in that: Generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence includes: Embedding the initial word sequence into the encoded vector to form a first vector, and generating a first set of word vectors for the summary sentence by decoding the first vector; Generating the i+1th group of word vectors according to the initial word sequence, the first i groups of word vectors, and the encoding vector includes: The word sequence consisting of the initial word sequence and the first i groups of word vectors is embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

10. The audio processing method according to claim 8 or 9, characterized in that: Generate the i+1th group of word vectors as follows: Determine a target probability of at least one word vector in a preset vocabulary, where the target probability is a probability that the word vector belongs to the i+1th group of word vectors; The (i+1)th group of word vectors is generated according to the target probability of the at least one word vector.

11. The audio processing method according to claim 7, wherein: Also includes: Extracting audio features of training audio in a training set, wherein the training audio is labeled with a category label and a summary label; Inputting the audio features and category labels of the training audio into the audio summary generation model, and the audio summary generation model outputting a predicted label of the training audio; A network loss value is determined according to the summary label and the predicted label, and network parameters of an encoding module and a decoding module of the audio summary generation model are adjusted according to the network loss value.

12. An audio processing device, characterized in that: include: A feature module, used to extract audio features of the audio to be processed; A classification module, configured to determine the audio category of the audio to be processed using an audio tag prediction model based on the audio features; A summary module, configured to generate a summary statement of the audio to be processed using a pre-trained audio summary generation model based on the audio features and the audio category; The category module is specifically used for: Inputting the audio feature into the audio tag prediction model, wherein the audio tag prediction model outputs a score of at least one audio tag in the tag library; selecting a preset number of audio tags from the at least one audio tag as retained tags based on the score; Determine, from a label library, a candidate audio category corresponding to each of the reserved labels; Determining an audio category as the audio to be processed from a preset number of candidate audio categories; When the category module is used to determine an audio category as the audio category to be processed from a preset number of candidate audio categories, it is specifically used to: Determine a comprehensive score for each candidate audio category, wherein the comprehensive score is the sum of scores of all subordinate audio tags corresponding to the candidate audio category; The candidate audio category with the highest comprehensive score is determined as the audio category of the audio to be processed.

13. The audio processing device according to claim 12, characterized in that The feature module is specifically used for: Sampling the audio to be processed using a preset frequency to obtain a sampling result; Feature extraction is performed on the sampling result to obtain a feature sequence as the audio feature.

14. The audio processing device according to claim 12, wherein: The category module is also used to: After outputting the score of at least one audio tag in the tag library, normalizing the score; The step of selecting a preset number of audio tags from the at least one audio tag as retained tags according to the score includes: A preset number of audio tags are selected from the at least one audio tag according to the normalized score as reserved tags.

15. The audio processing device according to claim 12, wherein: The category module is configured to select a preset number of audio tags from the at least one audio tag according to the score as reserved tags, specifically for: Among the at least one audio tag, retaining an audio tag having a score higher than a preset score threshold; In response to the number of the reserved audio tags being less than or equal to the preset number, taking all the reserved audio tags as reserved tags; In response to the number of the retained audio tags being greater than the preset number, a preset number of audio tags are selected from the retained audio tags in descending order of scores as retained tags.

16. The audio processing device according to claim 12, characterized in that The tag library includes a plurality of audio tags, wherein the plurality of audio tags have a tree-like hierarchical relationship, each audio tag has an upper-level and / or lower-level audio tag, and at least one audio tag in the tag library is pre-marked as a first-category tag; When the category module is used to determine the candidate audio category corresponding to each of the reserved tags from the tag library, it is specifically used to: In response to the reserved label being a first category label, determining the first category label as a candidate audio category corresponding to the reserved label; Otherwise, taking the reserved tag as the starting level, the upper-level audio tags are determined step by step in the tag library until the upper-level audio tag is a first-category tag, and the first-category tag is determined as the candidate audio category corresponding to the reserved tag.

17. The audio processing device according to claim 13 or 16, characterized in that: Also includes tagging modules for: At least one audio tag in the tag library is marked as a first-category tag.

18. The audio processing device according to claim 12, characterized in that The summary module is specifically used for: The encoding module of the audio summary generation model encodes the audio features to obtain an encoding vector; The decoding module of the audio summary generation model generates a summary sentence of the audio to be processed according to the encoding vector and the initial word sequence, wherein the initial word sequence includes the initial word vector and the label word vector of the audio category.

19. The audio processing device according to claim 18, characterized in that When the summary module is used to generate a summary statement of the audio to be processed based on the encoding vector and the initial word sequence, it is specifically used to: generating a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence; The i+1th group of word vectors is generated according to the initial word sequence and the word sequence composed of the first i groups of word vectors, and the encoding vector until a terminal word vector is generated, where i≥1 and each group of word vectors includes at least one word vector.

20. The audio processing device according to claim 19, characterized in that When the summary module is used to generate a first set of word vectors for the summary sentence based on the encoding vector and the initial word sequence, it is specifically used to: Embedding the initial word sequence into the encoded vector to form a first vector, and generating a first set of word vectors for the summary sentence by decoding the first vector; The summary module is used to generate the i+1th group of word vectors based on the initial word sequence, the word sequence composed of the first i groups of word vectors, and the encoding vector, specifically to: The word sequence consisting of the initial word sequence and the first i groups of word vectors is embedded in the encoding vector to form a second vector, and the i+1th group of word vectors is generated by decoding the second vector.

21. The audio processing device according to claim 19 or 20, characterized in that: The summary module is used to generate the i+1th group of word vectors in the following manner: Determine a target probability of at least one word vector in a preset vocabulary, where the target probability is a probability that the word vector belongs to the i+1th group of word vectors; The (i+1)th group of word vectors is generated according to the target probability of the at least one word vector.

22. The audio processing device according to claim 18, characterized in that Also includes training modules for: Extracting audio features of training audio in a training set, wherein the training audio is labeled with a category label and a summary label; Inputting the audio features and category labels of the training audio into the audio summary generation model, and the audio summary generation model outputting a predicted label of the training audio; A network loss value is determined according to the summary label and the predicted label, and network parameters of an encoding module and a decoding module of the audio summary generation model are adjusted according to the network loss value.

23. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

24. A computing device, characterized in that The computing device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method according to any one of claims 1 to 11 when executing the computer instructions.

Images