Audio classification method, device, equipment and medium based on text information

Abstract

The present application relates to the field of artificial intelligence technology, and proposes an audio classification method, device, electronic device, and storage medium based on text information. The method includes: converting the acquired audio data into audio text to obtain an audio word segmentation dataset; screening the word segmentation dataset to construct an audio keyword table; constructing a text sentence index dataset based on the audio text and the audio keyword table; training a custom text classification model based on the text sentence index dataset; obtaining text information features of the target audio based on the trained custom text classification model; screening the audio features and text information features based on an integrated learning model to obtain audio classification salient features; and classifying the target audio based on the audio classification salient features and the integrated learning model. The present application classifies audio data by integrating the audio features and text features of the audio data and incorporating keywords into the text, which can effectively improve the accuracy of audio classification.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and storage medium for audio classification based on text information. Background Art

[0002] The development of machine learning models such as deep learning and neural networks has made audio processing technology more accurate and reliable. Speech recognition technology is widely used in fields such as digital healthcare, fintech, and smart homes. When training or evaluating ASR models for smart devices, we need to collect large amounts of speech data and its ASR conversion results, and then manually annotate this data. Often, the data collection environment is less than ideal. Various background noises or the mis-triggered recording device when not recording may generate some meaningless data, such as noise or incorrectly collected conversations. Therefore, to reduce the cost of subsequent manual annotation, it is necessary to remove as much meaningless data as possible and retain the meaningful data.

[0003] Eliminating meaningless data can be considered a binary classification problem, classifying data into meaningful and meaningless categories. Existing audio classification methods typically first extract basic features from the audio and then feed these features into a classification model. The drawback of these methods is that they only utilize the audio features, not the content. While some meaningless audio may have features similar to meaningful audio, its content is meaningless, resulting in inaccurate and unreliable audio classification results. Summary of the Invention

[0004] In view of the above, it is necessary to propose an audio classification method, device, electronic device and storage medium based on text information to solve the technical problem of how to improve the accuracy of audio classification.

[0005] The present application provides a method for audio classification based on text information, the method comprising:

[0006] Convert the acquired audio data into audio text to obtain an audio word segmentation dataset;

[0007] Filtering the word segmentation dataset to construct an audio keyword table;

[0008] Constructing a text sentence index data set based on the audio text and the audio keyword table;

[0009] Training a custom text classification model based on the text sentence index dataset;

[0010] Obtain target audio, and obtain text information features of the target audio based on the trained custom text classification model;

[0011] Extracting audio features of the target audio, and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features;

[0012] The target audio is classified based on the audio classification salient features and the integrated learning model.

[0013] In some embodiments, screening the word segmentation dataset to construct an audio keyword table includes:

[0014] Counting the word frequency of each word in the word segmentation data set;

[0015] Filtering the word segmentation dataset based on the word frequency to obtain an initial word segmentation dataset;

[0016] The stop words in the initial word segmentation data set are filtered out according to a preset stop word list to obtain an audio keyword list.

[0017] In some embodiments, constructing a text sentence index dataset based on the audio text and the audio keyword table includes:

[0018] Adding a custom sentence start symbol and sentence end symbol to the beginning and end of a target sentence in the audio text to obtain a text target sentence, where the target sentence is any sentence in the audio text;

[0019] Matching the text target sentence with the audio keyword table to obtain target sentence keywords, and sequentially splicing all target sentence keywords to the text target sentence using a custom splicing operator to obtain a key target sentence;

[0020] Comparing the length of the key target sentence with a preset text sentence length threshold;

[0021] If the length of the key target sentence is not less than the text sentence length threshold, truncating the key target sentence so that the length of the truncated key target sentence is equal to the text sentence length threshold;

[0022] If the length of the key target sentence is less than the text sentence length threshold, then the splicing symbol is added to the end of the key target sentence to fill it up so that the length of the padded key target sentence is equal to the text sentence length threshold;

[0023] Segmenting a key target sentence whose length is equal to the text sentence length threshold to obtain a target segmentation table;

[0024] Obtaining a segmentation index of each segmentation word in the key target sentence based on the target segmentation table to generate a text sentence index array;

[0025] The audio text is traversed to obtain a text sentence index array corresponding to each sentence in the audio text, and all the text sentence index arrays are used as a text sentence index data set.

[0026] In some embodiments, training a custom text classification model based on the text sentence index dataset includes:

[0027] Build a custom text classification model based on the preset language model, bidirectional GRU neural network, attention mechanism and fully connected neural network;

[0028] Constructing a text classification training set and a text classification label set based on the text sentence index dataset;

[0029] Inputting the text classification training set into the language model to obtain a high-dimensional word vector set;

[0030] Input the high-dimensional word vector set into the bidirectional GRU network to obtain a low-dimensional word vector set and a text sentence vector set;

[0031] Performing an attention mechanism on the low-dimensional word vector set and the text sentence vector set to obtain a feature sentence vector set, where the feature sentence vectors in the feature sentence vector set are used to represent the importance of the corresponding text sentence in the audio text;

[0032] Inputting the feature sentence vector set into the fully connected neural network for classification to obtain a text classification set;

[0033] The classification loss between the text classification set and the text classification label set is calculated based on a cross entropy loss function to optimize the training of the custom text classification model.

[0034] In some embodiments, acquiring the target audio and acquiring text information features of the target audio based on the trained custom text classification model includes:

[0035] Classify the audio text of the target audio using the custom text classification model to obtain a meaningful text set and a meaningless text set;

[0036] The meaningful text set and the meaningless text set are used as text information features of the target audio.

[0037] In some embodiments, extracting audio features of the target audio and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features includes:

[0038] Extracting multiple audio features of the target audio, and using a meaningful text set and a meaningless text set in the text information features as audio features of the target audio;

[0039] Building a classification and regression tree based on the audio features, wherein the classification and regression tree corresponds to the audio features one by one;

[0040] Fit all classification and regression trees based on the ensemble learning model to obtain a significant classification and regression tree;

[0041] The audio features corresponding to each significant classification regression tree are used as the significant features of audio classification.

[0042] In some embodiments, classifying the target audio based on the audio classification salient features and the ensemble learning model includes:

[0043] Constructing a binary regression tree based on the audio classification salient features, wherein the binary regression tree corresponds to the audio classification salient features in a one-to-one manner;

[0044] Classify the target audio based on the binary regression tree to obtain a single-tree classification result;

[0045] The classification result of the target audio is obtained by superimposing the single-tree classification results of all binary regression trees after fitting based on the ensemble learning model.

[0046] The present application also provides an audio classification device based on text information, the device comprising a conversion module, a screening module, a construction module, a training module, an acquisition module, an extraction module, and a classification module:

[0047] The conversion module is used to convert the acquired audio data into audio text to obtain an audio segmentation dataset;

[0048] The screening module is used to screen the word segmentation data set to construct an audio keyword table;

[0049] The construction module is used to construct a text sentence index data set based on the audio text and the audio keyword table;

[0050] The training module is used to train a custom text classification model based on the text sentence index dataset;

[0051] The acquisition module is used to acquire target audio and obtain text information features of the target audio based on the trained custom text classification model;

[0052] The extraction module is used to extract audio features of the target audio, and screen the audio features and the text information features according to the ensemble learning model to obtain audio classification salient features;

[0053] The classification module is used to classify the target audio based on the audio classification salient features and the integrated learning model.

[0054] An embodiment of the present application further provides an electronic device, comprising:

[0055] a memory storing at least one instruction;

[0056] The processor executes the instructions stored in the memory to implement the audio classification method based on text information.

[0057] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the text information-based audio classification method.

[0058] This application converts the acquired audio data into audio text and incorporates keywords into the audio text to train a custom text classification model that can obtain text information features. The audio data is then classified by combining the acquired text information features and the extracted audio features, which can effectively improve the accuracy of audio classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flowchart of a preferred embodiment of the audio classification method based on text information involved in this application.

[0060] Figure 2 This is a functional module diagram of a preferred embodiment of the text information-based audio classification device involved in this application.

[0061] Figure 3 It is a structural diagram of an electronic device of a preferred embodiment of the audio classification method based on text information involved in this application. DETAILED DESCRIPTION

[0062] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application. The embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the described features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0065] An embodiment of the present application provides an audio classification method based on text information, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0066] An electronic device can be any electronic product that can interact with a user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, Internet Protocol Television (IPTV), smart wearable device, etc.

[0067] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0068] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0069] like Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the audio classification method based on text information of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0070] The text-based audio classification method provided in the embodiment of the present application can be applied to intelligent diagnosis and treatment, remote consultation, etc. in the field of digital medicine, and can also be applied to intelligent customer service in the field of financial technology, etc.

[0071] S10, converting the acquired audio data into audio text to obtain an audio word segmentation dataset.

[0072] In an optional embodiment, converting the acquired audio data into audio text to obtain an audio word segmentation dataset includes:

[0073] Collect voice and store it to obtain audio data;

[0074] Convert the acquired audio data into audio text based on the automatic speech recognition model;

[0075] The audio text is segmented according to a segmentation tool to obtain an audio segmentation dataset.

[0076] In this optional embodiment, a large amount of voice data can be collected by a recording device, and all the collected voice data can be stored in a preset voice database. In this solution, all the voice data stored in the voice database can be regarded as audio data.

[0077] In this optional embodiment, an automatic speech recognition model such as Kaldi can be used to convert the audio data into audio text, and the acquired audio text can be segmented using a word segmentation tool such as Jieba. In this solution, all the segmentations generated after the audio text is segmented are used as an audio segmentation dataset.

[0078] For example, assuming that the audio text is "check the weather in Shanghai", after word segmentation by the word segmentation tool, we get ["check", "Shanghai", "weather"].

[0079] In this way, the collected audio data can be quickly converted into text data through the automatic speech recognition model, and the text data can be quickly converted into word segments through the word segmentation tool, thereby providing data support for subsequent processes.

[0080] S11, screening the word segmentation dataset to construct an audio keyword table.

[0081] In an optional embodiment, screening the word segmentation dataset to construct an audio keyword table includes:

[0082] Counting the word frequency of each word in the word segmentation data set;

[0083] Filtering the word segmentation dataset based on the word frequency to obtain an initial word segmentation dataset;

[0084] The stop words in the initial word segmentation data set are filtered out according to a preset stop word list to obtain an audio keyword list.

[0085] In this optional embodiment, the frequency of each word segment in the word segment dataset can be counted, so as to screen the word segments in the word segment dataset, and retain a part of the word segments with the highest frequency as the initial word segment dataset. Exemplarily, 300 word segments with the highest word frequencies in the word segment dataset can be retained as the initial word segment dataset.

[0086] In this optional embodiment, the matching stop words in the initial word segment dataset can be screened out and removed by using a preset Chinese stop word list, such as the Harbin Institute of Technology Chinese stop word list. Exemplarily, the stop words are meaningless words in the text such as "you", "me", "generally speaking", "le", etc. In this solution, the initial word segment dataset without stop words is used as the audio keyword table.

[0087] In this way, important word segments in the word segment dataset can be screened out by word frequency, and meaningless stop words can be removed according to the stop word list, so as to retain more important word segments.

[0088] S12. Construct a text sentence index dataset based on the audio text and the audio keyword table.

[0089] In an optional embodiment, the constructing a text sentence index dataset based on the audio text and the audio keyword table includes:

[0090] Add a custom sentence start symbol and a sentence end symbol to the start and end of the target sentence in the audio text respectively to obtain the text target sentence, where the target sentence is any sentence in the audio text;

[0091] Match the text target sentence and the audio keyword table to obtain the target sentence keywords, and splice all the target sentence keywords to the back of the text target sentence in sequence through a custom splicing symbol to obtain the key target sentence;

[0092] Compare the length of the key target sentence with a preset text sentence length threshold;

[0093] If the length of the key target sentence is not less than the text sentence length threshold, truncate the key target sentence so that the length of the truncated key target sentence is equal to the text sentence length threshold;

[0094] If the length of the key target sentence is less than the text sentence length threshold, add the splicing symbol at the end of the key target sentence to make up the length so that the length of the key target sentence after being made up is equal to the text sentence length threshold;

[0095] Perform word segmentation on the key target sentence with a length equal to the text sentence length threshold to obtain a target word segmentation table;

[0096] Obtaining a segmentation index of each segmentation word in the key target sentence based on the target segmentation table to generate a text sentence index array;

[0097] The audio text is traversed to obtain a text sentence index array corresponding to each sentence in the audio text, and all the text sentence index arrays are used as a text sentence index data set.

[0098] In this optional embodiment, since the audio text includes a large number of text sentences, the audio text can be divided according to text sentences, and a custom sentence start symbol [CLS] is added to the beginning of the target sentence in the audio text, and a custom sentence end symbol [SEP] is added to the end of the sentence, where [CLS] indicates the beginning of the sentence and [SEP] indicates the end of the sentence. The target sentence is any sentence in the audio text.

[0099] In this optional embodiment, keywords in the text target sentence can be matched according to the audio keyword table, and the matched keywords are sequentially spliced to the end of the text target sentence using a custom splicing symbol [SEP] to form a key target sentence. For example, the text target sentence is "I want to see the weather in Shanghai today", where the keywords that match the audio keyword table are "today" and "weather", so the spliced key target sentence is [CLS]I want to see the weather in Shanghai today[SEP]today[SEP]weather[SEP].

[0100] In this optional embodiment, the length of the key target sentence can be adjusted to a fixed length using a preset text sentence length threshold. Specifically, the length of the key target sentence is compared with the preset text sentence length threshold. If the length of the key target sentence is not less than the text sentence length threshold, the key target sentence is truncated so that the length of the truncated key target sentence is equal to the text sentence length threshold. If the length of the key target sentence is less than the text sentence length threshold, the splicing symbol [PAD] is added to the end of the key target sentence to pad it so that the length of the padded key target sentence is equal to the text sentence length threshold. The preset text sentence length threshold in this solution can be 50.

[0101] In this optional embodiment, the key target sentence whose length is equal to the text sentence length threshold can be segmented using the vocabulary provided by HuggingFace Albert_tiny, and all the segmentations of the key target sentence can be used as the target segmentation table. At the same time, a text sentence index array can be generated based on the index of each segmentation in the target segmentation table.

[0102] For example, let the target sentence be "Who am I", and the corresponding keyword target sentence be "[CLS]Who am I[SEP][PAD][PAD]...", where the indexes of "[CLS]", "I", "Is", "Who", "[SEP]", and "[PAD]" in the target word segmentation table are 102, 2771, 3223, 6445, 103, and 1 respectively, and the corresponding text sentence index array is [102, 2771, 3223, 6445, 103, 1, 1, ...].

[0103] In this optional embodiment, the text sentence index array corresponding to each sentence in the audio text can be obtained by traversing the audio text. In this solution, all text sentence index arrays are used as a text sentence index data set.

[0104] In this way, the audio text and the keyword information in the audio text can be combined to generate a text sentence index array with a uniform fixed length, which is convenient for training the neural network in the subsequent process.

[0105] S13: training a custom text classification model based on the text sentence index dataset.

[0106] In an optional embodiment, the training of a custom text classification model based on the text sentence index dataset includes:

[0107] Build a custom text classification model based on the preset language model, bidirectional GRU neural network, attention mechanism and fully connected neural network;

[0108] Constructing a text classification training set and a text classification label set based on the text sentence index dataset;

[0109] Inputting the text classification training set into the language model to obtain a high-dimensional word vector set;

[0110] Input the high-dimensional word vector set into the bidirectional GRU network to obtain a low-dimensional word vector set and a text sentence vector set;

[0111] Performing an attention mechanism on the low-dimensional word vector set and the text sentence vector set to obtain a feature sentence vector set, where the feature sentence vectors in the feature sentence vector set are used to represent the importance of the corresponding text sentence in the audio text;

[0112] Inputting the feature sentence vector set into the fully connected neural network for classification to obtain a text classification set;

[0113] The classification loss between the text classification set and the text classification label set is calculated based on a cross entropy loss function to optimize the training of the custom text classification model.

[0114] In this optional embodiment, the custom text classification model is composed of a preset language model, a bidirectional GRU neural network, an attention mechanism and a fully connected neural network, wherein the language model can be an Albert-tiny model.

[0115] In this optional embodiment, since the language model Albert-tiny has three inputs in total, the text sentence index arrays in the text sentence index dataset can be expanded to three inputs: input_ids, token_type_ids, and attention_masks, where input_ids is the text sentence index array; token_type_ids is an array with the same length as input_ids but all zeros, and attention masks is 0 only when the word in the text sentence is [PAD], and 1 when the word is not [PAD]. For example, the attention masks corresponding to the text sentence [CLS] Who am I [SEP][PAD][PAD] are [1,1,1,1,1,0,0].

[0116] In this optional embodiment, each text sentence index array in the text sentence index data set is expanded and used as a text classification training set. At the same time, a label can be set for each text sentence index array in the text sentence index data set manually, wherein if the text sentence corresponding to the text sentence index array is meaningful, the label of the text sentence index array can be set to 1; if the text sentence corresponding to the text sentence index array is meaningless, the label of the text sentence index array can be set to 0. In this solution, the text sentence index data set with labels is used as a text classification label set.

[0117] In this optional embodiment, the expanded index arrays of each text sentence can first be input into the pre-trained language model Albert-tiny, so as to generate a 312-dimensional word vector for each word in the text sentence, and then the obtained 312-dimensional word vectors are input into the bidirectional GRU, and the bidirectional GRU will convert the input 312-dimensional word vector into a 156-dimensional word vector, and then all 156-dimensional word vectors are calculated to obtain a text sentence vector. In this solution, all 312-dimensional word vectors are used as high-dimensional word vector sets, all 156-dimensional word vectors are used as low-dimensional word vector sets, and all text sentence vectors are used as text sentence vector sets.

[0118] In this optional embodiment, the low-dimensional word vector and the corresponding text sentence vector corresponding to each text sentence can be simultaneously sent to the neural network where the attention mechanism part is located. The attention mechanism is used to calculate the weight of each word in the text sentence in the entire sentence, and can perform weighted summation on all words in the text sentence to obtain the feature sentence vector corresponding to the text sentence. The feature sentence vector is used to represent the importance of the corresponding text sentence in the entire audio text. In this solution, all feature sentence vectors are used as a feature sentence vector set.

[0119] In this optional embodiment, the feature sentence vectors in the feature sentence vector set obtained are further fed into the fully connected neural network, and the fully connected neural network is used to classify the text sentences in the audio text into two categories: meaningful and meaningless according to the input feature sentence vectors. In this solution, the text sentences obtained after classification are used as a text classification set.

[0120] In this optional embodiment, the classification loss between the classification category corresponding to the text sentence in the text classification set and the label category of the text sentence in the text classification label set can be calculated using a cross-entropy loss function, and the parameters in the custom text classification model are optimized and adjusted based on the obtained classification loss, so that the classification loss continues to decrease, and finally the training of the custom text classification model is completed when the classification loss is 0.

[0121] In this way, a feature sentence vector that can reflect the importance of the text sentence in the entire audio text can be obtained through the text sentence index dataset, so as to train the custom text classification model more accurately and effectively.

[0122] S14, obtaining target audio, and obtaining text information features of the target audio based on the trained custom text classification model.

[0123] In an optional embodiment, the step of obtaining target audio and obtaining text information features of the target audio based on the trained custom text classification model includes:

[0124] Classify the audio text of the target audio using the custom text classification model to obtain a meaningful text set and a meaningless text set;

[0125] The meaningful text set and the meaningless text set are used as text information features of the target audio.

[0126] In this optional embodiment, the acquired audio text of the target audio is classified according to the trained custom text classification model, and the text sentences in the audio text of the target audio can be divided into two categories: meaningful and meaningless. In this solution, all text sentences belonging to the meaningful category are taken as a meaningful text set, and all text sentences belonging to the meaningless category are taken as a meaningless text set. Finally, the meaningful text set and the meaningless text set are taken as text information features of the target audio.

[0127] In this way, the text information features of the target audio can be quickly obtained through the trained custom text classification model.

[0128] S15, extracting audio features of the target audio, and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features.

[0129] In an optional embodiment, extracting audio features of the target audio and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features includes:

[0130] Extracting multiple audio features of the target audio, and using a meaningful text set and a meaningless text set in the text information features as audio features of the target audio;

[0131] Building a classification and regression tree based on the audio features, wherein the classification and regression tree corresponds to the audio features one by one;

[0132] Fit all classification and regression trees based on the ensemble learning model to obtain a significant classification and regression tree;

[0133] The audio features corresponding to each significant classification regression tree are used as the significant features of audio classification.

[0134] In this optional embodiment, multiple audio features of the target audio can be extracted by performing pre-emphasis, framing, windowing, Fourier transform, filtering, discrete cosine transform and other operations on the target audio. The audio features can be common audio features such as zero-crossing rate, loudness, energy, energy entropy, spectral frequency center, spectral extension, spectral entropy, spectral flux, spectral roll-off point, Mel-cepstral coefficients, etc. In this solution, 34 audio features of the target audio can be selected for subsequent calculations.

[0135] In this optional embodiment, the meaningful text set and the meaningless text set in the text information features can also be used as audio features of the target audio, so that a total of 36 audio features of the target audio are finally obtained in this solution.

[0136] In this optional embodiment, a corresponding classification and regression tree can be constructed according to each audio feature, and all classification and regression trees can be fitted by an ensemble learning model to obtain a significant classification and regression tree. The ensemble learning model can be an Xgboost ensemble learning model. The specific fitting process is:

[0137] Xgboost uses the classification and regression tree (CART tree) as the base classifier and treats each audio feature as a classification and regression tree. Since all classification and regression trees need to be superimposed in the end, only one initial classification and regression tree is established in the actual fitting process, and other classification and regression trees are split as tree nodes of the initial classification and regression tree. Finally, the overall classification result of all classification and regression trees can be obtained by counting the classification results of each tree node.

[0138] In this optional embodiment, the initial tree node of the initial classification and regression tree divides the input audio sample into a left subtree and a right subtree. The principle of division is to find the optimal division point of the optimal feature so that the preset objective function is minimized. The tree nodes of the left subtree and the right subtree will continue to split to further divide the audio samples in the left subtree and the right subtree. Xgboost continuously generates new CART trees. Each new CART tree is learning a new objective function. The goal of the function is to fit the historical residuals of all leaf nodes. Finally, it stops when the loss function of all historical residuals is less than a certain threshold. In this scheme, 34 CART trees will be obtained. When classifying the input audio features, each tree will generate a score. At the same time, each CART tree will obtain a corresponding weight according to the number of splits of the tree node during the splitting process, and the final classification result is obtained by weighted summation of the scores generated by each CART tree and the corresponding weights. That is, the final Xgboost integrated learning model is the superposition of all generated CART trees, and the classification result is a binary classification form of 0 and 1.

[0139] For example, assuming that the audio has only three audio features: zero-crossing rate, loudness, and energy, the audio can be judged and scored based on the level of each audio feature, where a high zero-crossing rate of the audio is scored 90 points, a low zero-crossing rate is scored 60 points, a high loudness of the audio is scored 5 points, a low loudness of the audio is scored -5 points, a high energy of the audio is scored 1 point, and a low energy of the audio is scored -1 point.

[0140] There are three audios A, B, and C. Suppose the three audio features corresponding to each audio are (high, high, high), (low, high, high), and (high, low, low). After scoring the three audio features, the corresponding true scores are 100, 70, and 86 respectively. Each time the Xgboost ensemble learning model is fitted, a new classification and regression tree is constructed. The score values of the three audios A, B, and C after the zero-crossing rate audio feature are 90, 60, and 90. When constructing the second classification and regression tree, the residuals (100-90=10), (70-60=10), and (86-90=-4) between the true score and the score given by the first classification and regression tree are considered to construct a new classification and regression tree. Then, by minimizing the residual, a classification and regression tree constructed by the loudness audio feature can be fitted to obtain the predicted values of (90+5), (60+5), and (90-5). The next classification and regression tree is then constructed by the residuals of (100-95=5), (70-65=5), and (86-85=1). Similarly, when the loss function of all historical residuals is less than the threshold, it stops, and an Xgboost ensemble learning model that can simultaneously integrate the three audio features of zero-crossing rate, loudness, and energy to classify audio can be obtained.

[0141] In this optional embodiment, the importance of each audio feature is calculated based on the number of times it serves as a splitting point during the fitting process. The more times it serves as a splitting point, the more important the audio feature is. Therefore, based on the importance of the audio feature, the first 24 classification and regression trees with the most splitting points are ultimately selected as significant classification and regression trees, and the audio features corresponding to each significant classification and regression tree are used as significant features for audio classification. That is, a total of 24 audio features are ultimately selected from the 36 audio features of the target audio as significant features for audio classification.

[0142] In this way, more important audio classification features can be obtained by screening all audio features of the target audio, which is conducive to improving the accuracy of classifying the target audio based on them in the subsequent process.

[0143] S16: Classify the target audio based on the audio classification salient features and the integrated learning model.

[0144] In an optional embodiment, classifying the target audio based on the audio classification salient features and the ensemble learning model includes:

[0145] Constructing a binary regression tree based on the audio classification salient features, wherein the binary regression tree corresponds to the audio classification salient features in a one-to-one manner;

[0146] Classify the target audio based on the binary regression tree to obtain a single-tree classification result;

[0147] The classification result of the target audio is obtained by superimposing the single-tree classification results of all binary regression trees after fitting based on the ensemble learning model.

[0148] In this optional embodiment, the audio classification significant features of the target audio can be obtained through the steps described in S10 to S16 above, and a corresponding binary classification regression tree can also be constructed for each audio classification significant feature. The binary classification regression tree is used to perform binary classification on the target audio according to the corresponding audio classification significant features to obtain a single tree classification result. The single tree classification result is 1 and 0, where 1 indicates that the corresponding target audio is meaningful, and 0 indicates that the corresponding target audio is meaningless.

[0149] In this optional embodiment, the binary regression tree can still be used as the base classifier through Xgboost, and then the historical residuals of all leaf nodes are fitted, and finally the process stops when the loss function of all historical residuals is less than a certain threshold. At this time, the single-tree classification results of all the binary regression trees after fitting are superimposed and voted according to the integrated learning model, so that the single-tree classification result with the most votes is used as the final classification result of the target audio.

[0150] For example, the 24 significant features of audio classification correspond to a total of 24 binary classification regression trees. After fitting the integrated learning model, the single-tree classification results of 10 binary classification regression trees are 1, and the single-tree classification results of the remaining 14 binary classification regression trees are 0. After comprehensive superposition voting on all the single-tree classification results, it is found that the single-tree classification results of 0 correspond to more votes, so the final classification result of the target audio is meaningless.

[0151] In this optional embodiment, if the final classification result of the target audio is meaningless, the target audio is discarded to reduce useless redundant audio. If the final classification result of the target audio is meaningful, the target audio is retained.

[0152] In this way, by obtaining the audio classification features of the target audio, the target audio can be effectively classified into two categories, so that meaningless target audio can be discarded in a timely manner according to the classification results.

[0153] The above method can be applied to intelligent diagnosis and treatment in the field of digital medicine. It can effectively classify the audio during the diagnosis and treatment process, discard meaningless target audio in a timely manner, and help improve the diagnosis and treatment effect.

[0154] See Figure 2 , Figure 2This is a functional block diagram of a preferred embodiment of the text-based audio classification device of the present application. The text-based audio classification device 11 includes a conversion module 110, a screening module 111, a construction module 112, a training module 113, an acquisition module 114, an extraction module 115, and a classification module 116. The units / modules referred to herein are a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each unit / module will be described in detail in subsequent embodiments.

[0155] In an optional embodiment, the conversion module 110 is configured to convert the acquired audio data into audio text to obtain an audio segmentation dataset.

[0156] In an optional embodiment, the screening module 111 is configured to screen the word segmentation dataset to construct an audio keyword table.

[0157] In an optional embodiment, screening the word segmentation dataset to construct an audio keyword table includes:

[0158] Counting the word frequency of each word in the word segmentation data set;

[0159] Filtering the word segmentation dataset based on the word frequency to obtain an initial word segmentation dataset;

[0160] The stop words in the initial word segmentation data set are filtered out according to a preset stop word list to obtain an audio keyword list.

[0161] In an optional embodiment, the construction module 112 is configured to construct a text sentence index dataset based on the audio text and the audio keyword table.

[0162] In an optional embodiment, constructing a text sentence index dataset based on the audio text and the audio keyword table includes:

[0163] Adding a custom sentence start symbol and sentence end symbol to the beginning and end of a target sentence in the audio text to obtain a text target sentence, where the target sentence is any sentence in the audio text;

[0164] Matching the text target sentence with the audio keyword table to obtain target sentence keywords, and sequentially splicing all target sentence keywords to the text target sentence using a custom splicing operator to obtain a key target sentence;

[0165] Comparing the length of the key target sentence with a preset text sentence length threshold;

[0166] If the length of the key target sentence is not less than the text sentence length threshold, truncating the key target sentence so that the length of the truncated key target sentence is equal to the text sentence length threshold;

[0167] If the length of the key target sentence is less than the text sentence length threshold, then the splicing symbol is added to the end of the key target sentence to fill it up so that the length of the padded key target sentence is equal to the text sentence length threshold;

[0168] Segmenting a key target sentence whose length is equal to the text sentence length threshold to obtain a target segmentation table;

[0169] Obtaining a segmentation index of each segmentation word in the key target sentence based on the target segmentation table to generate a text sentence index array;

[0170] The audio text is traversed to obtain a text sentence index array corresponding to each sentence in the audio text, and all the text sentence index arrays are used as a text sentence index data set.

[0171] In an optional embodiment, the training module 113 is configured to train a custom text classification model based on the text sentence index dataset.

[0172] In an optional embodiment, the training of a custom text classification model based on the text sentence index dataset includes:

[0173] Build a custom text classification model based on the preset language model, bidirectional GRU neural network, attention mechanism and fully connected neural network;

[0174] Constructing a text classification training set and a text classification label set based on the text sentence index dataset;

[0175] Inputting the text classification training set into the language model to obtain a high-dimensional word vector set;

[0176] Input the high-dimensional word vector set into the bidirectional GRU network to obtain a low-dimensional word vector set and a text sentence vector set;

[0177] Performing an attention mechanism on the low-dimensional word vector set and the text sentence vector set to obtain a feature sentence vector set, where the feature sentence vectors in the feature sentence vector set are used to represent the importance of the corresponding text sentence in the audio text;

[0178] Inputting the feature sentence vector set into the fully connected neural network for classification to obtain a text classification set;

[0179] The classification loss between the text classification set and the text classification label set is calculated based on a cross entropy loss function to optimize the training of the custom text classification model.

[0180] In an optional embodiment, the acquisition module 114 is configured to acquire target audio and acquire text information features of the target audio based on the trained custom text classification model.

[0181] In an optional embodiment, the step of obtaining target audio and obtaining text information features of the target audio based on the trained custom text classification model includes:

[0182] Classify the audio text of the target audio using the custom text classification model to obtain a meaningful text set and a meaningless text set;

[0183] The meaningful text set and the meaningless text set are used as text information features of the target audio.

[0184] In an optional embodiment, the extraction module 115 is configured to extract audio features of the target audio, and screen the audio features and the text information features according to an integrated learning model to obtain significant features for audio classification.

[0185] In an optional embodiment, extracting audio features of the target audio and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features includes:

[0186] Extracting multiple audio features of the target audio, and using a meaningful text set and a meaningless text set in the text information features as audio features of the target audio;

[0187] Building a classification and regression tree based on the audio features, wherein the classification and regression tree corresponds to the audio features one by one;

[0188] Fit all classification and regression trees based on the ensemble learning model to obtain a significant classification and regression tree;

[0189] The audio features corresponding to each significant classification regression tree are used as the significant features of audio classification.

[0190] In an optional embodiment, the classification module 116 is configured to classify the target audio based on the audio classification salient features and the integrated learning model.

[0191] In an optional embodiment, classifying the target audio based on the audio classification salient features and the ensemble learning model includes:

[0192] Constructing a binary regression tree based on the audio classification salient features, wherein the binary regression tree corresponds to the audio classification salient features in a one-to-one manner;

[0193] Classify the target audio based on the binary regression tree to obtain a single-tree classification result;

[0194] The classification result of the target audio is obtained by superimposing the single-tree classification results of all binary regression trees after fitting based on the ensemble learning model.

[0195] It can be seen from the above technical solution that the present application can convert the acquired audio data into audio text and integrate keywords into the audio text to train a custom text classification model that can obtain text information features, and then classify the audio data by combining the acquired text information features and the extracted audio features, which can effectively improve the accuracy of audio classification.

[0196] See Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Electronic device 1 includes memory 12 and processor 13. Memory 12 is used to store computer-readable instructions, and processor 13 is used to execute the computer-readable instructions stored in the memory to implement the text-based audio classification method described in any of the above embodiments.

[0197] In an optional embodiment, the electronic device 1 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13 , such as an audio classification program based on text information.

[0198] Figure 3 Only the electronic device 1 having the memory 12 and the processor 13 is shown. It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and the electronic device 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0199] Combine Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement an audio classification method based on text information, and the processor 13 can execute the plurality of instructions to implement:

[0200] Convert the acquired audio data into audio text to obtain an audio word segmentation dataset;

[0201] Filtering the word segmentation dataset to construct an audio keyword table;

[0202] Constructing a text sentence index data set based on the audio text and the audio keyword table;

[0203] Training a custom text classification model based on the text sentence index dataset;

[0204] Obtain target audio, and obtain text information features of the target audio based on the trained custom text classification model;

[0205] Extracting audio features of the target audio, and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features;

[0206] The target audio is classified based on the audio classification salient features and the integrated learning model.

[0207] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0208] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may have a bus structure or a star structure. The electronic device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the electronic device 1 may also include input and output devices, network access devices, etc.

[0209] It should be noted that the electronic device 1 is only an example, and other existing or future electronic products that are suitable for this application should also be included in the scope of protection of this application and incorporated herein by reference.

[0210] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. The memory 12 can not only be used to store application software and various types of data installed on the electronic device 1, such as the code of an audio classification program based on text information, but can also be used to temporarily store data that has been output or is to be output.

[0211] In some embodiments, the processor 13 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, connecting the various components of the entire electronic device 1 using various interfaces and circuits. It executes or runs programs or modules stored in the memory 12 (e.g., executing a text-based audio classification program) and accesses data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0212] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the audio classification method based on text information, for example Figure 1 Steps shown.

[0213] Exemplarily, the computer program may be divided into one or more units / modules, which are stored in the memory 12 and executed by the processor 13 to implement the present application. The one or more units / modules may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a conversion module 110, a screening module 111, a construction module 112, a training module 113, an extraction module, and a classification module.

[0214] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to execute the portion of the text-based audio classification method described in various embodiments of the present application.

[0215] If the integrated unit / module of the electronic device 1 is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments.

[0216] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory, or other memory.

[0217] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0218] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0219] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The diagram is represented by only one arrow, but it does not mean that there is only one bus or one type of bus. The bus is configured to implement connection and communication between the memory 12 and at least one processor 13, etc.

[0220] An embodiment of the present application further provides a computer-readable storage medium (not shown), in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in an electronic device to implement the text-based audio classification method described in any of the above embodiments.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0222] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0223] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0224] Furthermore, it is clear that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. Multiple modules or devices described in the specification may also be implemented by a single module or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An audio classification method based on text information, characterized in that: The method comprises: Convert the acquired audio data into audio text to obtain an audio word segmentation dataset; Filtering the word segmentation dataset to construct an audio keyword table; Constructing a text sentence index dataset based on the audio text and the audio keyword table, including: adding a custom sentence start symbol and sentence end symbol to the beginning and end of a target sentence in the audio text respectively to obtain a text target sentence, wherein the target sentence is any sentence in the audio text; matching the text target sentence with the audio keyword table to obtain target sentence keywords, and sequentially splicing all target sentence keywords to the text target sentence through a custom splicing symbol to obtain a key target sentence; obtaining a segmentation index of each segmentation word in the key target sentence based on a target segmentation table to generate a text sentence index array; traversing the audio text to obtain a text sentence index array corresponding to each sentence in the audio text, and using all text sentence index arrays as a text sentence index dataset; Training a custom text classification model based on the text sentence index dataset; Obtain target audio, and obtain text information features of the target audio based on the trained custom text classification model; Extracting audio features of the target audio, and screening the audio features and the text information features according to an integrated learning model to obtain audio classification salient features; The target audio is classified based on the audio classification salient features and the integrated learning model.

2. The audio classification method based on text information according to claim 1, characterized in that The filtering of the word segmentation data set to construct an audio keyword table includes: Counting the word frequency of each word in the word segmentation data set; Filtering the word segmentation dataset based on the word frequency to obtain an initial word segmentation dataset; The stop words in the initial word segmentation data set are filtered out according to a preset stop word list to obtain an audio keyword list.

3. The audio classification method based on text information according to claim 1, characterized in that The constructing of a text sentence index data set based on the audio text and the audio keyword table includes: Comparing the length of the key target sentence with a preset text sentence length threshold; If the length of the key target sentence is not less than the text sentence length threshold, truncating the key target sentence so that the length of the truncated key target sentence is equal to the text sentence length threshold; If the length of the key target sentence is less than the text sentence length threshold, then the splicing symbol is added to the end of the key target sentence to fill it up so that the length of the padded key target sentence is equal to the text sentence length threshold; The key target sentences whose length is equal to the text sentence length threshold are segmented to obtain a target segmentation table.

4. The audio classification method based on text information according to claim 1, characterized in that The training of the custom text classification model based on the text sentence index dataset includes: Build a custom text classification model based on the preset language model, bidirectional GRU neural network, attention mechanism and fully connected neural network; Constructing a text classification training set and a text classification label set based on the text sentence index dataset; Inputting the text classification training set into the language model to obtain a high-dimensional word vector set; Input the high-dimensional word vector set into the bidirectional GRU network to obtain a low-dimensional word vector set and a text sentence vector set; Performing an attention mechanism on the low-dimensional word vector set and the text sentence vector set to obtain a feature sentence vector set, where the feature sentence vectors in the feature sentence vector set are used to represent the importance of the corresponding text sentence in the audio text; Inputting the feature sentence vector set into the fully connected neural network for classification to obtain a text classification set; The classification loss between the text classification set and the text classification label set is calculated based on a cross entropy loss function to optimize the training of the custom text classification model.

5. The audio classification method based on text information according to claim 1, characterized in that The step of obtaining target audio and obtaining text information features of the target audio based on the trained custom text classification model includes: Classify the audio text of the target audio using the custom text classification model to obtain a meaningful text set and a meaningless text set; The meaningful text set and the meaningless text set are used as text information features of the target audio.

6. The audio classification method based on text information according to claim 5, characterized in that: The extracting audio features of the target audio and screening the audio features and the text information features according to the ensemble learning model to obtain audio classification salient features includes: Extracting multiple audio features of the target audio, and using a meaningful text set and a meaningless text set in the text information features as audio features of the target audio; Building a classification and regression tree based on the audio features, wherein the classification and regression tree corresponds to the audio features one by one; Fit all classification and regression trees based on the ensemble learning model to obtain a significant classification and regression tree; The audio features corresponding to each significant classification regression tree are used as the significant features of audio classification.

7. The audio classification method based on text information according to claim 1, characterized in that The classifying the target audio based on the audio classification salient features and the integrated learning model includes: Constructing a binary regression tree based on the audio classification salient features, wherein the binary regression tree corresponds to the audio classification salient features in a one-to-one manner; Classify the target audio based on the binary regression tree to obtain a single-tree classification result; The classification result of the target audio is obtained by superimposing the single-tree classification results of all binary regression trees after fitting based on the ensemble learning model.

8. An audio classification device based on text information, characterized in that: The apparatus comprises a module for implementing the method according to any one of claims 1 to 7, the apparatus comprising a conversion module, a screening module, a construction module, a training module, an acquisition module, an extraction module, and a classification module: The conversion module is used to convert the acquired audio data into audio text to obtain an audio segmentation dataset; The screening module is used to screen the word segmentation data set to construct an audio keyword table; The construction module is used to construct a text sentence index data set based on the audio text and the audio keyword table; The training module is used to train a custom text classification model based on the text sentence index dataset; The acquisition module is used to acquire target audio and obtain text information features of the target audio based on the trained custom text classification model; The extraction module is used to extract audio features of the target audio, and screen the audio features and the text information features according to the ensemble learning model to obtain audio classification salient features; The classification module is used to classify the target audio based on the audio classification salient features and the integrated learning model.

9. An electronic device, characterized in that: The electronic device comprises: a memory storing computer-readable instructions; and A processor is configured to execute computer-readable instructions stored in the memory to implement the audio classification method based on text information according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the text information-based audio classification method according to any one of claims 1 to 7 is implemented.

Images