Dry sound classification model training methods, dry sound classification, electronic devices, and storage media

Abstract

This application discloses a method for training a dry sound classification model, a dry sound classification method, and related devices and media. The dry sound classification model training method includes: labeling the dry sound categories of audio (at least two categories); extracting audio features from the labeled audio and generating an audio training dataset using these features; inputting the audio training dataset into a multi-layer convolutional layer, then through a bidirectional LSTM network, and finally into a fully connected layer to obtain shared audio features, which participate in the gradient calculation of the multi-layer convolutional layer, the bidirectional LSTM, and the fully connected layer; inputting the shared audio features into multiple dry sound category task networks to obtain the classification results of the dry sound categories in the audio training dataset; and training a dry sound classification model based on the labeled dry sound categories and the classification results of the dry sound categories in the audio training dataset. This application improves the training efficiency of the dry sound classification model and increases the classification accuracy of the trained dry sound classification model.

Description

Technical Field

[0001] This application relates to the field of audio detection technology, and more specifically, to a dry sound classification model training method, a dry sound classification method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In dry sound classification methods of related technologies, a classification model detects a specific category. For example, a breath sound detection model can only distinguish between breath sounds and non-breath sounds. Figure 1 As shown, a cascaded model is constructed using a single-task concatenation approach. Input features are fed into four models (mode1, mode2, mode3, mode4), each executing a separate task (task1, task2, task3, task4) to achieve dry sound detection results for multiple categories. For example, task1 distinguishes between breath sounds and non-breath sounds, task2 distinguishes between sibilant and non-sibilant sounds, task3 distinguishes between popping and non-popping sounds, and task4 distinguishes between frying sounds and non-frying sounds. However, determining whether dry sound belongs to a different category requires using different detection models and training different models separately, resulting in low efficiency in model training and dry sound category detection.

[0003] Therefore, how to improve the training efficiency of dry sound classification models and the detection efficiency of dry sound categories are technical problems that need to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a dry sound classification model training method, a dry sound classification method, an electronic device, and a computer-readable storage medium, which improves the training efficiency of the dry sound classification model and the detection efficiency of dry sound categories.

[0005] To achieve the above objectives, the first aspect of this application provides a method for training a dry sound classification model, comprising:

[0006] Label the dry audio category, wherein the dry audio category is greater than or equal to 2;

[0007] The audio features of the labeled audio are extracted, and the audio training dataset is generated using the audio features. The audio training dataset includes multiple types of feature information.

[0008] The audio training dataset is input into multiple convolutional layers, then through a bidirectional LSTM network, and then into a fully connected layer to obtain shared audio features. The bidirectional LSTM network is used to ensure the temporal stability of the audio training dataset. The shared audio features participate in the gradient calculation of the multiple convolutional layers, bidirectional LSTM, and fully connected layers.

[0009] The shared audio features are input into multiple dry sound category task networks to obtain the classification results of the dry sound categories in the audio training dataset;

[0010] Based on the classification results of the dry sound categories in the audio annotation and the dry sound categories in the audio training dataset, a dry sound classification model is trained, which includes the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks.

[0011] The dry audio category of the labeled audio is preceded by:

[0012] The audio is sliced ​​to obtain multiple audio segments, and the volume of the multiple audio segments is normalized.

[0013] Accordingly, the dry audio categories of the labeled audio include:

[0014] Label the dry audio categories of multiple audio segments;

[0015] Accordingly, the extracted audio features of the labeled audio include:

[0016] Extract audio features from multiple audio segments.

[0017] The audio features extracted from the labeled audio include:

[0018] Speech activity features are extracted from the labeled audio as audio features.

[0019] And / or, extract the fundamental frequency features of the labeled audio as audio features.

[0020] And / or, extract the energy envelope of the labeled audio as an audio feature.

[0021] And / or, extract the frequency domain features of the labeled audio as audio features.

[0022] The dry sound category task network is a binary classification task network, and the classification result output by the dry sound category task network indicates whether the audio training dataset belongs to the dry sound category corresponding to the dry sound category task network.

[0023] The dry sound classification model trained based on the classification results of the dry sound categories in the audio annotation and the dry sound categories in the audio training dataset includes the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks, comprising:

[0024] The loss value of the loss function of each dry sound category task network is calculated based on the classification result output by each dry sound category task network and the dry sound category of the audio annotation.

[0025] Calculate the sum of losses for all the dry sound category task networks, and update the model parameters of the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and the multiple dry sound category task networks based on the sum of losses, until the sum of losses converges to obtain the trained dry sound classification model.

[0026] The loss function of the dry sound category task network is:

[0027]

[0028] Where n is the number of categories classified by the dry sound category task network, and y i The true distribution of the i-th category determined based on the dry sound categories of the training audio. The distribution of the i-th category output by the dry sound category task network.

[0029] The dry sound category includes any one or a combination of several of the following: breathing sounds, sibilance, popping sounds, and bubbling sounds.

[0030] To achieve the above objectives, a second aspect of this application provides a dry sound classification method, comprising:

[0031] Acquire the target audio and extract the target audio features from the target audio;

[0032] The target audio features are input into the dry audio classification model trained using the dry audio classification model training method described above, and the classification result of the target audio is obtained.

[0033] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is configured to implement the steps of the dry sound classification model training method or the dry sound classification method described above when executing the computer program.

[0036] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described dry sound classification model training method or the above-described dry sound classification method.

[0037] As can be seen from the above scheme, the dry sound classification model training method provided in this application includes: labeling the dry sound categories of audio, wherein the dry sound categories are greater than or equal to 2; extracting audio features of the labeled audio, generating an audio training dataset using the audio features, wherein the audio training dataset includes multi-class feature information; inputting the audio training dataset into a multi-layer convolutional layer, then through a bidirectional LSTM network, and then into a fully connected layer, finally obtaining shared audio features, wherein the bidirectional LSTM network is used to ensure the temporal stability of the audio training dataset, and the shared audio features participate in the gradient calculation of the multi-layer convolutional layer, the bidirectional LSTM, and the fully connected layer; inputting the shared audio features into multiple dry sound category task networks to obtain the classification results of the dry sound categories of the audio training dataset; training a dry sound classification model containing the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks based on the labeled dry sound categories of the audio and the classification results of the dry sound categories of the audio training dataset.

[0038] In this application, a dry sound classification model is used to distinguish different categories of audio. In this model, different dry sound category task networks are used to determine whether an audio item belongs to its corresponding dry sound category. Each dry sound category task network shares an encoder, multiple convolutional layers, neural network layers, and fully connected layers, improving the training efficiency and detection efficiency of the dry sound classification model. During training, information contained in one sub-task can be used to improve the accuracy of other sub-tasks, thus increasing the classification accuracy of the trained dry sound classification model. This application also discloses a dry sound classification method, an electronic device, and a computer-readable storage medium, which can achieve the same technical effects.

[0039] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0041] Figure 1 This is a structural diagram of a cascade model in related technologies;

[0042] Figure 2A flowchart illustrating a dry sound classification model training method provided in this application embodiment;

[0043] Figure 3 A schematic diagram illustrating a breathing sound provided in an embodiment of this application;

[0044] Figure 4 A schematic diagram illustrating the annotation of dental sounds provided in an embodiment of this application;

[0045] Figure 5 This is a schematic diagram illustrating a microphone spraying method provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram illustrating the annotation of a bubble sound in an embodiment of this application;

[0047] Figure 7 A structural diagram of a dry sound classification model provided in an embodiment of this application;

[0048] Figure 8 A flowchart illustrating another method for training a dry sound classification model provided in this application embodiment;

[0049] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] This application discloses a dry sound classification model training method, which improves the training efficiency of the dry sound classification model and the detection efficiency of dry sound categories.

[0052] See Figure 2 The flowchart of a dry sound classification model training method provided in this application embodiment is as follows: Figure 2 As shown, it includes:

[0053] S101: Label the dry audio category, wherein the dry audio category is greater than or equal to 2;

[0054] The execution entity in this embodiment can be a server, and the purpose is to train a dry voice classification model. In this step, training audio is collected based on real singing scenarios, and labeled by combining audio auditory performance and temporal features. The labeling needs to include the start time, end time, and whether it belongs to multiple dry voice categories (type). Dry voice categories can include breath sounds, dentistry, popping sounds, and vocal fry. Breath sounds refer to the sound of breathing during speaking or singing. Popping sounds refer to the unpleasant booming sound produced when the breath is exhaled too forcefully during the pronunciation of certain words, causing excessive airflow impact on the diaphragm of the recording equipment; this usually occurs at the beginning of words. Dental sounds are a type of alveolar consonant; when pronouncing them, the tip of the tongue touches the upper incisors. If the tip of the tongue is between the upper and lower incisors, it is called an interdental consonant, generally appearing on phonemes such as z, c, and sh. Vocal fry refers to a sound produced from the throat by a weak airflow impacting the vocal cords, somewhat like the sound of continuously blown bubbles bursting. It has a very low frequency, a very small airflow, and a very low glottal depressor, with the vocal cords almost unaffected by the airflow. These categories are mutually exclusive and highly distinguishable, making them relatively easy to label accurately using spectral visualization annotation tools.

[0055] Figure 3 This is a diagram illustrating the labeling of breathing sounds. Figure 4 This is a diagram illustrating the marking of dental consonants. Figure 5 This is a label indicating the microphone output. Figure 6 This is a schematic diagram of the annotation of vocal fry. By analyzing the differences between the annotation categories in the time and frequency domains, we can see that: the energy of breath sounds is relatively evenly distributed across the entire frequency band; the energy of sibilants is concentrated in the mid-to-high frequencies and changes rapidly; popping sounds have a significant envelope bulge in the time domain and, compared to normal unstressed consonants, their frequency domain energy is concentrated in the mid-to-low frequencies, with rapid energy changes; the time domain waveform of vocal fry has obvious periodicity, with a short period, no obvious harmonic structure in the spectrum, and weak energy. Breath sounds mainly occur between speech segments, while sibilants, popping sounds, and vocal fry mainly occur within speech segments, with sibilants and popping sounds mainly appearing at the beginning of words. These clearly distinguishable features make it feasible to use deep learning to model this classification task.

[0056] It should be noted that the training audio includes various categories of audio, and the audio sample data of each category is relatively evenly distributed. There should not be a situation where one type of audio signal is abundant while others are scarce. When selecting data, training data that simultaneously contains multiple audio features can be chosen to ensure that the model can capture the differences between different audio features in high-dimensional space during training.

[0057] S102: Extract the audio features of the labeled audio, and use the audio features to generate an audio training dataset, which includes multiple types of feature information;

[0058] In this step, multiple audio features are extracted from the labeled audio, and an audio training dataset is generated based on the extracted multiple audio features.

[0059] As one feasible implementation, the extraction of audio features from the labeled audio includes: extracting speech activity features from the labeled audio as audio features, and / or extracting fundamental frequency features from the labeled audio as audio features, and / or extracting energy envelopes from the labeled audio as audio features, and / or extracting frequency domain features from the labeled audio as audio features.

[0060] In practical implementation, Voice Activity Detection (VAD) can be used to extract speech activity features from audio. VAD detection provides voiced and unvoiced information, and since training audio is voiced, most invalid information can be filtered out. Pitch detection can be used to extract the pitch fundamental frequency features of training audio; breath sounds, sibilance, and popping sounds are signals without a fundamental frequency. The energy envelope of the audio can be extracted, reflecting the rate of energy change. Frequency domain features of the audio can be extracted based on the Mel spectrum, including the energy distribution of each frequency band. The Mel spectrum differs from the general cepstrum; its most significant feature is that the frequency bands on the Mel cepstrum are uniformly distributed across the Mel scale. In other words, such frequency bands are closer to the nonlinear human auditory system than the commonly seen linear cepstrum representation, and the number of frequency bands in the Mel spectrum is much smaller than that of the linear spectrum, making it more suitable for acoustic modeling. Audio features can include any one or more of the above-mentioned speech activity features, fundamental frequency features, energy envelope, and frequency domain features; no specific limitations are imposed here.

[0061] S103: The audio training dataset is input into a multi-layer convolutional layer, then through a bidirectional LSTM network, and then into a fully connected layer to finally obtain shared audio features. The bidirectional LSTM network is used to ensure the temporal stability of the audio training dataset. The shared audio features participate in the gradient calculation of the multi-layer convolutional layer, the bidirectional LSTM, and the fully connected layer.

[0062] S104: Input the shared audio features into multiple dry sound category task networks to obtain the classification results of the dry sound categories of the audio training dataset;

[0063] S105: Train a dry sound classification model based on the classification results of the dry sound categories in the audio annotation and the dry sound categories in the audio training dataset, including the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks.

[0064] In practice, a dry sound classification model is trained using an audio training dataset and labeled dry sound categories. The dry sound classification model consists of a sequentially connected encoder, multiple convolutional layers, a bidirectional LSTM (Long Short-Term Memory) network, fully connected layers, and a task network. For example, ... Figure 7 As shown, the encoder can use embedding. The audio training dataset input to the dry sound classification model is embedded and then fed into multiple convolutional layers: Convolutional Layer 1, Convolutional Layer 2, and Convolutional Layer 3. LSTM can provide the correlation of information between different parts of the vocal signal, ensuring temporal stability. Finally, the output passes through a fully connected layer. The shared audio features output by the fully connected layer are input to multiple dry sound category task networks (task1, task2, task3, and task4) in the task network, resulting in the classification result output by each dry sound category task network. The dry sound category task network can be a binary classification task network, and the classification result output by the dry sound category task network indicates whether the audio training dataset belongs to the corresponding dry sound category of the dry sound category task network. From embedding to shared features, the parameters in this part are shared by all dry sound category task networks and participate in the gradient calculation together. The shared features are passed as common inputs to each dry sound category task network.

[0065] As a feasible implementation, the training of a dry sound classification model based on the classification results of the dry sound categories in the audio annotation and the dry sound categories in the audio training dataset includes the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks. This includes: calculating the loss value of the loss function of each dry sound category task network based on the classification result output by each dry sound category task network and the dry sound category in the audio annotation; calculating the sum of losses corresponding to all dry sound category task networks; and updating the model parameters of the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and the multiple dry sound category task networks based on the sum of losses, until the sum of losses converges to obtain the trained dry sound classification model.

[0066] In practice, each dry sound category task network is equivalent to a binary classification task. Each dry sound category task network calculates a loss based on the output classification result and the labeled dry sound category. For example, if task 1 is a breathing sound judgment model, the shared feature is passed through a convolutional layer and then through an activation function, such as the sigmoid function, outputting a number in the range of 0-1 and the classification result. For example, the classification result can be represented as [0000111100000…], where 0 and 1 represent the two categories "is a breathing sound" and "is not a breathing sound," respectively. The loss function of the dry sound category task network is:

[0067]

[0068] Where n is the number of categories classified by the dry sound category task network, and y i The true distribution of the i-th category determined based on the dry sound categories of the training audio. The distribution of the i-th category output by the dry sound category task network.

[0069] The losses of the networks for each dry sound category task are summed to output a loss sum, which is then used to iteratively update the model parameters using the backpropagation gradient. When the dry sound classification model converges, the trained dry sound classification model is obtained. The convergence condition can be that the above loss sum is less than a preset value, or the number of iterations reaches a preset number.

[0070] Since the audio features are related in different dry sound category task networks, distinguishing the details of different feature combinations can improve the accuracy of classification and reduce overfitting. Therefore, the multi-task learning method adopted in this embodiment is more effective and accurate than the single-task cascade method of related technologies.

[0071] The trained dry audio classification model is used to classify the target audio. In practice, the audio features of the target audio are extracted and input into the trained dry audio classification model. At the same time, the start and end times of the breathing sounds, sibilance, popping sounds, and bubbling sounds contained therein are determined.

[0072] In this embodiment, a dry sound classification model distinguishes different categories of audio. In this model, different dry sound category task networks are used to determine whether an audio item belongs to its corresponding category. Each dry sound category task network shares an encoder, multiple convolutional layers, neural network layers, and fully connected layers, improving the training efficiency and detection efficiency of the dry sound classification model. During training, information contained in one sub-task can be used to improve the accuracy of other sub-tasks, thus increasing the classification accuracy of the trained dry sound classification model.

[0073] This application discloses a method for training a dry sound classification model. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0074] See Figure 8 The flowchart of another dry sound classification model training method provided in this application embodiment is as follows: Figure 8 As shown, it includes:

[0075] S201: The audio is sliced ​​to obtain multiple audio segments, and the volume of the multiple audio segments is normalized.

[0076] In this embodiment, to make the feature distinction of each type of signal more obvious, during data processing, all audio can be sliced, for example, 5-10 seconds as an audio segment, and then the volume of each audio segment is normalized, for example, its volume is normalized to -18dB.

[0077] S202: Label the dry audio categories of the multiple audio segments;

[0078] In this step, each audio segment is labeled, including the start time, end time, and dry sound type. The dry sound type can include breathing sounds, sibilance, popping sounds, and bubbling sounds. For example, 1 represents breathing sounds, 2 represents sibilance, 3 represents popping sounds, and 4 represents bubbling sounds.

[0079] S203: Extract audio features from multiple audio segments, and generate an audio training dataset corresponding to multiple audio segments using the audio features of the multiple audio segments, wherein the audio training dataset includes multiple types of feature information;

[0080] S204: The audio training datasets corresponding to the multiple audio segments are input into multiple convolutional layers, then into a bidirectional LSTM network, and finally into a fully connected layer to obtain shared audio features. The bidirectional LSTM network is used to ensure the temporal stability of the audio training datasets. The shared audio features participate in the gradient calculation of the multiple convolutional layers, the bidirectional LSTM, and the fully connected layer. The shared audio features are input into multiple dry sound category task networks to obtain the dry sound category classification results of the audio training datasets corresponding to the multiple audio segments. Based on the dry sound category of the multiple audio segments and the classification results of the dry sound category of the audio training datasets corresponding to the multiple audio segments, a dry sound classification model containing the multiple convolutional layers, the bidirectional LSTM network, the fully connected layer, and the multiple dry sound category task networks is trained.

[0081] In this step, audio features are extracted from each audio segment, which may include speech activity features, fundamental frequency features, energy envelope, and frequency domain features. Then, a dry sound classification model is trained using the audio features of each audio segment and the dry sound category. The dry sound classification model consists of an encoder, multiple convolutional layers, a bidirectional LSTM network, a fully connected layer, and a task network, all connected in sequence. The task network includes a dry sound category task network corresponding to each dry sound category, and the audio features output from the fully connected layer serve as the input to each dry sound category task network.

[0082] In practice, the audio training dataset corresponding to each audio segment is input into the dry sound classification model to obtain the classification result output by the dry sound category task network. Based on the classification result output by each dry sound category task network and the dry sound category of the audio segment, the loss value of the loss function of each dry sound category task network is calculated. The loss sum of the loss values ​​corresponding to all dry sound category task networks is calculated. Based on the loss sum, the model parameters of the dry sound classification model are updated, including the model parameters in the encoder, multiple convolutional layers, bidirectional LSTM network, fully connected layer and dry sound category task network, until the loss sum converges to obtain the trained dry sound classification model.

[0083] The trained dry audio classification model is used to classify the target audio. In practice, the target audio is sliced ​​to obtain multiple target audio segments, and then the volume of these segments is normalized. The audio features of each target audio segment are extracted and input into the trained dry audio classification model. Simultaneously, it is determined whether each target audio segment belongs to a breath sound, sibilance, popping sound, or bubbly sound, thereby determining the start and end times of any breath sounds, sibilance, popping sounds, or bubbly sounds contained in the target audio.

[0084] Therefore, in this embodiment, the training audio is preprocessed by slicing and volume normalization, which makes the signal features of each category more obvious, improves the accuracy of subsequent dry sound classification model training, and enhances the classification effect of the trained dry sound classification model.

[0085] This application discloses a dry sound classification method, including:

[0086] Acquire the target audio and extract the target audio features from the target audio;

[0087] The target audio features are input into the dry audio classification model trained using the dry audio classification model training method described above, and the classification result of the target audio is obtained.

[0088] In practice, the audio features of the target audio are extracted and input into the trained dry audio classification model. At the same time, the start and end times of the breathing sounds, sibilance, popping sounds, and bubbling sounds contained therein are determined.

[0089] The following describes a dry sound classification model training device provided in an embodiment of this application. The dry sound classification model training device described below can be referred to in conjunction with the dry sound classification model training method described above. The dry sound classification model training device provided in this embodiment of the application includes:

[0090] The annotation module is used to annotate the dry sound category of the audio, wherein the dry sound category is greater than or equal to 2;

[0091] The generation module is used to extract audio features from the labeled audio and generate an audio training dataset using the audio features. The audio training dataset includes multiple types of feature information.

[0092] The first input module is used to input the audio training dataset into a multi-layer convolutional layer, then through a bidirectional LSTM network, and then into a fully connected layer to finally obtain shared audio features. The bidirectional LSTM network is used to ensure the temporal stability of the audio training dataset. The shared audio features participate in the gradient calculation of the multi-layer convolutional layer, the bidirectional LSTM, and the fully connected layer.

[0093] The second input module is used to input the shared audio features into multiple dry sound category task networks to obtain the classification results of the dry sound categories of the audio training dataset;

[0094] The training module is used to train a dry sound classification model, which includes the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and multiple dry sound category task networks, based on the classification results of the dry sound categories labeled in the audio and the dry sound categories in the audio training dataset.

[0095] In this embodiment, a dry sound classification model distinguishes different categories of audio. In this model, different dry sound category task networks are used to determine whether an audio item belongs to its corresponding category. Each dry sound category task network shares an encoder, multiple convolutional layers, neural network layers, and fully connected layers, improving the training efficiency and detection efficiency of the dry sound classification model. During training, information contained in one sub-task can be used to improve the accuracy of other sub-tasks, thus increasing the classification accuracy of the trained dry sound classification model.

[0096] Based on the above embodiments, as a preferred embodiment, it further includes:

[0097] The normalization module is used to slice the audio to obtain multiple audio segments, and to normalize the volume of the multiple audio segments.

[0098] Accordingly, the annotation module is used to: annotate the dry sound category of multiple audio segments;

[0099] Accordingly, the generation module is used to: extract audio features from multiple audio segments.

[0100] Based on the above embodiments, as a preferred embodiment, the generation module is used to: extract the speech activity features of the labeled audio as audio features, and / or extract the fundamental frequency features of the labeled audio as audio features, and / or extract the energy envelope of the labeled audio as audio features, and / or extract the frequency domain features of the labeled audio as audio features.

[0101] Based on the above embodiments, as a preferred implementation, the dry sound category task network is a binary classification task network, and the classification result output by the dry sound category task network is whether the audio training dataset belongs to the dry sound category corresponding to the dry sound category task network.

[0102] Based on the above embodiments, as a preferred implementation, the training module is used to: calculate the loss value of the loss function of each dry sound category task network based on the classification result output by each dry sound category task network and the dry sound category of the audio annotation; calculate the loss sum of the loss values ​​corresponding to all dry sound category task networks; and update the model parameters of the multi-layer convolutional layer, the bidirectional LSTM network, the fully connected layer, and the multiple dry sound category task networks based on the loss sum, until the loss sum converges to obtain the trained dry sound classification model.

[0103] Based on the above embodiments, as a preferred implementation, the loss function of the dry sound category task network is:

[0104]

[0105] Where n is the number of categories classified by the dry sound category task network, and y i The true distribution of the i-th category determined based on the dry sound categories of the training audio. The distribution of the i-th category output by the dry sound category task network.

[0106] Based on the above embodiments, as a preferred implementation, the dry sound category includes any one or a combination of several of the following: breathing sounds, sibilance, popping sounds, and bubbling sounds.

[0107] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0108] This application also provides an electronic device, see [link to document]. Figure 9 The present application provides a structural diagram of an electronic device 90, as shown in the embodiment. Figure 9 As shown, it may include a processor 91 and a memory 92.

[0109] The processor 91 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 91 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 91 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 91 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 91 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0110] The memory 92 may include one or more computer-readable storage media, which may be non-transitory. The memory 92 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 92 is used to store at least the following computer program 921, which, after being loaded and executed by the processor 91, is capable of implementing the relevant steps in the dry sound classification model training method executed by the electronic device side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 92 may also include an operating system 922 and data 923, etc., and the storage method may be temporary storage or permanent storage. The operating system 922 may include Windows, Unix, Linux, etc.

[0111] In some embodiments, the electronic device 90 may further include a display screen 93, an input / output interface 94, a communication interface 95, a sensor 96, a power supply 97, and a communication bus 98.

[0112] certainly, Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than [other components]. Figure 9More or fewer components as shown, or combinations of certain components.

[0113] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the dry sound classification model training method performed by the electronic device of any of the above embodiments.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0115] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for training a dry sound classification model, characterized in that, include: Label the dry audio category, wherein the dry audio category is greater than or equal to 2; The audio features of the labeled audio are extracted, and the audio training dataset is generated using the audio features. The audio training dataset includes multiple types of feature information. The audio training dataset is input into a multi-layer convolutional layer, then through a bidirectional long short-term memory network, and then into a fully connected layer to finally obtain shared audio features. The bidirectional long short-term memory network is used to ensure the temporal stability of the audio training dataset. The shared audio features participate in the gradient calculation of the multi-layer convolutional layer, the bidirectional long short-term memory network, and the fully connected layer. The shared audio features are input into multiple dry sound category task networks to obtain the classification results of the dry sound categories in the audio training dataset; Based on the classification results of the dry voice categories in the audio annotation and the dry voice categories in the audio training dataset, a dry voice classification model is trained, which includes the multi-layer convolutional layer, the bidirectional long short-term memory network, the fully connected layer, and multiple dry voice category task networks.

2. The dry sound classification model training method according to claim 1, characterized in that, Before the dry audio category of the labeled audio, it also includes: The audio is sliced ​​to obtain multiple audio segments, and the volume of the multiple audio segments is normalized. Accordingly, the dry audio categories of the labeled audio include: Label the dry audio categories of multiple audio segments; Accordingly, the extracted audio features of the labeled audio include: Extract audio features from multiple audio segments.

3. The dry sound classification model training method according to claim 1, characterized in that, The extracted audio features of the labeled audio include: Speech activity features are extracted from the labeled audio as audio features. And / or, extract the fundamental frequency features of the labeled audio as audio features. And / or, extract the energy envelope of the labeled audio as an audio feature. And / or, extract the frequency domain features of the labeled audio as audio features.

4. The dry sound classification model training method according to claim 1, characterized in that, The dry sound category task network is a binary classification task network, and the classification result output by the dry sound category task network indicates whether the audio training dataset belongs to the dry sound category corresponding to the dry sound category task network.

5. The dry sound classification model training method according to claim 4, characterized in that, The training of the dry sound classification model based on the classification results of the dry sound categories in the audio annotation and the dry sound categories in the audio training dataset includes the multi-layer convolutional layer, the bidirectional long short-term memory network, the fully connected layer, and multiple dry sound category task networks, including: The loss value of the loss function of each dry sound category task network is calculated based on the classification result output by each dry sound category task network and the dry sound category of the audio annotation. Calculate the sum of losses for all the dry sound category task networks, and update the model parameters of the multi-layer convolutional layer, the bidirectional long short-term memory network, the fully connected layer, and the multiple dry sound category task networks based on the sum of losses, until the sum of losses converges to obtain the trained dry sound classification model.

6. The dry sound classification model training method according to claim 5, characterized in that, The loss function of the dry sound category task network is: ； Where n is the number of categories output by the dry sound category task network. This represents the true distribution of the i-th category determined based on the dry sound categories of the audio. The distribution of the i-th category output by the dry sound category task network.

7. The dry sound classification model training method according to claim 1, characterized in that, The dry sound category includes any one or a combination of several of the following: breath sounds, sibilance, popping sounds, and bubbling sounds.

8. A method for classifying dry sound, characterized in that, include: Acquire the target audio and extract the target audio features from the target audio; The target audio features are input into the dry audio classification model trained by the dry audio classification model training method as described in any one of claims 1 to 7 to obtain the classification result of the target audio.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the dry sound classification model training method as described in any one of claims 1 to 7 or the dry sound classification method as described in claim 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dry sound classification model training method as described in any one of claims 1 to 7 or the dry sound classification method as described in claim 8.

Images