Audio recognition system and audio classification device and method thereof
By sampling and extracting audio features through an audio classification device and recognition system, and training with a Gaussian mixture model, the problem of accurate audio type recognition in electronic devices under different environments is solved, ensuring timely execution of functions.
Patent Information
- Application Number
- CN202411474461.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2026-04-24
AI Technical Summary
Existing electronic devices lack accurate audio classification mechanisms, resulting in the inability to perform functions at the appropriate time.
Audio sampling and feature extraction are performed using an audio classification device. A Gaussian mixture model is used to train and generate classification features. These features are then compared with an audio recognition device to determine the audio type.
It achieves accurate recognition of different audio types, enabling electronic devices to perform appropriate functions in different environments.
Smart Images
Figure CN121922115A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to audio recognition technology, and more particularly to an audio recognition system and its audio classification apparatus and method. Background Technology
[0002] In some electronic devices, different functions are often executed based on the sounds of the surrounding environment. However, when different sound sources exist in the environment, the sounds received by the electronic device will also differ. If the electronic device does not have a precise audio classification mechanism to identify the type of audio, it will be unable to execute the default function at the appropriate time. Summary of the Invention
[0003] In view of the problems of the prior art, one of the objectives of the present invention is to provide an audio recognition system and its audio classification apparatus and method to improve the prior art.
[0004] This invention includes an audio classification device, comprising: a storage circuit and a classification processing circuit. The storage circuit is configured to store multiple training audio files classified into multiple audio types. The classification processing circuit is configured to: select any audio type as the corresponding audio type and extract the training audio files classified into the corresponding audio type; perform audio sampling and feature extraction on the training audio files to generate multiple training feature data; train the training feature data using a Gaussian mixture model (GMM) to generate multiple Gaussian curves that approximate the data distribution of the training feature data; and generate multiple curve parameters of each Gaussian curve as classification features for the corresponding audio type.
[0005] The present invention also includes an audio classification method, comprising: selecting any audio type as the corresponding audio type from multiple training audio files that are classified into multiple audio types, and extracting training audio files classified into the corresponding audio type to perform audio sampling and feature extraction on the training audio files, and generating multiple training feature data; training a Gaussian mixture model on the training feature data to generate multiple Gaussian curves that approximate the data distribution of the training feature data; and generating multiple curve parameters of each Gaussian curve as classification features of the corresponding audio type.
[0006] The present invention also includes an audio recognition system, comprising: an audio classification device and an audio recognition device. The audio classification device includes: a storage circuit and a classification processing circuit. The storage circuit is configured to store multiple training audio files classified into multiple audio types. The classification processing circuit is configured to: select any audio type as the corresponding audio type and extract the training audio files classified into the corresponding audio type; perform audio sampling and feature extraction on the training audio files to generate multiple training feature data; train a Gaussian mixture model on the training feature data to generate multiple Gaussian curves approximating the data distribution of the training feature data; and generate multiple curve parameters of each Gaussian curve as classification features for the corresponding audio type. The audio recognition device includes: an audio extraction circuit and a recognition processing circuit. The audio extraction circuit is configured to extract input audio. The recognition processing circuit is configured to: perform audio sampling and feature extraction on the input audio to generate multiple audio feature data; compare the audio feature data of multiple recognition segments of the input audio with the classification features of all audio types to determine the audio type to which each recognition segment belongs; and count the audio type to which the most recognition segments belong as the recognized audio type of the input audio.
[0007] The features, implementation, and effects of this application are described in detail below with reference to the accompanying drawings, showing preferred embodiments. Attached Figure Description
[0008] Figure 1 This shows a block diagram of an audio recognition system according to one embodiment of the present invention; Figure 2 This diagram illustrates a classification processing circuit sampling multiple sampled audio frames from a training audio archive, according to one embodiment of the present invention. Figure 3 This diagram illustrates the data distribution and Gaussian curve of the training feature data in one embodiment of the present invention; and Figure 4 This diagram shows a flowchart of an audio classification method according to one embodiment of the present invention. Detailed Implementation
[0009] One objective of this invention is to provide an audio recognition system and its audio classification device and method, which generates classification features by sampling and extracting features from training audio files of different audio types, and then compares and statistically analyzes the audio feature data of the recognition segment in the input audio with the classification features to determine the recognition audio type of the input audio.
[0010] Please refer to Figure 1 . Figure 1 This diagram shows a block diagram of an audio recognition system 100 according to an embodiment of the present invention. The audio recognition system 100 includes an audio classification device 110 and an audio recognition device 120.
[0011] The audio classification device 110 is configured to train audio files AA1~AA1 of different audio types. N AB1~AB M and AC1~AC P Training is performed to generate classification features CA~CC corresponding to different audio types.
[0012] The audio recognition device 120 is configured to capture input audio IA and identify it according to classification features CA~CC to determine the audio type corresponding to the input audio IA.
[0013] The structure and operation mechanism of the audio classification device 110 will be described below.
[0014] The audio classification device 110 includes a storage circuit 130 and a classification processing circuit 140.
[0015] Storage circuit 130 can be any circuit with a data storage mechanism, configured to store multiple training audio files AA1~AA1, categorized into multiple audio types. N AB1~AB M and AC1~AC P , where N, M and P can be positive integers that are the same or different from each other.
[0016] In one embodiment, the audio type includes music type, speech type, and ambient sound type, so that the training audio files AA1~AA N The training audio files AB1~AB were categorized to correspond to different music genres. M The audio files were differentiated to correspond to speech types, and the training audio files were AC1~AC1. P They are distinguished to correspond to ambient sound types. However, the present invention is not limited thereto.
[0017] The classification processing circuit 140 is electrically coupled to the storage circuit 130. In one embodiment, the classification processing circuit 140 may access, for example but not limited to, an application stored in the storage circuit 130, to perform audio classification processing.
[0018] The classification processing circuit 140 is configured to select any audio type as the corresponding audio type and extract training audio files that are classified into the corresponding audio types, so as to perform audio sampling and feature extraction on the training audio files and generate multiple training feature data (TCA).
[0019] For example, the classification processing circuit 140 can first select music genre as the corresponding audio genre, and then extract training audio files AA1~AA1 that are classified as music genres. N .
[0020] Next, the classification processing circuit 140 performs audio sampling based on the sampled audio frame size and the overlap size to generate multiple sampled audio frames, each with a sampled audio frame size. Each sampled audio frame contains an actual sampled audio frame and an overlapping portion with an overlap size, and the overlapping portion of each sampled audio frame has the same signal content as the front end of the next sampled audio frame.
[0021] Please refer to Figure 2 . Figure 2 In one embodiment of the present invention, the classification processing circuit 140 samples multiple sampled audio frames SF1~SF1 from the training audio file AA1. N A schematic diagram.
[0022] Sample audio frames SF1~SF K Each has its own sampled audio frame size. Figure 2 In this example, the sample audio frame size SZ is marked above the sampled audio frame SF1. In one embodiment, the sampled audio frame size SZ may be determined by the sampling rate and the number of sampling points in a unit sampled audio frame. In a numerical example, the sampling rate may be, for example, but not limited to, 16,000 sampling points per second, and a sampled audio frame (e.g., sampled audio frame SF1) may contain 2,048 sampling points. In such a case, the sampled audio frame size is 0.128 seconds.
[0023] Each sampled audio frame SF1~SF K It includes the actual sampled audio frame and an overlapping portion with a certain overlap size. Taking sampled audio frame SF1 as an example, Figure 2 Below the sampled audio frame SF1, the actual sampled audio frame CF1 is drawn. The overlapping portion OP in sampled audio frame SF1 is marked with a diagonal block, and the overlap size OS is marked above the corresponding overlapping portion OP. The next sampled audio frame after sampled audio frame SF1, namely sampled audio frame SF2, is also marked with a diagonal block to indicate its front-end portion FP, where the overlapping portion OP and the front-end portion FP have the same signal content.
[0024] The configuration of the overlap portion (OP) prevents incomplete feature capture when adjacent sampled audio frames are discontinuous (i.e., without any overlap). In a numerical example, the overlap portion (OP) can contain 512 sampling points. In this case, the overlap size (OS) is 0.032 seconds, and the actual sampled audio frame (CF) size is 0.096 seconds.
[0025] Other sampled audio frames SF2~SF K It can have the same structure as the sampled audio frame SF1. Figure 2 In the sampled audio frames SF1~SF2, only... KThe actual sampled audio frames CF1~CF1 are shown below. K However, no two adjacent sampled audio frames SF2~SF3 were drawn. K The overlapping parts and the front end.
[0026] The classification processing circuit 140 classifies each sampled audio frame SF1~SF2. K Actual sampled audio frames CF1~CF K Feature extraction is performed to generate training feature data TCA containing zero-crossing rate (ZCR) data, spectral contrast data, chroma short-time Fourier transform (STFT) data, Mel spectrogram data, or combinations thereof.
[0027] When each actual sampled audio frame CF1~CF K When each pair of adjacent sampling points is considered as a group of sampling points, the zero-crossing rate data is the proportion of groups of adjacent pairs of sampling points that cross from negative to positive values and from positive to negative values. This characteristic is used to determine the actual sampled audio frames CF1~CF2. K It belongs to either the speech type or the ambient sound type. For each actual sampled audio frame CF1~CF K The zero-crossing rate data will contain 1 data point.
[0028] The spectrum comparison data report includes each actual sampled audio frame CF1~CF K The difference between peaks and troughs in multiple frequency bands. This characteristic is used to determine the difference between the peaks and troughs of each actual sampled audio frame CF1~CF2. K Whether it belongs to a music genre or not. When the sampled audio frames are analyzed according to 7 frequency bands, for each actual sampled audio frame CF1~CF K The spectrum comparison data report contains 7 data points.
[0029] Chromaticity short-time Fourier transform data packet containing each actual sampled audio frame CF1~CF K The size of each interval when the spectrum is projected onto the intervals of the 12 semitones in the corresponding octave. This characteristic is used to determine the size of each actual sampled audio frame CF1~CF2. K Whether it belongs to a music genre or not. For each sampled audio frame, the chroma short-time Fourier transform data contains 12 data points.
[0030] Mel spectrum data is obtained by analyzing the frequency scale used to simulate human nonlinear auditory perception across actual sampled audio frames CF1~CF2. KThe sampling points in the data are analyzed. For each actual sampled audio frame CF1~CF2... K The chromaticity short-time Fourier transform data contains 128 data points.
[0031] In the example of training feature data TCA containing the above items, each actual sampled audio frame CF1~CF K The training feature data TCA will contain 1+7+12+128=148 data points. This will be used to train audio files AA1~AA2. N For example, if N is 50, and each training audio file is AA1~AA N If the length is 30 seconds, then the actual sampled audio frames CF1~CF2 are obtained. K The number of features will be 30 / 0.096=312, and the total number of TCA training feature data extracted will be 50x312x148=2,308,800.
[0032] However, it should be noted that the above-mentioned items and data numbers are merely examples. In other embodiments, the training feature data (TCA) for each sampled audio frame may contain different items and different data numbers. This invention is not limited thereto.
[0033] The classification processing circuit 140 performs Gaussian mixture model training on the training feature data TCA to generate multiple Gaussian curves that approximate the data distribution of the training feature data.
[0034] Please refer to Figure 3 . Figure 3 This diagram illustrates the data distribution DD of the training feature data TCA and the Gaussian curves GC1~GC3 in one embodiment of the present invention. Figure 3 In the diagram, the horizontal axis represents the zero-crossing rate, and the vertical axis represents the number of audio frames.
[0035] In one embodiment, Gaussian mixture model training includes a classification processing circuit 140 performing multiple iterative procedures on the training feature data TCA according to multiple default Gaussian curves to approximate the training feature data TCA.
[0036] like Figure 3 As shown, the data distribution DD of the training feature data TCA is a non-Gaussian curve. Each data point in the data distribution DD represents the number of audio frames with a corresponding zero-crossing rate. Taking data point PO as an example, it represents 1010 audio frames with a zero-crossing rate of 0.225.
[0037] The classification processing circuit 140 can initiate an iterative process based on, for example, but not limited to, three default Gaussian curves (not shown).
[0038] These default Gaussian curves may have default weights, default center positions, and default covariance matrices. The weights determine the highest height of these default Gaussian curves, the center position determines the location of the highest height, and the covariance matrix represents the dispersion of these default Gaussian curves. The classification processing circuit 140 can approximate the data distribution DD by calculating the difference between the default Gaussian curves and the data distribution DD and correcting the default Gaussian curves.
[0039] After multiple iterations of calculating differences and making corrections, the classification processing circuit 140 can generate, for example, the following: Figure 3 The diagram shows three Gaussian curves GC1 to GC3 that approximate the data distribution DD. The parameter settings and execution count of the iterative program will affect the degree to which Gaussian curves GC1 to GC3 approximate the data distribution DD.
[0040] The classification processing circuit 140 further generates multiple curve parameters for each of the Gaussian curves GC1 to GC3 as classification features (CA) for the corresponding audio type. In one embodiment, the curve parameters include the weights, center positions, and covariance matrices of these Gaussian curves GC1 to GC3.
[0041] It should be noted that the number of Gaussian curves used to approximate the data distribution DD described above is only an example. In other embodiments, different numbers of Gaussian curves can be set depending on the required accuracy or computational resources.
[0042] The above examples are training audio files AA1~AA that are differentiated according to music genres. N For example, the training audio files AB1~AB1 are distinguished according to the speech type. M And training audio files AC1~AC that are differentiated according to the type of ambient sound. P The same process can be used to generate corresponding training feature data TCB and TCC, and then obtain the corresponding classification features CB and CC.
[0043] The following will refer to again Figure 1 The structure and operation mechanism of the audio recognition device 120 are described.
[0044] The audio recognition device 120 includes: an audio acquisition circuit 150, a recognition processing circuit 160, and a functional circuit 170.
[0045] Audio capture circuit 150 is configured to capture input audio IA. In one embodiment, audio capture circuit 150 may be, for example, but not limited to, a microphone or other circuitry capable of capturing audio.
[0046] The recognition processing circuit 160 is configured to perform audio sampling and feature extraction on the input audio IA to generate multiple audio feature data ACD.
[0047] In one embodiment, the identification processing circuit 160 can be based on the classification processing circuit 140 corresponding to... Figure 2 The described technique performs audio sampling and feature extraction on the input audio IA to generate multiple sampled audio frames. Each sampled audio frame sequentially contains an actual sampled audio frame and an overlapping portion with a certain overlap size, which will not be elaborated further here. For example, if the length of the input audio IA is 10 seconds, and the recognition processing circuit 160 performs audio sampling based on the same sampled audio frame size and overlap size as the classification processing circuit 140 in the aforementioned example, the number of actual sampled audio frames will be 10 / 0.096 = 104, and the number of audio feature data (ACD) data for each actual sampled audio frame will be 148 entries.
[0048] The recognition processing circuit 160 compares the audio feature data ACD of multiple recognition segments of the input audio IA with the classification features CA~CC of all audio types to determine which audio type each recognition segment belongs to. In one embodiment, the recognition processing circuit 160 can access the classification features CA~CC from the audio classification device 110 in real time when comparison is needed, or it can obtain the classification features CA~CC from the audio classification device 110 in advance and store them in a storage circuit (not shown) included in the audio recognition device 120, and access the classification features CA~CC from the storage circuit when comparison is needed.
[0049] In one embodiment, each identification segment is the actual sampled audio frame contained in one of the sampled audio frames. For one of the identification segments to be identified, the identification processing circuit 160 calculates the probability density function (PDF) according to the classification features CA~CC of all audio types corresponding to the audio feature data ACD of the segment to be identified, so as to determine that the segment to be identified belongs to one of the audio types corresponding to the maximum probability density value.
[0050] For example, when the recognition processing circuit 160 calculates the probability density function of the segment to be recognized, and generates three probability density values corresponding to music type, speech type and ambient sound type, and the probability density value generated for music type is the largest, then the recognition processing circuit 160 will determine that the segment to be recognized belongs to the music type.
[0051] Next, the recognition processing circuit 160 counts the audio type corresponding to the most recognized segment as the recognized audio type AT of the input audio IA.
[0052] For example, when the recognition processing circuit 160 calculates 104 recognition segments and finds that 1 recognition segment belongs to the music type, 71 recognition segments belong to the speech type, and 32 recognition segments belong to the ambient sound type, the recognition processing circuit 160 determines that the recognition audio type AT of the input audio IA is the speech type.
[0053] Functional circuit 170 is configured to perform a default function based on the identified audio type AT. The operation mechanism of functional circuit 170 will be described in more detail below with respect to different embodiments of audio recognition device 120.
[0054] In one embodiment, the audio recognition device 120 is a hearing aid, and the functional circuit 170 is an equalization circuit, which performs a speech enhancement function when the audio type AT is identified as a speech type, an audio enhancement function when the audio type AT is identified as a music type, and a noise reduction function when the audio type AT is identified as an ambient sound type.
[0055] In another embodiment, the audio recognition device 120 is, for example, but not limited to, a smart watch, a smartphone, a tablet computer, or a smart vehicle system, and the functional circuit 170 is a control circuit that performs voice control function, speech-to-text function, or message notification function when the audio type AT is identified as a voice type, and does not perform voice control function, speech-to-text function, or message notification function when the audio type AT is identified as a music type or an ambient sound type.
[0056] For example, when the audio recognition device 120 is a smart vehicle system, the functional circuit 170 can first determine whether the SMS message contains important information upon receiving it. If the SMS message contains important information, it can then determine the recognition audio type AT of the input audio IA. The functional circuit 170 then performs a message notification function with a first broadcast voice when the audio type AT is identified as a voice type, prompting the user whether they need the message content to be read aloud during a conversation. When the audio type AT is identified as a music type or an ambient sound type, it performs a message notification function with a second broadcast voice, simply informing the user that an important SMS message has been received. However, the invention is not limited to this.
[0057] The audio recognition system and audio classification device of the present invention can generate classification features by sampling and extracting audio features from training audio files of different audio types, and then compare and statistically analyze the audio feature data of the recognition segment in the input audio with the classification features to determine the recognition audio type of the input audio.
[0058] Please refer to Figure 4 . Figure 4 This diagram shows a flowchart of an audio classification method 400 according to an embodiment of the present invention.
[0059] In addition to the aforementioned apparatus, the present invention also discloses an audio classification method 400, applicable to, for example, but not limited to, [other applications]. Figure 1 In the audio classification device 110, one of the audio classification methods 400 is implemented, for example... Figure 4 As shown, it includes the following steps.
[0060] In step S410, multiple training audio files AA1~AA1 are self-differentiated into multiple audio types. N AB1~AB M and AC1~AC P In this process, any audio type is selected as the corresponding audio type, and training audio files that are distinguished by the corresponding audio type are extracted to perform audio sampling and feature extraction on the training audio files, and generate multiple training feature data TCA~TCC.
[0061] In step S420, Gaussian mixture model training is performed on the training feature data TCA~TCC to generate multiple Gaussian curves GC1~GC3 that approximate the data distribution DD of the training feature data.
[0062] In step S430, Gaussian curves GC1~GC3 are generated, and each curve parameter is used as the classification feature CA~CC of the corresponding audio type.
[0063] It should be noted that the above-described implementation is merely an example. In other embodiments, those skilled in the art can make modifications without departing from the spirit of the invention.
[0064] In summary, the audio recognition system and its audio classification device and method of the present invention can generate classification features by sampling and extracting audio features from training audio files of different audio types, and then compare and statistically analyze the audio feature data of the recognition segment in the input audio with the classification features to determine the recognition audio type of the input audio.
[0065] Although the embodiments of this application are described above, these embodiments are not intended to limit this application. Those skilled in the art can make changes to the technical features of this application based on the express or implied content of this application. All such changes may fall within the scope of patent protection sought by this application. In other words, the scope of patent protection of this application shall be determined by the claims of this application.
[0066] [Symbol Explanation] 100: Audio Recognition System 110: Audio classification device 120: Audio recognition device 130: Storage circuit 140: Classification Processing Circuit 150: Audio capture circuit 160: Identification and processing circuit 170: Functional Circuit 400: Audio Classification Methods S410~S430: Steps AA1~AA N AB1~AB M AC1~AC P Training audio files ACD: Audio Feature Data AT: Identify audio type CA~CC: Classification features CF1~CF K Actual sampled audio frames DD: Data Distribution FP: Front-end GC1~GC3: Gaussian curves IA: Input audio OP: Overlapping parts OS: Overlap Size PO: Data Point SF1~SF K : Sample audio frames SZ: Sampled audio frame size TCA~TCC: Training Feature Data
Claims
1. An audio classification device, characterized in that, Include: A storage circuit configured to store multiple training audio files categorized into multiple audio types; and A classification processing circuit, configured with: Select any of the audio types as a corresponding audio type and extract the training audio files that are distinguished by the corresponding audio type, so as to perform an audio sampling and a feature extraction on the training audio files and generate multiple training feature data. A Gaussian mixture model is trained on the training feature data to generate multiple Gaussian curves that approximate a data distribution of the training feature data; and The multiple curve parameters of each Gaussian curve are generated as a classification feature for the corresponding audio type.
2. An audio classification method, characterized in that, Include: From multiple training audio files that are classified into multiple audio types, select any one of those audio types as a corresponding audio type and extract the training audio files classified into that corresponding audio type, so as to perform an audio sampling and a feature extraction on the training audio files and generate multiple training feature data. A Gaussian mixture model is trained on the training feature data to generate multiple Gaussian curves that approximate a data distribution of the training feature data; and The multiple curve parameters of each Gaussian curve are generated as a classification feature for the corresponding audio type.
3. The audio classification method as described in claim 2, characterized in that, Also includes: The audio is sampled according to a sampled audio frame size and an overlap size to generate multiple sampled audio frames, each of which has the sampled audio frame size. Each of the sampled audio frames sequentially includes an actual sampled audio frame and an overlap portion having the overlap size, and the overlap portion of each of the sampled audio frames has the same signal content as a front end portion of a subsequent sampled audio frame.
4. The audio classification method as described in claim 3, characterized in that, Also includes: The feature extraction is performed on the actual sampled audio frame for each of the sampled audio frames to generate training feature data that includes a zero-crossing rate data, a spectral contrast data, a chromatic short-time Fourier transform data, a Mel spectral data, or a combination thereof.
5. The audio classification method as described in claim 2, characterized in that, The training of the Gaussian mixture model involves performing multiple iterative procedures on the training feature data according to multiple default Gaussian curves to approximate the training feature data.
6. The audio classification method as described in claim 2, characterized in that, These curve parameters include a weight, a center position, and a common variation matrix for each Gaussian curve.
7. The audio classification method as described in claim 2, characterized in that, These audio types include a music type, a speech type, and an ambient sound type.
8. An audio recognition system, characterized in that, Include: An audio classification device, comprising: A storage circuit configured to store multiple training audio files categorized into multiple audio types; and A classification processing circuit, configured with: Select any of the audio types as a corresponding audio type and extract the training audio files that are distinguished by the corresponding audio type, so as to perform an audio sampling and a feature extraction on the training audio files and generate multiple training feature data. A Gaussian mixture model is trained on the training feature data to generate multiple Gaussian curves that approximate a data distribution of the training feature data; and The multiple curve parameters of each Gaussian curve are generated as a classification feature for the corresponding audio type; as well as An audio recognition device, comprising: An audio capture circuit configured to capture an input audio; and A recognition processing circuit, configured with: The input audio is sampled and the features are extracted to generate multiple audio feature data. The audio feature data of multiple recognition segments of the input audio are compared with the classification features of all audio types to determine which audio type each recognition segment belongs to; and The audio type to which the most identified segments belong is identified as one of the audio types of the input audio.
9. The audio recognition system as described in claim 8, characterized in that, For one of the recognition segments to be processed, the recognition processing circuit performs a probability density function calculation on the audio feature data of the segment to be processed according to the classification features of all audio types, so as to determine that the segment to be processed belongs to one of the audio types corresponding to a maximum probability density value.
10. The audio recognition system as described in claim 8, characterized in that, The audio recognition circuit also includes a functional circuit configured to perform at least one default function based on the recognized audio type; When the audio recognition device is a hearing aid, the functional circuit is an equalization circuit, which performs a speech enhancement function when the recognized audio type is a speech type, an audio enhancement function when the recognized audio type is a music type, and a noise reduction function when the recognized audio type is an ambient sound type; and When the audio recognition device is an intelligent electronic device, the functional circuit is a control circuit that performs a voice control function, a speech-to-text function, or a message notification function when the recognized audio type is a voice type, and does not perform the voice control function, the speech-to-text function, or the message notification function when the recognized audio type is a music type or an ambient sound type.