Audio recognition method, system and equipment based on MC-CNN and multi-task learning
Through the MC-CNN and multi-task learning methods, adaptive time-frequency decomposition and multi-stage feature extraction, the performance of speech recognition technology in noisy environments is optimized, solving the noise suppression, computational complexity and data dependency problems in existing technologies, and achieving efficient and robust speech recognition effects.
Patent Information
- Application Number
- CN202510039895.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-10
- Publication Date
- 2025-09-23
AI Technical Summary
Existing speech recognition technology has deficiencies in noise suppression, computational complexity, time series modeling capabilities, and data dependency, resulting in poor recognition accuracy and robustness. Performance is significantly reduced, especially in noisy environments. Furthermore, high computational resource requirements make it difficult to meet the real-time requirements of embedded devices.
A method based on MC-CNN and multi-task learning is adopted. Through adaptive time-frequency decomposition and multi-stage convolutional neural network feature extraction, combined with the main task and auxiliary task loss functions, it optimizes speech content recognition and noise classification, reduces the model's sensitivity to noise interference, improves robustness and computational efficiency, and enhances time series modeling capabilities.
It significantly improves the recognition accuracy and robustness in noisy environments, reduces computational complexity and resource consumption, enhances the utilization efficiency of limited annotated data, and improves the real-time performance of speech recognition and the ability to recognize complex speech sequences.
Smart Images
Figure CN120690176A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to an audio recognition method, system, device, and computer-readable storage medium based on a multi-stage convolutional neural network (MC-CNN) and multi-task learning. Background Art
[0002] Speech recognition, a key area in natural language processing, is widely used in human-computer interaction, smart devices, and information acquisition. With the advancement of artificial intelligence and big data technologies, the application of deep learning in speech recognition has significantly improved the accuracy and robustness of recognition. Models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) have significantly advanced speech recognition technology. However, current technologies still face key challenges, such as insufficient noise suppression, high computational complexity, and a strong dependence on labeled data.
[0003] However, existing speech recognition technology still has the following shortcomings: 1. High computational complexity. Model training and decoding require a large amount of computing resources, which limits performance when processing complex or large-scale data. 2. Poor robustness. Insufficient adaptability to environmental noise and accent changes leads to a significant reduction in recognition rate in noisy environments. 3. Insufficient temporal feature modeling capabilities. HMM cannot effectively capture the long temporal characteristics of complex speech sequences.
[0004] At the same time, related technologies for applying deep learning to speech recognition, such as deep neural networks, convolutional neural networks, and recurrent neural networks, have solved some of the limitations of traditional technologies, but the following technical problems still exist: poor noise suppression effect, especially in high-noise environments, the recognition accuracy of the model decreases significantly; poor real-time performance, and high computing resource requirements cannot meet the real-time requirements of embedded devices or mobile terminals; strong dependence on labeled data, and the high cost of obtaining large amounts of high-quality labeled data limits the universality of the application.
[0005] Therefore, the current relevant technologies and application research on multi-task learning audio recognition methods still need to be further improved. Summary of the Invention
[0006] In view of this, the present invention proposes an audio recognition method, system, device and computer-readable storage medium based on MC-CNN and multi-task learning, which solves the technical problems of existing speech recognition technology in noise suppression, computational complexity, time series modeling capabilities and data dependence.
[0007] The technical solution of the present invention is achieved as follows:
[0008] In one aspect, the present invention provides an audio recognition method based on MC-CNN and multi-task learning, comprising the following steps:
[0009] S1, input original speech signal;
[0010] S2. Using an adaptive time-frequency decomposition method, dynamically adjusting the time window and basis function according to the instantaneous characteristics of the speech signal, and outputting a multi-resolution, multi-scale time-frequency feature map;
[0011] S3, extracting features from the time-frequency feature map of step S2 according to a multi-stage convolutional neural network;
[0012] S4. Use the multi-task learning framework to optimize the speech content recognition and noise classification tasks and output the speech recognition results.
[0013] On the basis of this technical solution, it is further preferred that the step S3 extracts features from the time-frequency feature map of step S2 according to a multi-stage convolutional neural network, including:
[0014] Low-level feature extraction: input the time-frequency feature map of step S2 and extract basic features, including edges and fundamental frequencies;
[0015] Intermediate feature extraction: Based on the low-level feature extraction, intermediate features are extracted. The intermediate features include phoneme distribution.
[0016] High-level feature extraction integrates low-level features and mid-level features to generate deep feature representation.
[0017] On the basis of this technical solution, it is further preferred that the multi-task learning framework described in step S4 includes:
[0018] Main task loss function, used to optimize speech content recognition;
[0019] Auxiliary task loss function, used to optimize noise classification;
[0020] The total loss function combines the main task loss function and the auxiliary task loss function and is balanced by the weight coefficient.
[0021] On the basis of this technical solution, it is further preferred that the multi-stage convolutional neural network includes skip connections and residual blocks.
[0022] On the basis of this technical solution, it is further preferred that step S2 uses an adaptive time-frequency decomposition method, including:
[0023] According to the instantaneous characteristics of the speech signal, the width of the time window is adjusted, and then a fixed Fourier basis function is used in the traditional transformation to optimize the objective function and perform time-frequency decomposition at multiple scales.
[0024] On the basis of this technical solution, further preferably, the objective function optimization includes minimizing the joint loss of reconstruction error and recognition error.
[0025] In a second aspect, the present invention further provides an audio recognition system based on MC-CNN and multi-task learning, comprising:
[0026] An input module, used for inputting original speech signals;
[0027] A time-frequency decomposition module, which uses an adaptive time-frequency decomposition method to dynamically adjust the time window and basis function according to the instantaneous characteristics of the speech signal and output a multi-resolution and multi-scale time-frequency feature map;
[0028] A multi-stage convolutional neural network module, used to extract features from the time-frequency feature map of step S2 according to the multi-stage convolutional neural network;
[0029] A multi-task learning framework module that optimizes speech content recognition and noise classification tasks and outputs speech recognition results.
[0030] On the basis of this technical solution, it is further preferred that
[0031] The multi-task learning framework module also includes:
[0032] Main task loss function unit, used to optimize speech content recognition;
[0033] Auxiliary task loss function unit, used to optimize noise classification;
[0034] The total loss function unit combines the main task loss function and the auxiliary task loss function and balances them through weight coefficients.
[0035] In a third aspect, the present invention also provides an audio recognition device based on MC-CNN and multi-task learning, the device comprising a processor and a memory for storing processor-executable instructions, the processor executing the instructions to implement the audio recognition method based on MC-CNN and multi-task learning as described in any one of the aspects.
[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio recognition method based on MC-CNN and multi-task learning as described in any one of the aspects.
[0037] The audio recognition method, system, device, and computer-readable storage medium based on MC-CNN and multi-task learning described in the present invention have the following advantages over the prior art:
[0038] Through multi-stage MC-CNN processing of speech features, noise interference is suppressed, recognition accuracy in noisy environments is improved, and noise robustness is enhanced. At the same time, the optimized network structure design based on multi-task learning reduces the number of parameters, thereby reducing resource consumption, achieving efficient real-time performance, and reducing computational complexity. In addition, through time-frequency transformation and staged feature extraction, the deep temporal characteristics of speech signals are captured, which also improves the temporal modeling capability. In combination with the multi-task learning framework, the auxiliary tasks share features to improve the efficiency of limited annotated data.
[0039] Compared to DeepSpeech and hybrid model-based approaches, this invention significantly improves robustness in noisy environments, optimizes computational efficiency, and enhances the ability to recognize complex speech sequences. Its multi-stage feature extraction and multi-task learning framework provides an innovative solution to the existing speech recognition field and is expected to be widely applied in diverse and complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0041] Figure 1 This is a flow chart of the audio recognition method based on MC-CNN and multi-task learning according to the present invention;
[0042] Figure 2 This is a framework diagram of the audio recognition system based on MC-CNN and multi-task learning according to the present invention;
[0043] Figure 3 This is a framework diagram of the multi-task learning module in the audio recognition system based on MC-CNN and multi-task learning described in the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0045] This invention provides an audio recognition method, system, device, and computer-readable storage medium based on MC-CNN and multi-task learning. By introducing MC-CNN, the feature extraction process of speech signals is divided into multiple stages: a low-level stage extracts basic spectral features of speech, a mid-level stage captures higher-level speech patterns, and a high-level stage integrates multiple levels of features to generate a deep feature representation. This step-by-step feature extraction method not only more effectively captures the temporal characteristics of speech signals but also significantly reduces the model's sensitivity to noise interference, improving recognition robustness.
[0046] Building on this foundation, the present invention employs a multi-task learning framework that shares the model's low-level and mid-level features across multiple tasks, simultaneously optimizing both speech content recognition and noise classification, thereby further improving model performance. This framework leverages information from auxiliary tasks, reducing reliance on large-scale annotated data and improving the efficiency of utilizing limited annotated data.
[0047] Example 1
[0048] This embodiment provides an audio recognition method based on MC-CNN and multi-task learning. The flowchart is as follows: Figure 1 As shown, the following steps are included:
[0049] S1, input original speech signal x(t);
[0050] S2. Using an adaptive time-frequency decomposition method, dynamically adjusting the time window and basis function according to the instantaneous characteristics of the speech signal, and outputting a multi-resolution, multi-scale time-frequency feature map;
[0051] Specifically, step S2 uses an adaptive time-frequency decomposition method, including:
[0052] According to the instantaneous characteristics of the speech signal, the width of the time window is adjusted, and then the traditional transformation uses a fixed Fourier basis function to optimize the objective function and perform time-frequency decomposition at multiple scales;
[0053] More specifically, the adaptive time window selection, i.e., dynamically adjusting the time window width w(t) according to the instantaneous characteristics of the speech signal, is expressed as follows:
[0054]
[0055] Where x(t) is the speech signal, (x(t))· is its first-order inverse, and the function f is learned through a neural network.
[0056] More specifically, the traditional transform of the learnable basis function uses a fixed Fourier basis function or wavelet basis, while the basis function optimized in this embodiment is:
[0057] ψ(t;θ)=Aexp(-αt2 )cos(2πβt+φ)
[0058] Among them, θ = {α, β, φ} is a learnable parameter.
[0059] More specifically, the objective function optimization mainly refers to minimizing the joint loss of reconstruction error and recognition error, and the formula is:
[0060] L=L recon +λL task
[0061]
[0062] Among them, L recon The L2 norm is used to reconstruct the error, and L task Cross entropy loss for speech recognition tasks.
[0063] More specifically, performing time-frequency decomposition at multiple scales means performing time-frequency decomposition at multiple scales simultaneously and jointly analyzing to obtain a time-frequency graph. The formula is:
[0064]
[0065] Where S is the scale number, and ATFD(x(t),s) represents the time-frequency decomposition at the Sth scale.
[0066] Step S2 can adapt to different speech environments, such as noise and different speaking rates, by introducing dynamic time windows and learnable basis functions. Multi-scale processing helps capture more features and reduce redundancy. At the same time, the joint optimization of the objective function can also improve the accuracy and noise resistance of speech recognition, thereby significantly improving adaptability, efficiency, and robustness.
[0067] S3, extracting features from the time-frequency feature map of step S2 according to a multi-stage convolutional neural network;
[0068] In the multi-stage processing process, if the design is not appropriate, it may lead to information loss between the levels of feature extraction, especially in the connection between high-frequency information and low-frequency information.
[0069] The solution adopted in this embodiment is to introduce skip connections or residual blocks, which ensures efficient transmission of information between different stages. The specific formula is:
[0070] F(x)=X+G(x)
[0071] Among them, F(x) is the output with residual, G(x) is the feature extracted by the convolution layer, and X is the input speech signal.
[0072] In addition, the multi-stage design of MC-CNN may lead to an excessive number of model parameters and increased computational requirements; therefore, this embodiment uses depthwise separable convolution to reduce computational complexity.
[0073] The formula for depth-wise separable convolution is:
[0074] C depthwise =(D k *D k *C in *H*W)+(1*1*C in *C out *H*W)
[0075] Among them, D k is the convolution kernel size, C in ,C out are the number of input and output channels, respectively, and H and W are the feature sizes.
[0076] Specifically, the feature extraction described in step S3 includes:
[0077] Low-level feature extraction: input the time-frequency feature map of step S2 and extract basic features, including edges and fundamental frequencies;
[0078] More specifically, low-level feature extraction mainly involves performing convolution operations on the initial time-frequency image of the speech signal to extract low-level features such as edges and spectral bands. The algorithm formula is:
[0079] F1=σ(W1*X+b1)
[0080] Where X is the input time-frequency data, W1 is the convolution kernel, b1 is the bias term, and σ is the activation function;
[0081] Intermediate feature extraction: Based on the low-level feature extraction, intermediate features are extracted. The intermediate features include phoneme distribution.
[0082] More specifically, based on the feature graph of the low-level feature extraction stage, higher-level speech patterns, such as the phoneme distribution of speech, are extracted. The algorithm formula is:
[0083] F2=σ(W2*F1+b2)
[0084] Among them, b2 is the bias term, W2 is the convolution kernel, and σ is the activation function;
[0085] High-level feature extraction, integrating low-level features with mid-level features to generate deep feature representations;
[0086] More specifically, the advanced feature extraction integrates the features of the multiple feature extraction stages to generate a deep feature representation. The algorithm formula is:
[0087] F final =Pool(F3)
[0088] Among them, F3 represents the high-level feature extraction and integration of low-level features F1 and intermediate features F2, and Pool refers to the pooling operation, which is used for dimensionality reduction or feature aggregation.
[0089] S4. Use the multi-task learning framework to optimize the speech content recognition and noise classification tasks and output the speech recognition results.
[0090] Specifically, the multi-task learning framework described in step S4 is to utilize the multi-task learning framework to optimize multiple tasks simultaneously by sharing the low-level and mid-level features extracted in step S3, thereby improving the overall performance.
[0091] More specifically, it includes:
[0092] A main task loss function is used to optimize speech content recognition; wherein the main task is used for speech content recognition;
[0093] More specifically, the formula for the main task loss function is:
[0094]
[0095] Among them, y i represents the real labeled data in the speech content recognition task, that is, the actual distribution of the target category, and Represents the results predicted by the model.
[0096] Auxiliary task loss function, used to optimize noise classification; where auxiliary task, is used for noise classification;
[0097] More specifically, the formula for the auxiliary task loss function is:
[0098]
[0099] Among them, z j Represents the true labeled data in the auxiliary task (noise classification task).
[0100] The total loss function combines the main task loss function and the auxiliary task loss function and is balanced by the weight coefficient.
[0101] The formula for the total loss function is:
[0102] L total =λ main L main +λ aux L aux
[0103] Among them, λ main and λaux is the weight coefficient.
[0104] Example 2
[0105] This embodiment provides an audio recognition system based on MC-CNN and multi-task learning. Figure 2 Shown, including:
[0106] An input module, used for inputting original speech signals;
[0107] A time-frequency decomposition module, which uses an adaptive time-frequency decomposition method to dynamically adjust the time window and basis function according to the instantaneous characteristics of the speech signal and output a multi-resolution and multi-scale time-frequency feature map;
[0108] A multi-stage convolutional neural network module, used to extract features from the time-frequency feature map of step S2 according to the multi-stage convolutional neural network;
[0109] A multi-task learning framework module that optimizes speech content recognition and noise classification tasks and outputs speech recognition results.
[0110] In a preferred embodiment, Figure 3 As shown, the multi-task learning framework module also includes:
[0111] Main task loss function unit, used to optimize speech content recognition;
[0112] Auxiliary task loss function unit, used to optimize noise classification;
[0113] The total loss function unit combines the main task loss function and the auxiliary task loss function and balances them through weight coefficients.
[0114] In a preferred embodiment, an audio recognition device based on MC-CNN and multi-task learning is also included, wherein the device includes a processor and a memory for storing processor-executable instructions, and the processor executes the instructions to implement the audio recognition method based on MC-CNN and multi-task learning as described in Example 1.
[0115] In a preferred embodiment, a computer-readable storage medium is further included, on which a computer program is stored. When the computer program is executed by a processor, the audio recognition method based on MC-CNN and multi-task learning as described in Example 1 is implemented.
[0116] In summary, the present invention provides an audio recognition method, system, device, and computer-readable storage medium based on MC-CNN and multi-task learning. The multi-stage design ensures layer-by-layer optimization of features and avoids information loss. The feature integration mechanism of skip connections and multi-layer integration improves the expressiveness of deep features. The feature extraction method of adaptive time-frequency decomposition provides high-quality input features, making MC-CNN more adaptable to complex speech environments. The two jointly optimize the objective function and improve the performance of the overall system.
[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. The audio recognition method based on MC-CNN and multi-task learning is characterized by: The following steps are involved: S1, input original speech signal; S2. Using an adaptive time-frequency decomposition method, dynamically adjusting the time window and basis function according to the instantaneous characteristics of the speech signal, and outputting a multi-resolution, multi-scale time-frequency feature map; S3, extracting features from the time-frequency feature map of step S2 according to a multi-stage convolutional neural network; S4. Use the multi-task learning framework to optimize the speech content recognition and noise classification tasks and output the speech recognition results.
2. The audio recognition method based on MC-CNN and multi-task learning according to claim 1, characterized in that: Step S3 extracts features from the time-frequency feature graph of step S2 using a multi-stage convolutional neural network, including: Low-level feature extraction: input the time-frequency feature map of step S2 and extract basic features, including edges and fundamental frequencies; Intermediate feature extraction: Based on the low-level feature extraction, intermediate features are extracted. The intermediate features include phoneme distribution. High-level feature extraction integrates low-level features and mid-level features to generate deep feature representation.
3. The audio recognition method based on MC-CNN and multi-task learning according to claim 1, characterized in that Step S4 utilizes a multi-task learning framework, including: Main task loss function, used to optimize speech content recognition; Auxiliary task loss function, used to optimize noise classification; The total loss function combines the main task loss function and the auxiliary task loss function and is balanced by the weight coefficient.
4. The audio recognition method based on MC-CNN and multi-task learning according to claim 2, characterized in that The multi-stage convolutional neural network includes skip connections and residual blocks.
5. The audio recognition method based on MC-CNN and multi-task learning according to claim 1, characterized in that: Step S2 uses an adaptive time-frequency decomposition method, including: According to the instantaneous characteristics of the speech signal, the width of the time window is adjusted, and then a fixed Fourier basis function is used in the traditional transformation to optimize the objective function and perform time-frequency decomposition at multiple scales.
6. The audio recognition method based on MC-CNN and multi-task learning according to claim 5, characterized in that The objective function optimization includes minimizing the joint loss of reconstruction error and recognition error.
7. An audio recognition system based on MC-CNN and multi-task learning, characterized by: include: An input module, used for inputting original speech signals; A time-frequency decomposition module, which uses an adaptive time-frequency decomposition method to dynamically adjust the time window and basis function according to the instantaneous characteristics of the speech signal and output a multi-resolution and multi-scale time-frequency feature map; A multi-stage convolutional neural network module, used to extract features from the time-frequency feature map of step S2 according to the multi-stage convolutional neural network; A multi-task learning framework module that optimizes speech content recognition and noise classification tasks and outputs speech recognition results.
8. The audio recognition system based on MC-CNN and multi-task learning according to claim 7, characterized in that: The multi-task learning framework module also includes: Main task loss function unit, used to optimize speech content recognition; Auxiliary task loss function unit, used to optimize noise classification; The total loss function unit combines the main task loss function and the auxiliary task loss function and balances them through weight coefficients.
9. An audio recognition device based on MC-CNN and multi-task learning, characterized in that: The device includes a processor and a memory for storing processor-executable instructions, and the processor executes the instructions to implement the audio recognition method based on MC-CNN and multi-task learning according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the audio recognition method based on MC-CNN and multi-task learning as described in any one of claims 1 to 6 is implemented.