An audio recognition method and device, computer equipment and storage medium

Abstract

The application belongs to the technical field of audio processing, and relates to an audio recognition method and device, computer equipment and a storage medium, comprising: acquiring a collected sound signal collected by a sound collection device; pre-processing the collected sound signal to obtain a pre-processed sound signal; performing short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data; performing joint feature extraction processing on the pre-processed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; inputting the short-time feature data, the human voice feature data and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result and output the audio recognition result. The application uses a GMM-HMM joint compensation model to extract features capable of representing a speaker and environmental factors from a speech signal, can more comprehensively capture stable random factors in the speech signal, and thus improves the robustness and adaptability of the model.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio recognition method, apparatus, computer equipment, and storage medium. Background Art

[0002] Intelligent sound recognition technology plays an increasingly important role in modern information processing and artificial intelligence, and is widely used in voice assistants, smart homes, security monitoring, and other areas. Neural network models are particularly good at processing complex sound data.

[0003] There is an existing sound recognition method, which uses a deep neural network (DNN) to perform speech recognition on the short-term features of the collected audio signal, thereby achieving the purpose of sound recognition.

[0004] However, the applicant found that traditional voice recognition methods process speakers and environmental factors separately and are unable to capture stable random factors in speech signals, thereby reducing the robustness and adaptability of the model and, in turn, reducing the accuracy of voice recognition. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an audio recognition method, apparatus, computer device and storage medium to solve the problem of low recognition accuracy in traditional sound recognition methods.

[0006] In order to solve the above technical problems, the present application provides an audio recognition method, which adopts the following technical solutions:

[0007] Acquiring a sound signal collected by a sound collection device;

[0008] Preprocessing the collected sound signal to obtain a preprocessed sound signal;

[0009] Performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data;

[0010] Performing joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data;

[0011] Inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result;

[0012] The audio recognition result is output.

[0013] Furthermore, the step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps:

[0014] Performing noise removal processing on the collected sound signal according to the filter to obtain a denoised sound signal;

[0015] Obtaining signal strength information of the denoised audio signal, and determining whether the signal strength information is lower than a preset threshold;

[0016] If the signal strength information is lower than the preset threshold, performing gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal;

[0017] If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

[0018] Furthermore, the step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps:

[0019] Converting the time domain of the collected sound signal to obtain the frequency domain;

[0020] Segmenting the frequency domain to obtain a plurality of sub-frequency bands;

[0021] Performing howling detection processing on the current sub-frequency band to obtain a howling detection result;

[0022] If the howling detection result shows howling noise, suppressing the current sub-band according to a notch filter method to obtain a pre-processed sub-band;

[0023] If the howling detection result shows that there is no howling noise, using the current sub-band as a pre-processing sub-band;

[0024] After all the sub-frequency bands are subjected to the howling detection and suppression processes, all the pre-processed sub-bands are combined to obtain the pre-processed sound signal.

[0025] Furthermore, the step of performing howling detection processing on the current sub-frequency band to obtain a howling detection result specifically includes the following steps:

[0026] Calculating the phase characteristic of the current sub-frequency band, and determining whether the phase characteristic meets a preset phase determination condition;

[0027] If the phase characteristic satisfies the preset phase judgment condition, the value of the phase counter is increased by 1;

[0028] If the phase characteristic does not meet the preset phase judgment condition, resetting the value of the phase counter to 0;

[0029] Calculating the amplitude characteristic of the current sub-frequency band, and determining whether the amplitude characteristic meets a preset amplitude determination condition;

[0030] If the amplitude characteristic satisfies the preset amplitude judgment condition, the value of the amplitude counter is increased by 1;

[0031] If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0;

[0032] Determine whether the value of the phase detector is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter;

[0033] If the value of the phase detector is greater than or equal to the value of the phase detection time counter and the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter, confirming the howling detection result of the current sub-band as the presence of howling noise;

[0034] Otherwise, the howling detection result of the current sub-frequency band is confirmed as no howling noise exists.

[0035] Furthermore, after the step of inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result, the following step is also included:

[0036] The parameters of the deep neural network model are adaptively adjusted according to the reinforcement learning algorithm and the short-term feature data.

[0037] In order to solve the above technical problems, the present application also provides an audio recognition device, which adopts the following technical solution:

[0038] A signal acquisition module is used to acquire the sound signal collected by the sound collection device;

[0039] A preprocessing module, configured to preprocess the collected sound signal to obtain a preprocessed sound signal;

[0040] A short-time feature extraction module is used to perform short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data;

[0041] A joint feature extraction module is used to perform joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data;

[0042] An audio recognition module is used to input the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result;

[0043] The result output module is used to output the audio recognition result.

[0044] Furthermore, the preprocessing module includes:

[0045] A noise removal submodule, configured to perform noise removal processing on the collected sound signal according to a filter to obtain a denoised sound signal;

[0046] a signal strength determination submodule, configured to obtain signal strength information of the denoised audio signal and determine whether the signal strength information is lower than a preset threshold;

[0047] a first strength judgment result submodule, configured to perform gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal if the signal strength information is lower than the preset threshold;

[0048] The second strength judgment result submodule is configured to use the denoised sound signal as the pre-processed sound signal if the signal strength information is not lower than the preset threshold.

[0049] Furthermore, the preprocessing module includes:

[0050] A conversion submodule, configured to convert the time domain of the collected sound signal to obtain the frequency domain;

[0051] A segmentation submodule, configured to segment the frequency domain to obtain a plurality of sub-frequency bands;

[0052] The howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result;

[0053] A first detection result submodule is configured to, if the howling detection result contains howling noise, suppress the current sub-band according to a notch filter method to obtain a pre-processed sub-band;

[0054] A second detection result submodule, configured to use the current sub-band as a pre-processing sub-band if the howling detection result shows that there is no howling noise;

[0055] The pre-processing confirmation submodule is used to combine all pre-processed sub-bands to obtain the pre-processed sound signal after all the sub-frequency bands have been subjected to the howling detection and suppression processing.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0057] The system comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio recognition method described above when executing the computer-readable instructions.

[0058] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0059] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the audio recognition method described above.

[0060] The present application provides an audio recognition method, comprising: obtaining a collected sound signal collected by a sound collection device; preprocessing the collected sound signal to obtain a preprocessed sound signal; performing short-term feature extraction processing on the preprocessed sound signal to obtain short-term feature data; performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, which can more comprehensively capture stable random factors in the speech signal, thereby improving the robustness and adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 This is a flowchart of the implementation of the audio recognition method provided in Example 1 of the present application;

[0064] Figure 3 This is an average time graph of the continuous increase in signal amplitude during the howling process provided in Example 1 of the present application;

[0065] Figure 4 This is a schematic diagram of the structure of the audio recognition device provided in Example 2 of the present application;

[0066] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0071] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0073] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0074] It should be noted that the audio recognition method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the audio recognition device is generally set in the server / terminal device.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio recognition method according to the present application. The audio recognition method includes: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0077] In step S201, a sound signal collected by a sound collection device is obtained.

[0078] In an embodiment of the present application, the ambient sound signal can be collected by a high-sensitivity microphone.

[0079] In step S202, the collected sound signal is preprocessed to obtain a preprocessed sound signal.

[0080] In an embodiment of the present application, the collected sound signal may be pre-processed, including denoising, gain control, and normalization.

[0081] In step S203, short-time feature extraction is performed on the pre-processed sound signal to obtain short-time feature data.

[0082] In the embodiment of the present application, the deep neural network is mainly used to obtain short-term feature information in the sound signal.

[0083] In step S204, joint feature extraction processing is performed on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data.

[0084] In the embodiment of the present application, a GMM-HMM joint compensation model is used to extract features that can represent the speaker and environmental factors from the speech signal.

[0085] In the embodiment of the present application, the above-mentioned joint feature extraction process further includes:

[0086] 1. Short-time Fourier transform (STFT): Using short-time Fourier transform for time-frequency analysis in frequency domain conversion can capture the time-frequency characteristics of sound signals in more detail.

[0087] 2.Formula description:

[0088]

[0089] in: is the time-frequency representation; is the input signal; is a window function; For time; is the frequency.

[0090] In step S205, the short-term feature data, the human voice feature data, and the environmental feature data are input into a deep neural network model for audio recognition processing to obtain an audio recognition result.

[0091] In an embodiment of the present application, after obtaining short-time feature data, human voice feature data, and environmental feature data, these features are combined with the short-time features and input into a deep neural network for learning to implement audio recognition processing operations.

[0092] In an embodiment of the present application, a multimodal feature fusion model (audio + spectrum image) can be used to identify non-speech audio signals (such as environmental sounds, music signals, etc.), wherein the multimodal model combines the features of the audio signal and the spectrum image, processes the spectrum image through a convolutional neural network (CNN), extracts image-level features, and then fuses them with the audio features.

[0093] Audio and image feature fusion:

[0094]

[0095] in, is the audio feature; is the image feature; is the fusion coefficient.

[0096] Activation function for convolution processing: ReLU can be used:

[0097]

[0098] In the embodiment of this application, by introducing a multimodal model, the system's recognition rate for non-speech signals such as ambient sounds and music has increased by 20%. In experiments, for a set of outdoor ambient sound tests, the recognition accuracy rate increased from the original 70% to 85%.

[0099] In step S206, the audio recognition result is output.

[0100] In the embodiments of the present application, when the current system processes a large amount of real-time audio data, especially in a multi-tasking scenario, the processing speed may decrease. To address the problem of decreased processing speed, a parallel computing and distributed processing architecture can be adopted:

[0101] Parallel model training: Accelerate the training and prediction process through distributed neural network frameworks (such as TensorFlow or PyTorch's distributed computing capabilities);

[0102] Data parallel processing:

[0103]

[0104] in: is the i-th data block; Sub-models executed in parallel.

[0105] Parameter synchronization:

[0106]

[0107] This formula shows that the gradients calculated by each sub-model can be accumulated in parallel to update the global parameters.

[0108] In the embodiment of the present application, through parallel processing, the system reduced the processing time of large-scale audio streaming tasks by 40%. In the processing task of real-time audio streaming, the delay was reduced from 500ms to 300ms, significantly improving the response speed.

[0109] In an embodiment of the present application, an audio recognition method is provided, comprising: obtaining a collected sound signal collected by a sound collection device; preprocessing the collected sound signal to obtain a preprocessed sound signal; performing short-term feature extraction processing on the preprocessed sound signal to obtain short-term feature data; performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; inputting the short-term feature data, human voice feature data, and environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, which can more comprehensively capture stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0110] In some optional implementations of the embodiments of the present application, step S202 specifically includes the following steps:

[0111] Perform noise removal processing on the collected sound signal according to the filter to obtain a denoised sound signal;

[0112] Obtaining signal strength information of the de-noised audio signal and determining whether the signal strength information is lower than a preset threshold;

[0113] If the signal strength information is lower than a preset threshold, the denoised sound signal is subjected to gain adjustment processing to obtain a preprocessed sound signal;

[0114] If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

[0115] In the embodiment of the present application, since the ambient sound signal collected by the microphone often contains certain interference information, it is necessary to preprocess the collected sound signal to obtain a sound signal with fewer interference factors. Specifically:

[0116] First, the collected sound signal is preprocessed by removing noise above 8000 Hz and below 300 Hz through a bandpass filter to obtain a relatively clean preprocessed sound signal; next, based on the signal strength of the preprocessed sound signal, its average strength is calculated using a sliding window. Assuming that the set threshold is -20 decibels, if the signal strength is lower than this threshold, a linear gain function, such as G(x)=2x, is applied to adjust its gain.

[0117] In some optional implementations of the embodiments of the present application, a short-time Fourier transform (STFT) can be used to perform spectral analysis on the gain-adjusted sound signal, and a 256-point fast Fourier transform (FFT) window can be used to obtain a detailed distribution diagram of the frequency range; through the spectral analysis results, we can determine the main frequency components of the sound signal; for example, if the main frequency components are concentrated between 1000-3000 Hz, we can determine whether there are any abnormal frequency components. Assuming that an abnormal frequency of 400 Hz is detected, it is filtered out by a narrowband filter.

[0118] Obtain the filtered sound signal and calculate its signal-to-noise ratio (SNR). Assume that the set threshold is 30 decibels. If the SNR is lower than this threshold, spectral subtraction is used for noise suppression.

[0119] By analyzing the sound signal after noise suppression processing, its timbre characteristics are determined, for example, by extracting timbre characteristics through Mel-frequency cepstral coefficients (MFCCs), and these characteristics are used as reference parameters for subsequent processing.

[0120] If the sound signal contains speech, a speech recognition algorithm such as the Hidden Markov Model (HMM) is applied to extract speech feature parameters to obtain fundamental frequency (F0) and formant information.

[0121] Based on the fundamental frequency and formant information, speech enhancement technology such as Wiener filtering is used to optimize the sound signal and ultimately obtain a high-quality processed sound signal.

[0122] In the embodiment of the present application, this high-quality sound signal can be used for subsequent speech recognition, thereby realizing a more intelligent and efficient sound processing system.

[0123] In some optional implementations of the embodiments of the present application, an adaptive filtering algorithm can be introduced to propose an adaptive noise suppression method. The algorithm dynamically adjusts the filtering parameters according to different noise environments to more effectively remove noise. The algorithm is expressed as:

[0124]

[0125] in, is the denoised signal; is the input signal; is the estimated noise component.

[0126] In some optional implementations of the embodiments of the present application, step S202 specifically includes the following steps:

[0127] Convert the time domain of the collected sound signal to obtain the frequency domain;

[0128] The frequency domain is divided into several sub-bands;

[0129] Performing howling detection processing on the current sub-frequency band to obtain a howling detection result;

[0130] If the howling detection result shows howling noise, the current sub-band is suppressed according to the notch filter method to obtain a pre-processed sub-band;

[0131] If the howling detection result shows that there is no howling noise, the current sub-band is used as the pre-processing sub-band;

[0132] After all sub-bands have been subjected to the howling detection and suppression processes, all pre-processed sub-bands are combined to obtain a pre-processed sound signal.

[0133] In this embodiment, the process of "stopping the detection module and performing only signal suppression after howling is detected" is abandoned. Instead, "after howling is detected, howling detection continues while performing signal suppression." This method effectively shortens howling detection time. Regarding the howling suppression algorithm, the present invention uses a method of multiplying the howling subband by a small gain factor in the frequency domain, which is a notch filter method.

[0134] In some optional implementations of the embodiments of the present application, the step of performing howling detection processing on the current sub-band to obtain a howling detection result specifically includes the following steps:

[0135] Calculate the phase characteristics of the current sub-band and determine whether the phase characteristics meet the preset phase judgment conditions;

[0136] If the phase characteristic meets the preset phase judgment condition, the value of the phase counter is increased by 1;

[0137] If the phase characteristic does not meet the preset phase judgment condition, the value of the phase counter is reset to 0;

[0138] Calculate the amplitude characteristics of the current sub-band and determine whether the amplitude characteristics meet the preset amplitude determination conditions;

[0139] If the amplitude characteristic meets the preset amplitude judgment condition, the value of the amplitude counter is increased by 1;

[0140] If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0;

[0141] Determine whether the value of the phase detector is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter;

[0142] If the value of the phase detector is greater than or equal to the value of the phase detection time counter and the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter, the howling detection result of the current sub-band is confirmed as the presence of howling noise;

[0143] Otherwise, the howling detection result of the current sub-frequency band is determined as no howling noise exists.

[0144] In the embodiment of the present application, the original judgment condition of "continuous increase of signal energy" is abandoned, and the "ratio of sub-band signal energy to the sum of all sub-band energy values" is detected instead. Experimental statistics show that the phenomenon of "continuous increase of signal energy" only lasts for about 75 milliseconds when the system enters the oscillation state from the normal working state. The average result obtained on the waveform of the four-segment howling signal is as follows: Figure 3 Therefore, using this condition as the basis for determining howling cannot guarantee the reliability of the algorithm. As can be seen from the spectrogram of the original howling signal, most of the signal energy is concentrated in the howling sub-band. Therefore, using "the ratio of sub-band signal energy to the total energy value of all sub-bands" as a feature for detecting howling is more effective.

[0145] In some optional implementations of the embodiments of the present application, after step S205, the following steps are further included:

[0146] The parameters of the deep neural network model are adaptively adjusted based on the reinforcement learning algorithm and short-term feature data.

[0147] In the embodiment of the present application, the above-extracted features are used as input to the reinforcement learning algorithm, where:

[0148] (1) Design of reinforcement learning algorithm:

[0149] State (S): The feature vector of the current environment, including the time-frequency characteristics of the sound signal, the ambient noise level, etc.

[0150] Action (A): An operation to adjust model parameters, such as adjusting filter parameters, neural network weights, etc.

[0151] Reward (R): Instant feedback based on system performance, including recognition accuracy and howling suppression effectiveness. High accuracy and effective howling suppression will receive positive rewards, while negative rewards will be received otherwise.

[0152] Policy (π): A strategy for selecting actions based on the current state. Commonly used strategies include ε-greedy strategy and policy gradient method.

[0153] (2) Model training:

[0154] The model is trained using the reinforcement learning algorithm (Q-learning), which enables it to dynamically adjust parameters in different environments.

[0155] Q-learning: Select the optimal action by updating the state-action value function Q(s,a) to maximize the long-term cumulative reward.

[0156]

[0157] Among them, α is the learning rate, γ is the discount factor, r is the immediate reward, s′ is the next state, and a′ is the next action.

[0158] (3) Online adjustment:

[0159] During actual operation, the environmental adaptation module dynamically adjusts the model parameters based on the real-time collected environmental sound characteristics.

[0160] In the embodiment of the present application, by continuously collecting data and updating parameters, the system can gradually adapt to different environmental conditions and improve recognition accuracy and howling suppression effect.

[0161] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0162] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0163] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware using computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0164] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0165] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of an audio recognition device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0166] like Figure 4 As shown, the audio recognition device 200 of the embodiment of the present application includes:

[0167] The signal acquisition module 210 is used to acquire the sound signal collected by the sound collection device;

[0168] The preprocessing module 220 is used to preprocess the collected sound signal to obtain a preprocessed sound signal;

[0169] The short-term feature extraction module 230 is used to perform short-term feature extraction processing on the pre-processed sound signal to obtain short-term feature data;

[0170] A joint feature extraction module 240 is configured to perform joint feature extraction processing on the preprocessed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data;

[0171] The audio recognition module 250 is used to input the short-term feature data, the human voice feature data, and the environmental feature data into the deep neural network model for audio recognition processing to obtain an audio recognition result;

[0172] The result output module 260 is used to output the audio recognition result.

[0173] In an embodiment of the present application, an audio recognition device 200 is provided, comprising: a signal acquisition module 210 for acquiring a sound signal collected by a sound collection device; a preprocessing module 220 for preprocessing the sound signal to obtain a preprocessed sound signal; a short-term feature extraction module 230 for performing short-term feature extraction on the preprocessed sound signal to obtain short-term feature data; a joint feature extraction module 240 for performing joint feature extraction on the preprocessed sound signal based on a GMM-HMM joint model to obtain human voice feature data and environmental feature data; an audio recognition module 250 for inputting the short-term feature data, human voice feature data, and environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and a result output module 260 for outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, which can more comprehensively capture stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0174] In some optional implementations of the embodiments of the present application, the preprocessing module includes:

[0175] The noise removal submodule is used to perform noise removal processing on the collected sound signal according to the filter to obtain a denoised sound signal;

[0176] The signal strength determination submodule is used to obtain the signal strength information of the de-noised audio signal and determine whether the signal strength information is lower than a preset threshold;

[0177] A first strength judgment result submodule is configured to perform gain adjustment processing on the denoised sound signal to obtain a preprocessed sound signal if the signal strength information is lower than a preset threshold;

[0178] The second strength judgment result submodule is configured to use the denoised sound signal as a pre-processed sound signal if the signal strength information is not lower than a preset threshold.

[0179] In some optional implementations of the embodiments of the present application, the preprocessing module includes:

[0180] The conversion submodule is used to convert the time domain of the collected sound signal into the frequency domain;

[0181] The segmentation submodule is used to segment the frequency domain to obtain several sub-bands;

[0182] The howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result;

[0183] The first detection result submodule is configured to suppress the current sub-band according to a notch filter method to obtain a pre-processed sub-band if the howling noise is present in the howling detection result;

[0184] A second detection result submodule, configured to use the current sub-band as a pre-processing sub-band if the howling detection result shows that there is no howling noise;

[0185] The pre-processing confirmation submodule is used to combine all pre-processed sub-bands to obtain a pre-processed sound signal after all sub-bands have been subjected to the howling detection and suppression processing.

[0186] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device according to an embodiment of the present application.

[0187] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows the computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0188] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0189] The memory 310 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the audio recognition method. In addition, the memory 310 can also be used to temporarily store various data that has been output or is about to be output.

[0190] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to execute computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for executing the audio recognition method.

[0191] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0192] The computer device provided in this application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, and can more comprehensively capture the stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0193] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the audio recognition method as described above.

[0194] The computer-readable storage medium provided in this application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, which can more comprehensively capture the stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0195] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0196] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. An audio recognition method, characterized in that: The steps include: Acquiring a sound signal collected by a sound collection device; Preprocessing the collected sound signal to obtain a preprocessed sound signal; Performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; Performing joint feature extraction processing on the preprocessed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data, wherein the environmental feature data is sound feature data extracted from the speech signal; Inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; outputting the audio recognition result; The step of inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result specifically includes the following steps: Acquiring spectrum image data corresponding to the collected sound signal; Performing an image feature extraction operation on the spectrum image data according to a convolutional neural network of a multimodal feature fusion model to obtain image feature data; Performing a feature fusion operation on the short-term feature data, the human voice feature data, the environmental feature data, and the image feature data to obtain fused feature data; An audio recognition operation is performed on the fused feature data according to the deep neural network model to obtain the audio recognition result.

2. The audio recognition method according to claim 1, wherein: The step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps: Performing noise removal processing on the collected sound signal according to the filter to obtain a denoised sound signal; Obtaining signal strength information of the denoised audio signal, and determining whether the signal strength information is lower than a preset threshold; If the signal strength information is lower than the preset threshold, performing gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal; If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

3. The audio recognition method according to claim 1, wherein: The step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps: Converting the time domain of the collected sound signal to obtain the frequency domain; Segmenting the frequency domain to obtain a plurality of sub-frequency bands; Performing howling detection processing on the current sub-frequency band to obtain a howling detection result; If the howling detection result shows howling noise, suppressing the current sub-band according to a notch filter method to obtain a pre-processed sub-band; If the howling detection result shows that there is no howling noise, using the current sub-band as a pre-processing sub-band; After all the sub-frequency bands are subjected to the howling detection and suppression processes, all the pre-processed sub-bands are combined to obtain the pre-processed sound signal.

4. The audio recognition method according to claim 3, characterized in that The step of performing howling detection processing on the current sub-band to obtain a howling detection result specifically includes the following steps: Calculating the phase characteristic of the current sub-frequency band, and determining whether the phase characteristic meets a preset phase determination condition; If the phase characteristic satisfies the preset phase judgment condition, the value of the phase counter is increased by 1; If the phase characteristic does not meet the preset phase judgment condition, resetting the value of the phase counter to 0; Calculating the amplitude characteristic of the current sub-frequency band, and determining whether the amplitude characteristic meets a preset amplitude determination condition; If the amplitude characteristic satisfies the preset amplitude judgment condition, the value of the amplitude counter is increased by 1; If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0; Determining whether the value of the phase counter is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude counter is greater than or equal to the value of the amplitude detection time counter; If the value of the phase counter is greater than or equal to the value of the phase detection time counter and the value of the amplitude counter is greater than or equal to the value of the amplitude detection time counter, confirming the howling detection result of the current sub-band as the presence of howling noise; Otherwise, the howling detection result of the current sub-frequency band is confirmed as no howling noise exists.

5. The audio recognition method according to claim 1, wherein: After the step of inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result, the following step is also included: The parameters of the deep neural network model are adaptively adjusted according to the reinforcement learning algorithm and the short-term feature data.

6. An audio recognition device, characterized in that: include: A signal acquisition module is used to acquire the sound signal collected by the sound collection device; A preprocessing module, configured to preprocess the collected sound signal to obtain a preprocessed sound signal; A short-time feature extraction module is used to perform short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data; a joint feature extraction module, configured to perform joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data, wherein the environmental feature data is sound feature data extracted from the speech signal; An audio recognition module is used to input the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; A result output module, used to output the audio recognition result; The deep neural network model is a multimodal feature fusion model. The audio recognition module includes: a spectrum image acquisition submodule, an image feature extraction submodule, a feature fusion submodule and an audio recognition submodule, wherein: The spectrum image acquisition submodule is used to acquire spectrum image data corresponding to the collected sound signal; The image feature extraction submodule is used to perform an image feature extraction operation on the spectrum image data according to a convolutional neural network of a multimodal feature fusion model to obtain image feature data; The feature fusion submodule is used to perform a feature fusion operation on the short-term feature data, the human voice feature data, the environmental feature data, and the image feature data to obtain fused feature data; The audio recognition submodule is used to perform an audio recognition operation on the fused feature data according to the deep neural network model to obtain the audio recognition result.

7. The audio recognition device according to claim 6, characterized in that The pre-processing module comprises: A noise removal submodule, configured to perform noise removal processing on the collected sound signal according to a filter to obtain a denoised sound signal; a signal strength determination submodule, configured to obtain signal strength information of the denoised audio signal and determine whether the signal strength information is lower than a preset threshold; a first strength judgment result submodule, configured to perform gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal if the signal strength information is lower than the preset threshold; The second strength judgment result submodule is configured to use the denoised sound signal as the pre-processed sound signal if the signal strength information is not lower than the preset threshold.

8. The audio recognition device according to claim 7, characterized in that The pre-processing module comprises: A conversion submodule, configured to convert the time domain of the collected sound signal to obtain the frequency domain; A segmentation submodule, configured to segment the frequency domain to obtain a plurality of sub-frequency bands; The howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result; A first detection result submodule is configured to, if the howling detection result contains howling noise, suppress the current sub-band according to a notch filter method to obtain a pre-processed sub-band; A second detection result submodule, configured to use the current sub-band as a pre-processing sub-band if the howling detection result shows that there is no howling noise; The pre-processing confirmation submodule is used to combine all pre-processed sub-bands to obtain the pre-processed sound signal after all the sub-frequency bands have been subjected to the howling detection and suppression processing.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio recognition method according to any one of claims 1 to 5 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the audio recognition method according to any one of claims 1 to 5.

Images