Audio recognition method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of audio processing, and relates to an audio recognition method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a collection sound signal collected by sound collection equipment; preprocessing the collected sound signal to obtain a preprocessed sound signal; performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environment feature data; and inputting the short-time feature data, the human voice feature data and the environment feature data into a deep neural network model for audio recognition processing, and obtaining and outputting an audio recognition result. According to the method, the GMM-HMM joint compensation model is used, the features capable of representing the speaker and the environmental factors are extracted from the voice signals, the stable random factors in the voice signals can be captured more comprehensively, and therefore the robustness and adaptability of the model are improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio recognition method, device, computer equipment and storage medium. Background Art

[0002] Intelligent sound recognition technology plays an increasingly important role in the fields of modern information processing and artificial intelligence, and is widely used in voice assistants, smart homes, security monitoring, etc. Neural network models are particularly good at processing complex sound data.

[0003] There is an existing sound recognition method, which is to perform speech recognition on the short-term features of the collected audio signal through a deep neural network (DNN), so as to achieve the purpose of sound recognition.

[0004] However, the applicant has found that traditional voice recognition methods process speakers and environmental factors separately and fail to capture stable random factors in speech signals, thereby reducing the robustness and adaptability of the model and, in turn, reducing the accuracy of voice recognition. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an audio recognition method, apparatus, computer device and storage medium to solve the problem of low recognition accuracy in traditional sound recognition methods.

[0006] In order to solve the above technical problems, the present application provides an audio recognition method, which adopts the following technical solution: Acquire a sound signal collected by a sound collection device; Preprocessing the collected sound signal to obtain a preprocessed sound signal; Performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; Perform joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data; Inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; The audio recognition result is output.

[0007] Furthermore, the step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps: Performing noise removal processing on the collected sound signal according to the filter to obtain a de-noised sound signal; Obtaining signal strength information of the denoised audio signal, and determining whether the signal strength information is lower than a preset threshold; If the signal strength information is lower than the preset threshold, performing gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal; If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

[0008] Furthermore, the step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically includes the following steps: Converting the time domain of the collected sound signal to obtain the frequency domain; The frequency domain is segmented to obtain a plurality of sub-frequency bands; Performing howling detection processing on the current sub-frequency band to obtain a howling detection result; If the howling detection result contains howling noise, suppressing the current sub-frequency band according to a notch filter method to obtain a pre-processed sub-frequency band; If the howling detection result shows that there is no howling noise, using the current sub-frequency band as a pre-processing sub-frequency band; After all the sub-frequency bands are subjected to the howling detection and suppression processing, all the pre-processed sub-bands are combined to obtain the pre-processed sound signal.

[0009] Furthermore, the step of performing howling detection processing on the current sub-frequency band to obtain a howling detection result specifically includes the following steps: Calculating the phase characteristic of the current sub-frequency band, and determining whether the phase characteristic meets a preset phase determination condition; If the phase characteristic meets the preset phase judgment condition, the value of the phase counter is increased by 1; If the phase characteristic does not meet the preset phase judgment condition, resetting the value of the phase counter to 0; Calculating the amplitude characteristic of the current sub-frequency band, and determining whether the amplitude characteristic meets a preset amplitude determination condition; If the amplitude characteristic meets the preset amplitude judgment condition, the value of the amplitude counter is increased by 1; If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0; Determine whether the value of the phase detector is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter; If the value of the phase detector is greater than or equal to the value of the phase detection time counter and the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter, the howling detection result of the current sub-band is confirmed as the presence of howling noise; Otherwise, the howling detection result of the current sub-frequency band is confirmed as the absence of howling noise.

[0010] Furthermore, after the step of inputting the short-time feature data, the human voice feature data and the environmental feature data into the deep neural network model for audio recognition processing to obtain the audio recognition result, the following steps are also included: The parameters of the deep neural network model are adaptively adjusted according to the reinforcement learning algorithm and the short-term feature data.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides an audio recognition device, which adopts the following technical solution: A signal acquisition module is used to acquire the collected sound signal collected by the sound collection device; A preprocessing module, used for preprocessing the collected sound signal to obtain a preprocessed sound signal; A short-time feature extraction module is used to perform short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data; A joint feature extraction module, used for performing joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data; An audio recognition module, used for inputting the short-term feature data, the human voice feature data and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; The result output module is used to output the audio recognition result.

[0012] Furthermore, the preprocessing module includes: A noise removal submodule, configured to perform noise removal processing on the collected sound signal according to a filter to obtain a de-noised sound signal; A signal strength determination submodule, used to obtain the signal strength information of the de-noised sound signal and determine whether the signal strength information is lower than a preset threshold; A first strength judgment result submodule, configured to perform gain adjustment processing on the denoised sound signal to obtain the pre-processed sound signal if the signal strength information is lower than the preset threshold; The second strength judgment result submodule is used to use the denoised sound signal as the pre-processed sound signal if the signal strength information is not lower than the preset threshold.

[0013] Furthermore, the preprocessing module includes: A conversion submodule, used for converting the time domain of the collected sound signal to obtain the frequency domain; A segmentation submodule, used for segmenting the frequency domain to obtain a plurality of sub-frequency bands; A howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result; A first detection result submodule, configured to suppress the current sub-frequency band according to a notch filter method to obtain a pre-processed sub-frequency band if the howling detection result contains howling noise; A second detection result submodule, configured to use the current sub-frequency band as a pre-processing sub-frequency band if the howling detection result shows that there is no howling noise; The preprocessing confirmation submodule is used to combine all preprocessing subbands to obtain the preprocessing sound signal after all the sub-frequency bands have been subjected to the howling detection processing and suppression processing.

[0014] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio recognition method as described above when executing the computer-readable instructions.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio recognition method described above are implemented.

[0016] The present application provides an audio recognition method, including: obtaining a collected sound signal collected by a sound collection device; preprocessing the collected sound signal to obtain a preprocessed sound signal; performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; inputting the short-time feature data, the human voice feature data and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent speakers and environmental factors from speech signals, and can more comprehensively capture stable random factors in speech signals, thereby improving the robustness and adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is an exemplary system architecture diagram to which the present application may be applied; Figure 2 is a flowchart of the implementation of the audio recognition method provided in Example 1 of the present application; Figure 3 is an average time graph of the continuous increase in signal amplitude during the howling process provided in the first embodiment of the present application; Figure 4 is a structural diagram of an audio recognition device provided in Embodiment 2 of the present application; Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0020] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0022] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0024] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0025] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0026] It should be noted that the audio recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the audio recognition device is generally arranged in the server / terminal device.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0028] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio recognition method according to the present application. The audio recognition method comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0029] In step S201, a sound signal collected by a sound collection device is obtained.

[0030] In the embodiment of the present application, the ambient sound signal can be collected by a high-sensitivity microphone.

[0031] In step S202, the collected sound signal is preprocessed to obtain a preprocessed sound signal.

[0032] In an embodiment of the present application, the collected sound signal may be preprocessed, including denoising, gain control, and normalization processing.

[0033] In step S203, short-time feature extraction is performed on the pre-processed sound signal to obtain short-time feature data.

[0034] In the embodiment of the present application, the deep neural network is mainly used to obtain short-term feature information in the sound signal.

[0035] In step S204, joint feature extraction processing is performed on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data.

[0036] In the embodiment of the present application, a GMM-HMM joint compensation model is used to extract features that can represent the speaker and environmental factors from the speech signal.

[0037] In the embodiment of the present application, the above-mentioned joint feature extraction process further includes: 1. Short-time Fourier transform (STFT): Using short-time Fourier transform for time-frequency analysis in frequency domain conversion can capture the time-frequency characteristics of sound signals in more detail.

[0038] 2. Formula description:

[0039] in: is the time-frequency representation; is the input signal; is a window function; For time; is the frequency.

[0040] In step S205, the short-term feature data, the human voice feature data and the environmental feature data are input into the deep neural network model for audio recognition processing to obtain an audio recognition result.

[0041] In an embodiment of the present application, after acquiring short-time feature data, human voice feature data, and environmental feature data, these features are combined with the short-time features and input into a deep neural network for learning to implement audio recognition processing operations.

[0042] In an embodiment of the present application, a multimodal feature fusion model (audio + spectrum image) can be used to identify non-speech audio signals (such as environmental sounds, music signals, etc.), wherein the multimodal model combines the features of the audio signal and the spectrum image, processes the spectrum image through a convolutional neural network (CNN), extracts image-level features, and then fuses them with the audio features.

[0043] Audio and image feature fusion:

[0044] in, is the audio feature; is the image feature; is the fusion coefficient.

[0045] Activation function for convolution processing: ReLU can be used:

[0046] In the embodiment of the present application, by introducing a multimodal model, the system's recognition rate for non-speech signals such as ambient sound and music has been increased by 20%. In the experiment, in a test of a group of outdoor ambient sounds, the recognition accuracy rate increased from the original 70% to 85%.

[0047] In step S206, the audio recognition result is output.

[0048] In the embodiment of the present application, when the current system processes a large amount of real-time audio data, especially in the scenario of multi-tasking, the processing speed may decrease. To address the problem of decreased processing speed, parallel computing and distributed processing architecture can be used: Parallel model training: speed up the training and prediction process through distributed neural network frameworks (such as TensorFlow or PyTorch's distributed computing capabilities); Data parallel processing:

[0049] in: is the i-th data block; Sub-models that are executed in parallel.

[0050] Parameter synchronization:

[0051] This formula shows that the gradients calculated by each sub-model can be accumulated in parallel to update the global parameters.

[0052] In the embodiment of the present application, through parallel processing, the system reduces the processing time of large-scale audio stream tasks by 40%. In the processing task of real-time audio stream, the delay is reduced from 500ms to 300ms, which significantly improves the response speed.

[0053] In an embodiment of the present application, an audio recognition method is provided, including: obtaining a collected sound signal collected by a sound collection device; preprocessing the collected sound signal to obtain a preprocessed sound signal; performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; inputting the short-time feature data, human voice feature data and environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent speakers and environmental factors from speech signals, and can more comprehensively capture stable random factors in speech signals, thereby improving the robustness and adaptability of the model.

[0054] In some optional implementations of the embodiments of the present application, the above step S202 specifically includes the following steps: Perform noise removal processing on the collected sound signal according to the filter to obtain a de-noised sound signal; Obtaining signal strength information of the de-noised audio signal, and determining whether the signal strength information is lower than a preset threshold; If the signal strength information is lower than a preset threshold, the de-noised sound signal is subjected to gain adjustment processing to obtain a pre-processed sound signal; If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

[0055] In the embodiment of the present application, since the ambient sound signal collected by the microphone often contains certain interference information, it is necessary to pre-process the collected sound signal to obtain a sound signal with fewer interference factors, specifically: First, the collected sound signal is preprocessed by removing noise above 8000 Hz and below 300 Hz through a bandpass filter, thereby obtaining a relatively clean preprocessed sound signal; next, according to the signal strength of the preprocessed sound signal, its average strength is calculated using a sliding window. Assuming that the set threshold is -20 decibels, if the signal strength is lower than this threshold, a linear gain function, such as G(x)=2x, is applied to adjust its gain.

[0056] In some optional implementations of the embodiments of the present application, a short-time Fourier transform (STFT) can be used to perform spectral analysis on the gain-adjusted sound signal, and a 256-point fast Fourier transform (FFT) window can be used to obtain a detailed distribution diagram of the frequency range; through the spectral analysis results, we can determine the main frequency components of the sound signal; for example, if the main frequency components are concentrated between 1000-3000 Hz, we can determine whether there are any abnormal frequency components. Assuming that an abnormal frequency of 400 Hz is detected, it is filtered out by a narrowband filter.

[0057] Get the sound signal after filtering and calculate its signal-to-noise ratio (SNR). Assume that the set threshold is 30 decibels. If the SNR is lower than this threshold, spectral subtraction is used for noise suppression.

[0058] By analyzing the sound signal after noise suppression processing, its timbre characteristics are determined, for example, by extracting timbre characteristics through Mel-frequency cepstral coefficients (MFCCs), and these characteristics are used as reference parameters for subsequent processing.

[0059] If the sound signal contains speech, a speech recognition algorithm such as the Hidden Markov Model (HMM) is applied to extract speech feature parameters to obtain fundamental frequency (F0) and formants information.

[0060] Based on the fundamental frequency and formant information, speech enhancement technology such as Wiener filtering is used to optimize the sound signal and finally obtain a high-quality processed sound signal.

[0061] In the embodiment of the present application, this high-quality sound signal can be used for subsequent speech recognition, thereby realizing a more intelligent and efficient sound processing system.

[0062] In some optional implementations of the embodiments of the present application, an adaptive filtering algorithm may be introduced to propose an adaptive noise suppression method. The algorithm dynamically adjusts the filtering parameters according to different noise environments to more effectively remove noise. The algorithm is expressed as:

[0063] in, is the denoised signal; is the input signal; is the estimated noise component.

[0064] In some optional implementations of the embodiments of the present application, the above step S202 specifically includes the following steps: Convert the time domain of the collected sound signal to obtain the frequency domain; The frequency domain is divided into several sub-bands; Performing howling detection processing on the current sub-frequency band to obtain a howling detection result; If the howling detection result has howling noise, the current sub-band is suppressed according to the notch filter method to obtain a pre-processed sub-band; If the howling detection result shows that there is no howling noise, the current sub-band is used as a pre-processing sub-band; After all sub-bands are subjected to the howling detection and suppression processing, all pre-processed sub-bands are combined to obtain a pre-processed sound signal.

[0065] In the embodiment of the present application, the process of "stopping the detection module after detecting howling and only performing signal suppression" is abandoned, and it is changed to "after detecting howling, continuing howling detection while performing signal suppression". This method effectively shortens the time of howling detection. In the selection of howling suppression algorithm, the present invention adopts the method of multiplying the howling sub-band by a smaller gain factor in the frequency domain, which belongs to the notch filter method.

[0066] In some optional implementations of the embodiments of the present application, the step of performing howling detection processing on the current sub-frequency band to obtain a howling detection result specifically includes the following steps: Calculate the phase characteristics of the current sub-frequency band, and determine whether the phase characteristics meet the preset phase determination conditions; If the phase characteristic meets the preset phase judgment condition, the value of the phase counter is increased by 1; If the phase characteristic does not meet the preset phase judgment condition, the value of the phase counter is reset to 0; Calculate the amplitude characteristics of the current sub-frequency band, and determine whether the amplitude characteristics meet the preset amplitude determination conditions; If the amplitude characteristic meets the preset amplitude judgment condition, the value of the amplitude counter is increased by 1; If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0; Determine whether the value of the phase detector is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter; If the value of the phase detector is greater than or equal to the value of the phase detection time counter and the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter, the howling detection result of the current sub-band is confirmed as the presence of howling noise; Otherwise, the howling detection result of the current sub-frequency band is confirmed as the absence of howling noise.

[0067] In the embodiment of the present application, the original judgment condition of "signal energy continuously increases" is abandoned, and the "ratio of sub-band signal energy to the sum of all sub-band energy values" is detected instead. Experimental statistics show that the phenomenon of "signal energy continuously increases" only lasts for about 75 milliseconds when the system enters the oscillation state from the normal working state. The average result obtained on the waveform of the four-segment howling signal is as follows: Figure 3 As shown. Therefore, using this condition as the basis for judging howling cannot guarantee the reliability of the algorithm. From the spectrogram of the original howling signal, it can be seen that most of the signal energy is concentrated in the howling sub-band, so it is more effective to use "the proportion of sub-band signal energy to the sum of all sub-band energy values" as the feature for detecting howling.

[0068] In some optional implementations of the embodiments of the present application, after step S205, the following steps are further included: The parameters of the deep neural network model are adaptively adjusted according to the reinforcement learning algorithm and short-term feature data.

[0069] In the embodiment of the present application, the above extracted features are used as input to the reinforcement learning algorithm, where: (1) Design of reinforcement learning algorithm: State (S): The feature vector of the current environment, including the time-frequency characteristics of the sound signal, the ambient noise level, etc.

[0070] Action (A): An operation to adjust model parameters, such as adjusting filter parameters, neural network weights, etc.

[0071] Reward (R): Instant feedback based on system performance, including recognition accuracy, howling suppression effect, etc. High accuracy and effective howling suppression will receive positive rewards, and vice versa.

[0072] Policy (π): A strategy for selecting actions based on the current state. Commonly used ones include ε-greedy strategy and policy gradient method.

[0073] (2) Model training: The model is trained using a reinforcement learning algorithm (Q-learning) so that it can dynamically adjust parameters in different environments.

[0074] Q-learning: Select the optimal action by updating the state-action value function Q(s,a) to maximize the long-term cumulative reward.

[0075]

[0076] Among them, α is the learning rate, γ is the discount factor, r is the immediate reward, s′ is the next state, and a′ is the next action.

[0077] (3) Online adjustment: During actual operation, the environmental adaptation module dynamically adjusts the model parameters according to the real-time collected environmental sound characteristics.

[0078] In the embodiment of the present application, by continuously collecting data and updating parameters, the system can gradually adapt to different environmental conditions and improve the recognition accuracy and howling suppression effect.

[0079] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0080] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0081] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0082] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0083] Further references Figure 4, as a response to the above Figure 2 The present application provides an embodiment of an audio recognition device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0084] like Figure 4 As shown, the audio recognition device 200 of the embodiment of the present application includes: The signal acquisition module 210 is used to acquire the collected sound signal collected by the sound collection device; The preprocessing module 220 is used to preprocess the collected sound signal to obtain a preprocessed sound signal; The short-time feature extraction module 230 is used to perform short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data; A joint feature extraction module 240 is used to perform joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data; The audio recognition module 250 is used to input the short-term feature data, the human voice feature data and the environmental feature data into the deep neural network model for audio recognition processing to obtain an audio recognition result; The result output module 260 is used to output the audio recognition result.

[0085] In an embodiment of the present application, an audio recognition device 200 is provided, including: a signal acquisition module 210, for acquiring a collected sound signal collected by a sound collection device; a preprocessing module 220, for preprocessing the collected sound signal to obtain a preprocessed sound signal; a short-time feature extraction module 230, for performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; a joint feature extraction module 240, for performing joint feature extraction processing on the preprocessed sound signal according to a GMM-HMM joint model to obtain human voice feature data and environmental feature data; an audio recognition module 250, for inputting the short-time feature data, human voice feature data and environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; and a result output module 260, for outputting the audio recognition result. Compared with the prior art, the present application uses a GMM-HMM joint compensation model to extract features that can represent speakers and environmental factors from speech signals, and can more comprehensively capture stable random factors in speech signals, thereby improving the robustness and adaptability of the model.

[0086] In some optional implementations of the embodiments of the present application, the preprocessing module includes: The noise removal submodule is used to perform noise removal processing on the collected sound signal according to the filter to obtain a de-noised sound signal; The signal strength determination submodule is used to obtain the signal strength information of the de-noised sound signal and determine whether the signal strength information is lower than a preset threshold; A first strength judgment result submodule is used to perform gain adjustment processing on the de-noised sound signal to obtain a pre-processed sound signal if the signal strength information is lower than a preset threshold; The second strength judgment result submodule is used to use the denoised sound signal as a pre-processed sound signal if the signal strength information is not lower than a preset threshold.

[0087] In some optional implementations of the embodiments of the present application, the preprocessing module includes: The conversion submodule is used to convert the time domain of the collected sound signal to obtain the frequency domain; A segmentation submodule is used to segment the frequency domain to obtain a number of sub-frequency bands; A howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result; The first detection result submodule is used for suppressing the current sub-band according to the notch filter method to obtain a pre-processed sub-band if the howling noise exists in the howling detection result; A second detection result submodule, configured to use the current sub-frequency band as a pre-processing sub-frequency band if the howling detection result shows that there is no howling noise; The preprocessing confirmation submodule is used to combine all preprocessing subbands to obtain a preprocessing sound signal after all sub-frequency bands have been subjected to the howling detection and suppression processing.

[0088] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0089] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0090] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0091] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer readable instructions of the audio recognition method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or will be output.

[0092] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the audio recognition method.

[0093] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0094] The computer device provided in the present application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, and can more comprehensively capture stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0095] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the audio recognition method as described above.

[0096] The computer-readable storage medium provided in the present application uses a GMM-HMM joint compensation model to extract features that can represent the speaker and environmental factors from the speech signal, and can more comprehensively capture the stable random factors in the speech signal, thereby improving the robustness and adaptability of the model.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0098] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An audio recognition method, characterized in that: The steps include: Acquire a sound signal collected by a sound collection device; Preprocessing the collected sound signal to obtain a preprocessed sound signal; Performing short-time feature extraction processing on the preprocessed sound signal to obtain short-time feature data; Perform joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data; Inputting the short-term feature data, the human voice feature data, and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; The audio recognition result is output.

2. The audio recognition method according to claim 1, characterized in that: The step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically comprises the following steps: Performing noise removal processing on the collected sound signal according to the filter to obtain a de-noised sound signal; Obtaining signal strength information of the denoised audio signal, and determining whether the signal strength information is lower than a preset threshold; If the signal strength information is lower than the preset threshold, performing gain adjustment processing on the denoised sound signal to obtain the preprocessed sound signal; If the signal strength information is not lower than the preset threshold, the denoised sound signal is used as the preprocessed sound signal.

3. The audio recognition method according to claim 1, characterized in that: The step of preprocessing the collected sound signal to obtain a preprocessed sound signal specifically comprises the following steps: Converting the time domain of the collected sound signal to obtain the frequency domain; The frequency domain is segmented to obtain a plurality of sub-frequency bands; Performing howling detection processing on the current sub-frequency band to obtain a howling detection result; If the howling detection result contains howling noise, suppressing the current sub-frequency band according to a notch filter method to obtain a pre-processed sub-frequency band; If the howling detection result shows that there is no howling noise, using the current sub-frequency band as a pre-processing sub-frequency band; After all the sub-frequency bands are subjected to the howling detection and suppression processing, all the pre-processed sub-bands are combined to obtain the pre-processed sound signal.

4. The audio recognition method according to claim 3, characterized in that: The step of performing howling detection processing on the current sub-frequency band to obtain a howling detection result specifically includes the following steps: Calculating the phase characteristic of the current sub-frequency band, and determining whether the phase characteristic meets a preset phase determination condition; If the phase characteristic meets the preset phase judgment condition, the value of the phase counter is increased by 1; If the phase characteristic does not meet the preset phase judgment condition, resetting the value of the phase counter to 0; Calculating the amplitude characteristic of the current sub-frequency band, and determining whether the amplitude characteristic meets a preset amplitude determination condition; If the amplitude characteristic meets the preset amplitude judgment condition, the value of the amplitude counter is increased by 1; If the amplitude characteristic does not meet the preset amplitude judgment condition, the value of the amplitude counter is reset to 0; Determine whether the value of the phase detector is greater than or equal to the value of the phase detection time counter, and whether the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter; If the value of the phase detector is greater than or equal to the value of the phase detection time counter and the value of the amplitude detector is greater than or equal to the value of the amplitude detection time counter, the howling detection result of the current sub-band is confirmed as the presence of howling noise; Otherwise, the howling detection result of the current sub-frequency band is confirmed as the absence of howling noise.

5. The audio recognition method according to claim 1, characterized in that: After the step of inputting the short-time feature data, the human voice feature data and the environmental feature data into the deep neural network model for audio recognition processing to obtain the audio recognition result, the following step is also included: The parameters of the deep neural network model are adaptively adjusted according to the reinforcement learning algorithm and the short-term feature data.

6. An audio recognition device, characterized in that: include: A signal acquisition module is used to acquire the collected sound signal collected by the sound collection device; A preprocessing module, used for preprocessing the collected sound signal to obtain a preprocessed sound signal; A short-time feature extraction module is used to perform short-time feature extraction processing on the pre-processed sound signal to obtain short-time feature data; A joint feature extraction module, used for performing joint feature extraction processing on the pre-processed sound signal according to the GMM-HMM joint model to obtain human voice feature data and environmental feature data; An audio recognition module, used for inputting the short-term feature data, the human voice feature data and the environmental feature data into a deep neural network model for audio recognition processing to obtain an audio recognition result; The result output module is used to output the audio recognition result.

7. The audio recognition device according to claim 6, characterized in that: The preprocessing module comprises: A noise removal submodule, configured to perform noise removal processing on the collected sound signal according to a filter to obtain a de-noised sound signal; A signal strength determination submodule, used to obtain the signal strength information of the de-noised sound signal and determine whether the signal strength information is lower than a preset threshold; A first strength judgment result submodule, configured to perform gain adjustment processing on the denoised sound signal to obtain the pre-processed sound signal if the signal strength information is lower than the preset threshold; The second strength judgment result submodule is used to use the denoised sound signal as the pre-processed sound signal if the signal strength information is not lower than the preset threshold.

8. The audio recognition device according to claim 7, characterized in that: The preprocessing module comprises: A conversion submodule, used for converting the time domain of the collected sound signal to obtain the frequency domain; A segmentation submodule, used for segmenting the frequency domain to obtain a plurality of sub-frequency bands; A howling detection submodule is used to perform howling detection processing on the current sub-frequency band to obtain a howling detection result; A first detection result submodule, configured to suppress the current sub-frequency band according to a notch filter method to obtain a pre-processed sub-frequency band if the howling detection result contains howling noise; A second detection result submodule, configured to use the current sub-frequency band as a pre-processing sub-frequency band if the howling detection result shows that there is no howling noise; The preprocessing confirmation submodule is used to combine all preprocessing subbands to obtain the preprocessing sound signal after all the sub-frequency bands are subjected to the howling detection processing and suppression processing.

9. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio recognition method according to any one of claims 1 to 5 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio recognition method according to any one of claims 1 to 5 are implemented.

Images