Equipment state detection method and device based on audio analysis, equipment and storage medium

Abstract

The invention relates to an equipment state detection method and device based on audio analysis, equipment and a storage medium, and the method comprises the steps: carrying out the multi-dimensional feature extraction and Mel frequency feature extraction of a processed audio signal, and obtaining an audio signal feature and a Mel frequency feature; performing time sequence analysis according to the hardware information time sequence data to obtain hardware time sequence characteristics, and performing comprehensive prediction on the state of the current equipment according to the audio signal characteristics, the Mel frequency characteristics and the hardware time sequence characteristics by using the equipment state detection neural network to obtain a comprehensive prediction result; the comprehensive prediction result is a state evaluation result of the current equipment, and corresponding feedback is carried out according to the state evaluation result, so that real-time state detection of the sound control equipment is realized. By adopting the method, the requirements of accuracy and high efficiency in sound control equipment state detection can be met.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a device state detection method, apparatus, device and storage medium based on audio analysis. Background Art

[0002] With the rapid development of artificial intelligence technology, voice recognition has been widely used as an important interaction method for smart devices such as smart homes, car systems, smart speakers, and voice assistants. The voice control system can control the operation of the device by recognizing the user's voice commands, which greatly improves the user experience. However, the current voice control system still faces many challenges in practical applications, especially in equipment fault detection and audio signal quality assurance.

[0003] Device hardware failure is an important factor affecting the stability of voice control systems. Current voice control systems have obvious deficiencies in hardware fault detection. Most systems rely mainly on traditional hardware diagnostic tools, such as manual inspection, software restart, and other methods to troubleshoot hardware problems. These methods are not only inefficient, but also difficult to monitor the health of the hardware in real time. Common hardware failures, such as microphone failure, sound card problems, or connection line failures, are usually only discovered when the device fails to respond normally, resulting in the system failing to respond in time after the failure occurs.

[0004] At present, most fault detection methods lack dynamic monitoring of hardware health status and cannot evaluate the device status in real time. In particular, when the device status is abnormal but not completely failed, users are usually unable to detect and take measures. This delayed feedback not only affects the user experience, but may also cause the device to be in an unstable state for a long time, or even cause major failures. Summary of the invention

[0005] Based on this, it is necessary to provide a device status detection method, apparatus, device and storage medium based on audio analysis that can meet the requirements of accuracy and efficiency in response to the above technical problems.

[0006] A device state detection method based on audio analysis, the method being applied to a voice-controlled device, comprising:

[0007] Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal;

[0008] Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0009] Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics;

[0010] Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0011] Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0012] In one embodiment, the preprocessing of the ambient audio signal includes: performing noise suppression and gain control processing on the audio signal to obtain the processed audio signal.

[0013] In one embodiment, the time domain features include amplitude information and zero crossing rate;

[0014] The frequency domain feature is a spectrum envelope.

[0015] In one embodiment, when extracting Mel frequency features from the processed audio signal, the method includes:

[0016] After the processed audio signal is framed, short-time Fourier transform is performed on each frame signal to obtain frequency information;

[0017] The frequency information is processed by a Mel filter to obtain a Mel spectrum, and a logarithmic Mel spectrum feature of the Mel spectrum is extracted;

[0018] The Mel-frequency cepstral coefficients in the logarithmic Mel-frequency spectrum feature are extracted by using discrete cosine transform, and a preset number of Mel-frequency cepstral coefficients are selected as the Mel-frequency feature.

[0019] In one embodiment, the Mel filter obtains the Mel spectrum by weighting the spectrum of the signal using a set of filters on the Mel frequency scale on the frequency information;

[0020] The frequency information is mapped to the Mel frequency scale using the following formula:

[0021]

[0022] In the above formula, f represents the frequency information, and M(f) is the frequency information in the Mel scale.

[0023] In one embodiment, the Mel-frequency cepstrum coefficients in the logarithmic Mel-frequency spectrum feature are extracted, and the discrete cosine transform used is expressed as:

[0024]

[0025] In the above formula, C k represents the kth Mel frequency cepstrum coefficient, L m represents the mth value in the log-Mel spectrum, where M is the number of Mel filters.

[0026] In one embodiment, performing timing analysis according to the hardware information timing data to obtain hardware timing characteristics includes:

[0027] According to the hardware information timing data, the response time of the hardware is calculated and the response delay is obtained, and the response delay of the hardware is used as the hardware timing feature.

[0028] The present application also provides a device state detection device based on audio analysis, characterized in that the device comprises:

[0029] An audio signal acquisition and preprocessing acquisition module is used to acquire an audio signal, preprocess the ambient audio signal, and obtain a processed audio signal;

[0030] A multi-dimensional audio signal feature and Mel frequency feature extraction module, used to perform multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0031] A hardware feature extraction module is used to obtain hardware information timing data, and perform timing analysis on the hardware information timing data to obtain hardware timing features;

[0032] A comprehensive evaluation module, used to use a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0033] The status diagnosis and feedback module is used to provide corresponding feedback according to the comprehensive prediction result to realize real-time status detection of the voice control device.

[0034] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal;

[0036] Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0037] Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics;

[0038] Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0039] Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal;

[0042] Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0043] Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics;

[0044] Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0045] Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0046] The above-mentioned device state detection method, apparatus, device and storage medium based on audio analysis respectively extract multi-dimensional features and Mel frequency features from the processed audio signal to obtain audio signal features and Mel frequency features, perform time series analysis based on hardware information time series data to obtain hardware time series features, use the device state detection neural network to comprehensively predict the state of the current device based on the audio signal features, Mel frequency features and hardware time series features to obtain a comprehensive prediction result, and provide corresponding feedback based on the state evaluation result to achieve real-time state detection of the voice-controlled device. The use of this method can meet the requirements of accuracy and efficiency in voice-controlled device state detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of a device status detection method based on audio analysis in one embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of a device status detection method based on audio analysis in another embodiment;

[0049] Figure 3 is a structural block diagram of a device state detection apparatus based on audio analysis in one embodiment;

[0050] Figure 4 is a structural block diagram of a device status detection system based on audio analysis in one embodiment;

[0051] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The existing fault detection technologies for voice control systems mainly include: fault detection based on speech recognition error rate. Some systems infer the occurrence of faults by monitoring the changes in the accuracy of speech recognition; however, this method usually relies on the setting of high error rate thresholds, which makes it difficult to respond promptly in the early stages of a fault or when a minor fault occurs. Single audio processing method. Existing noise suppression and echo cancellation technologies are mostly optimized for specific types of noise or echo, and are difficult to cope with complex and changing sound environments. In addition, most existing technologies rely on static models and have poor adaptability to environmental changes or equipment failures; hardware monitoring. Existing hardware monitoring methods are mostly offline fault detection and fail to monitor the working status of the equipment in real time. Equipment failures are usually discovered when the system has already been affected. Such detection methods cannot respond to potential problems caused by hardware failures in a timely manner.

[0054] In response to the above problems, Figure 1 As shown, a device status detection method based on audio analysis is provided, which specifically includes the following steps:

[0055] Step S100: Acquire an audio signal, pre-process the ambient audio signal, and obtain a processed audio signal.

[0056] Step S110, performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features.

[0057] Step S120, acquiring hardware information timing data, and performing timing analysis based on the hardware information timing data to obtain hardware timing characteristics.

[0058] Step S130, using the device state detection neural network to comprehensively predict the state of the current device according to the audio signal characteristics, Mel frequency characteristics and hardware timing characteristics to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features of the audio signal characteristics, a long short-term memory network for extracting features of the Mel frequency characteristics and the hardware timing characteristics, and a category prediction network for comprehensively predicting the features output by the convolutional neural network and the long short-term memory network.

[0059] Step S140: providing corresponding feedback according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0060] In this application, through multi-dimensional audio signal feature extraction, combined with multiple features such as Mel-frequency cepstral coefficients (MFCC), the quality of the audio signal can be evaluated more comprehensively and accurately. At the same time, by real-time monitoring of the hardware in the voice control system to obtain timing characteristics, machine learning technology is further introduced to analyze the audio signal characteristics and the timing characteristics of the hardware to identify and predict possible faults in advance. At the same time, it can also actively adjust relevant parameters (such as gain control, noise suppression, etc.) according to the diagnosis results, or automatically issue a fault warning to prompt the user to take appropriate measures, thereby reducing equipment downtime and fault repair costs.

[0061] In step S100, an audio signal (which may include ambient sound) is collected by a high-sensitivity microphone array or a single microphone device. The audio collector collects ambient audio signals at a sampling frequency of 20kHz and saves the collected audio signals as files of 3 minutes each to ensure data integrity.

[0062] Specifically, assuming that the original signal collected is x(t), where t represents time and x(t) is the amplitude of the signal in the time domain, the original signal collected is first subjected to noise suppression to remove background noise or interference signals in the audio to ensure that the signal for subsequent processing is cleaner. Then, gain control is applied to uniformly adjust the amplitude of the audio signal to avoid affecting the processing effect due to volume differences.

[0063] Specifically, the gain control formula is as follows:

[0064] x processed (t) = G(t)·x(t)

[0065] In the above formula, G(t) represents the gain factor, which is usually adjusted according to the real-time environmental noise level and audio signal strength to ensure that the signal is within the appropriate range.

[0066] Furthermore, after noise reduction and gain control of the audio signal, endpoint detection is performed to accurately locate the starting and ending points of the effective signal in the audio, providing an accurate input basis for subsequent slicing and analysis. This sequence can maximize the efficiency and accuracy of processing. Finally, the processed audio signal is cut into independent segments of 8 seconds each to ensure that each audio segment contains effective signals.

[0067] In step S110, when performing multi-dimensional audio signal feature extraction on the pre-processed audio signal, time domain features and frequency domain features of the audio signal are extracted, wherein the time domain features include amplitude information and zero-crossing rate, and the frequency domain features are spectrum envelopes.

[0068] Specifically, time domain features include signal amplitude, waveform changes, etc. The stability of the audio signal is analyzed by calculating the signal energy and zero-crossing rate. Common time domain features include:

[0069] Signal amplitude calculation:

[0070] A peak =max(x(t))

[0071]

[0072] In the above formula, A peak is the maximum amplitude of the signal, A rsm represents the RMS amplitude of the signal, and N represents the signal length.

[0073] Furthermore, the zero-crossing rate (ZCR) measures the rate of change of the signal, which is particularly suitable for the analysis of speech signals. The calculation formula is:

[0074]

[0075] In the above formula, F(·) is an indicator function that returns 1 if the condition is met, otherwise it returns 0.

[0076] Specifically, the frequency domain feature converts the time domain signal into the frequency domain through Fourier transform to evaluate the frequency distribution and changes of the signal. The calculation formula of Fourier transform is:

[0077]

[0078] In the above formula, X(f) represents the frequency domain signal, f represents the frequency, and T represents the sampling point. Through Fourier transform, the frequency spectrum characteristics of the audio signal can be obtained to analyze the frequency response and distortion problems.

[0079] Furthermore, the spectrum envelope is used to analyze the low-frequency and high-frequency components in the signal to help detect signal attenuation or distortion. Specifically, by analyzing the amplitude and phase of the spectrum, it can be determined whether the audio signal has a frequency offset.

[0080] In this embodiment, the frequency domain feature data is normalized to ensure the consistency and robustness of the feature data. This includes normalizing the spectrum envelope and frequency domain data. The spectrum envelope is a frequency domain feature that represents the envelope curve of the frequency component changing with frequency. Normalizing it can adjust its amplitude range (for example, normalizing the maximum value to 1) in order to compare the spectrum shapes of different signals. Normalizing the frequency domain data involves the entire frequency domain features, including but not limited to the spectrum envelope. For example, it may also include features such as power spectrum, frequency centroid, bandwidth, etc. The purpose of normalization is to adjust all frequency domain features to a standardized range (such as [0,1] or standard normal distribution) to eliminate the impact of differences in feature value ranges.

[0081] In this embodiment, the Mel frequency feature extraction is also performed on the processed audio signal, including: firstly dividing the processed audio signal into frames, performing short-time Fourier transform on each frame signal to obtain frequency information, using a Mel filter to process the frequency information to obtain a Mel spectrum, extracting the logarithmic Mel spectrum feature of the Mel spectrum, using discrete cosine transform to extract the Mel frequency cepstral coefficients in the logarithmic Mel spectrum feature, and selecting a preset number of Mel frequency cepstral coefficients as the Mel frequency feature.

[0082] Mel-frequency cepstral coefficients (MFCC) can effectively extract the frequency characteristics of audio signals and simulate the auditory characteristics of the human ear, so they play an important role in the quality analysis of speech signals.

[0083] Specifically, the processed audio signal is first divided into frames, and each frame signal is short-time Fourier transformed to obtain a frequency domain representation. The spectrum of each frame contains the frequency distribution information of the audio signal in that time period. Then the frequency response of the human ear is simulated by mapping the spectrum to the Mel scale. The Mel scale adjusts the frequency resolution to be more in line with the perception of the human ear, with finer distinctions for low frequencies and coarser distinctions for high frequencies. Among them, the following formula is used to map the frequency information to the Mel frequency scale:

[0084]

[0085] In the above formula, f represents the frequency information, and M(f) is the frequency information in the Mel scale.

[0086] In this embodiment, a Mel filter is used to obtain a Mel spectrum according to frequency information, and a set of filters (Mel filter bank) is used on the Mel frequency scale to weight the spectrum of the signal. Each Mel filter weights different frequency bands of the signal and outputs the weighted spectrum. Through this process, the Mel spectrum of each frame of the signal can be obtained.

[0087] Next, a logarithmic transformation is applied to the Mel spectrum to simulate the way the human ear perceives sound. The logarithmic transformation can compress the dynamic range of the signal and highlight the important features in the speech. The formula for calculating the logarithmic Mel spectrum is:

[0088] L(t,f)=log(M(t,f))

[0089] In the above formula, L(t,f) represents the logarithmic Mel spectrum, and M(t,f) is the spectrum after Mel filtering, that is, the Mel spectrum.

[0090] Furthermore, the logarithmic Mel spectrum is subjected to discrete cosine transform (DCT) to extract the main information in the spectrum. DCT can effectively compress information and remove redundancy. The Mel frequency cepstrum coefficients in the logarithmic Mel spectrum feature are extracted, and the discrete cosine transform used is expressed as:

[0091]

[0092] In the above formula, C k represents the kth Mel frequency cepstrum coefficient, L m represents the mth value in the log-Mel spectrum, where M is the number of Mel filters.

[0093] Preferably, the first 12 or 13 MFCC coefficients (generally 12 or 13) are selected as the final Mel frequency features.

[0094] In step S120, the working timing information of the microphone, sound card and other hardware in the voice control system is obtained in real time, and the hardware timing features of interest are raised based on the working timing information, including: calculating the hardware response time and obtaining the response delay according to the hardware information timing features, and taking the hardware delayed response as the hardware timing feature.

[0095] In step S130, the device status detection neural network includes a convolutional neural network and a long short-term memory network for feature extraction, and a category prediction network for fusion prediction based on the features. The previously extracted time domain features and frequency domain features are input into the convolutional neural network, and the hardware timing features and Mel frequency features are input into the long short-term memory network for further feature learning. Finally, the features learned by the two networks are spliced ​​to complete feature fusion.

[0096] Specifically, the convolutional neural network includes an input layer, a convolution layer, a pooling layer, and a fully connected layer. The input layer receives the time-frequency graph or mel-spectrogram from the audio signal. The signal data of each frame is converted into a two-dimensional matrix, which represents the frequency and time information of the audio signal. The convolution layer extracts local features of the input data through convolution kernels (filters) and outputs a feature map (Feature Map). The convolution operation formula is:

[0097]

[0098] In the above formula, x m,n Represents the mth and nth elements of the input matrix, i.e., the two-dimensional matrix, ω i,j represents the convolution kernel, and b represents the bias term.

[0099] Specifically, the pooling layer is used to reduce the dimension of the feature map, reduce the computational complexity, and retain the most important features. The commonly used pooling operation is Max Pooling:

[0100] pool(x)=max(x 1 ,x 2 ,…,x n )

[0101] In the above formula, x 1 ,x 2 ,…,x n Represents the value within the pooling window, and the pooling operation outputs the maximum value within the window.

[0102] After multiple convolution and pooling operations, the feature map is flattened and sent to the fully connected layer for nonlinear mapping to the output layer.

[0103] In order to process time series data and capture the timing dependency of signals, LSTM can capture long-term dependencies and effectively avoid the gradient vanishing problem. Therefore, in this embodiment, a long short-term memory network (LSTM) is used to extract Mel frequency features and hardware timing features. The LSTM network can capture the timing pattern of the signal, predict the system state based on the time series data, and then judge and warn the fault.

[0104] Specifically, the LSTM unit consists of an input gate, a forget gate, an output gate, and a memory unit. The calculation formula of the input gate is expressed as:

[0105] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0106] The calculation formula of the forget gate is expressed as:

[0107] f t =σ(W t ·[h t-1 ,x t ]+b f )

[0108] The calculation formula of the output gate is expressed as:

[0109] O t =σ(W o ·[h t-1 ,x t ]+b o )

[0110] The calculation formula for memory unit update is expressed as:

[0111] c t =f t ·c t-1 +it tanh(W c ·[h t-1 ,x t ]+b c )

[0112] The calculation formula of the output value is expressed as:

[0113] h t =o t ·tanh(c t )

[0114] In the above formula, i t is the output gate, f t For the forget gate, O t is the output gate, c t is the memory unit, h t is the output value, W and b are the network weights and bias, σ is the Sigmoid function, and tanh is the hyperbolic tangent function.

[0115] In this embodiment, finally, the category prediction network fuses the features output by the convolutional neural network and the long short-term memory network, and then performs a comprehensive prediction based on the fused features, wherein the category prediction network uses the Softmax function for state classification to predict the probability of each device state. The Softmax function is expressed as:

[0116]

[0117] In the above formula, z j is the score of each fault category, C is the total number of fault categories, P(y=c i x) is the predicted category c i probability.

[0118] In this embodiment, when training the device status detection neural network, its training data is the audio features and hardware information timing features extracted from historical data, and are annotated according to their corresponding real device status, for example, the corresponding audio features include noise interference and echo interference, and the corresponding hardware information timing features include microphone failure, sound card failure, etc.

[0119] In step S140, the comprehensive prediction result of the network prediction is the current state of the device, and feedback is given according to the specific device state.

[0120] In this embodiment, if the current device status is poor audio quality caused by noise interference or echo interference, the noise suppression and gain control related parameters are adaptively adjusted in step S100. If the current device status is a hardware failure, a maintenance warning is fed back and suggestions for fault repair are given.

[0121] In one embodiment, the entire process of the device status detection method based on audio analysis can also be as follows: Figure 2 shown.

[0122] In the above-mentioned device state detection method based on audio analysis, the device state detection method, apparatus, device and storage medium based on audio analysis respectively extract multi-dimensional features and Mel frequency features from the processed audio signal to obtain audio signal features and Mel frequency features, perform timing analysis based on the hardware information timing data to obtain hardware timing features, use the device state detection neural network to comprehensively predict the state of the current device based on the audio signal features, Mel frequency features and hardware timing features to obtain a comprehensive prediction result, and provide corresponding feedback based on the state evaluation result to realize real-time state detection of the voice-controlled device. This method can meet the requirements of accuracy and efficiency in voice-controlled device state detection.

[0123] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0124] In one embodiment, Figure 3 As shown, a device state detection device based on audio analysis is provided, including: an audio signal acquisition and preprocessing acquisition module 200, a multi-dimensional audio signal feature and Mel frequency feature extraction module 210, a hardware feature extraction module 220, a comprehensive evaluation module 230 and a state diagnosis and feedback module 240, wherein:

[0125] The audio signal acquisition and preprocessing acquisition module 200 is used to acquire the audio signal, preprocess the ambient audio signal, and obtain the processed audio signal;

[0126] A multi-dimensional audio signal feature and Mel frequency feature extraction module 210 is used to perform multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0127] The hardware feature extraction module 220 is used to obtain hardware information timing data, and perform timing analysis on the hardware information timing data to obtain hardware timing features;

[0128] A comprehensive evaluation module 230 is used to use a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0129] The status diagnosis and feedback module 240 is used to provide corresponding feedback according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0130] For the specific definition of the device state detection device based on audio analysis, please refer to the definition of the device state detection method based on audio analysis above, which will not be repeated here. Each module in the above-mentioned device state detection device based on audio analysis can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] like Figure 4 As shown, a device status detection system based on audio analysis is also proposed, including an audio collector, a server and an alarm device, wherein the audio collector is used to collect audio signals and transmit them to the server, and the server uses the above-mentioned device status detection method to analyze the audio signals to obtain a comprehensive prediction result of the voice-controlled device, and transmits the comprehensive prediction result to the alarm device, which then issues an alarm.

[0132] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a device status detection method based on audio analysis is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0133] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0135] Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal;

[0136] Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0137] Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics;

[0138] Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0139] Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0140] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0141] Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal;

[0142] Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features;

[0143] Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics;

[0144] Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network;

[0145] Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A device status detection method based on audio analysis, characterized in that: The method is applied to a voice control device, comprising: Acquire an audio signal, and preprocess the audio signal to obtain a processed audio signal; Performing multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features; Acquire hardware information timing data, and perform timing analysis based on the hardware information timing data to obtain hardware timing characteristics; Using a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features, and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and the hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network; Corresponding feedback is provided according to the comprehensive prediction result to achieve real-time status detection of the voice control device.

2. The device status detection method according to claim 1, characterized in that: The preprocessing of the ambient audio signal includes: performing noise suppression and gain control processing on the audio signal to obtain the processed audio signal.

3. The device status detection method according to claim 2, characterized in that: The time domain features include amplitude information and zero-crossing rate; The frequency domain feature is a spectrum envelope.

4. The device status detection method according to claim 3, characterized in that: When extracting Mel frequency features from the processed audio signal, the method includes: After the processed audio signal is framed, short-time Fourier transform is performed on each frame signal to obtain frequency information; The frequency information is processed by a Mel filter to obtain a Mel spectrum, and a logarithmic Mel spectrum feature of the Mel spectrum is extracted; The Mel-frequency cepstral coefficients in the logarithmic Mel-frequency spectrum feature are extracted by using discrete cosine transform, and a preset number of Mel-frequency cepstral coefficients are selected as the Mel-frequency feature.

5. The device status detection method according to claim 4, characterized in that: The Mel filter obtains the Mel spectrum by weighting the spectrum of the signal using a set of filters on the Mel frequency scale on the frequency information; The frequency information is mapped to the Mel frequency scale using the following formula: In the above formula, f represents the frequency information, and M(f) is the frequency information in the Mel scale.

6. The device status detection method according to claim 5, characterized in that: The Mel-frequency cepstrum coefficients in the logarithmic Mel-frequency spectrum feature are extracted, and the discrete cosine transform used is expressed as: In the above formula, C k represents the kth Mel frequency cepstrum coefficient, L m represents the mth value in the log-Mel spectrum, where M is the number of Mel filters.

7. The device status detection method according to claim 6, characterized in that: The performing timing analysis according to the hardware information timing data to obtain hardware timing characteristics includes: According to the hardware information timing data, the response time of the hardware is calculated and the response delay is obtained, and the response delay of the hardware is used as the hardware timing feature.

8. A device state detection device based on audio analysis, characterized in that: The device comprises: An audio signal acquisition and preprocessing acquisition module is used to acquire an audio signal, preprocess the ambient audio signal, and obtain a processed audio signal; A multi-dimensional audio signal feature and Mel frequency feature extraction module, used to perform multi-dimensional feature extraction and Mel frequency feature extraction on the processed audio signal to obtain audio signal features and Mel frequency features, wherein the audio signal features include time domain features and frequency domain features; A hardware feature extraction module is used to obtain hardware information timing data, and perform timing analysis on the hardware information timing data to obtain hardware timing features; A comprehensive evaluation module, used to use a device state detection neural network to comprehensively predict the state of the current device according to the audio signal features, Mel frequency features and hardware timing features to obtain a comprehensive prediction result, wherein the device state detection neural network includes a convolutional neural network for extracting features from the audio signal features, a long short-term memory network for extracting features from the Mel frequency features and hardware timing features, and a category prediction network for comprehensively predicting features output by the convolutional neural network and the long short-term memory network; The status diagnosis and feedback module is used to provide corresponding feedback according to the comprehensive prediction result to realize real-time status detection of the voice control device.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images