Infant cry detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a baby cry detection method and device, electronic equipment and a storage medium. According to the method, the amplitude spectrum of an audio signal is used for generating logarithmic Mel spectrum features, preliminary prediction is carried out on baby cry through a full convolutional neural network, the time-frequency features of a cry signal are analyzed on the basis, the mean value of amplitude spectrum peaks of the cry signal is calculated by using the harmonic characteristics of voiced sound, and the mean value is stored in a buffer area and updated in real time; and then the harmonic energy in the buffer area is averaged by using a sliding window with a short length, so that short cry is prevented from being omitted, and secondary verification can be carried out on the preliminary prediction result. According to the embodiment of the invention, preliminary prediction is carried out only by means of the small-size neural network, secondary verification is carried out on the prediction result of the neural network by means of a traditional time-frequency signal analysis method, and a depth method and a traditional algorithm are effectively combined, so that the neural network is simple in structure and more efficient in operational analysis, the accuracy of cry detection can be improved, and the accuracy of cry detection is improved. And the false alarm rate is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of audio detection, and in particular to a method, device, electronic device and storage medium for detecting a baby's cry. Background Art

[0002] With the development of the concept of smart home, more and more home devices are joining the trend of intelligence, such as smart TVs, smart refrigerators, smart speakers, etc. The development of intelligence has further given rise to the personalized needs of users. Baby Crying Detection (BCD) is a user demand that was born in this context. When the parents of newborns are doing housework or are entangled with other things, they cannot always be with the baby, but the baby at this stage needs parents to respond to their demands in a timely manner, such as feeding, changing diapers, etc. The only way for babies to communicate with the outside world is through crying. Therefore, developing a baby crying detection algorithm in home monitoring can well meet the user needs of this scenario.

[0003] At present, some equipment manufacturers have integrated the baby crying monitoring function into their latest home monitoring, using deep learning algorithms to monitor the crying of babies in the room in real time. Once the baby cries, the monitoring equipment will detect the crying and send a notification or alarm to the baby's parents' mobile phone. However, in actual application, the applicant found that the existing methods rely entirely on neural networks, use more and more complex input features for feature fusion, and then send them to the neural network together, or use larger and more redundant network structures, trying to use large networks to solve all problems encountered, but relying solely on such features cannot significantly improve the accuracy of detection, so there is still a certain false alarm rate. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage medium for detecting baby crying. The method performs preliminary prediction of audio signals through a neural network, and on this basis, performs secondary verification of the prediction results through a sliding window in combination with the spectrum of the audio signal, thereby effectively improving the accuracy of crying detection and reducing the false alarm rate.

[0005] The present application provides a method for detecting a baby crying, comprising:

[0006] Acquire an audio signal, and generate a logarithmic Mel-spectrogram according to the amplitude spectrum of the audio signal;

[0007] Inputting the logarithmic Mel-frequency spectrum graph into a crying prediction model for processing to obtain an initial prediction result;

[0008] Obtaining a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculating a first energy mean thereof, generating a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram;

[0009] If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame within the sliding window in the buffer is calculated, the positive detection result is verified according to the second energy mean, and whether the positive detection result is a false detection is determined according to the verification result.

[0010] In one embodiment, obtaining an audio signal and generating a logarithmic Mel-frequency spectrum diagram according to an amplitude spectrum of the audio signal includes:

[0011] Performing a short-time Fourier Transform (STFT) on the audio signal to obtain an amplitude spectrum of the audio signal;

[0012] The power spectrum corresponding to the amplitude spectrum is calculated, and the power spectrum is processed by Mel filtering and logarithm processing to obtain the logarithmic Mel spectrum graph.

[0013] In one embodiment, searching for a plurality of maximum frequency points in the amplitude spectrum of the audio signal and calculating a first energy mean of the plurality of maximum frequency points comprises:

[0014] Finding a set of maximum frequency points in the amplitude spectrum of the audio signal;

[0015] A preset number of maximum frequency points in the maximum frequency point set are obtained, and the logarithmic means of the preset number of maximum frequency points are calculated according to the order of the maximum frequency points to serve as the first energy mean.

[0016] In one embodiment, the generating a buffer according to the first energy mean, wherein the buffer has the number of frames of the logarithmic Mel-spectrogram as the length, comprises:

[0017] Obtaining first energy means corresponding to the current frame and the previous N-1 frames of the logarithmic Mel-spectrogram, respectively, to generate the buffer, where N is the number of frames of the logarithmic Mel-spectrogram;

[0018] When a new frame arrives, the data in the buffer is updated according to the first energy mean value of the new frame.

[0019] In one embodiment, the calculating the second energy mean of each frame in the sliding window in the buffer includes:

[0020] Determining a sliding interval according to the value of the sliding window;

[0021] Calculate the number of times the sliding window slides in the buffer zone according to the sliding interval and the length of the buffer zone;

[0022] According to the sliding times, the second energy mean of the frames in each window is calculated, and a second energy mean set is generated.

[0023] In one embodiment, verifying the positive detection result using the second energy mean value includes:

[0024] Obtaining a preset energy value, and determining whether there is a sample in the second energy mean value set that is greater than the preset energy value;

[0025] If it exists, retain the positive detection result and output a prompt message;

[0026] If not, it is determined that the positive detection result is a false detection.

[0027] In one embodiment, the initial prediction result includes a plurality of prediction results corresponding to a plurality of logarithmic Mel-spectrograms respectively, and the method further includes:

[0028] Obtaining a prediction threshold, and searching for positive quantity information among the multiple prediction results;

[0029] When the quantity information exceeds the prediction threshold, determining that the initial prediction result is a positive detection result;

[0030] When the quantity information does not exceed the prediction threshold, the initial prediction result is determined to be a negative detection result, and the step of acquiring the audio signal is continued.

[0031] In one embodiment, before inputting the logarithmic Mel-frequency spectrogram into the crying prediction model for processing, the method further includes:

[0032] Collecting a sample set of baby crying sounds, wherein the sample set includes positive sample data and negative sample data;

[0033] Performing data augmentation processing on the positive sample data and the negative sample data respectively;

[0034] The crying prediction model is trained according to the augmented sample set and the corresponding labels to correct the parameters in the crying prediction model.

[0035] The present application also provides a baby crying detection device, comprising:

[0036] An acquisition module, used for acquiring an audio signal and generating a logarithmic Mel spectrum diagram according to an amplitude spectrum of the audio signal;

[0037] A detection module, used for inputting the logarithmic Mel-frequency spectrum into a crying prediction model for processing to obtain an initial prediction result;

[0038] a calculation module, configured to obtain a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculate a first energy mean thereof, and generate a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram;

[0039] A verification module is used to define a sliding window according to the number of frames when the initial prediction result is a positive detection result, calculate the second energy mean of each frame in the sliding window in the buffer, verify the positive detection result according to the second energy mean, and determine whether the positive detection result is a false detection according to the verification result.

[0040] The present application also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of any one of the baby crying detection methods provided in the present application by calling the computer program stored in the memory.

[0041] The present application also provides a computer-readable storage medium, in which a plurality of computer programs are stored. The computer programs can be loaded by a processor to execute the steps in any one of the baby crying detection methods provided in the present application.

[0042] The method for detecting infant crying provided by the present application uses the amplitude spectrum of the audio signal to generate a logarithmic Mel spectrum feature, performs a preliminary prediction of the infant crying through a fully convolutional neural network, and analyzes the time-frequency characteristics of the crying signal on this basis, calculates the mean of the peak of its amplitude spectrum using the voiced harmonic characteristics, stores it in a buffer and updates it in real time, and then uses a shorter sliding window to average the harmonic energy of the buffer, which can avoid missing shorter crying sounds and can realize a secondary verification of the preliminary prediction results of the neural network, thereby reducing the false alarm rate. The embodiment of the present application only relies on a small-sized neural network to make a preliminary prediction, and at the same time, with the help of a traditional time-frequency signal analysis method, performs a secondary verification on the prediction results of the neural network, effectively combining the deep method with the traditional algorithm, not only the structure of the neural network is simple, but also the operation and analysis are more efficient, which can improve the accuracy of crying detection and reduce the false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a first flow chart of a method for detecting a baby cry provided in an embodiment of the present application;

[0045] Figure 2 This is a second flow chart of the method for detecting a baby crying provided in an embodiment of the present application;

[0046] Figure 3 This is a third flow chart of the method for detecting a baby crying sound provided in an embodiment of the present application;

[0047] Figure 4 This is a fourth flow chart of the method for detecting a baby crying sound provided in an embodiment of the present application;

[0048] Figure 5 This is a fifth flow chart of the method for detecting a baby crying provided in an embodiment of the present application;

[0049] Figure 6 This is a sixth flow chart of the method for detecting a baby crying sound provided in an embodiment of the present application;

[0050] Figure 7 This is a structural schematic diagram of a baby crying detection device provided in an embodiment of the present application;

[0051] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0054] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0055] It should be noted that, in this article, step codes such as 101, 102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute 102 first and then 101, etc., but these should all be within the scope of protection of this application.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] An embodiment of the present application provides a method for detecting a baby crying. The execution subject of the method for detecting a baby crying may be a baby crying detection device provided in the embodiment of the present application, or a server integrated with the baby crying detection device, wherein the baby crying detection device may be implemented in hardware or software.

[0058] like Figure 1 As shown, Figure 1 This is a schematic diagram of the first process of the baby crying detection method provided in the embodiment of the present application. The specific process of the baby crying detection method can be as follows:

[0059] 101. Obtain an audio signal, and generate a logarithmic Mel spectrum graph according to the amplitude spectrum of the audio signal.

[0060] In one embodiment, the audio signal may be a continuous analog signal obtained from a baby crying environment through an audio acquisition device such as a microphone, and these continuous analog signals are then converted into digital signals through sampling and quantization so that they can be processed and analyzed by a computer or a digital signal processor. Sampling refers to dividing a temporally continuous analog signal into a series of discrete sample points. The sampling rate determines the accuracy of signal restoration, for example, the sampling rate can be 16KHz. Quantization refers to converting the analog amplitude of each sample point into the closest digital value.

[0061] Furthermore, the audio signal obtained above is processed to generate a logarithmic Mel spectrogram, wherein the logarithmic Mel spectrogram is a feature obtained by converting the audio signal from the time domain to the frequency domain and processing it using the Mel frequency scale. This feature extraction method combines the characteristics of the logarithmic transformation and the Mel frequency scale, and can effectively extract important information from the audio signal while having a certain robustness to noise. Specifically, the audio signal can be first divided into multiple short time periods (frames) by short-time Fourier transform, and Fourier transform is applied to each frame to obtain its spectrum. After the above Fourier transform, a complex matrix is ​​obtained, and each element in the complex matrix represents the amplitude and phase of the spectral component of the corresponding frequency and time point. The amplitude spectrum is the amplitude part of this complex matrix, which represents the intensity of each frequency component in time. Then, according to the number of Mel filters (Mel-Filter Bank, MFB) required (usually 64 to 128) and the sampling rate of the audio signal, a Mel filter bank is constructed. The Mel filter bank is a group of filters uniformly distributed on the Mel scale. Finally, the amplitude spectrum is passed through the Mel filter bank to obtain the energy of each Mel frequency band, and a logarithmic transformation is applied to the output of the Mel filter bank to compress the dynamic range and enhance smaller signal components. The result is a two-dimensional array (or matrix), in which each row represents a time frame and each column represents the logarithmic energy of a Mel frequency band. This array is the logarithmic Mel spectrum map.

[0062] In one embodiment, before generating the above-mentioned logarithmic Mel-frequency spectrum, the audio signal may be further subjected to noise reduction and silent segment removal processing to improve the efficiency and accuracy of subsequent processing, and then the logarithmic Mel-frequency spectrum may be further generated according to the amplitude spectrum of the audio signal. For details, please refer to Figure 2 , that is, the above step 101 may include:

[0063] 1011. Perform short-time Fourier transform on the audio signal to obtain an amplitude spectrum of the audio signal;

[0064] 1012. Calculate the power spectrum corresponding to the amplitude spectrum, and process the power spectrum through Mel filtering and logarithm processing to obtain a logarithmic Mel spectrum graph.

[0065] In computers, audio is processed in the form of time domain signals. Specifically, we can define:

[0066] S=(s1,s2,...,s n ),n∈Z +

[0067] Where S represents a discrete signal in the time domain, n represents a sampling sequence number, and a short-time Fourier transform is performed on the time domain signal S, and the absolute value is taken to obtain the amplitude spectrum M(t,f) of S:

[0068] M(t,f)=|STFT(S)|

[0069] Where t and f represent the frame index in the time domain and the frequency index in the frequency domain respectively, then the power spectrum P(t,f) of S is:

[0070] P(t,f)=|STFT(S)| 2 =M(t,f) 2

[0071] Then apply the Mel filter to the power spectrum P and take the logarithm to get the logarithmic Mel spectrum LMS:

[0072] LMS(mt,mf)=10·log10(MFB(P(t,f)))

[0073] Where MFB(·) is the Mel filter bank, mt and mf represent the frame index and frequency index in the Mel domain, respectively, and LMS is the input feature of the neural network model used for crying prediction.

[0074] 102. Input the logarithmic Mel-frequency spectrum into the crying prediction model for processing to obtain an initial prediction result.

[0075] In one embodiment, a fully convolutional neural network can be selected as a crying prediction model. A fully convolutional neural network is a deep learning algorithm that is particularly suitable for processing data with a grid structure, such as images and audio. In the detection of infant crying, the fully convolutional neural network can automatically extract high-level abstract features from the logarithmic Mel spectrum features, which can be used to distinguish infant crying from other sounds.

[0076] Specifically, the above-mentioned logarithmic Mel-spectrogram is used as input and passed to the trained crying prediction model, which is a fully convolutional crying prediction model. Inside the crying prediction model, the logarithmic Mel-spectrogram will undergo multiple layers of convolution operations. Each layer of convolution operation will extract different features of the input data, which will gradually become abstract and complex. After passing through the convolution layer, the pooling layer (such as maximum pooling) is connected to reduce the dimension of the data while retaining important feature information. After multiple layers of convolution and pooling, the crying prediction model will output a feature vector, which contains the key information related to the input audio signal and crying. Then, the crying prediction model will convert the feature vector into a probability distribution through a fully connected layer (or dense layer) and a softmax function. The probability distribution represents the probability that the input audio signal belongs to the crying category. Finally, according to the output of the softmax function, the category with the highest probability is selected as the prediction result, which is the above-mentioned initial prediction result. For example, if the probability of the crying category exceeds a preset threshold (such as 0.5), it is determined that the input audio signal contains a baby crying, and the initial prediction result is a positive detection result; otherwise, it is confirmed that it does not contain a baby crying, and the initial prediction result is a negative detection result.

[0077] It should be noted that the above softmax function is used to convert the feature vector output by the crying prediction model into a probability distribution. In the crying prediction model, the softmax function can convert the original output (real number or real number vector) of the crying prediction model for whether the input audio signal belongs to the crying category into a probability value between 0 and 1. This probability value represents the confidence that the input audio signal belongs to the crying category. In addition, no matter what the original value of the crying prediction model output is, after being processed by the softmax function, the sum of the probabilities of all predicted output categories is always 1. In the crying prediction model, it is ensured that the sum of the probabilities of the crying category and the non-crying category is 1.

[0078] In one embodiment, the initial prediction result may include multiple prediction results corresponding to multiple logarithmic Mel-spectrograms respectively. Therefore, after obtaining the above-mentioned multiple prediction results, a prediction threshold can also be obtained, and the positive quantity information in the multiple prediction results can be searched. When the quantity information exceeds the prediction threshold, the initial prediction result is determined to be a positive detection result. When the quantity information does not exceed the prediction threshold, the initial prediction result is determined to be a negative detection result.

[0079] 103. Obtain a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculate the first energy mean thereof, generate a buffer according to the first energy mean, and use the number of frames of the logarithmic Mel-spectrogram as the length of the buffer.

[0080] In one embodiment, considering the difficulty in obtaining baby crying data and issues such as privacy protection, the data set is often not as sufficient as other classification tasks. Therefore, the generalization of neural networks in the task of baby crying detection is usually not as ideal as other classification tasks, and some biased adjustments need to be made to the prediction results of the neural network. In engineering practice, the prediction output of the neural network itself is often biased towards positive examples, but this will lead to an increase in false alarms, and users are extremely sensitive to false alarms. In order to solve this problem, this application starts from the signal characteristics of baby crying and further verifies the initial prediction results of the above-mentioned neural network for a second time, thereby effectively reducing the false alarm rate. For details, please continue to refer to Figure 3 In the above step 103, the step of searching for multiple maximum frequency points in the amplitude spectrum of the audio signal and calculating the first energy mean of the multiple maximum frequency points may specifically include:

[0081] 1031. Find a maximum frequency point set in the amplitude spectrum of the audio signal;

[0082] 1032. Obtain a preset number of maximum frequency points from the maximum frequency point set, and calculate the logarithmic mean of the preset number of maximum frequency points according to the order of the maximum frequency points to serve as the first energy mean.

[0083] Furthermore, the step of generating a buffer according to the first energy mean and using the number of frames of the logarithmic Mel-spectrogram as the length in step 103 may specifically include:

[0084] 1033. Obtain first energy means corresponding to the current frame and the previous N-1 frames of the logarithmic Mel-spectrogram to generate a buffer, where N is the number of frames of the logarithmic Mel-spectrogram;

[0085] 1034. When a new frame arrives, update the data in the buffer according to the first energy mean value of the new frame.

[0086] Specifically, in actual applications, the crying sound of a baby shows strong voiced characteristics, with obvious harmonics and fundamental frequency, and the spectrum peak spacing is larger than that of normal voiced speech, the peak-to-valley energy difference is more significant, and the harmonic point position is stable in a short time. Based on these traditional signal characteristics, this embodiment can perform the following steps:

[0087] First, define the maximum frequency point set of the amplitude spectrum M(t,f) of the audio signal as P φ , the P φ Calculated by the following formula:

[0088] P φ ={p i |M(t,i)>M(t,i-1)and M(t,i)≥M(t,i+1),i∈F}

[0089] Wherein, F=fs / 2, fs is the sampling rate.

[0090] Then from the above set P φ The preset number of maximum frequency points ranked top in descending order is obtained. In this embodiment, the preset number is k, and the k maximum frequency points are recorded as P φ,k , the calculation process is as follows:

[0091] P φ,k =maxk(P φ )={p j |M(t,p j )≥M(t,p j-1 )≥...≥M(t,p j-k+1 ),j∈F}

[0092] Among them, p j is the frequency index after descending sorting, and the logarithmic mean (in dB) of these k maximum frequency points is:

[0093]

[0094] Among them, ε is the minimum value.

[0095] The size of the input feature LMS used by the neural network is N×MF, where N represents the time dimension of the input feature, that is, the number of frames. For example, if each frame is 10ms, then a 3-second audio will contain 300 frames. MF represents the Mel spectrum dimension of the input feature, that is, the number of Mel filters, which can be selected between 64 and 128. In order to perform a secondary verification on the output result of the neural network, this embodiment can set a buffer of length N. Define as follows:

[0096]

[0097] Among them, t is still the time frame, that is, the buffer What is saved in the buffer is always the first energy mean of the preset number of maximum frequency points of the logarithmic amplitude spectrum of the current frame and the previous N-1 frames, where k is the preset number. The update can be performed through a "first-in, first-out" queue mode, that is, whenever a new frame arrives, the first energy mean corresponding to the oldest frame is removed, and the first energy mean corresponding to the new frame is added.

[0098] 104. If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame in the sliding window in the buffer is calculated, the positive detection result is verified according to the second energy mean, and whether the positive detection result is a false detection is determined according to the verification result.

[0099] In one embodiment, considering that the duration of infant cries varies, according to experience, the duration of shorter infant cries is about 0.5 seconds to 1 second, while the input feature duration used by the existing neural network-based infant cry detection algorithm is often greater than 3 seconds. Therefore, the length N of the buffer zone can be set to 0.5 seconds or a multiple of 1 second. The first energy mean of the maximum frequency point in the buffer zone cannot be directly averaged, otherwise it will interfere with the detection of some short cries. In order to retain the positive prediction results of the neural network for short cries and reduce false alarms as much as possible, the embodiment of the present application can average the first energy mean of the buffer zone by a sliding window method. For details, please continue to refer to Figure 4 , the step of calculating the second energy mean of each frame in the sliding window in the buffer in step 104 may specifically include:

[0100] 1041. Determine a sliding interval according to a value of the sliding window;

[0101] 1042. Calculate the number of times the sliding window slides in the buffer according to the sliding interval and the length of the buffer;

[0102] 1043. Calculate the second energy mean of the frames in each window according to the number of sliding times, and generate a second energy mean set.

[0103] Furthermore, the step of verifying the positive detection result with the second energy mean in step 104 may specifically include:

[0104] 1044. Obtain a preset energy value, and determine whether there is a sample in the second energy mean value set that has an energy value greater than the preset energy value;

[0105] 1045. If it exists, retain the positive detection result and output a prompt message;

[0106] 1046. If not, it is determined that the positive detection result is a false detection.

[0107] Specifically, we can first define the size of the sliding window as W, where N>>W, >> is much larger than, so the sliding interval of the sliding window is W / 2, and accordingly, it can be calculated by the following formula:

[0108]

[0109] Where L is the sliding window in the buffer The number of slides, It is rounded up and then the second energy mean of each frame in the window is calculated according to the number of sliding times, and the second energy mean set is generated The calculation formula is as follows:

[0110]

[0111] Further, the second energy mean value set is determined Is there a sample greater than or equal to the preset energy value in Then it is considered that the sliding window has passed the secondary verification of the neural network prediction result, and the positive prediction result of the neural network is retained and the user is prompted. Otherwise, the positive prediction of the neural network is considered to be a false positive, where θ E The above-mentioned prompting method may include sounding an alarm through a speaker, or sending a notification push through an application on a mobile device or computer.

[0112] Starting from the time-frequency signal of baby crying, this application deeply studies its spectral characteristics, and abstractly summarizes the traditional signal characteristics of baby crying, such as strong voiced characteristics, obvious harmonics and fundamental frequency, and the spectrum peak spacing is larger than that of normal voiced speech, the peak-to-valley energy difference is more significant, and the harmonic point position is stable in a short time, etc., and uses these characteristics to design a method based on sliding window energy verification to perform secondary verification on the predicted output of the neural network. Please continue to refer to Figure 5 First, the amplitude spectrum M(t,f) of the audio signal is calculated, and the logarithmic Mel spectrum LMS calculated based on the amplitude spectrum M(t,f) is used as input, and processed by the crying prediction model to obtain the initial prediction result. When the initial prediction result is a negative detection result, the current process is stopped and the audio signal is acquired again. When the initial prediction result is a positive detection result, the k maximum frequency points in the maximum frequency point set are further calculated based on the above amplitude spectrum M(t,f), which are recorded as P φ,k , calculate the logarithmic mean of these k maximum frequency points and generate a buffer of length N Finally, the second energy mean set of frames in each window is calculated And determine the second energy mean set Is there a value greater than or equal to the preset energy value θ E If the sample exists, then the positive prediction result is confirmed to be correct, that is, there is currently a baby crying sound. If not, then the positive prediction result is confirmed to be a false alarm, and there is currently no baby crying sound. The embodiment of the present application realizes the combination of traditional signal processing algorithms and neural network algorithms, effectively reduces the computing pressure of the neural network, and has strong universality, which can effectively reduce the number of false alarms of the neural network.

[0113] As described above, the method for detecting infant crying proposed in the embodiment of the present application can use the amplitude spectrum of the audio signal to generate a logarithmic Mel spectrum feature, make a preliminary prediction of the infant crying through a fully convolutional neural network, and on this basis analyze the time-frequency characteristics of the crying signal, use the voiced harmonic characteristics to calculate the mean of the peak of its amplitude spectrum, store it in a buffer and update it in real time, and then use a shorter sliding window to average the harmonic energy of the buffer, so as to avoid missing shorter crying sounds, and realize secondary verification of the prediction results of the neural network, thereby reducing the false alarm rate. The embodiment of the present application only relies on a small-sized neural network to make a preliminary prediction, and at the same time, with the help of a traditional time-frequency signal analysis method, the prediction results of the neural network are secondary verified, effectively combining the deep method with the traditional algorithm, not only the structure of the neural network is simple, but also the operation and analysis are more efficient, which can improve the accuracy of crying detection and reduce the false alarm rate.

[0114] The method described in the above embodiment will be further described below.

[0115] See also Figure 6 , Figure 6 This is a second flow chart of the method for detecting the crying sound of a baby provided in the embodiment of the present application. The method comprises:

[0116] 201. Obtain an audio signal, and generate a logarithmic Mel-spectrogram according to the amplitude spectrum of the audio signal.

[0117] In one embodiment, the audio signal may be a continuous analog signal obtained from a baby crying environment through an audio acquisition device such as a microphone, and then the audio signal is converted from the time domain to the frequency domain and processed using the Mel frequency scale to obtain a logarithmic Mel spectrum.

[0118] 202. Build a crying prediction model and collect a sample set of baby crying sounds, where the sample set includes positive sample data and negative sample data.

[0119] 203. Perform data augmentation processing on the positive sample data and the negative sample data respectively.

[0120] 204. The crying prediction model is trained according to the augmented sample set and the corresponding labels to correct the parameters in the crying prediction model.

[0121] In one embodiment, a crying prediction model can be constructed by a fully convolutional neural network, which only contains simple operators such as convolution, BatchNorm, MaxPool, etc., has no complex structure, and the convolution kernel size does not exceed 3x3, which is convenient for quantitative deployment, has a small overall amount of computation, and has low power consumption on the end side, which effectively reduces resource overhead while realizing baby crying detection. Specifically, a convolutional neural network model can be constructed using convolutional layers, pooling layers, activation functions (such as ReLU), fully connected layers, etc. These layers are stacked to extract high-level features from input features. The positive logarithmic Mel spectrogram and the negative logarithmic Mel spectrogram are used as inputs respectively (or after merging) and forward propagated through the CNN model. In each layer, the data is processed and passed to the next layer until the output layer generates a prediction result. Based on the output predicted by the model and the true label (i.e., the label corresponding to the positive sample data and the negative sample data), the loss value of the model is calculated by a loss function (such as cross entropy loss CrossEntropy Loss). The smaller the loss value, the more accurate the prediction of the model. According to the calculated loss value, the model parameters are modified using an optimization algorithm (such as gradient descent, Adam, etc.). This process is repeated for multiple iterations until a certain stop condition is met (such as the loss value no longer decreases significantly or reaches a preset number of iterations), thereby obtaining a trained crying prediction model.

[0122] In one embodiment, data augmentation processing for positive sample data and negative sample data may specifically include: expanding the crying data set by time shift (Time-Shift), spectrum enhancement (SpecAugment), adding noise, adding reverberation, etc. for the positive sample data, and expanding the negative sample data by superposition or intensity change. These expansion processing methods can be used alone or in combination to produce a more diverse sample set. By using different processing methods to increase the diversity and complexity of the data, and by training and evaluating on these processed test sets, the performance and robustness of the crying detection model can be significantly improved.

[0123] In one embodiment, the above-mentioned convolutional neural network algorithm includes at least one of an AlexNet network algorithm, a ResNet network algorithm, an Inception network algorithm, and a convolutional recurrent neural network algorithm.

[0124] 205. Input the logarithmic Mel-frequency spectrum into the trained crying prediction model for processing to obtain an initial prediction result.

[0125] Furthermore, the above-mentioned logarithmic Mel-spectrogram is used as the input of the convolutional neural network. Before the detection, the convolutional neural network has been trained using a large amount of infant crying data as a sample set. It can automatically extract high-level abstract features from the logarithmic Mel-spectrogram to determine whether the input signal is crying, which is the above-mentioned initial prediction result.

[0126] 206. Obtain a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculate the first energy mean thereof, and generate a buffer according to the first energy mean and with the number of frames of the logarithmic Mel-spectrogram as the length.

[0127] 207. If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame in the sliding window in the buffer is calculated, the positive detection result is verified with the second energy mean, and whether the positive detection result is a false detection is determined based on the verification result.

[0128] The specific contents of the above steps 206 and 207 can refer to the description of the relevant parts in the previous embodiment, and this embodiment will not be further elaborated here. For example, taking a sampling rate of 16KHz as an example, the input of the crying prediction model is the LMS feature extracted from a 5-second audio clip. When calculating, the number of points nfft=1024 used for fast Fourier transform (FFT), the number of samples between sliding windows (or frames) hop_len=350, the number of Mel filter groups n_mels=225, and a feature map of size 225x226 is obtained, which is the input of the neural network. The length of the buffer is the same as the length of the LMS feature map, that is, N=226, and the length of the sliding window W=46. After calculating the above data, the operation result set of the sliding window can be calculated and verified.

[0129] The embodiment of the present application starts from the baby cry signal itself, analyzes the traditional time-frequency characteristics of the signal, uses the voiced harmonic characteristics to calculate the average of the strongest k amplitude spectrum peaks, stores them in the buffer and updates them in real time, and then uses a shorter sliding window to average the harmonic energy in the buffer. This can avoid missing shorter cries and can achieve secondary verification of the positive prediction results of the neural network, which can effectively reduce false alarms. In the task of baby cry detection, previous technical solutions almost completely rely on neural networks, or use complex input features to fuse features such as logarithmic Mel spectrum, MFCC, amplitude spectrum, etc. and send them to the network, which directly increases the computational complexity of the neural network, causing the network size to be too large, causing a large loss of accuracy in the engineering quantitative deployment stage, and also has high requirements for the computing power of the end-side chip. However, this application only relies on a simple neural network to make preliminary predictions, and uses traditional time-frequency signal analysis methods to perform secondary verification on the prediction results of the neural network, effectively combining deep methods with traditional algorithms. Not only is the neural network small in size and simple in structure, but the analysis of time domain features also further improves the accuracy of crying predictions, reducing the difficulty of missed and false alarms in baby crying detection at an extremely low cost.

[0130] As described above, the method for detecting infant crying proposed in the embodiment of the present application adopts a data augmentation method to expand the positive crying samples and the negative non-crying samples respectively, so that more abundant samples can be used to train the full convolutional neural network model, so that the prediction results of the model are more accurate. In addition, the amplitude spectrum of the audio signal is used to generate the logarithmic Mel spectrum feature, and the infant crying is preliminarily predicted by the full convolutional neural network, and on this basis, the time-frequency characteristics of the crying signal are analyzed, and the mean of the amplitude spectrum peak is calculated by using the voiced harmonic characteristics, which is stored in the buffer and updated in real time, and then the harmonic energy of the buffer is averaged using a shorter sliding window, so that the omission of shorter crying can be avoided, and the preliminary prediction results of the neural network can be verified twice, thereby reducing the false alarm rate. The embodiment of the present application only relies on a small-sized neural network to make a preliminary prediction, and at the same time, with the help of the traditional time-frequency signal analysis method, the prediction results of the neural network are verified twice, which effectively combines the deep method with the traditional algorithm, not only the structure of the neural network is simple, but also the operation analysis is more efficient, which can improve the accuracy of crying detection and reduce the false alarm rate.

[0131] In order to implement the above method, the embodiment of the present application further provides a baby crying detection device, which can be integrated into a terminal device such as a mobile phone, a tablet computer and the like.

[0132] For example, Figure 7 As shown, Figure 7 : is a schematic diagram of a structure of a baby crying detection device provided in an embodiment of the present application. The baby crying detection device may include:

[0133] An acquisition module 301 is used to acquire an audio signal and generate a logarithmic Mel spectrum diagram according to the amplitude spectrum of the audio signal;

[0134] A detection module 302 is used to input the logarithmic Mel-frequency spectrum into the crying prediction model for processing to obtain an initial prediction result;

[0135] A calculation module 303 is used to obtain a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculate the first energy mean thereof, and generate a buffer according to the first energy mean, wherein the buffer has the number of frames of the logarithmic Mel-spectrogram as the length;

[0136] The verification module 304 is used to define a sliding window according to the number of frames when the initial prediction result is a positive detection result, calculate the second energy mean of each frame in the sliding window in the buffer, verify the positive detection result according to the second energy mean, and determine whether the positive detection result is a false detection according to the verification result.

[0137] As can be seen from the above, the baby crying detection device proposed in the embodiment of the present application uses the amplitude spectrum of the audio signal to generate the logarithmic Mel spectrum feature, and makes a preliminary prediction of the baby crying through the fully convolutional neural network. The convolution kernel size does not exceed 3x3, which is convenient for quantitative deployment, with low overall computational complexity and low power consumption on the end side. On this basis, the time-frequency characteristics of the crying signal are analyzed, and the prediction results of the neural network can be verified twice, thereby reducing the false alarm rate. Not only is the structure of the neural network used in the embodiment of the present application simple, but the computational analysis is more efficient, which can improve the accuracy of crying detection and reduce the false alarm rate.

[0138] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0139] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0140] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any of the baby crying detection methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:

[0141] Acquire an audio signal, and generate a logarithmic Mel-spectrogram according to the amplitude spectrum of the audio signal;

[0142] Inputting the logarithmic Mel-frequency spectrum graph into a crying prediction model for processing to obtain an initial prediction result;

[0143] Obtaining a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculating a first energy mean thereof, generating a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram;

[0144] If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame within the sliding window in the buffer is calculated, the positive detection result is verified according to the second energy mean, and whether the positive detection result is a false detection is determined according to the verification result.

[0145] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0146] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0147] Since the computer program stored in the storage medium can execute the steps in any one of the baby crying detection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the baby crying detection methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0148] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the device equipped with the chip executes the methods in various possible implementation modes as described above.

[0149] For example, the computer device mentioned above may be a terminal device with corresponding functions such as a mobile phone, a tablet computer, a personal computer, a cloud computer, etc. Figure 8 , Figure 8 A schematic diagram of the structure of a computer provided in an embodiment of the present application.

[0150] The computer device 400 may include components such as a memory 401 and a processor 402. Those skilled in the art will appreciate that Figure 8 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0151] The memory 401 may be used to store applications and data. The applications stored in the memory 401 include executable codes. The applications may be composed of various functional modules. The processor 402 executes various functional applications and data processing by running the applications stored in the memory 401.

[0152] In this embodiment, the processor 402 in the computer device will load the executable code corresponding to the process of one or more application programs into the memory 401 according to the following instructions, and the processor 402 will run the application program stored in the memory 401 to execute:

[0153] Acquire an audio signal, and generate a logarithmic Mel-spectrogram according to the amplitude spectrum of the audio signal;

[0154] Inputting the logarithmic Mel-frequency spectrum graph into a crying prediction model for processing to obtain an initial prediction result;

[0155] Obtaining a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculating a first energy mean thereof, generating a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram;

[0156] If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame within the sliding window in the buffer is calculated, the positive detection result is verified according to the second energy mean, and whether the positive detection result is a false detection is determined according to the verification result.

[0157] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The modules in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs.

[0158] The various technical features of the technical solution of the present application can be arbitrarily combined. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0159] The above is a detailed introduction to a baby crying detection method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting a baby crying, characterized in that: include: Acquire an audio signal, and generate a logarithmic Mel-spectrogram according to the amplitude spectrum of the audio signal; Inputting the logarithmic Mel-frequency spectrum graph into a crying prediction model for processing to obtain an initial prediction result; Obtaining a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculating a first energy mean thereof, generating a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram; If the initial prediction result is a positive detection result, a sliding window is defined according to the number of frames, a second energy mean of each frame within the sliding window in the buffer is calculated, the positive detection result is verified according to the second energy mean, and whether the positive detection result is a false detection is determined according to the verification result.

2. The method for detecting a baby crying as claimed in claim 1, wherein: The step of acquiring an audio signal and generating a logarithmic Mel-frequency spectrum diagram according to an amplitude spectrum of the audio signal includes: Performing short-time Fourier transform on the audio signal to obtain an amplitude spectrum of the audio signal; The power spectrum corresponding to the amplitude spectrum is calculated, and the power spectrum is processed by Mel filtering and logarithm processing to obtain the logarithmic Mel spectrum graph.

3. The method for detecting a baby crying as claimed in claim 1, wherein: The step of searching for a plurality of maximum frequency points in the amplitude spectrum of the audio signal and calculating a first energy mean of the plurality of maximum frequency points comprises: Finding a set of maximum frequency points in the amplitude spectrum of the audio signal; A preset number of maximum frequency points in the maximum frequency point set are obtained, and the logarithmic means of the preset number of maximum frequency points are calculated according to the order of the maximum frequency points to serve as the first energy mean.

4. The method for detecting a baby crying as claimed in claim 1, characterized in that: The step of generating a buffer according to the first energy mean, wherein the buffer has the number of frames of the logarithmic Mel-spectrogram as a length, comprises: Obtaining first energy means corresponding to the current frame and the previous N-1 frames of the logarithmic Mel-spectrogram, respectively, to generate the buffer, where N is the number of frames of the logarithmic Mel-spectrogram; When a new frame arrives, the data in the buffer is updated according to the first energy mean value of the new frame.

5. The method for detecting a baby crying as claimed in claim 1, wherein: The calculating the second energy mean of each frame in the sliding window in the buffer comprises: Determining a sliding interval according to the value of the sliding window; Calculate the number of times the sliding window slides in the buffer zone according to the sliding interval and the length of the buffer zone; According to the sliding times, the second energy mean of the frames in each window is calculated, and a second energy mean set is generated.

6. The method for detecting a baby crying as claimed in claim 5, characterized in that: The verifying the positive detection result by using the second energy mean value includes: Obtaining a preset energy value, and determining whether there is a sample in the second energy mean value set that is greater than the preset energy value; If it exists, retain the positive detection result and output a prompt message; If not, it is determined that the positive detection result is a false detection.

7. The method for detecting a baby crying sound according to any one of claims 1 to 6, characterized in that: The initial prediction result includes a plurality of prediction results corresponding to a plurality of logarithmic Mel-spectrograms respectively, and the method further includes: Obtaining a prediction threshold, and searching for positive quantity information among the multiple prediction results; When the quantity information exceeds the prediction threshold, determining that the initial prediction result is a positive detection result; When the quantity information does not exceed the prediction threshold, the initial prediction result is determined to be a negative detection result.

8. The method for detecting a baby crying sound according to any one of claims 1 to 6, characterized in that: Before inputting the logarithmic Mel-frequency spectrogram into the crying prediction model for processing, the method further includes: Collecting a sample set of baby crying sounds, wherein the sample set includes positive sample data and negative sample data; Performing data augmentation processing on the positive sample data and the negative sample data respectively; The crying prediction model is trained according to the augmented sample set and the corresponding labels to correct the parameters in the crying prediction model.

9. A baby crying detection device, characterized in that: include: An acquisition module, used for acquiring an audio signal and generating a logarithmic Mel spectrum diagram according to an amplitude spectrum of the audio signal; A detection module, used for inputting the logarithmic Mel-frequency spectrum into a crying prediction model for processing to obtain an initial prediction result; a calculation module, configured to obtain a preset number of maximum frequency points from the amplitude spectrum of the audio signal and calculate a first energy mean thereof, and generate a buffer according to the first energy mean, wherein the buffer has a length of the number of frames of the logarithmic Mel-spectrogram; A verification module is used to define a sliding window according to the number of frames when the initial prediction result is a positive detection result, calculate the second energy mean of each frame in the sliding window in the buffer, verify the positive detection result according to the second energy mean, and determine whether the positive detection result is a false detection according to the verification result.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the steps in the baby crying detection method according to any one of claims 1 to 8 by calling the computer program stored in the memory.

11. A computer-readable storage medium, characterized in that The storage medium stores a plurality of computer programs, which can be loaded by a processor to execute the steps in the method for detecting a baby cry as described in any one of claims 1 to 8.

Images