Infant cry detection method, device and system and storage medium

Abstract

The invention discloses an infant cry detection method, device and system and a storage medium. The detection method comprises the following steps: acquiring an audio sampling signal; processing the audio sampled signal to generate a logarithmic Mel spectrum feature; the logarithmic Mel spectrum features are input into a trained neural network algorithm model to be detected, a cry detection result is obtained, the detection model comprises a convolutional neural network algorithm, a prediction function and a judgment threshold value, the judgment threshold value can be adjusted, the convolutional neural network algorithm generates a multi-classification task result according to the logarithmic Mel spectrum features, and the multi-classification task result is subjected to multi-classification. The prediction function determines a classification result according to a multi-classification task result, and the judgment threshold is used for comparing with the classification result to obtain a cry detection result. According to the detection method, the input logarithmic Mel spectrum features are detected through the detection model to generate the multi-classification task result, and the cry detection result is obtained by comparing the multi-classification task result with the adjustable threshold value, so that the problem of how to balance false alarms and missing alarms is solved.

Description

Technical Field

[0001] The present application relates to the technical field of sound detection, and in particular to a detection method, a detection device, a detection system and a readable storage medium based on baby crying. Background Art

[0002] As wearable devices and household products become more and more intelligent, people have higher and higher requirements for smart home products, such as televisions, surveillance and other hardware devices.

[0003] People hope that home monitoring equipment can monitor the baby's status at all times. Once the baby cries or has other abnormal feedback, the monitoring facilities can remind parents in time, and the demand for baby crying detection (BCD) arises. However, the crying detection function of related monitoring products is still imperfect. The frequency and accuracy of crying alarms, as well as the reduction of false alarm rate of crying, are determined by equipment manufacturers during the development stage and cannot be determined by users themselves based on their personal preferences. On the one hand, the monitoring products on the current market have the problem of mistaking sounds that are not baby crying as crying, and on the other hand, there is the problem of missing crying, which leads to the failure to meet the baby's needs in time. Different customers have different needs. Some users hope to minimize false alarms, while some users hope to avoid missing alarms as much as possible. How to adjust according to user needs and balance false alarms and missed alarms according to actual needs has become a difficult problem. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application provides a method, a detection device, a detection system and a computer-readable storage medium for detecting a baby's cry.

[0005] The method for detecting infant crying of the embodiment of the present application includes: obtaining an audio sampling signal; processing the audio sampling signal to generate a logarithmic Mel spectrum feature; inputting the logarithmic Mel spectrum feature into a trained detection model for detection to obtain a crying detection result, wherein the detection model includes a convolutional neural network algorithm, a prediction function and a determination threshold, the determination threshold is adjustable, the convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature, the prediction function determines the classification result based on the multi-classification task result, and the determination threshold is used to compare with the classification result to obtain the crying detection result.

[0006] In some embodiments, processing the audio sampling signal to generate a logarithmic Mel spectrum feature includes: intercepting the audio sampling signal with a preset window size to obtain a target audio segment; processing the target audio segment to obtain the logarithmic Mel spectrum feature.

[0007] In some embodiments, the crying detection result includes a positive detection result and a negative detection result, and the determination threshold is used to compare with the classification result to obtain the crying detection result, including: when the classification result is less than the determination threshold, generating the negative detection result; when the classification result is greater than or equal to the determination threshold, generating the positive detection result.

[0008] In certain embodiments, the logarithmic Mel-spectrogram feature is input into a trained detection model for detection to output a crying detection result, including: inputting a plurality of the logarithmic Mel-spectrogram features into the detection model in sequence to generate a plurality of prediction results, each logarithmic Mel-spectrogram feature corresponding to a prediction result; determining the crying detection result based on the plurality of prediction results and a maximum positive prediction threshold, wherein the maximum positive prediction threshold is adjustable.

[0009] In certain embodiments, the crying detection result includes a positive detection result and a negative detection result, and the crying detection result is determined based on multiple prediction results and a maximum positive prediction threshold, including: when the number of positive results in the prediction results is greater than or equal to the maximum positive prediction threshold, determining that the detection result is the positive detection result; when the number of positive results in the prediction results is less than the maximum positive prediction threshold, determining that the detection result is the negative detection result.

[0010] In some embodiments, the detection method also includes: obtaining a test set of infant crying, the test set including a positive sample test set and a negative sample test set, the positive sample test set including positive crying sample data, and the negative sample test set including negative crying sample data; processing the positive crying sample data and the negative crying sample data respectively to obtain positive logarithmic Mel spectrum features of the positive crying sample data and negative logarithmic Mel spectrum features of the negative crying sample data; using the constructed detection model, processing the positive logarithmic Mel spectrum features and the negative logarithmic Mel spectrum features respectively in the training stage to obtain a training output result; based on the training output result, calculating the loss value of the constructed detection model through a loss function; and correcting the parameters of the constructed detection model according to the loss value to obtain the trained detection model.

[0011] In some embodiments, the detection method further includes: performing expansion processing on the test set, and the expansion processing method includes at least one of time offset processing, spectrum enhancement processing, noise addition processing or reverberation addition processing.

[0012] In some embodiments, the convolutional neural network algorithm includes at least one of an AlexNet network algorithm, a ResNet network algorithm, an Inception network algorithm, and a convolutional recurrent neural network algorithm.

[0013] In some implementations, the detection method further includes: generating a reminder message according to the crying detection result; and sending the reminder message.

[0014] The embodiment of the present application also provides a device for detecting the crying of an infant, and the detection device includes: an acquisition module, a processing module and a detection module. The acquisition module is used to acquire an audio sampling signal of the crying of an infant; the processing module is used to process the audio sampling signal to obtain a logarithmic Mel spectrum feature; the detection module is used to input the logarithmic Mel spectrum feature into a trained detection model for detection to obtain a crying detection result. Among them, the detection model includes a convolutional neural network algorithm, a prediction function and a judgment threshold, the judgment threshold can be adjusted, the convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature, the prediction function determines the classification result based on the multi-classification task result, and the judgment threshold is used to compare with the classification result to obtain the crying detection result.

[0015] An embodiment of the present application also provides a system for detecting a baby's cry, comprising a processor, a memory and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method for detecting a baby's cry described in any one of the above embodiments.

[0016] The present application also provides a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by a processor, the processor executes the method for detecting a baby's cry as described in any one of the above embodiments.

[0017] In the detection method, detection device, detection system and computer-readable storage medium of the present application, the sampling signal of the baby's cry is processed to generate a logarithmic Mel spectrum feature, and then the convolutional neural network algorithm is used to apply the idea of ​​multi-classification tasks to the detection model. The detection model generates a multi-classification task result for the input logarithmic Mel spectrum feature detection, and obtains the crying detection result by comparing it with an adjustable threshold. In addition, due to the use of simple logarithmic Mel spectrum features and neural networks, the cost is extremely low. In this way, the false positives and false negatives of crying detection can be adjusted according to the actual needs of the user.

[0018] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 This is one of the flow charts of a method for detecting a baby's cry in certain embodiments of the present application;

[0021] Figure 2 This is the second flow chart of the method for detecting the crying sound of a baby in certain embodiments of the present application;

[0022] Figure 3 This is the third flow chart of the method for detecting the crying sound of a baby according to some embodiments of the present application;

[0023] Figure 4 This is a fourth flow chart of a method for detecting a baby's cry in certain embodiments of the present application;

[0024] Figure 5 This is a fifth flow chart of a method for detecting a baby's cry in certain embodiments of the present application;

[0025] Figure 6 This is the sixth flow chart of the method for detecting the crying sound of a baby according to some embodiments of the present application;

[0026] Figure 7 FIG7 is a flow chart of a method for detecting a baby crying sound according to certain embodiments of the present application;

[0027] Figure 8 It is a module schematic diagram of a baby crying detection device in certain embodiments of the present application.

[0028] Main component reference numbers:

[0029] Detection device 100 , acquisition module 10 , processing module 20 , detection module 30 , and calculation module 40 . DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the embodiments of the present application.

[0031] See also Figure 1 The present application provides a method for detecting a baby's crying sound, the method comprising the steps of:

[0032] 01, obtain audio sampling signal;

[0033] 02. Process the audio sampling signal to generate logarithmic Mel spectrum features;

[0034] 03. Input the logarithmic Mel spectrum feature into the trained detection model for detection to obtain the crying detection result. The detection model includes a convolutional neural network algorithm, a prediction function and a judgment threshold. The judgment threshold can be adjusted. The convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature. The prediction function determines the classification result based on the multi-classification task result. The judgment threshold is used to compare with the classification result to obtain the crying detection result.

[0035] In the detection method of the present application, the sampling signal of the baby's cry is processed to generate a logarithmic Mel spectrum feature, and then the convolutional neural network algorithm is used to apply the idea of ​​multi-classification tasks to the detection model. The detection model generates a multi-classification task result for the input logarithmic Mel spectrum feature detection, and uses an adjustable judgment threshold to compare with the classification result to generate a crying detection result, or uses an adjustable maximum positive prediction threshold to compare with the statistical result of the prediction result to generate a crying detection result. In addition, due to the use of simple logarithmic Mel spectrum features and neural networks, the cost is extremely low. In this way, the difficult problem of how to balance false positives and false negatives in baby crying detection is solved at a very low cost.

[0036] It should be noted that the audio sampling signal mainly refers to the continuous analog signal obtained from the baby crying environment through audio acquisition devices such as microphones. These signals are then sampled and quantized and converted into digital form for processing and analysis by computers or digital signal processors. Sampling refers to dividing the continuous analog signal into a series of discrete sample points in time. The sampling rate determines the accuracy of signal restoration, and the sampling rate can be 16kHz. Quantization refers to converting the analog amplitude of each sample point into the closest digital value.

[0037] Logarithmic Mel spectrum feature is a common feature extraction method in audio signal processing, and is widely used in speech recognition, environmental sound event detection and other fields. Logarithmic Mel spectrum feature is a feature obtained by converting the audio signal from the time domain to the frequency domain and processing it using the Mel frequency scale. This feature extraction method combines the characteristics of logarithmic transformation and Mel frequency scale, and can effectively extract important information from audio signals while being robust to noise.

[0038] There are two major advantages to using the logarithmic Mel spectrum feature instead of the logarithmic amplitude spectrum. First, the design of the Mel scale is more in line with the characteristics of the human ear being more sensitive to low-frequency components. Therefore, in the logarithmic Mel spectrum feature map, the low 4K frequency band can account for more than 75% of the entire feature map, while the high 4K frequency band accounts for less than 25%. The network trained with this feature has a greater weight in the low-frequency area; second, surveillance cameras will be deployed and installed in a higher position to obtain a better monitoring field of view, such as the corner of the ceiling, which is far away from the baby, at 3 meters or even more than 3 meters, and the signal energy attenuation is more serious, especially the degree of attenuation of high-frequency components will be greater than that of low-frequency components. Therefore, if the logarithmic amplitude spectrum is used, the corresponding feature map will show serious loss of high 4K frequency band information, and the high 4K frequency band accounts for 50% of the logarithmic amplitude spectrum feature map, which will eventually reduce the prediction accuracy of the convolutional neural network.

[0039] Convolutional neural network is a deep learning algorithm that is particularly suitable for processing data with grid structure, such as images and audio. In the detection of baby crying, convolutional neural network can automatically extract high-level abstract features from the logarithmic Mel spectrum features, which can be used to distinguish baby crying from other sounds. Convolutional neural network can select a network with simple structure, high prediction accuracy and low time consumption from existing technical solutions.

[0040] The prediction function is used to convert the multi-classification task results generated by the convolutional neural network into the final classification results. In the case of baby cry detection, the prediction function can be a normalized exponential function, or softmax function. In the multi-classification task of baby cry detection, the convolutional neural network outputs a vector, each element of which corresponds to a possible category (such as different types of baby cries). However, these raw output values ​​may be arbitrary real numbers and cannot be directly interpreted as probabilities, because the probability values ​​must be between 0 and 1, and the sum of the probabilities of all possible categories must be 1. The role of the softmax function is to convert these raw output values ​​into probability values ​​that meet the above conditions. The softmax function applies an exponential function to each raw output value, making all values ​​positive and amplifying the numerical differences, and then normalizes these exponential values, that is, dividing them by the sum of all exponential values ​​to obtain the probability of each category. In this way, the softmax function converts the raw output of the convolutional neural network into a probability distribution for each possible cry type, based on which the category with the highest probability can be selected as the final prediction result.

[0041] Multi-classification tasks are tasks that divide input data into multiple mutually exclusive categories. In the case of baby cry detection, multi-classification tasks may include classifying the audio signal into multiple categories such as "baby cry", "adult voice", "environmental noise", etc. Convolutional neural networks are trained to learn how to map the input logarithmic Mel spectrum features to different categories. During the training process, the network adjusts its weights and biases based on the input features and corresponding labels to minimize the classification error rate.

[0042] The decision threshold is used to compare the output of the prediction function with a preset threshold to generate the final crying detection result. If the confidence (or probability) of the prediction result is higher than the decision threshold, it is considered that the baby crying is detected; otherwise, it is considered that the baby crying is not detected. The decision threshold can be adjusted according to actual needs.

[0043] In one embodiment, in a hospital nursery, the monitoring equipment captures audio and samples it at a sampling rate of 16 kHz.

[0044] In the computer, first, the audio is processed as a time domain signal:

[0045] Specifically, define:

[0046] X=(x1,x2,...,x n ),n∈Z + (1)

[0047] Where X represents the time domain signal, n represents the sampling sequence number, and S represents the spectrum of the signal, then:

[0048] S(t,f)=STFT(X) (2)

[0049] Where STFT(·) represents short-time Fourier transform (STFT), t and f represent time frame and frequency respectively. The Mel-Spectrum M(t,f) of the signal is defined as:

[0050] P(t,f)=|S(t,f) 2 (3)

[0051] M(t,f)=MFB(P(t,f)) (4)

[0052] Where P(t,f) is the power spectrum of the signal, MFB(·) is the Mel filter bank, so the Log-Mel Spectrogram (LMS) used in the present invention can be expressed as:

[0053] LMS(t,f)=10·log10(M(t,f)) (5)

[0054] When the number of sampling points used in the calculation is nfft=1024, the frame shift hop_len=350, and the number of Mel spectra n_mels=225, a feature map of size 225x226 is obtained and input into the convolutional neural network detection.

[0055] The computer uses the above feature map as the input of the convolutional neural network. Before the detection, the convolutional neural network has been trained with a large amount of infant crying data as a training set, which can automatically extract high-level abstract features from the logarithmic Mel spectrum feature map to determine whether the input signal is crying.

[0056] In the related art, the convolutional neural network treats the detection of baby crying as a binary classification task, and only outputs a single value, such as 0.47, which corresponds to the probability of the convolutional neural network to determine whether a certain signal is crying. The closer this probability is to 1, the more it is considered to be crying, and the closer it is to 0, the less it is considered to be crying. The developer gives a rigid threshold value, such as 0.8, based on experience, which cannot be changed by the user. If the value output by the neural network is greater than the threshold, it is considered to be crying, and if it is less than this threshold, it is not considered to be crying. 0.47 is less than 0.8, so it is considered not to be crying here. The size of this threshold is directly related to the false alarm rate and the missed alarm rate. Some users want to minimize false alarms and need to increase the threshold. Some users want to minimize missed alarms and need to lower the threshold. The rigid threshold cannot be changed after it is set, resulting in baby crying detection not meeting the actual needs of users.

[0057] In this application, the convolutional neural network detects baby crying as a multi-classification task, including two categories: crying and non-crying, and outputs a predicted one-hot vector p i is the probability of the i-th category, crying is category 1, and non-crying is category 2. For example, output a This application uses the maximum value of 0.53 in the convolutional neural network prediction vector to compare with the judgment threshold to determine the category. The judgment threshold can be adjusted according to the actual needs of the user, and the user adjusts the balance between missed reports and false alarms according to his own needs, and the range of user adjustment has a certain boundary, such as limited to 0.9 and 1. Within this range, no matter how to adjust, missed reports and false alarms are within a reasonable range, and the adjustment has a certain effect on false alarms and missed reports. In this way, adjusting the threshold only reflects the user's bias. If the user wants to miss as little crying as possible, it can be adjusted to the side that is biased towards 0.9, for example, it can be 0.901; if the user wants to have as few false alarms as possible, it can be adjusted to the side that is biased towards 1, for example, it can be 0.971, which has little impact on the overall index. Therefore, the value of the judgment threshold is only a manifestation of bias, rather than a key factor that has a significant impact on the results. Different from the related technical solutions in which the developer solidifies the threshold in the development stage, this application can open this threshold to the user for self-adjustment, which can improve the interactivity of the product and enhance the user experience on the basis of balancing missed reports and false alarms.

[0058] Compared with the above-mentioned binary classification task prediction single value output and the rigid threshold given by the developer, the multi-classification task prediction vector output and the judgment threshold given by the user in the present application can adjust the threshold according to the actual needs of the user, reflecting the bias of the user's choice, and can also improve the interactivity of the detection device 100 on the basis of balancing missed alarms and false alarms, thereby enhancing the user experience.

[0059] See also Figure 2 In some embodiments, step 02 includes the sub-steps of:

[0060] 021, intercepting the audio sampling signal with a preset window size to obtain a target audio segment;

[0061] 022, process the target audio segment and obtain the logarithmic Mel spectrum feature.

[0062] Step 021 sequentially captures multiple target audio segments from the starting position of the original audio sampling signal according to the preset step frequency and window size. These audio segments are used as target audio units for subsequent step 022 processing.

[0063] The step frequency (or step length) refers to the number of sample points skipped on the original audio signal each time the audio segment is intercepted. A smaller step frequency will result in more overlap, which may capture more details, but will also increase the amount of calculation. A larger step frequency will reduce overlap and reduce the amount of calculation, but may miss some important transient information.

[0064] The window size refers to the number of sample points contained in each audio segment. This size should be determined based on the characteristics of the audio signal and the requirements of subsequent processing. For example, in crying detection, a window size that can contain a complete crying event is preferred.

[0065] In the process of detecting the crying of a baby, it is not necessary to judge whether the baby is crying in real time and continuously, but it is necessary to accurately identify whether this sound phenomenon exists within a period of time. The above processing method is more in line with the actual use scenario of users, that is, to detect crying within a certain time window, rather than requiring the system to respond without delay. As long as the system can detect the crying of a baby in a short time (such as within a few seconds) and give corresponding feedback or response, it can meet the basic needs of users.

[0066] In one embodiment, the number of sample points of audio data with a duration of 1 second is used as the step size, and the window size is set to the number of sample points with a duration of 5 seconds, that is, the first target audio segment is given in the first 5 seconds of the audio data, and a target audio segment is given every subsequent second. The sliding window has a 4-second overlap. Therefore, starting from the data of the 5th second in the audio stream, the data of each second will be used for subsequent signal processing and logarithmic transformation 5 times in succession, and will also be input into the detection module 30 5 times to obtain 5 prediction results.

[0067] Step 022 generates a logarithmic Mel spectrum feature for each target audio segment through signal processing and logarithmic transformation. These features will be used for subsequent crying detection or classification tasks.

[0068] In some embodiments, the crying detection result includes a positive detection result and a negative detection result, and the determination threshold is used to compare with the classification result to obtain the crying detection result, including:

[0069] When the classification result is less than the decision threshold, a negative detection result is generated;

[0070] When the classification result is greater than or equal to the determination threshold, a positive detection result is generated.

[0071] Specifically, a positive detection result represents a category that the user is interested in, that is, the crying of a baby is successfully detected. A negative detection result represents a category that the user is not interested in, that is, the crying of a baby is not detected, which may be environmental noise, adult voice or other non-crying audio.

[0072] The decision threshold is used to compare with the classification result (usually a probability value) to obtain the detection result. If the classification result is less than the decision threshold, it means that the audio signal is more likely to belong to the "negative class", that is, the baby crying is not detected, so a negative detection result should be generated. If the classification result is greater than or equal to the decision threshold, it means that the audio signal is more likely to belong to the "positive class", that is, the baby crying is detected, so a positive detection result should be generated.

[0073] In some embodiments, the detection result can be directly sent to the customer as a crying detection result.

[0074] See also Figure 3 In detail, step 022 includes:

[0075] 0221, applying short-time Fourier transform to each target audio segment, converting it from the time domain to the frequency domain to obtain a spectrogram;

[0076] 0222, applying a Mel filter bank to filter the spectrogram to obtain a Mel spectrogram;

[0077] 0223, take the logarithm of each frequency band of the Mel spectrum graph to generate logarithmic Mel spectrum features.

[0078] See also Figure 4 In some embodiments, step 03 includes the sub-steps of:

[0079] 031, sequentially inputting multiple logarithmic Mel spectrum features into the detection model to generate multiple prediction results, each logarithmic Mel spectrum feature corresponding to a prediction result;

[0080] 032, determine the crying detection result according to multiple prediction results and the maximum positive prediction threshold, and the maximum positive prediction threshold can be adjusted.

[0081] Step 031 inputs the logarithmic Mel spectrum features generated in step 02 into the trained detection model in sequence.

[0082] The detection model includes convolutional neural network algorithms, prediction functions, and judgment thresholds. The convolutional neural network has been introduced before and will not be repeated here. The detection model is a deep learning model that has been trained with a large amount of labeled data. It can process the input logarithmic Mel spectrum features and output a prediction result. This prediction result can be a probability value, indicating the possibility that the input audio segment belongs to the "baby crying" category. For each input logarithmic Mel spectrum feature, the detection model will generate a corresponding prediction result. Therefore, for the crying detection at a specific moment, that is, a certain second, if multiple logarithmic Mel spectrum features are input, multiple prediction results will be generated.

[0083] In this embodiment, the prediction result obtained by the detection model can also be used as an intermediate value (not directly output), and the statistical result is obtained after statistical processing of the prediction result, and finally the statistical result is output to the user as the detection result. Compared with directly outputting the prediction result obtained by the detection model to the user, it avoids errors caused by inaccurate single prediction results and improves the accuracy of crying detection.

[0084] See also Figure 5In some embodiments, the crying detection result includes a positive detection result and a negative detection result, and step 032 includes sub-steps:

[0085] 0321, when the number of positive prediction results counted is greater than or equal to the maximum positive prediction threshold, the detection result is determined to be a positive detection result;

[0086] 0322, when the number of positive prediction results counted is less than the maximum positive prediction threshold, the detection result is determined to be a negative detection result.

[0087] The maximum positive prediction threshold is a preset threshold used to convert the prediction result into the final crying detection result. When the prediction result exceeds this threshold, the detection model considers that the baby crying is detected; otherwise, it is considered that it is not detected. This threshold can be adjusted by the user according to actual needs to achieve the user's ideal detection effect.

[0088] After obtaining multiple prediction results, the detection model needs to determine the final crying detection result based on these prediction results and the maximum positive prediction threshold. To do this, the judgment of multiple prediction results is completed by calculating the statistical function, and the value of the statistical function is the cumulative sum of all the prediction results. Since the prediction results are output one by one, in the actual process, it is not necessary to count after all the prediction results are output, but to output a prediction result, count it once, and when the maximum positive prediction threshold is met, the result can be output.

[0089] The statistical function value is compared with the maximum positive prediction threshold. When the statistical function value is greater than or equal to the maximum positive prediction threshold, the detection result is determined to be a positive detection result, and it is considered that the baby crying sound is detected in this second of the entire audio signal. When the statistical function value is less than the maximum positive prediction threshold, the detection result is determined to be a negative detection result, and it is considered that the baby crying sound is not detected in this second.

[0090] In one embodiment, a square represents 1 second of audio data, and the neural network has a sliding window of 5 seconds. Assuming that the audio activation area is detected by the Voice Activity Detection (VAD) technology at the 7th second, that is, at the moment v=7, to determine whether the activation area is crying, the statistical function MP() starts counting from the first time the sliding window of the neural network includes the activation area until the sliding window includes the activation area for the last time, and the maximum predicted number of positive samples is obtained, that is, Then the maximum positive prediction threshold determined by the user finally gives the event level judgment result. is the prediction result of the convolutional neural network of the sliding window starting from a certain moment, and is defined as is the output sequence predicted by the convolutional neural network on the time axis. If 4 out of 5 prediction results are positive, the maximum prediction threshold is 3, and 4 is greater than 3, the detection result is determined to be a positive detection result, and it is considered that the baby crying sound is detected at the 7th second in the entire audio signal.

[0091] See also Figure 6 In certain embodiments, the detection method further comprises:

[0092] 001, obtain a test set of baby crying sounds, the test set includes a positive sample test set and a negative sample test set, the positive sample test set includes positive crying sample data, and the negative sample test set includes negative crying sample data;

[0093] 002, respectively processing the crying sound positive sample data and the crying sound negative sample data to obtain the positive logarithmic Mel spectrum features of the crying sound positive sample data and the negative logarithmic Mel spectrum features of the crying sound negative sample data;

[0094] 003, through the constructed detection model, the positive logarithmic Mel spectrum features and the negative logarithmic Mel spectrum features are processed respectively in the training stage to obtain the training output result;

[0095] 004, based on the training output results, the loss value of the constructed detection model is calculated by the loss function;

[0096] 005. Modify the parameters of the constructed detection model according to the loss value to obtain a trained detection model.

[0097] The above methods can be implemented by computers and deep learning frameworks. Deep learning frameworks are software platforms that support the implementation and operation of deep learning algorithms. They provide various tools and libraries required to build and train neural networks, allowing developers to design, test and deploy deep learning models more conveniently. Common deep learning frameworks include TensorFlow, PyTorch, Keras, etc. In deep learning frameworks, convolutional neural network (CNN) models are constructed using convolutional layers, pooling layers, activation functions (such as ReLU), fully connected layers, etc. These layers are stacked to extract high-level features from input features. The positive logarithmic Mel spectrum features and negative logarithmic Mel spectrum features are used as inputs separately (or after merging) and forward propagated through the CNN model. In each layer, the data is processed and passed to the next layer until the output layer produces a prediction result. Based on the output predicted by the model and the true label (i.e., positive sample or negative sample), the loss value of the model is calculated through a loss function (such as cross entropy loss, etc.). The smaller the loss value, the more accurate the model's prediction. Based on the calculated loss value, the model parameters are modified using an optimization algorithm (such as gradient descent, Adam, etc.). This process is repeated for multiple iterations until a certain stopping condition is met (such as the loss value no longer decreases significantly, the preset number of iterations is reached, etc.).

[0098] In this way, a trained detection model can be obtained, which can accurately identify the crying sound of a baby.

[0099] In some embodiments, the test set is extended, and the extension processing method includes at least one of time shift processing, spectrum enhancement processing, noise addition processing, or reverberation addition processing.

[0100] Specifically, time shift processing refers to shifting the crying signal on the time axis to simulate the slight difference in time of the signal. This processing can make the model unaffected by the time when the signal appears during detection, thus improving the temporal robustness of the model.

[0101] Spectral enhancement processing simulates the changes of signals in different environments or devices by enhancing certain frequency components of crying signals. This processing can help the model better identify crying signals with changes in the spectrum and improve the spectral robustness of the model.

[0102] Noise processing is to add different types of noise (such as Gaussian noise, salt and pepper noise, etc.) to the crying signal to simulate the interference that may exist in the real environment. This processing helps the model to accurately identify crying signals in a noisy environment and improves the model's noise resistance.

[0103] Reverberation refers to the phenomenon that sound is produced after multiple reflections in a closed or semi-enclosed space. Adding reverberation processing can simulate the propagation of crying signals in different spatial environments, so that the model can better adapt to the crying detection task in different spatial environments.

[0104] These extended processing methods can be used individually or in combination to produce more diverse test sets. Using different processing methods to increase the diversity and complexity of the data, and training and evaluating on these processed test sets can significantly improve the performance and robustness of the cry detection model.

[0105] In some embodiments, the convolutional neural network algorithm includes at least one of an AlexNet network algorithm, a ResNet network algorithm, an Inception network algorithm, and a convolutional recurrent neural network algorithm.

[0106] Specifically, the AlexNet network consists of five convolutional layers and three fully connected layers, with a relatively simple but efficient structure. The ReLU (Rectified Linear Unit) activation function was successfully applied in large networks to solve the gradient vanishing problem of traditional activation functions (such as sigmoid and tanh). Local response normalization (LRN) was added after the ReLU layer to improve the generalization ability of the network. Overlapping pooling can reduce overfitting and improve performance compared to traditional non-overlapping pooling methods.

[0107] The ResNet network algorithm introduces residual blocks, which allows the network to be trained deeper and solves the degradation problem in deep neural network training. The skip connections in the residual modules allow the gradient to propagate directly, avoiding the gradient vanishing problem in deep networks. The ResNet network can train very deep networks (such as ResNet-50, ResNet-101, etc.), performs well in multiple visual tasks, and is the basic model for many tasks.

[0108] The Inception network algorithm uses convolution kernels of the same scale to process input images in parallel, which can extract features of different scales and enhance the network's feature extraction capabilities. By using 1x1 convolution kernels to reduce and combine features, the number of model parameters and computational complexity is reduced, while improving the performance of the model, making it suitable for a variety of image processing tasks.

[0109] Convolutional recurrent neural networks, such as ConvLSTM, are a type of network that combines convolution and recurrent properties. They are mainly used to process data with spatiotemporal dependencies (such as videos, time series images, etc.). Convolutional recurrent neural networks extract spatial features through convolution operations, while using the recurrent properties of long short-term memory (LSTM) networks to capture temporal dependencies. Convolutional recurrent neural networks are particularly suitable for video processing tasks such as video frame prediction, video analysis, and action recognition. Compared with simple convolutional neural networks or recurrent neural networks, convolutional recurrent neural networks have higher model complexity and require more computing resources.

[0110] In the present embodiment, the convolutional neural network adopted is a simplified AlexNet network. The AlexNet network can automatically extract high-level feature representations from the original sound data through multi-layer convolution and pooling operations. This is helpful to capture more subtle and abstract features for the complex sound signal of baby crying, thereby improving the accuracy of detection. The simplified AlexNet network only contains convolutional layer, fully connected layer, maximum pooling layer and normalization layer, and the final loss function also uses the common cross entropy loss (CrossEntropy Loss). The simple structure means lower computational complexity and faster reasoning speed, which is an advantage for the baby crying detection task with higher real-time requirements. In addition, the model structure of AlexNet has certain scalability and flexibility. According to the specific needs of the baby crying detection task, the model can be appropriately modified and expanded, such as adding more convolutional layers or fully connected layers, or changing the size and quantity of convolution kernels, etc. This flexibility helps to better adapt to different application scenarios and data sets.

[0111] See also Figure 7 In certain embodiments, the detection method further comprises:

[0112] 04. Generate a reminder message based on the crying detection result;

[0113] 05. Send a reminder message.

[0114] In addition to the core crying recognition and detection functions, it also includes the function of generating and sending reminder messages based on the detection results. This step is very important for practical application scenarios because it allows the system to convey the detected crying information to relevant personnel in a timely manner so that they can take appropriate actions. The reminder message can contain content such as "Baby crying detected, please check immediately" and can be accompanied by additional information such as time and location. The sending method can be to send an audible alarm through a speaker to attract attention or to send a notification push through an application on a mobile device or computer. The sending method can also be to send a text message or make a call to a preset mobile phone number.

[0115] See also Figure 8 The embodiment of the present application also provides a baby crying detection device 100, and the detection device 100 includes an acquisition module 10, a processing module 20, a detection module 30 and a calculation module 40.

[0116] In some embodiments, the detection device 100 may be a surveillance camera, a baby monitor, a baby care robot, or a crying detector. This embodiment is described by taking the detection device 100 as a surveillance camera as an example, that is, the detection device 100 is applied to but not limited to surveillance cameras.

[0117] The acquisition module 10 is mainly responsible for collecting the audio of the baby as an audio sampling signal. The acquisition module 10 can be a baby monitor, a mobile phone, a voice recorder or other recording equipment. The acquisition module 10 needs to be close enough to the sound source to ensure that the quality of the collected audio signal is high enough for subsequent processing and analysis. The acquisition module 10 receives the audio signal through a microphone or an audio input interface and converts it into a digital signal. These digital signals are then stored as audio files or transmitted to the processing module 20 in real time.

[0118] The processing module 20 is responsible for preprocessing and feature extraction of the acquired audio sampling signal. Preprocessing may include noise reduction, echo removal, filtering and other operations to improve the quality of the audio signal. Feature extraction is to extract features useful for baby crying detection from the processed audio signal, such as logarithmic Mel spectrum features. Logarithmic Mel spectrum features are a commonly used audio feature that simulates the nonlinear perception characteristics of the human ear to frequency. Logarithmic Mel spectrum features can be obtained by framing, windowing, short-time Fourier transform (STFT), Mel filter bank processing and logarithmic operation of the audio signal.

[0119] The detection module 30 inputs the logarithmic Mel spectrum features extracted by the processing module 20 into the trained detection model to output the crying detection result. The detection model contains a convolutional neural network algorithm, a prediction function, and a judgment threshold, etc., which are used to classify and judge the input features.

[0120] The calculation module 40 is used to process the positive logarithmic Mel-spectrogram features and the negative logarithmic Mel-spectrogram features respectively during the training phase through the constructed detection model to obtain the training output results; based on the training output results, the loss value of the constructed detection model is calculated through the loss function; according to the loss value, the parameters of the constructed detection model are corrected to obtain the trained detection model.

[0121] In some embodiments, step 01 can be implemented by the acquisition module 10, step 02 can be implemented by the processing module 20, and step 03 can be implemented by the detection module 30. In other words, the acquisition module 10 can be used to acquire an audio sampling signal of a baby's cry; the processing module 20 can be used to process the audio sampling signal to generate a logarithmic Mel spectrum feature; the detection module 30 can be used to input the logarithmic Mel spectrum feature into a trained detection model for detection to output a crying detection result. Among them, the detection model includes a convolutional neural network algorithm, a prediction function and a determination threshold, the determination threshold can be adjusted, the convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature, the prediction function determines the classification result based on the multi-classification task result, and the determination threshold is used to compare with the classification result to obtain a crying detection result.

[0122] In some embodiments, sub-steps 021 and 022 can be implemented by the processing module 20, or in other words, the processing module 20 can be used to intercept the audio sampling signal in sequence with a preset step frequency and a preset window size to obtain multiple target audio segments; and process all the target audio segments separately to generate multiple logarithmic Mel-spectrogram features, each target audio segment corresponding to a logarithmic Mel-spectrogram feature.

[0123] In some embodiments, step 031 and step 032 can be implemented by the detection module 30, or in other words, the detection module 30 can be used to sequentially input multiple logarithmic Mel-spectrogram features into the detection model to generate multiple detection results, each logarithmic Mel-spectrogram feature corresponds to a detection result; and determine the crying detection result based on the multiple detection results and the maximum positive prediction threshold, and the maximum positive prediction threshold can be adjusted.

[0124] In some embodiments, steps 0321, 0322, and 0323 may be implemented by the detection module 30, or in other words, the detection module 30 may be used to count the number of positives in the detection results; when the number of positives in the detection results is greater than or equal to the maximum positive prediction threshold, the detection result is determined to be a positive detection result; when the number of positives in the detection results is less than the maximum positive prediction threshold, the detection result is determined to be a negative detection result.

[0125] In some embodiments, step 001 can be implemented by the acquisition module 10, step 002 can be implemented by the processing module 20, and steps 003, 004 and 005 can be implemented by the calculation module 40. In other words, the acquisition module 10 can be used to obtain a test set of infant crying, the test set includes a positive sample test set and a negative sample test set, the positive sample test set includes positive crying sample data, and the negative sample test set includes negative crying sample data. The positive sample test set contains real infant crying data (i.e., positive crying sample data), while the negative sample test set contains non-crying data (i.e., negative crying sample data), which may be environmental noise, adult voices or other non-infant crying audio. The processing module 20 can be used to process the positive crying sample data and the negative crying sample data respectively to obtain the positive logarithmic Mel spectrum features of the positive crying sample data and the negative logarithmic Mel spectrum features of the negative crying sample data. The calculation module 40 includes a deep learning framework. Step 003, step 004 and step 005 can be implemented by the deep learning framework of the calculation module 40. The calculation module 40 can be used to process the positive logarithmic Mel spectrum features and the negative logarithmic Mel spectrum features respectively in the training stage through the constructed detection model to obtain the training output result; based on the training output result, the loss value of the constructed detection model is calculated through the loss function; according to the loss value, the parameters of the constructed detection model are corrected to obtain the trained detection model.

[0126] The embodiment of the present application also provides a system for detecting a baby's crying. The detection system may include a processor, which can be used to implement the above-mentioned detection method, or in other words, the processor is used to obtain an audio sampling signal of a baby's crying; and process the audio sampling signal to generate a logarithmic Mel-spectrogram feature; the processor can also be used to input the logarithmic Mel-spectrogram feature into a trained detection model for detection to obtain a crying detection result.

[0127] In some embodiments, the detection system may be computer hardware or software adapted to the detection device 100, and may execute the method for detecting the crying of a baby when the detection device 100 is started. The detection system may be a discrete component assembled in a certain manner to have the aforementioned functions, or a chip having the aforementioned functions in the form of an integrated circuit, or a computer software code segment that enables the computer to have the aforementioned functions when running on the computer.

[0128] The embodiments of the present application also provide a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by a processor, the processor executes the method for detecting a baby's cry in any one of the above embodiments.

[0129] In some embodiments, the non-volatile computer-readable storage medium may be a storage medium built into the detection device 100, such as a memory, or a storage medium that can be plugged into the detection device 100, such as an SD card.

[0130] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "certain embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0132] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for detecting a baby's crying sound, characterized in that: The detection method comprises: Get audio sampling signal; Processing the audio sampling signal to generate a logarithmic Mel spectrum feature; The logarithmic Mel spectrum feature is input into a trained detection model for detection to obtain a crying detection result, wherein the detection model includes a convolutional neural network algorithm, a prediction function and a determination threshold, the determination threshold is adjustable, the convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature, the prediction function determines the classification result based on the multi-classification task result, and the determination threshold is used to compare with the classification result to obtain the crying detection result.

2. The detection method according to claim 1, characterized in that: Processing the audio sampling signal to generate a logarithmic Mel spectrum feature includes: Intercepting the audio sampling signal with a preset window size to obtain a target audio segment; The target audio segment is processed to obtain the logarithmic Mel spectrum feature.

3. The detection method according to claim 2, characterized in that: The crying sound detection result includes a positive detection result and a negative detection result. The determination threshold is used to compare with the classification result to obtain the crying sound detection result, including: generating the negative detection result when the classification result is less than the determination threshold; When the classification result is greater than or equal to the determination threshold, the positive detection result is generated.

4. The detection method according to claim 2, characterized in that: The logarithmic Mel spectrum feature is input into the trained detection model for detection to output the crying sound detection result, including: Inputting the plurality of logarithmic Mel spectrum features into the detection model in sequence to generate a plurality of prediction results, wherein each logarithmic Mel spectrum feature corresponds to one prediction result; The crying detection result is determined according to the plurality of prediction results and a maximum positive prediction threshold, and the maximum positive prediction threshold is adjustable.

5. The detection method according to claim 4, characterized in that: The crying detection result includes a positive detection result and a negative detection result, and determining the crying detection result according to the plurality of prediction results and a maximum positive prediction threshold includes: In a case where the number of positive prediction results counted is greater than or equal to the maximum positive prediction threshold, determining that the detection result is the positive detection result; When the number of positive prediction results counted is less than the maximum positive prediction threshold, the detection result is determined to be the negative detection result.

6. The detection method according to claim 1, characterized in that: The detection method further comprises: Acquire a test set of infant crying sounds, wherein the test set includes a positive sample test set and a negative sample test set, wherein the positive sample test set includes positive crying sample data, and the negative sample test set includes negative crying sample data; Processing the positive crying sound sample data and the negative crying sound sample data respectively to obtain positive logarithmic Mel-spectrogram features of the positive crying sound sample data and negative logarithmic Mel-spectrogram features of the negative crying sound sample data; Through the constructed detection model, the positive logarithmic Mel spectrum feature and the negative logarithmic Mel spectrum feature are processed respectively in the training stage to obtain a training output result; Based on the training output result, calculating the loss value of the constructed detection model through a loss function; The parameters of the constructed detection model are modified according to the loss value to obtain the trained detection model.

7. The detection method according to claim 6, characterized in that: The detection method further comprises: The test set is extended, and the extension processing method includes at least one of time offset processing, spectrum enhancement processing, noise addition processing or reverberation addition processing.

8. The detection method according to claim 1, characterized in that: The convolutional neural network algorithm includes at least one of an AlexNet network algorithm, a ResNet network algorithm, an Inception network algorithm, and a convolutional recurrent neural network algorithm.

9. The detection method according to any one of claims 1 to 8, characterized in that: The detection method further comprises: generating a reminder message according to the crying sound detection result; The reminder message is issued.

10. A device for detecting a baby's crying, characterized in that: The detection device comprises: An acquisition module, used for acquiring an audio sampling signal of a baby crying; A processing module, used for processing the audio sampling signal to generate a logarithmic Mel spectrum feature; A detection module is used to input the logarithmic Mel spectrum feature into a trained detection model for detection to obtain a crying detection result, wherein the detection model includes a convolutional neural network algorithm, a prediction function and a determination threshold, the determination threshold is adjustable, the convolutional neural network algorithm generates a multi-classification task result based on the logarithmic Mel spectrum feature, the prediction function determines the classification result based on the multi-classification task result, and the determination threshold is used to compare with the classification result to obtain the crying detection result.

11. A system for detecting a baby's crying, characterized in that: The invention comprises a processor, a memory and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method for detecting the crying sound of a baby according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to execute the method for detecting a baby's cry according to any one of claims 1 to 9.

Images