Environment-adaptive baby cry detection method and system, storage medium and device

Abstract

The present invention relates to an environment-adaptive baby cry detection method and system, a storage medium and a device. The environment-adaptive baby cry detection method and system, the storage medium and the device aim to address the technical shortcomings of existing newborn and baby cry detection and recognition methods of performance deterioration in new environments, poor generalization ability, insufficient human calibration and training data, and difficulties in achieving sufficient detection performance in real clinical environments, which are often manifested as low precision and high recall rate. The method comprises the following steps: S1, environment simulation data augmentation based on speech linear superposition characteristics: collecting speech public data sets and collecting public baby cry data, using public labeled data to perform data augmentation, and, by using samples from the environment simulation data augmentation, training a classifier, so as to learn cry characteristics in synthetic chaotic environments; and S2, mutual learning for environment adaptation: by using an exponential moving average method and a pseudo label training mode, establishing an environment-adaptive cry detection learning closed loop, continuously deriving cry data distribution in new environments, and correcting the cry characteristics obtained in step S1, thus implementing environment-adaptive cry detection.

Description

Method, system, storage medium and device for detecting baby crying based on scene adaptation Technical Field

[0001] The present invention relates to speech recognition and artificial intelligence monitoring technology, and in particular to a scene-adaptive infant cry detection method, system, storage medium and device, which improves the newborn baby cry detection and recognition method and belongs to the field of application technology of computer technology combined with artificial intelligence in the medical and health field. Background Art

[0002] As a biomarker, infant crying can reveal information at multiple levels of abstraction, including a baby's physical condition, needs, and emotions. Clinical studies have shown that acoustic analysis of infant crying can provide valuable medical support, such as understanding the cause of crying, assessing the development of premature infants, and diagnosing specific diseases. Therefore, developing an accurate automated method for detecting infant crying is crucial for neonatal healthcare.

[0003] Audio-based machine learning methods for automatic infant cry detection have been widely studied recently. However, limitations remain for neonatal cry detection. Specifically, most methods struggle to achieve adequate detection performance in real-world clinical settings, often exhibiting low precision and high recall. These methods are evaluated in controlled environments, which differ from real-world clinical settings, characterized by a variety of background noises, such as conversation, footsteps, and alarms. This additional background noise degrades the performance of most methods in real-world settings. Furthermore, these methods overly rely on threshold-based background noise filtering. In many cases, the threshold is dependent on the location of the acquisition device, room size, and the variability of various sound sources, meaning that most methods must manually adjust the threshold to adapt to new environments. However, collecting and annotating speech data for each new environment and adjusting the threshold is impractical, especially in clinical settings. These practical factors hinder the rapid application of infant cry detection in the real world. Technical issues

[0004] In summary, the purpose of the present invention is to solve the problems of existing newborn baby cry detection and recognition methods, such as deteriorating performance in new environments, poor generalization ability, insufficient manual calibration and training data, difficulty in achieving sufficient detection performance in real clinical environments, and often exhibiting technical deficiencies of low precision and high recall rate. Instead, a scene-adaptive baby cry detection method, system, storage medium and device are proposed. Technical Solutions

[0005] A scene-adaptive baby crying detection method comprises the following steps:

[0006] Step S1, scene simulation data enhancement based on the linear addition characteristic of speech: collect public speech datasets and public infant crying data, use the publicly annotated data for data enhancement, and use the scene simulation data-enhanced samples to train the classifier to learn the crying characteristics in the synthetic chaotic scene;

[0007] Step S2, scene-adaptive mutual learning; the exponential moving average method and pseudo-label training are used to establish a scene-adaptive crying detection learning loop, continuously derive the crying data distribution in the new environment, and correct the crying characteristics obtained in step S1 to achieve environment-adaptive crying detection.

[0008] A scene-adaptive baby crying detection system, the system comprising:

[0009] The scene simulation data augmentation unit based on the linear addition property of speech is used to collect public speech datasets and public infant crying data, perform data augmentation using publicly annotated data, and train a classifier using samples augmented with scene simulation data to learn the characteristics of crying in synthetic chaotic scenes.

[0010] A scene-adaptive mutual learning unit is used to establish a scene-adaptive crying detection learning loop using the exponential moving average method and pseudo-label training. This unit continuously derives the crying data distribution in new environments and corrects the crying characteristics obtained in the scene simulation data enhancement unit based on the linear addition characteristics of speech, thereby achieving environment-adaptive crying detection.

[0011] A storage medium stores a program capable of implementing the above-mentioned scene-adaptive baby crying detection method.

[0012] An electronic device includes: a voice acquisition module, a processor, and a storage medium storing a program. The voice acquisition module acquires ambient audio in real time and provides it to the processor. The storage medium stores a program capable of implementing the above-mentioned scene-adaptive infant crying detection method. The processor performs signal processing on the ambient audio acquired in real time by the voice acquisition module according to the computer program stored in the storage medium.

[0013] The features of the technical solution that further define the present invention include:

[0014] In step S1, multiple random factors are used during data augmentation to simulate audio clips that may appear in the scene. The multiple random factors include:

[0015] (i) Sound source random factor: The sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that emit synchronous sounds in the scene;

[0016] (ii) Sound length random factor: The sample selected in (i) is cropped using the sound length random factor to simulate audio segments of different lengths;

[0017] (iii) Starting random factor: The starting random factor is used to determine the addition position between samples, so as to achieve the addition combination between different speech samples, thereby simulating the different characteristics of speech utterance time in real environments.

[0018] In step S2, establishing a scene-adaptive crying detection learning closed loop includes the following sub-steps:

[0019] Step 2-1: Create a new classifier using the classifier structure and model parameters obtained in step S1, and define it as an EMA classifier;

[0020] Step S2-2: Freeze the EMA classifier parameters, make predictions for the data in the new scenario, and generate pseudo labels;

[0021] Step S2-3: Unfreeze the EMA classifier parameters and use the parameter update method of the EMA classifier to transfer the weights in the classifier to the EMA classifier to improve the quality of the pseudo labels generated by the EMA classifier;

[0022] Step S2-4: By continuously iterating steps S2-2 and S2-3, the classifier continuously derives the crying data distribution in the new environment and corrects the crying characteristics obtained in step S1 to achieve environment-adaptive crying detection.

[0023] In step S1, when collecting public speech data sets, use an audio download tool to collect public speech data sets from a public audio download website, and download the original audio data set fragments containing environmental sounds during collection; when collecting public baby crying data, download the baby crying data from a website containing baby crying data.

[0024] The public speech data set uses set duration tags to capture relevant speech segments.

[0025] The label is a second-level label.

[0026] The public speech dataset and the baby crying data are preprocessed to obtain a unified data paradigm.

[0027] The preprocessing is to segment the collected audio data according to the set sampling frequency and the set duration to obtain data samples.

[0028] The sampling frequency is set to downsample to 8 kHz.

[0029] The segmentation process is performed using a non-overlapping sliding window.

[0030] The set duration is 2.5 seconds.

[0031] In step S2-2, samples with high confidence and corresponding pseudo labels are selected by the confidence threshold method to train the classifier.

[0032] In step S2-2, when training the classifier, public data is used to preliminarily train the classifier, and multiple random factors are used to collaborate to simulate speech data in clinical scenarios. Then, a new classifier is created for training, and the structure of the classifier adopts a deep learning model.

[0033] The deep learning model is a fully connected neural network, a convolutional neural network or a recurrent neural network.

[0034] In step S2-2, when training the classifier, the data used also includes collected clinical data.

[0035] When collecting clinical data, a single-channel microphone is used to collect real-time audio in a clinical environment and the same preprocessing is performed.

[0036] In the scene simulation data enhancement unit based on the linear addition characteristic of speech, multiple random factors are used during data enhancement to simulate audio clips that may appear in the scene. The multiple random factors include:

[0037] (i) Sound source random factor: The sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that emit synchronous sounds in the scene;

[0038] (ii) Sound length random factor: The sample selected in (i) is cropped using the sound length random factor to simulate audio segments of different lengths;

[0039] (iii) Starting random factor: The starting random factor is used to determine the addition position between samples, so as to achieve the addition combination between different speech samples, thereby simulating the different characteristics of speech utterance time in real environments.

[0040] In the scene-adaptive mutual learning unit, establishing a scene-adaptive crying detection learning closed loop includes:

[0041] Classifier creation unit: used to create a new classifier based on the classifier structure and model parameters obtained in the scene simulation data enhancement unit based on the linear addition characteristic of speech, and defined as an EMA classifier;

[0042] EMA classifier parameter freezing unit: used to freeze the EMA classifier parameters, predict data in new scenarios, and generate pseudo labels;

[0043] EMA classifier parameter solution unit: used to unfreeze the EMA classifier parameters and use the EMA classifier parameter update method to transfer the weights in the classifier to the EMA classifier, thereby improving the quality of the pseudo labels generated by the EMA classifier;

[0044] Iteration unit: It is used to continuously iterate the scene simulation data enhancement unit based on the linear addition characteristics of speech and the EMA classifier parameter solution unit, so that the classifier can continuously deduce the crying data distribution in the new environment, and correct the crying characteristics obtained in the scene simulation data enhancement unit based on the linear addition characteristics of speech, so as to realize environment-adaptive crying detection. Through the continuous iteration of the mutual learning method, the performance of the classifier in the new environment is improved, thereby realizing scene-adaptive plug-and-play crying detection.

[0045] In the scene simulation data enhancement unit based on the linear addition characteristics of speech, when collecting public speech data sets, an audio download tool is used to collect public speech data sets from a public audio download website, and the original audio data set fragments containing environmental sounds are downloaded during the collection; when collecting public baby crying data, the baby crying data is downloaded from a website containing baby crying data.

[0046] The public speech data set uses set duration tags to capture relevant speech segments.

[0047] The label is a second-level label.

[0048] The public speech dataset and the baby crying data are preprocessed to obtain a unified data paradigm.

[0049] The preprocessing is to segment the collected audio data according to the set sampling frequency and the set duration to obtain data samples.

[0050] The sampling frequency is set to downsample to 8 kHz.

[0051] The segmentation process is performed using a non-overlapping sliding window.

[0052] The set duration is 2.5 seconds.

[0053] In the EMA classifier parameter freezing unit, samples with high confidence and corresponding pseudo labels are selected by the confidence threshold method to train the classifier.

[0054] In the EMA classifier parameter freezing unit, when training the classifier, public data is used to preliminarily train the classifier, multiple random factors are used to collaborate to simulate speech data in clinical scenarios, and then a new classifier is created for training. The structure of the classifier adopts a deep learning model.

[0055] The deep learning model is a fully connected neural network, a convolutional neural network or a recurrent neural network.

[0056] In the EMA classifier parameter freezing unit, when training the classifier, the data used also includes collected clinical data.

[0057] When collecting clinical data, a single-channel microphone is used to collect real-time audio in a clinical environment and the same preprocessing is performed. Beneficial effects

[0058] Based on the linear addition properties of speech, the present invention performs scene-simulation data enhancement on existing data to obtain speech data similar to clinical scenarios, forcing the classifier to learn the characteristics of infant cries in the synthesized chaotic scenes. Unsupervised mutual learning methods are then used to continuously derive the distribution of crying data in new clinical scenarios, and the learned infant crying characteristics in the synthesized chaotic scenes are corrected to achieve scene-adaptive crying detection, improving the robustness and generalization ability of the crying detection model. The present invention fully utilizes existing public data sets to improve the performance of infant crying detection, and has good robustness for crying detection in clinical environments, maintaining recall rate while improving precision. The algorithm is effective for any deep classifier and can achieve an adaptive process for clinical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] FIG1 is a diagram of a plug-and-play baby crying detection framework based on scene adaptation according to the present invention;

[0060] FIG2 is a schematic diagram of scene simulation data enhancement based on speech addition characteristics according to the present invention. Best Mode for Carrying Out the Invention

[0061] The baby crying detection method of the present invention will be further described below in conjunction with the accompanying drawings and preferred specific embodiments of the present invention.

[0062] As shown in Figures 1 and 2, the scene-adaptive infant cry detection method claimed in the present invention is mainly divided into two stages: the first stage is a scene simulation data enhancement method based on the linear addition characteristics of speech, and the second stage is a scene-adaptive mutual learning method. It can fully utilize existing public datasets to improve infant cry detection performance, has good robustness for cry detection in clinical environments, maintains recall rate, and improves precision. It is effective for any deep classifier and can realize the adaptive process of clinical scenarios.

[0063] The present invention specifically includes the following steps:

[0064] Step S1: Scenario simulation data enhancement based on the linear addition property of speech: Collect public speech datasets and public infant crying data, use the rich annotated data in the public datasets for data enhancement, and use multiple random factors to simulate the audio clips that may appear in the scene. The random factors specifically include:

[0065] (i) Sound source random factor: Since multiple sound sources often occur almost simultaneously in clinical scenarios, a sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that sound synchronously in the scene;

[0066] (ii) Sound Length Random Factor: Since sounds from different sound sources in real environments usually have different durations, the sound length random factor is used to crop the samples selected in step (i) to simulate audio clips of different durations.

[0067] (iii) Starting random factor: Since the starting positions of different sounds are different, the starting random factor is used to determine the addition position between samples, so as to achieve the additive combination between different speech samples, thereby simulating the different characteristics of speech utterance time in real environments.

[0068] The present invention uses samples enhanced by scene simulation data to train a classifier and learn the crying characteristics in a synthetic chaotic scene.

[0069] In the specific implementation process, there are many methods for collecting public speech datasets. You can use existing data in public databases in the prior art, or you can use a self-built database to collect data and build a database. In this embodiment, the specific method used to collect public speech datasets is: use youtube_dl to download the original AudioSet audio clips from Youtube, where Youtube is a public website that provides various video and audio download services. In addition to Youtube, you can also collect public speech datasets from public audio and video download websites such as Vimeo, Dailymotion, and Facebook. youtube_dl is a Python library that provides command-line tools and programmable APIs for downloading videos and audio from various websites (including YouTube, Vimeo, Dailymotion, etc.). In addition to youtube_dl, in the specific implementation, you can also use: PyTube, youtube-dl-py, youtube-dl-gui or 4K Video Audio download tools such as AudioDownloader collect public speech datasets from public databases. AudioSet is a widely used dataset primarily used for audio classification and annotation tasks. The present invention collects public speech datasets to leverage the rich annotated data for data enhancement. AudioSet is a preferred implementation of the present invention. Similar datasets such as UrbanSound (primarily covering traffic noise, construction sounds, and sound events) or ESC-50 (primarily covering animal sounds, natural sounds, and household sounds) can also be used for specific implementations.

[0070] For the collected public speech dataset, relevant speech segments are intercepted using a set duration tag. Since the present invention is a method for detecting infant crying, a cycle of infant crying is usually 1 second to several seconds. If the duration of the tag is set too short or too long, it will not have much practical significance for the present invention. Therefore, in this embodiment, the duration tag is set to a second-level tag, that is, the relevant speech segments are intercepted using a second-level tag.

[0071] In the specific implementation process, there are also many methods for collecting baby crying data. You can use existing data in public databases in the prior art, or you can use a self-built database to collect data and build a database. In this embodiment, the specific method used for collecting baby crying data is: download baby crying data from the ESC-50 official website. The ESC-50 dataset is a collection of 2,000 labeled environmental audio recordings, which is suitable for benchmarking methods for environmental sound classification, including baby crying data. In specific implementation, baby crying data can also be collected from other public datasets, such as: UrbanSound Dataset (The UrbanSound dataset is a widely used audio dataset that contains various audio clips from urban environments, including baby crying), MIVIA Audio Events Dataset (MIVIA is a research project for multimedia information and visual analysis. They provide an audio event dataset that contains baby crying audio), etc.

[0072] All collected audio data are preprocessed to obtain a unified data paradigm. In this embodiment, the collected data include the collected public voice data set and the collected public baby crying data. Data preprocessing mainly includes segmenting the collected audio data according to the set sampling frequency and the set duration to obtain data samples. In this embodiment, the sampling frequency is set to downsample to 8kHz. Since the sound frequency perceived by the human body is usually between 20Hz and 20,000Hz, the common human vocal frequencies are: low male voice (Bass) is about 80Hz to 150Hz, middle male voice (Baritone) is about 100Hz to 200Hz, high male voice (Tenor) is about 150Hz to 300Hz, low female voice (Alto) is about 150Hz to 400Hz, middle female voice (Mezzo-soprano) is about 200Hz to 500Hz, and high female voice (Soprano) is about 300Hz to 1,000Hz. Although the occurrence frequency of infants is higher than that of adults, downsampling to 8kHz is not enough. Hz is also sufficient. More frequencies are mostly undisputed noises. Discarding them will reduce the amount of computation of the entire system. In specific implementation, the downsampling frequency can also be set to 7kHz or 10kHz, or a specific value can be selected according to actual needs.

[0073] When performing audio data segmentation processing, in this embodiment, a non-overlapping sliding window is used for segmentation to avoid excessive overlapping parts and high data repetition rate. The duration of the sliding window is preferably 2.5s to obtain the most optimal data duration. During specific implementation, the duration of the sliding window can also be selected as other values ​​according to actual needs.

[0074] Step S2: Scenario-adaptive mutual learning: This method uses the exponential moving average (EMA) method and pseudo-label training to establish a scenario-adaptive crying detection learning loop. This includes the following sub-steps:

[0075] Step 2-1: Create a new classifier using the classifier structure and model parameters obtained through training in step S1, which is defined as an EMA classifier in the present invention;

[0076] Step S2-2: Freeze the EMA classifier parameters, predict the data in the new scenario, and generate pseudo-labels. Due to the differences between the new and old environments, the performance of the EMA classifier is not sufficient to generate accurate crying detection results. Therefore, in this embodiment, a confidence threshold method is introduced to select samples with high confidence and corresponding pseudo-labels to train the classifier. Through this method, the classifier can reliably learn the distribution of crying data in the new environment;

[0077] In this embodiment, public data is used to preliminarily train the classifier so that it can grasp the characteristics of crying in a synthetic chaotic scene. The public data is then enhanced with scene simulation data based on speech features. Multiple random factors are used to collaborate to simulate speech data in a clinical scene. Then, a new classifier is created for training. The structure of the classifier can adopt any deep learning model, including but not limited to: fully connected neural network, convolutional neural network and recurrent neural network.

[0078] In this embodiment, when training the classifier, the data used also includes collected clinical data, and scene adaptation is achieved through a mutual learning method. In specific implementation, a single-channel microphone is used to collect real-time audio in a clinical environment, and the same preprocessing is performed to ensure that the data paradigm is the same.

[0079] Step S2-3: Unfreeze the EMA classifier parameters and use the parameter update method of the EMA classifier to transfer the weights of the classifier trained by the confidence threshold method in step S2-2 to the EMA classifier to improve the quality of the pseudo labels generated by the EMA classifier;

[0080] Step S2-4: By continuously iterating steps S2-2 and S2-3, the classifier continuously derives the distribution of crying data in the new environment and corrects the crying characteristics obtained in step S1 to achieve environment-adaptive crying detection. Through continuous iteration of the mutual learning method, the performance of the classifier in the new environment is improved, thereby achieving scene-adaptive plug-and-play crying detection.

[0081] The present invention also protects a scene-adaptive baby crying detection system, which mainly includes:

[0082] 1. Data augmentation unit for scene simulation based on the linear addition property of speech: This unit collects public speech datasets and public infant crying data, uses the rich annotated data in the public datasets for data augmentation, and uses multiple random factors to simulate audio clips that may appear in the scene. The random factors include:

[0083] (i) Sound source random factor: Since multiple sound sources often occur almost simultaneously in clinical scenarios, a sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that sound synchronously in the scene;

[0084] (ii) Sound Length Random Factor: Since sounds from different sound sources in real environments usually have different durations, the sound length random factor is used to crop the samples selected in step (i) to simulate audio clips of different durations.

[0085] (iii) Starting random factor: Since the starting positions of different sounds are different, the starting random factor is used to determine the addition position between samples, so as to achieve the additive combination between different speech samples, thereby simulating the different characteristics of speech utterance time in real environments.

[0086] The present invention uses samples enhanced by scene simulation data to train a classifier and learn the crying characteristics in a synthetic chaotic scene.

[0087] In the specific implementation process, there are many methods for collecting public speech datasets. You can use existing data in public databases in the prior art, or you can use a self-built database to collect data and build a database. In this embodiment, the specific method used to collect public speech datasets is: use youtube_dl to download the original AudioSet audio clips from Youtube, where Youtube is a public website that provides various video and audio download services. In addition to Youtube, you can also collect public speech datasets from public audio and video download websites such as Vimeo, Dailymotion, and Facebook. youtube_dl is a Python library that provides command-line tools and programmable APIs for downloading videos and audio from various websites (including YouTube, Vimeo, Dailymotion, etc.). In addition to youtube_dl, in the specific implementation, you can also use: PyTube, youtube-dl-py, youtube-dl-gui or 4K Video Audio download tools such as AudioDownloader collect public speech datasets from public databases. AudioSet is a widely used dataset primarily used for audio classification and annotation tasks. The present invention collects public speech datasets to leverage the rich annotated data for data enhancement. AudioSet is a preferred implementation of the present invention. Similar datasets such as UrbanSound (primarily covering traffic noise, construction sounds, and sound events) or ESC-50 (primarily covering animal sounds, natural sounds, and household sounds) can also be used for specific implementations.

[0088] For the collected public speech dataset, relevant speech segments are intercepted using a set duration tag. Since the present invention is a method for detecting infant crying, a cycle of infant crying is usually 1 second to several seconds. If the duration of the tag is set too short or too long, it will not have much practical significance for the present invention. Therefore, in this embodiment, the duration tag is set to a second-level tag, that is, the relevant speech segments are intercepted using a second-level tag.

[0089] In the specific implementation process, there are also many methods for collecting baby crying data. You can use existing data in public databases in the prior art, or you can use a self-built database to collect data and build a database. In this embodiment, the specific method used for collecting baby crying data is: download baby crying data from the ESC-50 official website. The ESC-50 dataset is a collection of 2,000 labeled environmental audio recordings, which is suitable for benchmarking methods for environmental sound classification, including baby crying data. In specific implementation, baby crying data can also be collected from other public datasets, such as: UrbanSound Dataset (The UrbanSound dataset is a widely used audio dataset that contains various audio clips from urban environments, including baby crying), MIVIA Audio Events Dataset (MIVIA is a research project for multimedia information and visual analysis. They provide an audio event dataset that contains baby crying audio), etc.

[0090] All collected audio data are preprocessed to obtain a unified data paradigm. In this embodiment, the collected data include the collected public voice data set and the collected public baby crying data. Data preprocessing mainly includes segmenting the collected audio data according to the set sampling frequency and the set duration to obtain data samples. In this embodiment, the sampling frequency is set to downsample to 8kHz. Since the sound frequency perceived by the human body is usually between 20Hz and 20,000Hz, the common human vocal frequencies are: low male voice (Bass) is about 80Hz to 150Hz, middle male voice (Baritone) is about 100Hz to 200Hz, high male voice (Tenor) is about 150Hz to 300Hz, low female voice (Alto) is about 150Hz to 400Hz, middle female voice (Mezzo-soprano) is about 200Hz to 500Hz, and high female voice (Soprano) is about 300Hz to 1,000Hz. Although the occurrence frequency of infants is higher than that of adults, downsampling to 8kHz is not enough. Hz is also sufficient. More frequencies are mostly undisputed noises. Discarding them will reduce the amount of computation of the entire system. In specific implementation, the downsampling frequency can also be set to 7kHz or 10kHz, or a specific value can be selected according to actual needs.

[0091] When performing audio data segmentation processing, in this embodiment, a non-overlapping sliding window is used for segmentation to avoid excessive overlapping parts and high data repetition rate. The duration of the sliding window is preferably 2.5s to obtain the most optimal data duration. During specific implementation, the duration of the sliding window can also be selected as other values ​​according to actual needs.

[0092] 2. Scenario-Adaptive Mutual Learning Unit: This unit uses the exponential moving average (EMA) method and pseudo-label training to establish a scenario-adaptive crying detection learning loop. It includes the following subunits:

[0093] 2.1 Classifier creation unit: used to create a new classifier using the classifier structure and model parameters obtained in the scene simulation data enhancement unit based on the linear addition characteristic of speech. This is defined as an EMA classifier in the present invention;

[0094] 2.2 EMA classifier parameter freezing unit: This unit is used to freeze the EMA classifier parameters, predict the data in the new scenario, and generate pseudo-labels. Due to the differences between the new and old environments, the performance of the EMA classifier is not sufficient to generate accurate crying detection results. Therefore, in this embodiment, a confidence threshold method is introduced to select samples with high confidence and corresponding pseudo-labels to train the classifier. Through this method, the classifier can reliably learn the distribution of crying data in the new environment.

[0095] In this embodiment, public data is used to preliminarily train the classifier so that it can grasp the characteristics of crying in a synthetic chaotic scene. The public data is then enhanced with scene simulation data based on speech features. Multiple random factors are used to collaborate to simulate speech data in a clinical scene. Then, a new classifier is created for training. The structure of the classifier can adopt any deep learning model, including but not limited to: fully connected neural network, convolutional neural network and recurrent neural network.

[0096] In this embodiment, when training the classifier, the data used also includes collected clinical data, and scene adaptation is achieved through a mutual learning method. In specific implementation, a single-channel microphone is used to collect real-time audio in a clinical environment, and the same preprocessing is performed to ensure that the data paradigm is the same.

[0097] 2.3 EMA classifier parameter unfreezing unit: This unit unfreezes the EMA classifier parameters and uses the EMA classifier parameter update method to transfer the weights of the classifier trained by the confidence threshold method in the scene simulation data augmentation unit based on the linear addition characteristics of speech to the EMA classifier, thereby improving the quality of the pseudo labels generated by the EMA classifier.

[0098] 2.4 Iteration unit: used to continuously iterate the scene simulation data enhancement unit and the EMA classifier parameter solution unit based on the linear addition characteristics of speech, so that the classifier can continuously deduce the crying data distribution in the new environment and correct the crying characteristics obtained in step S1 to achieve environment-adaptive crying detection. Through continuous iteration of the mutual learning method, the performance of the classifier in the new environment is improved, thereby achieving scene-adaptive plug-and-play crying detection.

[0099] The present invention collects clinical data and achieves scene adaptation through a mutual learning method. A single-channel microphone can be used to collect real-time audio in a clinical environment and perform the same preprocessing as when enhancing scene simulation data to ensure the same data paradigm. Through continuous iteration of the mutual learning method, the performance of the classifier in the new environment is improved, thereby achieving scene-adaptive plug-and-play crying detection. The plug-and-play based on scene adaptation has good robustness for crying detection in clinical environments, maintaining the recall rate while improving the accuracy.

[0100] The present invention also protects a storage medium storing a program capable of implementing the above-mentioned scene-adaptive baby crying detection method.

[0101] The present invention also protects an electronic device, including: a voice acquisition module, a processor and a storage medium storing a program, the voice acquisition module acquires ambient audio in real time and provides it to the processor, the storage medium stores a program that can implement a scene-adaptive baby crying detection method, and the processor performs signal processing on the ambient audio acquired in real time by the voice acquisition module according to the program stored in the storage medium.

Claims

1. A scene-adaptive baby crying detection method, characterized in that: The method comprises the following steps: Step S1, scene simulation data enhancement based on speech linear addition characteristics: collect public speech data sets and public baby crying data, use public labeled data for data enhancement, and use scene simulation data enhanced samples to train the classifier to learn the crying characteristics in the synthetic chaotic scene; Step S2: scene-adaptive mutual learning; The exponential moving average method and pseudo-label training method are used to establish a scene-adaptive crying detection learning closed loop, continuously derive the crying data distribution in the new environment, and correct the crying characteristics obtained in step S1 to achieve environment-adaptive crying detection.

2. The scene-adaptive baby crying detection method according to claim 1, characterized in that: In step S1, multiple random factors are used during data enhancement to simulate audio clips that may appear in the scene. The multiple random factors include: (i) Sound source random factor: The sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that emit synchronous sounds in the scene; (ii) Sound length random factor: The sample selected in (i) is cropped using the sound length random factor to simulate audio clips of different lengths; (iii) Starting random factor: The starting random factor is used to determine the addition position between samples, so as to achieve the addition combination between different speech samples, thereby simulating the different characteristics of speech utterance time in the real environment.

3. The scene-adaptive baby crying detection method according to claim 1, characterized in that: In the step S2, establishing a scene-adaptive crying detection learning closed loop includes the following sub-steps: Step 2-1: Use the classifier structure and model parameters obtained in step S1 to create a new classifier, which is defined as an EMA classifier; Step S2-2: Freeze the EMA classifier parameters, predict the data in the new scenario, and generate pseudo labels; Step S2-3: unfreeze the EMA classifier parameters, and use the parameter update method of the EMA classifier to transfer the weights in the classifier to the EMA classifier to improve the quality of the pseudo labels generated by the EMA classifier; Step S2-3: By continuously iterating steps S2-2 and S2-3, the classifier continuously derives the crying data distribution in the new environment, and corrects the crying characteristics obtained in step S1, thereby realizing environmentally adaptive crying detection.

4. The scene-adaptive baby crying detection method according to claim 1, characterized in that: In the step S1, when collecting the public speech data set, the public speech data set is collected from the public audio download website using an audio download tool. When collecting, the original audio data set segments containing the ambient sound are downloaded. The public speech data set uses the set time length label to intercept the relevant speech segments, and the label uses the second-level label; When collecting public baby crying data, the baby crying data was downloaded from the website containing the baby crying data. The public speech data set and the baby crying data were preprocessed to obtain a unified data paradigm. The preprocessing was to segment the collected audio data according to the set sampling frequency and the set duration to obtain data samples. The sampling frequency was set to downsample to 8kHz. A non-overlapping sliding window was used for segmentation, and the set duration was 2.5s. 5.

5. The scene-adaptive baby crying detection method according to claim 3, characterized in that: In the step S2-2, samples with high confidence and corresponding pseudo labels are selected through the confidence threshold method to train the classifier. When training the classifier, public data is used to preliminarily train the classifier, and multiple random factors are used to collaborate to simulate speech data in clinical scenarios. Then, a new classifier is created for training. The structure of the classifier adopts a deep learning model, and the deep learning model is a fully connected neural network, a convolutional neural network or a recurrent neural network. When training the classifier, the data used also includes collected clinical data. When collecting clinical data, a single-channel microphone is used to collect real-time audio in a clinical environment, and the same preprocessing is performed.

6. A scene-adaptive baby crying detection system, characterized in that: The system comprises: Scene simulation data enhancement unit based on speech linear addition characteristics: used to collect public speech data sets and public baby crying data, use public annotated data for data enhancement, and use scene simulation data enhanced samples to train the classifier to learn the crying characteristics in synthetic chaotic scenes; A scene-adaptive mutual learning unit; used to establish a scene-adaptive crying detection learning loop using the exponential moving average method and pseudo-label training, continuously derive the crying data distribution in the new environment, and correct the crying characteristics obtained in the scene simulation data enhancement unit based on the linear addition characteristics of speech, so as to realize environment-adaptive crying detection.

7. The scene-adaptive baby crying detection system according to claim 6, characterized in that: In the scene simulation data enhancement unit based on the linear addition characteristic of speech, multiple random factors are used to simulate the audio clips that may appear in the scene during data enhancement. The multiple random factors include: (i) Sound source random factor: The sound source random factor is used to randomly select samples from the public dataset to simulate the number of sound sources that emit synchronous sounds in the scene; (ii) Sound length random factor: The sample selected in (i) is cropped using the sound length random factor to simulate audio clips of different lengths; (iii) Starting random factor: The starting random factor is used to determine the addition position between samples, so as to achieve the addition combination between different speech samples, thereby simulating the different characteristics of speech utterance time in the real environment.

8. The scene-adaptive baby crying detection system according to claim 6, characterized in that: In the scene-adaptive mutual learning unit, establishing a scene-adaptive crying detection learning closed loop includes: Classifier creation unit: used to create a new classifier using the classifier structure and model parameters obtained in the scene simulation data enhancement unit based on the linear addition characteristics of speech, and defined as an EMA classifier; EMA classifier parameter freezing unit: used to freeze the EMA classifier parameters, predict the data in the new scenario, and generate pseudo labels; EMA classifier parameter solution unit: used to unfreeze the EMA classifier parameters and use the parameter update method of the EMA classifier to transfer the weights in the classifier to the EMA classifier, thereby improving the quality of pseudo labels generated by the EMA classifier; Iteration unit: used to continuously iterate the scene simulation data enhancement unit based on the linear addition characteristics of speech and the EMA classifier parameter solution unit, so that the classifier can continuously derive the crying data distribution in the new environment, and correct the crying characteristics obtained in the scene simulation data enhancement unit based on the linear addition characteristics of speech, so as to realize environment-adaptive crying detection. Through the continuous iteration of the mutual learning method, the performance of the classifier in the new environment is improved, thereby realizing scene-adaptive plug-and-play crying detection.

9. The scene-adaptive baby crying detection system according to claim 6, characterized in that: In the scene simulation data enhancement unit based on the linear addition characteristic of speech, when collecting the public speech data set, the public speech data set is collected from the public audio download website using an audio download tool, and the original audio data set fragments containing environmental sounds are downloaded during the collection. The public speech data set uses a set time length label to intercept relevant speech fragments, and the label uses a second-level label; When collecting public baby crying data, the baby crying data was downloaded from the website containing the baby crying data. The public speech data set and the baby crying data were preprocessed to obtain a unified data paradigm. The preprocessing was to segment the collected audio data according to the set sampling frequency and the set duration to obtain data samples. The sampling frequency was set to downsample to 8kHz. A non-overlapping sliding window was used for segmentation, and the set duration was 2.5s.

10. The scene-adaptive baby crying detection system according to claim 8, characterized in that: In the EMA classifier parameter freezing unit, samples with high confidence and corresponding pseudo labels are selected by the confidence threshold method to train the classifier. When training the classifier, public data is used to preliminarily train the classifier, and multiple random factors are used to collaborate to simulate speech data in clinical scenarios. Then, a new classifier is created for training. The structure of the classifier adopts a deep learning model, and the deep learning model is a fully connected neural network, a convolutional neural network or a recurrent neural network. When training the classifier, the data used also includes collected clinical data. When collecting clinical data, a single-channel microphone is used to collect real-time audio in a clinical environment, and the same preprocessing is performed.

Images