Method and system for infant cry need classification based on deep neural networks

Abstract

This invention discloses a method and system for classifying infant cries based on deep neural networks, belonging to the field of infant care technology. The method includes acquiring raw audio of an infant's cries, preprocessing and performing time-frequency transformation to obtain a multi-scale time-frequency representation set, adaptively selecting the optimal time-frequency representation mode based on the current signal quality, generating a feature map of the selected mode, inputting it into a multi-scale attention network to extract deep spatiotemporal features, inputting the deep spatiotemporal features into a classification network, outputting the demand category corresponding to the infant's current cry and its confidence level, and generating personalized soothing instructions based on the demand category and confidence level, combined with preset infant physiological parameters and crying respiratory cycle features, and sending the soothing instructions to a smart soothing device to execute the corresponding operation. This invention solves the technical problems of poor environmental adaptability and insufficient classification accuracy in existing methods, achieving highly robust and high-precision infant cry demand recognition and intelligent soothing response.

Description

Technical Field

[0001] This invention relates to the field of infant care technology, specifically to a method and system for classifying infant crying needs based on deep neural networks. Background Technology

[0002] Before infants can speak, crying is their primary means of communication to express their needs and discomfort. Different types of cries often correspond to different physiological or psychological needs, such as hunger, drowsiness, the need for a diaper change, physical discomfort, or a search for comfort. Accurately identifying an infant's crying needs is of significant practical importance for new parents in scientific childcare, reducing caregiving anxiety, and assisting medical staff in making initial assessments of the condition in hospital pediatric settings.

[0003] Traditionally, judging an infant's cries for help relies primarily on the caregiver's subjective experience. However, due to individual differences in infants and the complexity and variability of cries, even experienced parents or medical professionals find it difficult to make 100% accurate judgments. With the development of speech signal processing and artificial intelligence technologies, some studies have attempted to apply audio recognition technology to infant cry analysis. Existing methods typically collect audio recordings of infant cries, extract acoustic features such as Mel-frequency cepstral coefficients, and then use support vector machines or shallow neural networks for classification.

[0004] However, existing technologies have the following significant drawbacks: First, the feature extraction methods are limited, mainly relying on manually designed acoustic features such as MFCC, which makes it difficult to fully capture the non-stationary time-frequency characteristics contained in the crying signal; second, the model capacity is limited, and the generalization ability to complex and non-stationary infant crying signals is insufficient, resulting in a significant drop in recognition accuracy in real-world home environments with low signal-to-noise ratios; third, existing models lack dynamic adaptability and cannot adaptively adjust the feature modes used according to signal quality, leading to poor robustness in noisy environments; fourth, the classification results lack interpretability and cannot provide quantitative basis for subsequent responses.

[0005] To address the aforementioned issues, researchers have explored various improvement schemes. Some studies employ hybrid models combining CNNs and RNNs to extract spatiotemporal features, but these models have fixed structures and cannot dynamically adjust processing strategies based on input signal characteristics. Other studies introduce attention mechanisms to enhance the attention given to key regions, but the calculation of attention weights relies on a single feature representation. Recent research indicates that multi-armed slot machine algorithms can be used for dynamic feature mode selection, achieving significant performance improvements in noisy environments; however, existing methods decouple feature selection from feature extraction, failing to achieve end-to-end joint optimization. Furthermore, the Discrete S-Transform, as an advanced time-frequency analysis method, offers the advantage of adaptive time-frequency resolution compared to STFTs and MFCCs, but it has not yet been effectively integrated with dynamic feature selection mechanisms.

[0006] Therefore, how to construct an infant cry demand classification system that can adaptively handle different signal-to-noise ratio environments, dynamically fuse multimodal features, and is interpretable has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for classifying infant cries based on deep neural networks. By constructing a multi-level adaptive feature learning and dynamic modality fusion framework, it solves the technical problems of insufficient feature extraction, poor environmental adaptability, and insufficient classification accuracy in existing methods.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for infant cry demand classification based on deep neural networks includes: Step S1: Collect the raw crying audio of the baby through the data acquisition module, and preprocess and perform time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; Step S2: Based on the dynamic feature selection module, the optimal time-frequency representation mode is adaptively selected according to the current signal quality, and the selected mode feature map is generated; Step S3: Based on the multi-scale spatiotemporal feature extraction module, the selected modal feature map is input into the multi-scale attention network constructed based on the improved residual shrinkage network and the bidirectional gated recurrent unit to extract deep spatiotemporal features; Step S4: Based on the demand classification module, input the deep spatiotemporal features into the classification network, and output the demand category corresponding to the baby's current cry and its confidence level; Step S5: Based on the intervention decision module, according to the need category and its confidence level, combined with the preset infant physiological parameters and crying breathing cycle characteristics, a personalized soothing instruction is generated and sent to the smart soothing device to perform the corresponding operation.

[0009] Further, in step S1, the original crying audio is preprocessed and subjected to time-frequency transformation to obtain a multi-scale time-frequency representation set, specifically: The original crying audio is processed by delay summation beamforming. By estimating the direction of the sound source at the location of the baby's head, the signals of each channel are delayed and weighted and summed to obtain the enhanced audio signal. Using an endpoint detection algorithm based on short-time energy and zero-crossing rate, infant cry segments are automatically detected and extracted from the enhanced audio signal to obtain several cry samples. Each cry sample is then segmented by respiratory cycle to divide the cry signal into exhalation and inhalation segments. Multimodal time-frequency transformation is performed on the segmented expiratory segment signal to obtain a multi-scale time-frequency representation set.

[0010] Furthermore, the multi-scale time-frequency representation set includes three complementary time-frequency representations: short-time Fourier transform spectrum, Mel frequency cepstral coefficient feature map, and discrete S-transform time-frequency map.

[0011] Further, in step S2, the dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates the selected mode feature map, specifically: A dynamic modality selector based on a multi-armed slot machine algorithm is used to calculate the selection value of each arm for each input crying sample according to the confidence interval upper bound strategy, and the arm with the highest value is selected as the optimal modality of the current sample to generate the selected modality feature map.

[0012] Further, in step S2, after selecting the optimal mode of the current sample, the corresponding selected mode feature map is input into the lightweight quality assessment network to calculate the signal-to-noise ratio estimate and signal quality score of the current crying sample. The instantaneous reward is calculated based on the local classification correctness and the signal quality score, and the expected reward estimate of the selected arm is updated using the incremental update rule.

[0013] Further, in step S3, the selected modal feature map is input into a multi-scale attention network constructed based on an improved residual shrinkage network and a bidirectional gated recurrent unit, based on the multi-scale spatiotemporal feature extraction module, to extract deep spatiotemporal features, specifically: The selected modal feature map is concatenated with its corresponding signal quality score, and the concatenated data is input into the multi-scale spatiotemporal feature extraction module. The multi-scale spatiotemporal feature extraction module includes a multi-scale convolutional feature extraction layer, an improved residual contraction attention module, a bidirectional gated recurrent unit layer, and a temporal attention aggregation layer. Based on the multi-scale convolutional feature extraction layer, features are extracted from the concatenated data to obtain a multi-scale fused feature map. The multi-scale fused feature map is then input into the improved residual contraction attention module for soft thresholding. The processed multi-scale fused feature map is reshaped into a time step sequence and input into the bidirectional gated recurrent unit layer to obtain a fused bidirectional temporal dependent feature sequence. Based on the temporal attention aggregation layer, the fused bidirectional temporal dependent feature sequence is weighted and aggregated to obtain a deep spatiotemporal feature vector.

[0014] Further, in step S4, the deep spatiotemporal features are input into the classification network based on the demand classification module, and the demand category corresponding to the baby's current cry and its confidence level are output, specifically: The deep spatiotemporal feature vector is input into the demand classification module. Through two fully connected layers and a Softmax output layer, the probability distribution of five demand categories is output. The category corresponding to the highest probability is the demand category of the baby's current cry, and the highest probability itself is the confidence level.

[0015] The present invention also provides a system for classifying infant cry needs based on deep neural networks, which is applied to the above-mentioned method for classifying infant cry needs based on deep neural networks, including: a data acquisition module, a dynamic feature selection module, a multi-scale spatiotemporal feature extraction module, a needs classification module and an intervention decision module connected in sequence; The data acquisition module acquires the raw crying audio of the infant, and preprocesses and performs time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; The dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates a feature map of the selected mode. The multi-scale spatiotemporal feature extraction module is used to extract deep spatiotemporal features from selected modality feature maps by constructing a multi-scale attention network based on an improved residual shrinkage network and a bidirectional gated recurrent unit. The demand classification module is used to input deep spatiotemporal features into the classification network and output the demand category and its confidence level corresponding to the baby's current cry. The intervention decision module is used to generate personalized soothing instructions based on the need category and its confidence level, combined with preset infant physiological parameters and crying breathing cycle characteristics, and send the soothing instructions to the smart soothing device to perform the corresponding operation.

[0016] Furthermore, the data acquisition module collects the baby's original crying audio through a circular array of four microphones deployed around the crib.

[0017] In summary, the present invention has at least one of the following beneficial technical effects: First, this invention overcomes the limitations of fixed feature resolution in traditional MFCC and STFT by introducing the Discrete S-Transform and a multimodal time-frequency representation set. The Discrete S-Transform uses a window width inversely proportional to the frequency, providing high frequency resolution to capture the fundamental harmonic structure in the low-frequency band and high time resolution to capture transient changes in the high-frequency band, significantly improving the ability to characterize the non-stationary characteristics of crying sounds.

[0018] Second, this invention innovatively integrates a multi-armed slot machine algorithm with a deep feature extraction network to construct a dynamic mode selection mechanism. This mechanism can adaptively select the optimal time-frequency representation mode based on the quality and characteristics of the input signal, solving the problem of poor robustness of traditional fixed feature sets in noisy environments. Experiments show that the dynamic selection strategy based on the multi-armed slot machine can improve classification accuracy by approximately 10% compared to static feature fusion methods.

[0019] Third, this invention designs an improved residual contraction attention module, which automatically suppresses task-irrelevant noise channels through adaptive thresholding, enhancing the model's robustness in low signal-to-noise ratio environments. This mechanism eliminates the need for explicit noise estimation, achieving end-to-end adaptive noise suppression.

[0020] Fourth, this invention employs a structure combining multi-scale convolution and bidirectional gated recurrent units to simultaneously capture the local time-frequency patterns and long-term temporal dependencies of the crying signal. The temporal attention mechanism further focuses on the most discriminative time segments, enhancing the interpretability of the model.

[0021] Fifth, this invention introduces respiratory cycle segmentation and a multimodal decision fusion tree, incorporating the respiratory physiological characteristics of crying into the intervention decision-making process. Studies have shown that the ratio of expiratory to inspiratory duration varies significantly under different demand states, and the introduction of this feature improves the personalization and accuracy of decision-making.

[0022] Sixth, this invention constructs a complete closed loop from "cry collection - modality selection - feature extraction - demand classification - soothing execution", which solves the technical problem that the existing technology only stops at recognition and lacks linkage with intervention measures, and realizes truly intelligent infant care. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] like Figure 2 As shown, the present invention provides a system for infant cry demand classification based on deep neural networks, comprising: a data acquisition module, a dynamic feature selection module, a multi-scale spatiotemporal feature extraction module, a demand classification module, and an intervention decision module connected in sequence; The data acquisition module acquires the raw crying audio of the infant, and preprocesses and performs time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; The dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates a feature map of the selected mode. The multi-scale spatiotemporal feature extraction module is used to extract deep spatiotemporal features from selected modality feature maps by constructing a multi-scale attention network based on an improved residual shrinkage network and a bidirectional gated recurrent unit. The demand classification module is used to input deep spatiotemporal features into the classification network and output the demand category and its confidence level corresponding to the baby's current cry. The intervention decision module is used to generate personalized soothing instructions based on the need category and its confidence level, combined with preset infant physiological parameters and crying breathing cycle characteristics, and send the soothing instructions to the smart soothing device to perform the corresponding operation.

[0026] like Figure 1 As shown, the present invention also provides a method for classifying infant cry needs based on deep neural networks, applied to the aforementioned system for classifying infant cry needs based on deep neural networks, comprising: Step S1: Collect the raw crying audio of the baby through the data acquisition module, and preprocess and perform time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; Step S2: Based on the dynamic feature selection module, the optimal time-frequency representation mode is adaptively selected according to the current signal quality, and the selected mode feature map is generated; Step S3: Based on the multi-scale spatiotemporal feature extraction module, the selected modal feature map is input into the multi-scale attention network constructed based on the improved residual shrinkage network and the bidirectional gated recurrent unit to extract deep spatiotemporal features; Step S4: Based on the demand classification module, input the deep spatiotemporal features into the classification network, and output the demand category corresponding to the baby's current cry and its confidence level; Step S5: Based on the intervention decision module, according to the need category and its confidence level, combined with the preset infant physiological parameters and crying breathing cycle characteristics, a personalized soothing instruction is generated and sent to the smart soothing device to perform the corresponding operation.

[0027] In step S1, the raw crying audio of the infant is acquired through the data acquisition module, and the raw crying audio is preprocessed and subjected to time-frequency transformation to obtain a multi-scale time-frequency representation set, specifically: A circular array of four microphones is deployed around the crib, with a preset sampling frequency. Continuous acquisition of ambient sound signals yields multi-channel raw audio signals. ;in, Indicates a time index. Indicates the first Each microphone in time The collected audio amplitude values; For multi-channel raw audio signals Delay-sum beamforming is performed to enhance the signal in the direction of the sound source and suppress ambient noise, resulting in an enhanced audio signal. : ; in, Indicates the first Time delay compensation for each microphone relative to the sound source Indicates the weighting coefficient; Endpoint detection algorithms based on short-time energy and zero-crossing rate are used to enhance audio signals. The system detected and extracted segments of infant cries, obtaining several cry samples. , Indicates the sample index; For each crying sample Perform respiratory cycle segmentation to divide the crying signal into expiratory phases. and inhalation The respiratory cycle segmentation uses an algorithm based on Hilbert envelope and zero-crossing rate detection. The expiratory segment corresponds to the main crying sound, and the inspiratory segment corresponds to a brief inspiratory sound. For the segmented expiratory signal Multimodal time-frequency transformation is performed to obtain three complementary time-frequency representations, forming a multi-scale time-frequency representation set. ;in: The short-time Fourier transform spectrum is plotted using a Hamming window with a window length of [missing information]. Frame shift ; To generate the Mel frequency cepstral coefficient feature map, 40 Mel filter banks were used to extract 13-dimensional MFCCs and their first and second order differences, forming a 39-dimensional feature map. This is the time-frequency plot of the discrete S-transform; the formula for calculating the discrete S-transform is: ; in, Indicates a time index. Indicates frequency index, Indicates signal length. The imaginary unit is represented; the discrete S-transform uses a window width inversely proportional to the frequency, providing high frequency resolution in the low-frequency band and high time resolution in the high-frequency band.

[0028] In step S2, the dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates the selected mode feature map, specifically: The core of the dynamic feature selection module is a dynamic modality selector built based on the multi-armed slot machine algorithm, whose input is a set of multi-scale time-frequency representations. The output is the selected modality feature map. This module aims to adaptively select the optimal time-frequency representation mode based on the quality of the current input signal, thereby improving the robustness of the model in complex acoustic environments. The following section provides a detailed explanation of this module: 1. Modal arm definition The three time-frequency representation modes are defined as three independent arms: (1) Arm 1: Short-time Fourier transform spectrum ,in For time frames, (Corresponding to the one-sided spectrum of 512-point FFT).

[0029] (2) Arm 2: Mel frequency cepstral coefficient characteristic map It includes 13-dimensional static MFCC, 13-dimensional first-order difference, and 13-dimensional second-order difference.

[0030] (3) Arm 3: Discrete S-transform time-frequency diagram The frequency range is 0-8kHz, and a window width inversely proportional to the frequency is used.

[0031] 2. Confidence Interval Upper Bound Strategy

[0032] Each arm Associated with an estimated expected reward This represents the expected contribution of the modality to the classification task. For the current time step... (i.e., the currently input crying sample), the selector selects the optimal arm according to the following UCB formula. : ; in: : No. The current estimated expected reward for each modality is initialized to 0.

[0033] Exploration coefficient: controls the balance between exploration and utilization. In this embodiment... .

[0034] : Current time step (counting from 1).

[0035] : No. The modality has been selected the number of times, initially set to 0.

[0036] To prevent division by zero, take the small constant value. .

[0037] 3. Quality assessment network

[0038] Select modal arm Then, the corresponding modal feature maps Input a lightweight quality assessment network, which is used to estimate the signal quality score. The quality assessment network structure is as follows: enter: The size is (Single channel).

[0039] First layer: Two-dimensional convolutional layer, convolutional kernel Step size 1, padding 1, output channels 16, followed by batch normalization and ReLU activation.

[0040] Second layer: Global average pooling, output size .

[0041] The third layer is a fully connected layer with 16 inputs and 8 outputs, activated by ReLU.

[0042] Fourth layer: Fully connected layer, input 8, output 1, Sigmoid activation, to obtain signal quality score. .

[0043] Meanwhile, the quality assessment network also estimates the signal-to-noise ratio. As an intermediate feature (not directly output), it is used for subsequent reward calculations.

[0044] 4. Instant Reward Calculation

[0045] Instant rewards are calculated based on the local classification correctness and signal quality score of the current sample. : ; in: : The local classification correctness of the current sample on the subsequent classifier; if the classification is correct, then ,otherwise This value is obtained during the training phase after forward propagation (and is not involved in gradient calculation).

[0046] : Quality assessment network output signal quality score.

[0047] Balance coefficient, in this embodiment .

[0048] 5. Expected Rewards Updated

[0049] The expected reward estimate for the selected arm is updated using an incremental update rule: ; in In this embodiment, the learning rate is... .

[0050] 6. Dynamically weighted feature map generation

[0051] Select modal feature map With signal quality score By concatenating the features, a dynamically weighted feature map is obtained. : The size is The first channel is the original feature map. The second channel is Extend to Quality score map with the same spatial dimensions (achieved by repeated filling).

[0052] This feature map will serve as input for the subsequent multi-scale spatiotemporal feature extraction module.

[0053] In step S3, the selected modal feature map is input into a multi-scale attention network constructed based on an improved residual shrinkage network and a bidirectional gated recurrent unit, based on the multi-scale spatiotemporal feature extraction module, to extract deep spatiotemporal features, specifically: This section provides a detailed introduction to multi-scale attention networks: The multi-scale spatiotemporal feature extraction module receives dynamically weighted feature maps. Deep spatiotemporal features are extracted through multi-scale convolution, residual shrinkage attention, bidirectional GRU, and temporal attention aggregation. .

[0054] 1. Multi-scale convolutional feature extraction layer

[0055] Three parallel convolutional branches are used to extract local time-frequency patterns at different scales. The specific parameters of each branch are as follows: Branch 1: Kernel Size Step size 1, padding 1, output channels 64.

[0056] Branch 2: Kernel Size Step size 1, fill size 2, output channels 64.

[0057] Branch 3: Kernel Size Step size 1, padding 3, output channels 64.

[0058] The operation for each branch is as follows: ; ; ; in Represents two-dimensional convolution. Indicates batch normalization, This represents the ReLU activation function.

[0059] The output feature map size of all three branches is (Assuming convolution does not change the spatial dimensions, padding is used to preserve them). The three feature maps are concatenated along the channel dimension to obtain a multi-scale fused feature map. .

[0060] 2. Improved residual shrinkage attention module

[0061] This module is used to adaptively suppress noise-related feature channels, and its structure includes the following steps: Global average pooling: for Global average pooling is performed in the spatial dimensions (time and frequency) to obtain the channel description vector. : ; The first fully connected layer: Mapped to a 64-dimensional hidden layer and activated by ReLU: ; in .

[0062] The second fully connected layer: Mapping back to 192 dimensions yields the original threshold vector. : ; in , .

[0063] Threshold generation: The threshold is generated using the Sigmoid function. Normalization to Multiply by The maximum absolute value is used to obtain the adaptive threshold for each channel. : ; in Take the maximum absolute value (scalar) of all spatial locations and expand it into a vector with the same number of channels.

[0064] Soft thresholding: Perform a soft thresholding operation on the feature map of each channel, setting features with absolute values ​​less than the threshold to zero and retaining features with values ​​greater than the threshold. ; in For symbolic functions, This is the ReLU function; subtraction is a broadcast operation.

[0065] The denoised feature map is obtained after soft thresholding. .

[0066] 3. Two-way gated loop unit layer

[0067] Will Remodeling into a time step sequence This involves flattening the frequency and channel dimension of each time step into a one-dimensional vector. This represents the number of time frames.

[0068] Bidirectional GRU consists of two independent GRU layers: Forward GRU: In chronological order arrive Process the sequence and output the hidden state. , As a hidden state dimension, in this invention .

[0069] Reverse GRU: Reverse order by time arrive Process the sequence and output the hidden state. .

[0070] Each time step The hidden state of the splicing is: ; in This indicates a splicing operation.

[0071] 4. Temporal Attention Aggregation Layer

[0072] This layer focuses on the most discriminative time segments, assigning attention weights to each time step. First, the attention score for each time step is calculated. : ; in: This is the attention weight matrix. For the attention hidden layer dimension, in this invention .

[0073] For bias.

[0074] This is the attention vector.

[0075] Then, the attention weights are obtained through Softmax normalization. : ; Finally, weighted aggregation yields the deep spatiotemporal feature vector. : .

[0076] In step S4, the deep spatiotemporal features are input into the classification network based on the demand classification module, and the demand category corresponding to the baby's current cry and its confidence level are output, specifically: The demand classification module receives deep spatiotemporal feature vectors. The probability distribution of demand categories is output through two fully connected layers and Softmax.

[0077] 1. First fully connected layer

[0078] Will Mapped to the hidden space and activated by ReLU: ; in , The hidden layer dimension is 256, and Dropout (dropout rate 0.5) is introduced to prevent overfitting.

[0079] 2. Second fully connected layer

[0080] Map hidden features to demand category score vectors : ; in .

[0081] 3. Softmax output layer

[0082] Transform the score vector into a probability distribution : ; Final demand category Choose the category with the highest probability and the confidence level. .

[0083] The specific training process of the model involved in steps S2-S4 is as follows: 1. Construct the training dataset This invention uses multi-source datasets for model training to ensure data diversity and coverage, specifically including the following four parts: The publicly available dataset, Donate-a-Cry Corpus, contains approximately 2000 recordings of infant cries at a sampling rate of 44.1 kHz, with some cries labeled with their type (e.g., hunger, discomfort). This invention selects valid samples from this dataset, resamples them to 16 kHz, and manually reviews the labeling quality.

[0084] The publicly available dataset, Baby Chillanto, contains approximately 1500 infant cries, labeled as painful or non-painful, and was provided by the University of Costa Rica. This invention categorizes painful samples as "colic," and non-painful samples are further subdivided based on other information (and merged into "seeking comfort" if necessary).

[0085] The publicly available ICSD dataset contains approximately 3,000 infant cries, covering various needs, and is provided by the International Baby Cries Database Project. This dataset is well-annotated, including cries of hunger, drowsiness, wet diapers, and discomfort.

[0086] Self-collected data: Approximately 3,500 infant cries in real-world environments were collected through collaboration with the pediatric departments of local tertiary hospitals and community maternal and infant centers. Data collection equipment included professional recorders and smartphones, with a sampling rate of 48kHz. Data was processed uniformly after collection. Labeling was completed collaboratively by at least two professional pediatric nurses and one parent, categorizing cries into five types: hunger, drowsiness, diaper discomfort, colic, and seeking comfort. Controversial samples were discussed and their labels determined or removed.

[0087] Data preprocessing: All original audio files undergo the following preprocessing: Resampling: Use an audio processing library to uniformly resample audio at different sampling rates to 16kHz.

[0088] Channel merging: Multi-channel recordings (such as stereo) are merged into a single channel (average value).

[0089] Silent excision: An endpoint detection algorithm based on short-time energy and zero-crossing rate is used to excise the first and last silent segments while preserving the effective crying portion.

[0090] Length normalization: Each crying sample is truncated or padded to a fixed length of 2 seconds (corresponding to 32,000 sampling points). If the original sample length is less than 2 seconds, zeros are padded at the end; if it exceeds 2 seconds, a 2-second segment is taken (1 second before and after the peak energy point).

[0091] Amplitude normalization: Normalize the audio amplitude to the range [-1, 1] by dividing by the maximum absolute value.

[0092] Labeling and category distribution: The final dataset contains five demand categories, distributed as follows (after data augmentation): Hunger: 2000; Sleepiness: 2100; Diaper discomfort: 1900 pieces; Intestinal colic: 2200 cases; Seeking appeasement: 1800; A total of 10,000 samples were collected, and the categories were basically balanced.

[0093] Dataset partitioning: All samples were randomly divided into a training set (8000 samples), a validation set (1000 samples), and a test set (1000 samples) in a ratio of 8:1:1. The division was performed while ensuring that the proportion of each category in each set was consistent with the overall population.

[0094] 2. Perform data augmentation on the dataset.

[0095] To improve the model's generalization ability and robustness, the following augmentation operations (randomly combined) are applied online to each batch of data during training: (1) Time scaling With random factors The audio is time-stretched or compressed, while the labeled frame information is adjusted (this task involves classifying the entire segment, so only the labels need to remain unchanged). This is achieved using librosa's time_stretch function.

[0096] (2) Pitch shift

[0097] With random semitones This function shifts the pitch of audio without changing its duration. It uses the `pitch_shift` function from librosa.

[0098] (3) Add background noise

[0099] A noise segment is randomly selected from a pre-built noise library (containing common household noises such as fan noise, air conditioner noise, people talking, television noise, and toy noise), and the signal-to-noise ratio is randomly selected. This noise is superimposed on the crying signal. The noise intensity is adjusted proportionally to the energy level to ensure the signal-to-noise ratio remains within the specified range.

[0100] (4) Random gain

[0101] With random factors Scaling the audio amplitude simulates volume changes.

[0102] (5) Random cover

[0103] Randomly mask the time or frequency dimension of the time-frequency plot (after transformation): zero out several consecutive frames of the time axis with a probability of 0.3 (masking length up to 10 frames), or zero out several consecutive frequency bands of the frequency axis with a probability of 0.3 (masking width up to 20 frequency bands), to simulate partial information loss and enhance the robustness of the model.

[0104] 3. Training Hyperparameters and Optimization Details

[0105] (1) Optimizer

[0106] The Adam optimizer is used, with the following parameter settings: Initial learning rate: ; Weight decay: ; First-order moment exponential decay rate Second-order moment exponential decay rate ; (2) Learning rate scheduling A cosine annealing strategy is used, and the learning rate is updated after each training epoch. ; in .

[0107] (3) Batch size and training rounds

[0108] Batch size: 32; Maximum number of training rounds: 100; Early stopping strategy: If the validation set loss does not decrease for 10 consecutive rounds, stop training and restore the optimal model parameters.

[0109] (4) Parameter initialization

[0110] The weights of the convolutional and fully connected layers are initialized using He (kaiming_normal), and the biases are initialized to 0.

[0111] The weights of the GRU layer are initialized orthogonally, and the bias is initialized to 0.

[0112] The scaling factor of the batch normalization layer is initialized to 1, and the bias is initialized to 0.

[0113] (5) Regularization

[0114] Add a Dropout layer after the fully connected layer (the first layer of the classification module) to reduce the dropout rate. .

[0115] Weight decay (L2 regularization) coefficient .

[0116] Finally, to summarize the above, a detailed training process is presented, in which the following steps are performed for each training batch: Data Loading and Augmentation: Load a batch (32 samples) of raw audio data and labels from the training set, and perform random augmentation operations (time stretching, pitch shifting, noise addition, random gain) independently on each sample. The augmented audio remains a time-domain waveform with a uniform length of 32,000 points.

[0117] Multimodal time-frequency transform: For each enhanced sample, calculate three time-frequency representations respectively: STFT spectrum: window length 512, frame shift 128, resulting in the size (Depending on the signal length).

[0118] MFCC feature map: Extract 13-dimensional MFCCs and their first and second order differences to obtain the size. .

[0119] DST time-frequency diagram: Using Discrete S-transform, with 256 frequency points, the dimensions are obtained. .

[0120] All feature maps are aligned in the time dimension. (If the lengths are the same, they are adjusted by linear interpolation; if the lengths are inconsistent, they are adjusted by linear interpolation).

[0121] Dynamic mode selection: For each sample, the expected reward is calculated based on the current estimates for each arm. (Initialized to 0 at the start of training), modal arms are selected using the UCB formula. .

[0122] Extract feature maps of the corresponding modes Input the signal into the quality assessment network to obtain the signal quality score. .

[0123] Will and spliced ​​together .

[0124] Forward propagation: Will The input is processed by a multi-scale spatiotemporal feature extraction module, which then performs multi-scale convolution, residual shrinkage attention, bidirectional GRU, and temporal attention to obtain deep features. .

[0125] Will Input the data into the classification module to obtain the predicted probability. .

[0126] Loss Calculation: Calculate the classification cross-entropy loss .

[0127] Determine the accuracy of local classification based on the prediction results. (i.e., whether the predicted category matches the true label), and Combined calculation of instant rewards .

[0128] Accumulated reward losses for this batch ,in .

[0129] Calculate total loss .

[0130] Backpropagation and parameter update: Clear the gradient and call loss.backward() to calculate the gradient.

[0131] Gradient clipping: Clips the global gradient norm to its maximum value of 1.0 to prevent gradient explosion.

[0132] The optimizer performs parameter updates.

[0133] Update Q value: For each selected arm Update its estimated expected reward according to the incremental formula: .

[0134] Update the count of each arm being selected. .

[0135] Log the following metrics: average loss, accuracy, number of modal selections for this batch.

[0136] After each training epoch (i.e., after traversing all training samples), the model performance is evaluated on the validation set, and the validation loss and accuracy are calculated. If the validation loss does not improve for 10 consecutive epochs, training is stopped, and the model parameters with the lowest validation loss are loaded as the final model.

[0137] After training is complete, a final evaluation is performed on the test set to calculate various classification metrics: Overall accuracy; Precision, recall, and F1 score for each category; Confusion matrix; Meanwhile, in order to verify the effectiveness of dynamic mode selection, the relationship between the selected frequency of each mode and the signal-to-noise ratio of the test set was calculated to analyze the adaptability of the model.

[0138] The final model is exported in ONNX format and optimized for embedded devices (such as Raspberry Pi and Jetson Nano) using quantization (INT8 quantization) to reduce inference latency and memory footprint. During deployment, a dynamic modality selector... The value is fixed at the value at the end of training and will not be updated further; it will be directly based on... Select the optimal mode.

[0139] In step S5, the intervention decision module generates personalized soothing instructions based on the need category and its confidence level, combined with preset infant physiological parameters and crying-breathing cycle characteristics. These instructions are then sent to the smart soothing device to execute the corresponding operation. Specifically: The process of generating personalized soothing instructions based on the need category and its confidence level, combined with preset infant physiological parameters and crying-breathing cycle characteristics, includes the following steps: Retrieve infant physiological parameters from the database Among them, physiological parameters include age. ,weight Last feeding time Last sleep time ; Extracting respiratory cycle features from respiratory cycle segmentation results Including the average duration of the expiratory phase Average duration of inhalation Breathing-inhalation cycle ratio The respiratory cycle characteristics of infants vary significantly under different needs. According to demand category Confidence level Physiological parameters and respiratory cycle characteristics The comprehensive reassurance action score was calculated using a multimodal decision fusion tree. , This represents the total number of possible soothing actions in this invention. It corresponds to four functions: intelligent cradle soothing, intelligent music playback, intelligent feeding reminder, and intelligent pacifier. The multimodal decision fusion tree is constructed based on the gradient boosting decision tree algorithm and trained using historical interaction data; each leaf node of the tree corresponds to an action score vector. The overall rating of reassurance actions will be used. The highest-rated action is used as the target soothing action. ; Soothing actions based on the target Generate corresponding reassurance instructions, wherein the reassurance instructions include an instruction type field. and instruction parameter fields For example, if the target action is to soothe the baby in a smart cradle, then , Includes swing frequency and swing amplitude .

[0140] Next, we will introduce the intelligent soothing device: The intelligent soothing device includes an intelligent cradle, an intelligent music player, an intelligent feeding reminder, and an intelligent pacifier; wherein: After receiving a soothing command, the smart cradle will adjust its settings according to the command parameters. Adjust the swing frequency and amplitude; After receiving the soothing command, the smart music player will then proceed according to the command parameters. Play preset soothing music or white noise; After receiving a soothing instruction, the smart feeding reminder will adjust its settings according to the instruction parameters. Send a feeding reminder notification to the bound terminal; After receiving a soothing command, the smart pacifier will adjust its settings according to the command parameters. Activate the vibration soothing mode.

[0141] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for infant cry need classification based on deep neural networks, characterized in that, include: Step S1: Collect the raw crying audio of the baby through the data acquisition module, and preprocess and perform time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; Step S2: Based on the dynamic feature selection module, the optimal time-frequency representation mode is adaptively selected according to the current signal quality, and the selected mode feature map is generated; Step S3: Based on the multi-scale spatiotemporal feature extraction module, the selected modal feature map is input into the multi-scale attention network constructed based on the improved residual shrinkage network and the bidirectional gated recurrent unit to extract deep spatiotemporal features; Step S4: Based on the demand classification module, input the deep spatiotemporal features into the classification network, and output the demand category corresponding to the baby's current cry and its confidence level; Step S5: Based on the intervention decision module, according to the need category and its confidence level, combined with the preset infant physiological parameters and crying breathing cycle characteristics, a personalized soothing instruction is generated and sent to the smart soothing device to perform the corresponding operation.

2. The method for classifying infant cry needs based on deep neural networks according to claim 1, characterized in that, In step S1, the original crying audio is preprocessed and subjected to time-frequency transformation to obtain a multi-scale time-frequency representation set, specifically: The original crying audio is processed by delay summation beamforming. By estimating the direction of the sound source at the location of the baby's head, the signals of each channel are delayed and weighted and summed to obtain the enhanced audio signal. Using an endpoint detection algorithm based on short-time energy and zero-crossing rate, infant cry segments are automatically detected and extracted from the enhanced audio signal to obtain several cry samples. Each cry sample is then segmented by respiratory cycle to divide the cry signal into exhalation and inhalation segments. Multimodal time-frequency transformation is performed on the segmented expiratory segment signal to obtain a multi-scale time-frequency representation set.

3. The method for classifying infant cry needs based on deep neural networks according to claim 2, characterized in that, The multi-scale time-frequency representation set includes three complementary time-frequency representations: short-time Fourier transform spectrum, Mel frequency cepstral coefficient feature map, and discrete S-transform time-frequency map.

4. The method for classifying infant cry needs based on deep neural networks according to claim 2, characterized in that, In step S2, the dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates the selected mode feature map, specifically: A dynamic modality selector based on a multi-armed slot machine algorithm is used to calculate the selection value of each arm for each input crying sample according to the confidence interval upper bound strategy, and the arm with the highest value is selected as the optimal modality of the current sample to generate the selected modality feature map.

5. The method for classifying infant cry needs based on deep neural networks according to claim 4, characterized in that, In step S2, after selecting the optimal mode of the current sample, the corresponding selected mode feature map is input into the lightweight quality assessment network to calculate the signal-to-noise ratio estimate and signal quality score of the current crying sample. The instantaneous reward is calculated based on the local classification correctness and signal quality score, and the expected reward estimate of the selected arm is updated using the incremental update rule.

6. The method for classifying infant cry needs based on deep neural networks according to claim 5, characterized in that, In step S3, the selected modal feature map is input into a multi-scale attention network constructed based on an improved residual shrinkage network and a bidirectional gated recurrent unit, based on the multi-scale spatiotemporal feature extraction module, to extract deep spatiotemporal features, specifically: The selected modal feature map is concatenated with its corresponding signal quality score, and the concatenated data is input into the multi-scale spatiotemporal feature extraction module. The multi-scale spatiotemporal feature extraction module includes a multi-scale convolutional feature extraction layer, an improved residual contraction attention module, a bidirectional gated recurrent unit layer, and a temporal attention aggregation layer. Based on the multi-scale convolutional feature extraction layer, features are extracted from the concatenated data to obtain a multi-scale fused feature map. The multi-scale fused feature map is then input into the improved residual contraction attention module for soft thresholding. The processed multi-scale fused feature map is reshaped into a time step sequence and input into the bidirectional gated recurrent unit layer to obtain a fused bidirectional temporal dependent feature sequence. Based on the temporal attention aggregation layer, the fused bidirectional temporal dependent feature sequence is weighted and aggregated to obtain a deep spatiotemporal feature vector.

7. The method for classifying infant cry needs based on deep neural networks according to claim 6, characterized in that, In step S4, the deep spatiotemporal features are input into the classification network based on the demand classification module, and the demand category corresponding to the baby's current cry and its confidence level are output, specifically: The deep spatiotemporal feature vector is input into the demand classification module. Through two fully connected layers and a Softmax output layer, the probability distribution of five demand categories is output. The category corresponding to the highest probability is the demand category of the baby's current cry, and the highest probability itself is the confidence level.

8. A system for classifying infant cry needs based on deep neural networks, applied to the method for classifying infant cry needs based on deep neural networks as described in any one of claims 1-7, characterized in that, include: The data acquisition module, dynamic feature selection module, multi-scale spatiotemporal feature extraction module, demand classification module, and intervention decision-making module are connected sequentially. The data acquisition module acquires the raw crying audio of the infant, and preprocesses and performs time-frequency transformation on the raw crying audio to obtain a multi-scale time-frequency representation set; The dynamic feature selection module adaptively selects the optimal time-frequency representation mode based on the current signal quality and generates a feature map of the selected mode. The multi-scale spatiotemporal feature extraction module is used to extract deep spatiotemporal features from selected modality feature maps by constructing a multi-scale attention network based on an improved residual shrinkage network and a bidirectional gated recurrent unit. The demand classification module is used to input deep spatiotemporal features into the classification network and output the demand category and its confidence level corresponding to the baby's current cry. The intervention decision module is used to generate personalized soothing instructions based on the need category and its confidence level, combined with preset infant physiological parameters and crying breathing cycle characteristics, and send the soothing instructions to the smart soothing device to perform the corresponding operation.

9. The system for classifying infant cry needs based on deep neural networks according to claim 8, characterized in that, The data acquisition module collects the baby's original crying audio through a circular array of four microphones placed around the crib.

Images