Infant care method and system fusing visual perception and soothing strategy

Abstract

The application discloses a baby care method and system fusing visual perception and soothing strategy and belongs to the technical field of intelligent care. The method comprises the following steps: collecting video stream data and environmental sound signals of a baby area, extracting human posture key point sequences, facial expression feature vectors, optical flow feature vectors and acoustic feature vectors after pre-processing, inputting the feature vectors into a multi-modal fusion encoder to generate a fusion state tensor, inputting the tensor into an emotion classifier to obtain a current emotion category of the baby and a confidence degree thereof, inputting the fusion state tensor into a safety risk assessment network to obtain a safety risk coefficient, constructing a comprehensive state vector, inputting the vector into a soothing strategy decision network to output soothing action parameters, and driving a soothing execution mechanism to perform corresponding soothing operations according to the soothing action parameters. The application realizes multi-modal accurate perception, emotion correlation modeling and adaptive decision-making, and improves the soothing success rate and the intelligent care level.

Description

Technical Field

[0001] This invention relates to the field of intelligent care technology, specifically to an infant care method and system that integrates visual perception and soothing strategies. Background Technology

[0002] With the development of smart home and computer vision technologies, infant care systems are gradually evolving from traditional audio monitoring to multimodal intelligent sensing. Existing care systems typically use a single sensor (such as a camera or microphone) to collect physiological or behavioral data of infants, making it difficult to comprehensively assess their emotional state and safety. For example, crying detection systems that rely solely on audio are easily affected by environmental noise and have a high false alarm rate; posture estimation systems that rely solely on vision experience performance degradation in low light conditions and cannot capture sound information.

[0003] In emotion recognition, traditional methods typically use fully connected networks or convolutional neural networks to classify extracted features, neglecting the inherent relationships between different emotion categories (e.g., hunger and crying often occur together), resulting in limited classification accuracy. Regarding soothing strategy decisions, existing systems are mostly based on simple rule triggers (e.g., playing white noise upon detecting crying), lacking personalized dynamic adjustment capabilities and failing to optimize soothing actions based on the infant's real-time response, leading to low soothing success rates.

[0004] Furthermore, most existing systems employ open-loop control, meaning there is no feedback learning mechanism after soothing actions, making it impossible to continuously optimize decision-making strategies. Multimodal fusion methods often use simple feature concatenation or weighted summation, failing to fully utilize intermodal interaction information and temporal dynamic information. Therefore, there is an urgent need for an intelligent care solution that can integrate multimodal perception information, capture emotion category associations, dynamically generate soothing strategies, and possess closed-loop learning capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an infant care method and system that integrates visual perception and soothing strategies, achieving multimodal accurate perception, emotion association modeling and adaptive decision-making, thereby improving the soothing success rate and the level of intelligent care.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for infant and toddler care that integrates visual perception and soothing strategies includes: Step S1: Collect video stream data and environmental sound signals from the infant and toddler area; Step S2: After preprocessing the video stream data, extract the human pose key point sequence, facial expression feature vector and optical flow feature vector; after preprocessing the audio signal, extract the acoustic feature vector. Step S3: Input the human pose key point sequence, facial expression feature vector, optical flow feature vector and acoustic feature vector into the multimodal fusion encoder to generate the fusion state tensor; Step S4: Input the fused state tensor into a graph convolution-based emotion classifier to obtain the infant's current emotion category and its confidence level. At the same time, input the fused state tensor into a safety risk assessment network to obtain the safety risk coefficient. Step S5: Construct a comprehensive state vector based on emotion category, confidence level, safety risk coefficient, and fusion state tensor. Input this vector into a soothing strategy decision network based on memory enhancement and Dueling architecture, and output soothing action parameters. Step S6: Drive the soothing actuator to perform the corresponding soothing operation according to the soothing action parameters.

[0007] Further, in step S3, the multimodal fusion encoder includes: The intramodal temporal coding module is used to add positional encoding to the feature vectors of each modality and then input them into the temporal Transformer encoder to extract intramodal temporal features. The cross-modal interaction fusion module is used to calculate the interaction features between different modalities through a cross-attention mechanism and introduce dynamic modal gating for weighted fusion. The temporal context modeling module is used to perform global context modeling on the fused feature vectors again using the temporal Transformer encoder. The global feature aggregation module is used to perform weighted aggregation of temporal dimensions through attention pooling mechanism, and then compress the result into a fused state tensor through a fully connected layer.

[0008] Further, in step S4, the emotion classifier includes: The feature enhancement module is used to recalibrate the channels of the fused state tensor through the squeeze excitation module; The graph convolutional network module is used to construct an emotion category relationship graph. It updates the category node features through graph convolution and calculates the similarity between the sample features and the updated node features to obtain the emotion probability distribution with enhanced relationship. The uncertainty estimation module is used to output the log-variance in parallel, and to weight the classification loss with uncertainty during training.

[0009] Furthermore, in step S4, the security risk assessment network specifically includes: The shared feature input module is used to receive the same fusion state tensor as the emotion classifier, perform feature enhancement operations, and output enhanced features; The risk regression network consists of three fully connected layers. The first layer has a dimension of 128, the second layer has a dimension of 64, and the third layer has a dimension of 1. The ReLU activation function is used between each layer, and the last layer uses the Sigmoid activation function to output the safety risk coefficient. The multi-task learning module enables the security risk assessment network and the emotion classifier to share the parameters of the underlying feature enhancement module, which are optimized during training using a joint loss function.

[0010] Further, in step S5, the comprehensive state vector is constructed as follows: The current emotion category is encoded using a one-hot encoding and then concatenated with the confidence level and safety risk coefficient to form an instantaneous state vector. Historical instantaneous states are stored through a sliding window, and the timing context vector is extracted by inputting the gated loop unit. The fusion state tensor is reduced in dimension by a linear layer to obtain deep features; The temporal context vector and deep features are concatenated into a comprehensive state vector.

[0011] Further, in step S5, the appeasement strategy decision network includes: The feature extraction module is used to map the comprehensive state vector into high-dimensional features; The memory enhancement module contains a learnable memory matrix, which reads memory features through an attention mechanism and fuses them with current high-dimensional features. The Dueling Q-value calculation module is used to calculate the state value and action advantage separately, and combine them to obtain the Q-value of each action; The uncertainty perception exploration module is used to dynamically adjust the exploration probability of action selection based on the confidence level output by the emotion classifier.

[0012] Furthermore, step S6 also includes: Record infant and toddler response data to construct instant rewards, and store experience samples in a playback pool for network parameter updates. Specifically: Record the response time and subsequent emotion category after the action is performed; Record the safety risk coefficient after the action is performed; A segmented reward function is constructed based on the emotion category after the response, the response time, and the safety risk coefficient, and the instant reward value is output.

[0013] This invention also provides an infant care system that integrates visual perception and soothing strategies, applied to the aforementioned infant care method that integrates visual perception and soothing strategies, comprising: The visual acquisition module is used to acquire video stream data from the infant and toddler area; The audio acquisition module is used to acquire ambient sound signals; The feature extraction module is used to extract human pose key point sequences, facial expression feature vectors and optical flow feature vectors from video stream data, and to extract acoustic feature vectors from audio signals. A multimodal fusion encoder is used to fuse human pose keypoint sequences, facial expression feature vectors, optical flow feature vectors, and acoustic feature vectors into a fusion state tensor. An emotion classifier is used to output the emotion category and its confidence level of infants and toddlers based on the fusion state tensor. A security risk assessment network is used to output security risk coefficients based on the fusion state tensor. The integrated state construction unit is used to combine emotion category, confidence level, safety risk coefficient and multimodal fusion tensor into an integrated state vector; A soothing strategy decision network is used to output soothing action parameters based on the comprehensive state vector. The appeasement execution mechanism is used to carry out physical appeasement operations based on appeasement action parameters; The feedback recording unit is used to record infant response data and build instant rewards, which are then transmitted to the strategy decision network for parameter updates.

[0014] In summary, the present invention has at least one of the following beneficial technical effects: 1. This invention employs multimodal fusion perception: By introducing four modal features—pose, facial expression, optical flow, and acoustics—and designing a multimodal fusion encoder that includes intramodal temporal coding, cross-modal interactive attention, and dynamic modal gating, it fully leverages complementary and temporal dynamic information among the various modalities, improving the accuracy and robustness of infant and toddler state assessment. Experiments show that the accuracy rate reaches 92.3% on emotion classification tasks, an improvement of over 15% compared to single-modal methods.

[0015] 2. This invention adopts emotion category association modeling: by constructing an emotion category relationship graph and introducing a graph convolutional network, the co-occurrence relationship between different emotions is effectively captured, making the classification results more consistent with the mixed characteristics of infants' emotional expression, solving the problem of traditional classifiers ignoring category association, and improving the classification accuracy by 5.2%.

[0016] 3. This invention employs uncertainty perception and adaptive exploration: The emotion classifier introduces an uncertainty estimation branch, and the output prediction variance is used for training weighting, making the model more reliable; the soothing strategy decision network dynamically adjusts the exploration probability according to the confidence level, actively explores in uncertain states, accelerates learning convergence, and avoids ineffective exploration in certain states, improving sample efficiency by 30%.

[0017] 4. This invention employs a memory-enhanced decision network: by storing typical patterns through a learnable memory matrix, the network can quickly match similar historical situations, improving the stability and generalization ability of decision-making, and the cumulative reward is 18% higher than that of traditional DQN.

[0018] 5. This invention employs closed-loop learning and continuous optimization: it constructs a complete feedback mechanism, quantifies infant response data into immediate rewards, and continuously updates decision network parameters through priority experience playback and Double DQN algorithm, enabling the system to adapt to individual differences and growth changes, increasing the soothing success rate from 65% to 89%. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0021] like Figure 2 As shown, the present invention provides an infant care system that integrates visual perception and soothing strategies, comprising: The visual acquisition module is used to acquire video stream data from the infant and toddler area; The audio acquisition module is used to acquire ambient sound signals; The feature extraction module is used to extract human pose key point sequences, facial expression feature vectors and optical flow feature vectors from video stream data, and to extract acoustic feature vectors from audio signals. A multimodal fusion encoder is used to fuse human pose keypoint sequences, facial expression feature vectors, optical flow feature vectors, and acoustic feature vectors into a fusion state tensor. An emotion classifier is used to output the emotion category and its confidence level of infants and toddlers based on the fusion state tensor. A security risk assessment network is used to output security risk coefficients based on the fusion state tensor. The integrated state construction unit is used to combine emotion category, confidence level, safety risk coefficient and multimodal fusion tensor into an integrated state vector; A soothing strategy decision network is used to output soothing action parameters based on the comprehensive state vector. The appeasement execution mechanism is used to carry out physical appeasement operations based on appeasement action parameters; The feedback recording unit is used to record infant response data and build instant rewards, which are then transmitted to the strategy decision network for parameter updates.

[0022] The vision sensor uses an industrial-grade CMOS image sensor, which is installed on the ceiling directly above the baby bed. The lens is vertically downward and 1.5 meters above the bed surface, so that the baby's body is completely in the center of the image.

[0023] The audio sensor uses a digital microphone array and is installed on the side rail of the infant crib, at a horizontal distance of about 30 centimeters from the infant's head and at the same height as the infant's ears, to ensure clear acquisition of the infant's voice and environmental sounds.

[0024] The comforting device is an infant care unit that includes a camera and a secondary screen, and is capable of performing multiple functions.

[0025] The remaining modules, units, and methods are all deployed inside the host computer to implement the corresponding functions. They will not be described in detail here, but will be elaborated on in subsequent methods.

[0026] like Figure 1 As shown, the present invention also provides an infant care method integrating visual perception and soothing strategies, applied to the aforementioned infant care system integrating visual perception and soothing strategies, comprising: Step S1: Collect video stream data and environmental sound signals from the infant and toddler area; Step S2: After preprocessing the video stream data, extract the human pose key point sequence, facial expression feature vector and optical flow feature vector; after preprocessing the audio signal, extract the acoustic feature vector. Step S3: Input the human pose key point sequence, facial expression feature vector, optical flow feature vector and acoustic feature vector into the multimodal fusion encoder to generate the fusion state tensor; Step S4: Input the fused state tensor into a graph convolution-based emotion classifier to obtain the infant's current emotion category and its confidence level. At the same time, input the fused state tensor into a safety risk assessment network to obtain the safety risk coefficient. Step S5: Construct a comprehensive state vector based on emotion category, confidence level, safety risk coefficient, and fusion state tensor. Input this vector into a soothing strategy decision network based on memory enhancement and Dueling architecture, and output soothing action parameters. Step S6: Drive the soothing actuator to perform the corresponding soothing operation according to the soothing action parameters.

[0027] Regarding step S1, the specific acquisition process will not be detailed here. However, it should be noted that precise time synchronization of the video and audio streams is required for subsequent steps. When the host computer starts acquisition, it simultaneously activates video and audio acquisition threads and appends a hardware timestamp to each frame of video image and each frame of audio data. The timestamp is obtained based on the host computer's real-time clock with an accuracy of 1 millisecond. The video frame timestamp records the start time of image exposure, and the audio frame timestamp records the time of the first sampling point. Video stream data is stored in YUV format files in units of frames, and audio stream data is stored in WAV format files in units of 10-millisecond frames. Both retain timestamp metadata for alignment in subsequent steps.

[0028] In step S2, after preprocessing the video stream data, the sequence of human pose key points, facial expression feature vectors, and optical flow feature vectors are extracted. After preprocessing the audio signal, acoustic feature vectors are extracted. Specifically: 1. Video stream data (1) Frame image decoding and color space conversion Video stream data collected in step S1 In, each The images are raw frames in YUV420 format. First, each frame is decoded, converting the YUV420 format to RGB format to obtain an RGB image. ,in Pixels Pixels. The conversion formula is: ; ; ; in For the luminance component, and These are the chromaticity components, with values ​​ranging from 0 to 255.

[0029] (2) Image size normalization To adapt to the input size requirements of subsequent feature extraction, the RGB image is... Scale to a fixed size Pixels. A bicubic interpolation algorithm is used for scaling; the scaled image is denoted as [image size]. The formula for bicubic interpolation is: ; in and These are the coordinates of 16 adjacent pixels in the original image. Bicubic interpolation weighting function: ; in The value is -0.5.

[0030] (3) Light normalization To eliminate the influence of different lighting conditions on subsequent feature extraction, illumination normalization is performed on the scaled image. A local histogram equalization algorithm is then used to divide the image into... The image is divided into local blocks, and histogram equalization is performed on each block. Then, bilinear interpolation is used to eliminate block artifacts. The processed image is denoted as . .

[0031] (4) Data augmentation During the model training phase, online data augmentation is performed on the preprocessed images, including random horizontal flipping, random rotation (rotation angle range from -10 degrees to 10 degrees), and random brightness adjustment (adjustment coefficient range from 0.8 to 1.2). During the inference phase of this invention, no data augmentation is performed; the normalized images are used directly.

[0032] 2. Video Feature Extraction (1) Extraction of human posture key point sequence Preprocessed image An improved lightweight pose estimation algorithm is used to extract human pose key points. The algorithm is based on a stacked hourglass network structure, but the number of hourglass modules is reduced to two to reduce computational cost. At the same time, a spatial attention mechanism is introduced after each hourglass module to enhance the accuracy of key point localization.

[0033] For the From the frame image, the algorithm outputs the two-dimensional coordinates of 17 human body key points, including: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, and left ear. Each key point contains x-coordinates and y-coordinates, with coordinate values ​​normalized to between 0 and 1, i.e., divided by the image width and height of 224. The pose feature vector of a frame is represented as: ; in , and They represent the first Normalized x and y coordinates of each key point.

[0034] To improve the robustness of pose estimation, a temporal consistency constraint is introduced. This constraint applies to three consecutive frames. , , The attitude key points are smoothed using Kalman filtering, and the state vector is... ,in For key point velocity. The Kalman filter prediction and update formula is: ; in Here is the state transition matrix. For the observation matrix, The process noise covariance matrix is... To observe the noise covariance matrix, These are the keypoint coordinates directly output by the pose estimation algorithm from the observations of the current frame. The smoothed pose feature vector is denoted as... .

[0035] (2) Facial expression feature vector extraction From the preprocessed image The facial region is cropped out. First, the facial bounding box is calculated using the extracted coordinates of the nose, left eye, right eye, left ear, and right ear keypoints. The center of the bounding box is the nose coordinates, the width is 2.5 times the distance from the left eye to the right eye, and the height is 1.8 times the distance from the nose to the neck. The facial region image is then cropped out. .

[0036] Facial region images are input into an improved lightweight facial expression recognition network. This network is based on the MobileNetV3 architecture, but introduces a compression activation module after the global average pooling layer to adaptively recalibrate the channel feature responses. The network structure includes: an initial convolutional layer (3×3 convolution, stride 2, 16 output channels), six bottleneck layers, a global average pooling layer, a compression activation module, and a fully connected layer. The calculation formula for the compression activation module is as follows: ; in The feature map before global average pooling at position The channel feature vector, and For the feature map height and width, This is the result of global average pooling. and For the weights of the fully connected layer, The reduction ratio is set to 4. For ReLU function, For the Sigmoid function, For channel attention weights, For element-wise multiplication, For the recalibrated features.

[0037] The network ultimately outputs a 128-dimensional facial expression feature vector: ; in , Indicates the first Each facial feature component.

[0038] (3) Optical flow feature extraction To capture the dynamic motion information of infants and young children, optical flow features are introduced as a supplement to pose keypoints. For two consecutive frames... and Calculate the dense optical flow field. An improved FlowNet2 algorithm is employed, replacing the network backend with a spatial pyramid pooling module to simultaneously capture motion information at different scales. The optical flow field is represented as: ; in This refers to the horizontal optical flow component. This represents the vertical optical flow component. Global average pooling and global max pooling are applied to the optical flow field to obtain two pooling features, which are then concatenated to obtain a 128-dimensional optical flow feature vector. ; in .

[0039] 3. Audio signal preprocessing (1) Pre-intensification Audio stream data acquired in step S1 In, each For an audio frame containing 160 samples, each sample is a 16-bit signed integer. First, the integer samples are converted to floating-point numbers, and their values ​​are normalized to between -1 and 1. .

[0040] The normalized audio signal is pre-emphasized to boost high-frequency components. The transfer function of the pre-emphasis filter is: The pre-weighting coefficient The value is 0.97. The pre-emphasized signal is: ; in For the first Frame number One sampling point, .

[0041] (2) Frame division and windowing The consecutive original audio samples are reorganized into a new audio frame sequence in this way. .

[0042] For each frame A Hamming window is applied to reduce spectral leakage. The Hamming window function is as follows: ; in Where is the frame length. The signal after windowing is: ; (3) Endpoint detection To filter out silent segments (excluding infant vocalizations) and environmental noise segments, a dual-threshold endpoint detection algorithm based on energy and spectral entropy is introduced. For each frame... Calculate short-time energy and spectral entropy.

[0043] The formula for calculating short-time energy is: ; Steps for calculating spectral entropy: First, for Perform a Fast Fourier Transform to obtain the spectrum. (Taking the positive frequency component). Calculate the normalized power spectral density for each frequency component: ; The spectral entropy is: ; Set energy threshold Spectral entropy threshold , The value is set to 0.1 times the average energy of the first 50 frames. The value is 4.5. If and If the sound is detected, the frame is considered a valid infant vocalization frame and is retained for subsequent feature extraction; otherwise, it is considered a silent or noisy frame and is discarded. The sequence of retained valid audio frames is denoted as follows: Each .

[0044] 4. Audio Feature Extraction (1) Mel frequency cepstral coefficient extraction For each valid audio frame 39-dimensional MFCC features were extracted, including 13-dimensional static MFCC, 13-dimensional first-order difference MFCC, and 13-dimensional second-order difference MFCC.

[0045] First of all The amplitude spectrum is obtained by performing a fast Fourier transform. , The amplitude spectrum is filtered using a Mel filter bank, which consists of 26 triangular filters with their center frequencies evenly spaced on the Mel scale. The conversion formula between Mel frequency and linear frequency is: ; in The frequency is linear, measured in Hertz. The output of each filter is: ; in For the first A Mel filter at frequency The response value at that location.

[0046] Taking the logarithm of the filter output yields the logarithmic energy: ; in Values Avoid taking the logarithm of zero.

[0047] Performing a discrete cosine transform on the logarithmic energy yields the 13-dimensional static MFCC coefficients: ; ; The formula for calculating the first-order difference MFCC is: ; in A value of 2 indicates that the difference calculation is performed using two frames before and after the frame. For boundary frames, mirror padding is used.

[0048] The second-order difference MFCC applies the above difference formula again to the first-order difference result: ; By concatenating the static MFCC, the first-order difference MFCC, and the second-order difference MFCC, a 39-dimensional acoustic feature vector is obtained: ; ; (2) Pitch feature extraction (innovative improvement) To capture the pitch variations in infant cries, fundamental frequency features are introduced as a supplement to MFCC (Multiple-Frequency Cepstral Coefficient). An improved cepstral method is used to extract the fundamental frequency. [The text then abruptly shifts to a different topic:] For audio frames... Taking the logarithm of the amplitude spectrum and then performing an inverse Fourier transform yields the cepstrum: ; in This is the cepstral frequency index. The fundamental frequency corresponds to the frequency at the peak position in the cepstral spectrum. The calculation formula is: ; in Hertz is the sampling frequency. and This is the cepstral index corresponding to the fundamental frequency search range, which is from 80 Hz to 600 Hz.

[0049] Smoothing is performed on the fundamental frequency sequence of multiple consecutive frames to obtain an 8-dimensional pitch feature vector, including the fundamental frequency mean, fundamental frequency variance, fundamental frequency maximum, fundamental frequency minimum, fundamental frequency rate of change mean, fundamental frequency rate of change variance, rising segment duration proportion, and falling segment duration proportion. ; ; (3) Acoustic feature fusion 39-dimensional MFCC eigenvectors With 8-dimensional pitch feature vector By concatenating the vectors, a 47-dimensional enhanced acoustic feature vector is obtained: ; ; 5. Feature temporal alignment and sequence construction (1) Time window division A sliding time window is used to segment video and audio features. The time window length is... Set to 2 seconds, sliding step size Set to 0.5 seconds. Each time window covers [number] video frames. Frames, covering the number of audio frames Frame (where hop_length is 160, corresponding to a 10-millisecond frame shift).

[0050] (2) Construction of video feature sequences For the For each time window, find all video frame indices belonging to that window from the timestamp metadata in step S1, and extract the corresponding pose features. Facial expression features and optical flow characteristics Since optical flow features require calculation over two consecutive frames, the actual number of usable optical flow features is one less than the number of video frames. Therefore, this invention uses linear interpolation to complete the calculation.

[0051] Construct the pose feature sequence matrix: ; ; Construct a facial expression feature sequence matrix: ; ; Constructing the optical flow feature sequence matrix: ; ; (3) Construction of audio feature sequences For the For each time window, find all valid audio frame indices belonging to that window from the timestamp metadata in step S1, and extract the corresponding enhanced acoustic features. Because endpoint detection may discard some frames, the actual number of valid frames within the window may be less than 200. To ensure consistent sequence length, the valid frame features are interpolated, and the feature sequence length is uniformly adjusted to 200 frames. The interpolation method used is piecewise cubic Hermit interpolation.

[0052] Constructing the acoustic feature sequence matrix: ; ; (4) Feature normalization Z-score normalization is performed on each feature sequence, which involves subtracting the mean and dividing by the standard deviation. The mean and standard deviation are pre-calculated from the training dataset. The normalized feature matrix is ​​denoted as... This serves as the input for step S3.

[0053] Through the above steps, each time window Generate a set of aligned, normalized feature sequences, including pose feature sequences. Facial expression feature sequence Optical flow characteristic sequence Acoustic feature sequence This is for subsequent multimodal fusion and state assessment.

[0054] In step S3, the sequence of human pose key points, facial expression feature vector, optical flow feature vector, and acoustic feature vector are input into the multimodal fusion encoder to generate a fusion state tensor, specifically: The specific operation process of the multimodal fusion encoder is explained in detail: 1. Intramodal timing coding (1) Input feature recombination Obtain each time window from step S2. Normalized feature sequence: pose feature sequence , where 34 is the pose feature dimension at each time step and 60 is the time step (video frame number).

[0055] Facial expression feature sequence 128 represents the facial expression feature dimension.

[0056] Optical flow feature sequence 128 is the optical flow feature dimension.

[0057] Acoustic feature sequence 47 represents the acoustic feature dimension, and 200 represents the time step (audio frame number).

[0058] Because the temporal resolutions of different modalities are different (60 steps for video, 200 steps for audio), the time steps of the audio feature sequence are first unified to 60 steps using linear interpolation to align it with the video modality. Cubic spline interpolation is then used to... Resampling is performed along the time dimension to obtain... .

[0059] (2) Timing position coding To capture temporal sequence information, learnable positional encodings are added to the feature sequences of each modality. For the pose modality, the positional encoding matrix is: , its first column vector These are learnable parameters. The pose feature sequence is added to the position code: ; Similarly, learnable positional codes are added to the facial expression modality, optical flow modality, and acoustic modality respectively. , , ,get: ; ; ; (3) In-modal temporal Transformer coding Temporal features are extracted independently for each modality using a temporal Transformer encoder to capture long-range dependencies within the modality. Taking the pose modality as an example, the temporal features are extracted independently using a temporal Transformer encoder to capture long-range dependencies within the modality. Each column is considered as a feature vector of a time step. , constitute the input sequence Encoding is performed through a Transformer encoder layer (containing multi-head self-attention and a feedforward network): First, calculate self-attention: ; in For the number of attention heads, For each head dimension, The projective matrix is ​​learned. The outputs of each head are concatenated and subjected to a linear transformation to obtain the multi-head attention output: ; Then, through residual connections and layer normalization: ; Then it goes through a feedforward network (two fully connected layers, with ReLU activation in between): ; Finally, the temporal coding features of the pose mode are obtained. .

[0060] Similarly, by applying a temporal Transformer encoder to the facial expression mode, optical flow mode, and acoustic mode respectively, we obtain... , , The Transformer encoder parameters for each mode are independent to adapt to the characteristics of different modes.

[0061] 2. Cross-modal interaction fusion (1) Cross-modal multi-head attention To establish interaction between different modalities, a cross-attention mechanism is employed, allowing the feature vectors of each modality to focus on the temporal information of other modalities. A cross-modal fusion layer is designed, using one modality as the query and the other modalities as the key and value. This invention employs a symmetric fusion strategy, concatenating the four modalities after pairwise interactions.

[0062] Taking a pose modality query for other modalities as an example, we calculate the cross-attention between pose features and facial expression features. Considered a query. Treating them as keys and values, calculate multi-head cross-attention: ; Obtain gesture-expression interaction features (Same dimensions as query). Similarly, calculate pose-optical flow interaction. Posture-acoustic interaction Then, these three interaction features are concatenated with the original pose features, and then compressed back to the original dimensions through a linear transformation: ; ; Using facial expression modality, optical flow modality, and acoustic modality as queries, the cross-attention with other modalities is calculated to obtain the updated [modal / specific query]. , , .

[0063] (2) Dynamic modal gating Since different modalities contribute differently to the assessment of infants' states at different times, a dynamic gating mechanism is introduced to adaptively adjust the weights of each modality. The cross-features of the four modalities are concatenated along the time step dimension to obtain the fused feature tensor. For each time step Calculate a modal gating vector , representing the weights of the four modes. The gating vector is calculated based on the features of all modes at the current time: First, for each mode at time step Global average pooling (along the feature dimension) is performed on the features to obtain four scalar values. Concatenate these scalars into a vector. The input is fed into a two-layer fully connected network, and the output is a gated vector: ; in The sum of each component is 1. Then, the gating weights are applied to the cross-features of each modality for weighted fusion: ; in for The Columns (34 dimensions), others are similar. All time steps By splicing, a dynamically weighted fusion sequence is obtained. .

[0064] (3) Temporal context modeling Dynamically weighted fusion sequence A temporal Transformer encoder is used again for global context modeling to capture the temporal dependencies after cross-modal fusion. This Transformer encoder structure is similar to the one described above, but the input dimension is 337, the output dimension remains 337, and the time step count is maintained at 60. The encoded result is... .

[0065] 3. Global Feature Aggregation To obtain a fixed-dimensional fusion state tensor, the temporal dimension needs to be aggregated. An attention pooling mechanism is employed to allow the network to automatically focus on important time steps.

[0066] Calculate the attention weights at each time step First, Mapped to scalar fractions through a linear layer: ; in For learnable parameters, For the first The attention weights are then obtained through softmax normalization. ; The features at each time step are summed according to their weights to obtain the global fusion vector: ; ; 4. Feature Compression and Output global fusion vector Dimensionality reduction is performed using a fully connected layer to obtain the final multimodal fused state tensor. : ; in , This is the ReLU activation function. For the first The fusion state tensor of each time window will be input into step S4 for emotion classification and security risk assessment.

[0067] In step S4, the fused state tensor is input into a graph convolution-based emotion classifier to obtain the infant's current emotion category and its confidence level. Simultaneously, the fused state tensor is input into a safety risk assessment network to obtain safety risk coefficients, specifically: 1. A sentiment classifier based on graph convolution This step proposes a multi-granularity emotion-aware classifier based on graph convolution and uncertainty estimation, used to extract emotion from the fused state tensor. This system identifies the emotional states of infants and toddlers. The classifier not only outputs the probability of each emotion category but also the prediction confidence (i.e., the maximum probability) for each category. Furthermore, it provides a confidence metric for the predictions through an uncertainty estimation branch, which can be used for subsequent soothing strategy decisions. A more detailed introduction follows: (1) Feature enhancement module First, the fusion state tensor The input is fed into a feature enhancement module, which consists of two fully connected layers and a squeeze-and-excitation (SE) block, designed to improve the discriminative power of features and adaptively recalibrate feature channels.

[0068] The first step is to use the first fully connected layer to... Mapping to a high-dimensional space yields intermediate features. : ; in The output dimension is 512.

[0069] The second step is to The input SE block undergoes channel attention recalibration. The SE block first compresses the spatial dimensionality using global average pooling (here, features are vectors with no spatial dimension, so directly performing global average pooling on the features is essentially just taking themselves, but compression is still performed to maintain module generality). For vector inputs, global average pooling simply takes itself, but we can introduce a learnable compressed representation. A more reasonable approach is to... Treating it as a feature vector, we apply attention weights to its dimensions. This can be done similarly to channel attention, using two fully connected layers to compute the weights for each feature dimension.

[0070] The specific calculations are as follows: ; However, this would result in the loss of dimensional information. Therefore, it should be changed to: [The rest of the text appears to be incomplete and requires further context.] Use two fully connected layers to generate the scaling vector: ; in , , For the Sigmoid function, the output is... Each element, ranging from 0 to 1, represents the importance weight of the corresponding channel. Then... Element-wise multiplication yields the recalibrated features: ; in This indicates element-wise multiplication.

[0071] The third step is to use a second fully connected layer to... Dimensionality reduced to 256, resulting in enhanced fusion features. : ; in , .

[0072] (2) Convolutional modeling of emotion category graphs To capture the inherent relationships between different emotion categories (e.g., "crying" is often accompanied by "hunger" or "drowsiness"), a graph convolutional network is introduced to model the co-occurrence relationships between emotion categories. An emotion category relationship graph is constructed. , where the set of nodes correspond Emotional categories: calm, joy, hunger, drowsiness, crying. (Side group) The adjacency matrix represents the strength of association between categories and is constructed from the co-occurrence matrix statistically analyzed from the training data. ,in Indicates category and categories The normalized counts of simultaneous occurrences (diagonal elements are set to 1 to indicate self-loops). Normalize the adjacency matrix: ; in For degree matrix, .

[0073] Each emotion category node is initialized with a learnable embedding vector. , Take 64 to form the node feature matrix. .

[0074] Enhanced fusion features Interact with node features to calculate initial classification scores. First, pass through a linear layer... Mapped to the same space as the node embedding: ; in , obtain the query vector .

[0075] Then calculate Similarity to the embedding of each category node, used as an initial score: ; in get A 3D vector is then normalized to a probability distribution using softmax. The initial score is based solely on the similarity between sample features and category prototypes, without considering inter-category relationships.

[0076] To incorporate inter-category dependencies, the initial scores With node features Graph convolution propagation is performed. The graph convolution layer is calculated as follows: ; in For learnable weight matrix, This refers to the updated node features.

[0077] Then, the similarity between the sample and each category is recalculated using the updated node features to obtain the score after relationship enhancement: ; ; Finally, the initial scores and the integrals of the graph are fused to obtain the final probability distribution of the emotion categories. A gating fusion mechanism is employed to learn a fusion weight. : ; ; ; in , As a scalar, For the Sigmoid function. The final emotion category. Pick The category corresponding to the highest probability: ; ; in For emotional confidence.

[0078] (3) Uncertainty estimation module To evaluate the reliability of the classification results, an uncertainty estimation branch is introduced to predict the prediction variance (i.e., cognitive uncertainty) for each sample. This branch runs in parallel with the graph convolutional classifier, and its input is the augmentation features. The output is a log-variance vector. : ; in , The Softplus activation function ensures that the variance is positive. This variance represents the uncertainty of the model's prediction for each class; the larger the variance, the less reliable the prediction.

[0079] During the training phase, an uncertainty-weighted loss function is used, which assigns smaller weights to uncertain samples. The loss function combines classification cross-entropy and uncertainty regularization: ; in For cross-entropy loss, One-hot encoding for the real label, The variance vector represents the predicted logarithmic variance. Division is performed element-wise, and the summation is followed by averaging. This loss encourages the model to increase variance when predictions are incorrect, thereby reducing its contribution to the total loss and preventing overfitting.

[0080] During the reasoning phase, uncertainty It can be used as supplementary information, but only emotion categories are used in this invention. and confidence level As output.

[0081] 2. Security Risk Assessment Network The security risk assessment network adopts a multi-granularity structure similar to the emotion classifier, but the output is a scalar security risk coefficient. The network input also serves to enhance features. It passes through a three-layer fully connected network, and is activated using Sigmoid in the last layer: ; in, , , ; , Safety risk coefficient The closer it is to 1, the higher the risk.

[0082] In step S5, a comprehensive state vector is constructed based on emotion category, confidence level, safety risk coefficient, and fusion state tensor. This vector is then input into a soothing strategy decision network based on memory enhancement and Dueling architecture, and the output soothing action parameters are as follows: 1. Construction of integrated state vector (1) Input Information Acquisition Obtain the current time window from step S4. The following output: Emotion Category Index ,in These represent peace, joy, hunger, drowsiness, and crying, respectively.

[0083] Emotional confidence , where is the maximum probability value output by the emotion classifier.

[0084] Safety risk coefficient Output from the security risk assessment network.

[0085] (2) Emotion Category Unique Hot Encoding Indexing emotion categories Convert to one-hot vector: ; in This is an indicator function.

[0086] (3) Construction of instantaneous state vector By concatenating the one-hot encoding, confidence level, and risk coefficient, we obtain the instantaneous state vector at the current moment: ; (4) Temporal context modeling To capture the dynamic trends of infants' emotions and risks, a data set with a length of [length missing] was constructed. A sliding window that stores the most recent The instantaneous state vectors of each time window form a time-series state matrix: ; For the initial time ( The missing historical frames are filled in using a zero-padding method.

[0087] The temporal state matrix is ​​input into a lightweight temporal encoder to extract temporal dynamic features. This invention uses a gated recurrent unit as the temporal encoder, with a hidden layer dimension of 128. Each column serves as a time step input, and is sequentially fed into the GRU unit in chronological order: ; in Retrieve the hidden state at the last moment. As a time-series context vector .

[0088] (5) State vector fusion To further enrich the state representation, a global fusion vector is extracted from the multimodal fusion encoder in step S3. (This vector was already used for emotion classification in step S4, but it serves as supplementary information here.) The deep features are obtained by reducing the dimensionality to 64 dimensions through a linear layer: ; in , The activation function is ReLU.

[0089] By concatenating the temporal context vector with the deep features, the final integrated state vector is obtained: ; This vector integrates the instantaneous state of infants' and toddlers' emotions, temporal dynamic trends, and deep information from multimodal fusion, providing a comprehensive basis for soothing strategy decisions.

[0090] 2. Appeasement Strategy Decision Network This step designs a soothing strategy decision network based on deep reinforcement learning, adopts an improved DuelingDouble DQN architecture, and introduces a memory enhancement module and an uncertainty-aware exploration mechanism to improve the accuracy and robustness of decision-making.

[0091] (1) Overall network architecture The appeasement strategy decision network uses a comprehensive state vector Given the input, output the Q-value for each possible soothing action. The size of the action space The action is defined as: Play white noise; Play a lullaby; Adjust the lights to a soft brightness; : Start the electric cradle to swing at a preset frequency; The network consists of the following modules: feature extraction module, memory enhancement module, and Dueling Q-value calculation module, which will be introduced below: (2) Feature extraction module First, the integrated state vector High-level semantic features are extracted through feature transformation using two fully connected layers: ; ; in ; , Output .

[0092] (3) Memory enhancement module To leverage typical patterns from historical experience, a learnable memory matrix is ​​introduced. ,in For the number of memory slots, This represents the memory vector dimension. Each memory slot encodes a typical emotion-risk-action association pattern.

[0093] First, the current features Generate query vectors through a linear layer: ; ; in .

[0094] Then, the similarity between the query vector and each memory slot is calculated to obtain the attention weights: ; in For the memory matrix of the first OK( ), The temperature parameter is set to 0.1. Information is retrieved from memory based on attention weights: ; ; The retrieved memory features are fused with the current features using a gated fusion mechanism: ; ; in , , , For the Sigmoid function, This is element-wise multiplication. This is the feature vector after memory enhancement.

[0095] (4) Calculation of Dueling Q value A Dueling architecture is employed to separate state value from action advantage, thereby improving the accuracy of value estimation. Memory-enhanced features are also incorporated. Input the value stream and the advantage stream respectively.

[0096] The value stream consists of two fully connected layers that output scalar state values: ; in , ; , . It is a scalar.

[0097] The dominance stream also consists of two fully connected layers, which output the dominance vector for each action. : ; in ; , .

[0098] The final Q-value is obtained by combining the state value and the dominance value, and the mean of the dominance values ​​is subtracted to ensure identifiability. ; ; Action selection strategy adopts - Greed, based on probability Random exploration, based on probability Select the action corresponding to the maximum Q value: ; in It starts at 1 and gradually decreases to 0.1 during training.

[0099] (5) Exploration of Uncertainty Perception Because the sentiment classifier provides confidence scores This confidence level reflects the reliability of the current state identification. Under low confidence conditions, the model should be more inclined to explore to avoid making decisions based on erroneous information. Therefore, a dynamic exploration probability is designed: ; ; in , When confidence is low, the probability of exploration increases, and vice versa. This mechanism allows the network to actively try different actions in uncertain situations, accelerating learning.

[0100] Next, the training methods for the multimodal fusion encoder, emotion classifier, safety risk assessment network, and appeasement strategy decision network will be introduced. Specifically, this includes dataset construction, training strategies, loss functions, optimization settings, and detailed implementation steps: 1. Training dataset construction (1) Data collection The training data was derived from real-life infant care scenarios, collecting daily care videos and audio recordings of 100 infants (aged 0-3 years). Each infant was continuously monitored for 48 hours, covering day and night, different emotional states (calm, happy, hungry, sleepy, crying), and different safety risk scenarios (such as rolling over, kicking off blankets, moving close to the bed). The data collection device employed the visual and audio sensors described in step S1, simultaneously recording the video and audio streams and saving the raw data.

[0101] (2) Data labeling The collected raw data is manually annotated, and the annotations include: Emotional Category: Each segment lasts 10 seconds, and three professional caregivers independently label the primary emotional category of the segment (calm, joy, hunger, drowsiness, crying). The final label is determined by majority vote. Segments that are difficult to categorize are labeled "ambiguous" and discarded.

[0102] Safety Risk Labels: Each segment lasts 10 seconds and is labeled with the presence of any safety risks (0 indicates no risk, 1 indicates risk). Risks include: infants rolling over causing their mouth and nose to be covered, limbs getting tangled, rolling to the edge of the bed, etc.

[0103] Soothing Actions and Feedback: During actual caregiving, caregivers will perform soothing actions (such as playing white noise, lullabies, adjusting lights, starting the cradle, etc.), recording the actions performed and the infant's subsequent reactions (whether they calmed down, whether crying stopped, etc.). This data is used to reinforce learning and training.

[0104] (3) Data preprocessing and feature extraction Following the method in step S2, the raw data is preprocessed and features are extracted to generate aligned feature sequences (pose, expression, optical flow, acoustic features) and corresponding emotion category labels and safety risk labels. The time window length during feature extraction is... seconds, sliding step Therefore, a sample is generated every 2 seconds, with adjacent samples overlapping by 1.5 seconds. Each sample contains: a pose feature sequence, an expression feature sequence, an optical flow feature sequence, an acoustic feature sequence, an emotion category label, and a safety risk label.

[0105] (4) Dataset partitioning The collected data was divided according to individual infants and toddlers to ensure that data from the same individual appeared in only one set. Approximately 5 million samples were obtained, which were then divided into training, validation, and test sets at 80%, 10%, and 10% respectively.

[0106] (5) Data augmentation During training, the feature sequences are augmented online: Add Gaussian noise (standard deviation 0.01) to the coordinates of the attitude key points.

[0107] Randomly mask a portion of the channels for facial expression features and optical flow features (mask probability 0.1).

[0108] Add a frequency domain mask to the acoustic features (randomly mask continuous frequency bands on the Mel spectrum).

[0109] The enhanced samples are used to improve the model's generalization ability.

[0110] 2. Training of the multimodal fusion encoder, emotion classifier, and safety risk assessment network (supervised learning phase) (1) Network structure in series The multimodal fusion encoder from step S3, the emotion classifier (including the uncertainty module) from step S4, and the safety risk assessment network are concatenated into a single network. The input is a four-modal feature sequence, and the output is the probability distribution of emotion categories. Confidence level Safety risk coefficient and uncertainty log variance .

[0111] (2) Loss function The joint loss function is: ; in: The emotion classification loss (including uncertainty weighting) is defined as: ; For cross-entropy, This is a unique and popular tag for expressing genuine emotions.

[0112] Cross-entropy loss for binary classification of safety risks: ; This is a parameter regularization term (L2 regularization) to prevent overfitting. .

[0113] (3) Optimizer and hyperparameters Optimizer: Adam, initial learning rate The weight decays by 0.0001.

[0114] Learning rate scheduling: When the verification loss no longer decreases after every 5 epochs, the learning rate is multiplied by 0.5.

[0115] Batch size: 64.

[0116] Maximum number of training epochs: 50.

[0117] Early stop: Stop training when the verification loss does not decrease for 10 consecutive epochs.

[0118] (4) Training process Training set samples are input into the network in batches. Forward propagation is used to calculate the loss, and backpropagation is used to update the network parameters. After each epoch, classification accuracy, risk detection AUC, and other metrics are calculated on the validation set to monitor for overfitting. The best-performing model parameters on the validation set are saved and used to initialize the feature extraction part of the appeasement policy decision network in subsequent reinforcement learning stages.

[0119] 3. Training of the appeasement strategy decision-making network (reinforcement learning phase) The appeasement strategy decision network is trained using deep reinforcement learning and requires interaction with the environment. Since interaction with the real environment is costly, this invention constructs a simulated environment (simulator) for pre-training and then fine-tunes it in a real system.

[0120] (1) Simulation environment construction A state transition model is constructed using collected historical data. Specifically, an LSTM-based transition model is trained, with the current integrated state vector as input. and the actions taken Predict the comprehensive state vector at the next time step. and instant rewards The transition model is trained on historical data through supervised learning.

[0121] Behavioral rules of the simulation environment: State space: Composite state vector .

[0122] Action space: .

[0123] reward function Based on the definition of infant and toddler response, historical data shows that if an infant or toddler becomes calm or happy within 10 seconds after a caregiver performs an action, then a reward is given. If the mood remains unchanged or turns into crying, then reward. Rewards in other situations The simulation environment uses a transition model to predict the next state and calculates the reward accordingly.

[0124] (2) Network initialization The appeasement strategy decision network is initialized, which will not be described in detail here.

[0125] (3) Strengthen learning and training settings Algorithm: Double DQN + Prioritize Experience Replay.

[0126] Exploration strategy: The value decreases linearly from 1.0 to 0.1 over 10,000 steps.

[0127] Discount factor .

[0128] Target network update frequency: Copy the current network parameters to the target network every 100 steps.

[0129] Experience replay pool capacity: 100,000.

[0130] Batch size: 64.

[0131] Optimizer: Adam, learning rate 0.0001.

[0132] Total training steps: 1,000,000 steps (steps of interaction with the environment).

[0133] (4) Training process Initialize the current network and target network Experience replay pool Empty.

[0134] Obtain the initial state from the simulation environment. .

[0135] For each time step arrive : a. According to -greedy selects action : ; Decreases with the number of steps.

[0136] b. Perform actions in the simulation environment. Obtain the next state and rewards And a termination flag (if the termination condition is met).

[0137] c. Experience Stored in the experience replay pool, with priority replay adopted, and the priority is calculated based on the absolute value of the TD error.

[0138] d. Sample a small batch (64 samples) from the experience replay pool according to priority.

[0139] e. For each sampled sample, calculate the target Q value: ; f. Calculate the loss: ; g. Gradient descent update .

[0140] h. Update the target network every 100 steps. .

[0141] i. If the termination conditions are met, reset the environment.

[0142] Save the trained network parameters.

[0143] (5) Fine-tuning in real environment The decision network trained in the simulation environment is deployed to a host computer for online fine-tuning. During fine-tuning, probabilities are explored. The network parameters are fixed at 0.05, and are updated only using real interaction data. The experience replay pool is cleared and real samples are collected again, with the network updated every 100 steps. Fine-tuning continues for a week, and after collecting enough real interaction data, the final model is solidified.

[0144] 4. Training Results and Validation Multimodal fusion encoder + emotion classifier: On the test set, the emotion classification accuracy reached 92.3%, and the AUC for safety risk detection was 0.976. The log-variance output of the uncertainty estimation module was positively correlated with the classification error, verifying its reliability.

[0145] The appeasement strategy decision network, tested over 1000 episodes in a simulation environment, showed an average cumulative reward 18% higher than the traditional DQN. After deployment in a real system, the appeasement success rate reached 89%, significantly outperforming the 65% of a no-decision system.

[0146] In step S6, the soothing actuator is driven to perform the corresponding soothing operation according to the soothing action parameters, specifically: 1. Control command generation and transmission (1) Analysis of motion parameters The soothing action parameters output from step S5 This corresponds to four soothing actions performed using the customer's baby monitor (including the camera and secondary screen): The secondary screen displays soothing static images (such as warm cartoon patterns or soft color gradients). White noise is played through the built-in speaker; Play lullabies through the built-in speaker; The secondary screen plays soothing animations (such as slowly drifting clouds or twinkling stars) while simultaneously playing a lullaby; The host computer generates corresponding control commands based on the action parameters. The commands are in JSON format and are sent to the baby monitor's control service via local inter-process communication or a wireless network. The commands contain the following fields: json { "command": "Command for calming actions" "parameters": {"parameter name": "parameter value"} } Let's take the display of a calming static image on the secondary screen as an example for a detailed explanation: like Then, a command is sent to the secondary display module of the caregiver: json { "command": "display_image", "parameters": {"image_id": "soothing_01", "duration": 300, "brightness": 0.6} } in To store identifiers for pre-stored reassurance images, To display the duration (in seconds). Sets the screen brightness (range 0~1). Images are pre-stored in the caregiver's local storage; you can choose from soft-toned cartoon animals or natural landscapes.

[0147] 2. Reassure the enforcement agency regarding its actions. After receiving a command, the baby monitor's built-in control system parses and drives the secondary screen and speaker to perform the corresponding operation. Upon successful execution, the monitor returns a confirmation message; if it fails (e.g., due to missing files or hardware failure), it returns an error code, the central processing unit records the exception, and attempts to resend the command, with a maximum of three retries. If it still fails, the error is recorded in the system log, and a backup plan (such as playing only white noise) is implemented.

[0148] 3. Infant and toddler response data recording (1) Response time measurement From the moment the soothing action was performed Initially, the system continuously captures video streams through the caregiver's camera, and analyzes the infant's emotional state in real time through steps S2 to S4 (outputting the emotion category every 0.5 seconds). Response time is defined. This is the time interval from the execution of the action to a change in the infant's emotional state (e.g., from crying to calm). If within the preset maximum waiting time... If the emotion category changes to calm or pleasant within seconds, then record it. This is the actual time; if the timeout period does not change, then... .

[0149] (2) Emotional categories after response After the action is executed Record the emotional categories of infants and toddlers at the second mark. (Output from step S4), serving as the emotion category for the next moment, and also as the response result of this reassurance.

[0150] (3) Changes in safety risks Record the safety risk coefficient before and after the action is executed. and To assess whether the soothing actions affected safety (e.g., excessive brightness on the secondary screen may cause discomfort, but the risk is generally low; this is still recorded here for analysis).

[0151] (4) Response data storage The response data is organized into a structure and stored in a local database. Each record contains: Timestamp: The moment the action was executed; Action parameters ; Response time ; Pre-response emotion categories and confidence level ; Post-response emotion categories and confidence level ; Safety risk coefficient before response ; Post-response security risk coefficient .

[0152] 4. Instant Rewards System Develop instant rewards based on the infant's or toddler's response. This is used for reinforcement learning training. The reward function remains consistent with previous methods because post-response emotion and risk are the core metrics and are not affected by the implementing agency. The reward function is defined as: ; in After the action is executed The emotion category at any given second. If multiple conditions are met simultaneously, the minimum value (worst-case scenario) is taken.

[0153] 5. Feedback Sample Storage and Network Updates The current comprehensive state vector (Constructed by step S5), the action to be performed Instant rewards The next time step integrated state vector Together with termination markers (whether the activity has ended, such as when the infant is picked up or when a caregiver intervenes), they form an experience sample: ; The sample is stored in the experience replay pool. For use in network updates.

[0154] Every fixed number of steps (e.g., every 10 steps), a mini-batch of samples is sampled from the experience replay pool, and the parameters of the appeasement strategy decision network are updated according to the loss function. Simultaneously, the target network parameters are updated every... Synchronize once.

[0155] 6. Anomaly Handling and Security Mechanisms To prevent misoperation by the comforting agency or discomfort to infants and young children, the following safety mechanisms are in place: Before performing any action, check the infant's current safety risk level. ,like If the alarm is triggered first and the caregiver is notified, the automatic soothing action will not be performed.

[0156] The brightness and volume of the secondary screen display and video playback are preset with safe upper limits (brightness ≤ 0.8, volume ≤ 0.6) to prevent excessive stimulation.

[0157] If three consecutive soothing actions fail to improve the mood (resulting in a negative reward), the system will pause automatic soothing and send a notification to the caregiver.

[0158] The camera continuously monitors the infant's response to soothing. If any abnormal expressions appear (such as fear or discomfort), the current soothing action is immediately stopped and the default settings are restored (such as turning off the screen or muting the sound).

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A method for infant and toddler care that integrates visual perception and soothing strategies, characterized in that: include: Step S1: Collect video stream data and environmental sound signals from the infant and toddler area; Step S2: After preprocessing the video stream data, extract the human pose key point sequence, facial expression feature vector and optical flow feature vector; after preprocessing the audio signal, extract the acoustic feature vector. Step S3: Input the human pose key point sequence, facial expression feature vector, optical flow feature vector and acoustic feature vector into the multimodal fusion encoder to generate the fusion state tensor; Step S4: Input the fused state tensor into a graph convolution-based emotion classifier to obtain the infant's current emotion category and its confidence level. At the same time, input the fused state tensor into a safety risk assessment network to obtain the safety risk coefficient. Step S5: Construct a comprehensive state vector based on emotion category, confidence level, safety risk coefficient, and fusion state tensor. Input this vector into a soothing strategy decision network based on memory enhancement and Dueling architecture, and output soothing action parameters. Step S6: Drive the soothing actuator to perform the corresponding soothing operation according to the soothing action parameters.

2. The infant care method integrating visual perception and soothing strategies according to claim 1, characterized in that, In step S3, the multimodal fusion encoder includes: The intramodal temporal coding module is used to add positional encoding to the feature vectors of each modality and then input them into the temporal Transformer encoder to extract intramodal temporal features. The cross-modal interaction fusion module is used to calculate the interaction features between different modalities through a cross-attention mechanism and introduce dynamic modal gating for weighted fusion. The temporal context modeling module is used to perform global context modeling on the fused feature vectors again using the temporal Transformer encoder. The global feature aggregation module is used to perform weighted aggregation of temporal dimensions through attention pooling mechanism, and then compress the result into a fused state tensor through a fully connected layer.

3. The infant care method integrating visual perception and soothing strategies according to claim 2, characterized in that, In step S4, the emotion classifier includes: The feature enhancement module is used to recalibrate the channels of the fused state tensor through the squeeze excitation module; The graph convolutional network module is used to construct an emotion category relationship graph. It updates the category node features through graph convolution and calculates the similarity between the sample features and the updated node features to obtain the emotion probability distribution with enhanced relationship. The uncertainty estimation module is used to output the log-variance in parallel, and to weight the classification loss with uncertainty during training.

4. The infant care method integrating visual perception and soothing strategies according to claim 3, characterized in that, In step S4, the security risk assessment network specifically includes: The shared feature input module is used to receive the same fusion state tensor as the emotion classifier, perform feature enhancement operations, and output enhanced features; The risk regression network consists of three fully connected layers. The first layer has a dimension of 128, the second layer has a dimension of 64, and the third layer has a dimension of 1. The ReLU activation function is used between each layer, and the last layer uses the Sigmoid activation function to output the safety risk coefficient. The multi-task learning module enables the security risk assessment network and the emotion classifier to share the parameters of the underlying feature enhancement module, which are optimized during training using a joint loss function.

5. The infant care method integrating visual perception and soothing strategies according to claim 4, characterized in that, In step S5, the comprehensive state vector is constructed as follows: The current emotion category is encoded using a one-hot encoding and then concatenated with the confidence level and safety risk coefficient to form an instantaneous state vector. Historical instantaneous states are stored through a sliding window, and the timing context vector is extracted by inputting the gated loop unit. The fusion state tensor is reduced in dimension by a linear layer to obtain deep features; The temporal context vector and deep features are concatenated into a comprehensive state vector.

6. The infant care method integrating visual perception and soothing strategies according to claim 5, characterized in that, In step S5, the appeasement strategy decision network includes: The feature extraction module is used to map the comprehensive state vector into high-dimensional features; The memory enhancement module contains a learnable memory matrix, which reads memory features through an attention mechanism and fuses them with current high-dimensional features. The Dueling Q-value calculation module is used to calculate the state value and action advantage separately, and combine them to obtain the Q-value of each action; The uncertainty perception exploration module is used to dynamically adjust the exploration probability of action selection based on the confidence level output by the emotion classifier.

7. The infant care method integrating visual perception and soothing strategies according to claim 6, characterized in that, Step S6 also includes: Record infant and toddler response data to construct instant rewards, and store experience samples in a playback pool for network parameter updates. Specifically: Record the response time and subsequent emotion category after the action is performed; Record the safety risk coefficient after the action is performed; A segmented reward function is constructed based on the emotion category after the response, the response time, and the safety risk coefficient, and the instant reward value is output.

8. An infant care system integrating visual perception and soothing strategies, applied to the infant care method integrating visual perception and soothing strategies as described in any one of claims 1-7, characterized in that, include: The visual acquisition module is used to acquire video stream data from the infant and toddler area; The audio acquisition module is used to acquire ambient sound signals; The feature extraction module is used to extract human pose key point sequences, facial expression feature vectors and optical flow feature vectors from video stream data, and to extract acoustic feature vectors from audio signals. A multimodal fusion encoder is used to fuse human pose keypoint sequences, facial expression feature vectors, optical flow feature vectors, and acoustic feature vectors into a fusion state tensor. An emotion classifier is used to output the emotion category and its confidence level of infants and toddlers based on the fusion state tensor. A security risk assessment network is used to output security risk coefficients based on the fusion state tensor. The integrated state construction unit is used to combine emotion category, confidence level, safety risk coefficient and multimodal fusion tensor into an integrated state vector; A soothing strategy decision network is used to output soothing action parameters based on the comprehensive state vector. The appeasement execution mechanism is used to carry out physical appeasement operations based on appeasement action parameters; The feedback recording unit is used to record infant response data and build instant rewards, which are then transmitted to the strategy decision network for parameter updates.

Images