A multi-modal information fusion decision method for infant care

Abstract

This invention discloses a multimodal information fusion decision-making method for infant care. The method includes: synchronously collecting and monitoring environmental images, infant posture images, sound, and foot pedal force data through various modules deployed in the perception layer of a stroller, and globally timestamping the sensor data stream; preprocessing the multimodal data, and performing environmental perception, infant state perception, and voice intent processing through deep learning and signal processing algorithms; constructing a three-layer fusion architecture with adaptive weighted fusion, three-level hierarchical decision-making, and priority coordination rules, and deeply fusing the outputs of the environmental perception, infant state perception, and voice intent modules to output a unified control command; after the decision layer generates the control command vector, the actuator performs physical actions according to the command and returns to the perception and monitoring stage to form a closed-loop monitoring cycle. This invention achieves deep fusion of multi-source information, collaborative control in complex scenarios, and proactive safety protection.

Description

Technical Field

[0001] This invention belongs to the field of automatic control and artificial intelligence technology, and in particular relates to a multimodal information fusion decision-making method for infant and toddler care. Background Technology

[0002] Currently, while the global smart stroller market boasts numerous products, the intelligence level of most remains at a rudimentary stage, limited to basic electric propulsion and Bluetooth remote control, lacking proactive safety protection capabilities based on computer vision. Among publicly available technical solutions, although some literature mentions utilizing visual monitoring or voice interaction to achieve assistive functions, these solutions are mostly simple portings of general algorithms, failing to adapt and optimize for the unique dynamic scenarios of strollers, exhibiting the following key shortcomings.

[0003] First, the perception dimension is limited and the accuracy is low. Existing technologies mainly rely on sensors such as infrared and ultrasonic sensors to obtain obstacle distance information, but cannot identify object categories, thus failing to achieve risk classification and strategic obstacle avoidance. Second, there is a lack of proactive perception of the infant's state. Existing methods cannot perform non-contact monitoring and analysis of the infant's real-time state inside the stroller, resulting in significant blind spots in safety protection. Especially when the stroller is in motion, random swaying of the stroller causes key point positions to jump, and general attitude estimation algorithms easily misinterpret passive swaying as an attempt to stand; the failure rate of key point detection is high when the infant's upper limbs are obstructed; in addition, normal behavior and attempts to stand are highly similar in geometric features, making it difficult for existing methods to effectively distinguish them, resulting in a high false alarm rate. Third, the functional modules are independent and lack a collaborative decision-making mechanism. Existing technologies treat environmental obstacle avoidance, infant state monitoring, and voice interaction as three independent functions, developed separately. There is no information sharing or collaboration between the modules, making it impossible to handle complex scenarios such as "an infant attempting to stand while in motion." When environmental hazards and changes in the infant's state occur simultaneously, the system lacks a clear priority coordination strategy, resulting in the inability to provide unified control commands when functional conflicts occur. Finally, the human-computer interaction method is mechanical and passive. Existing voice interaction methods mostly adopt a simple command-response model, lacking the ability to proactively interact based on context and infant state.

[0004] In summary, existing technologies have significant shortcomings in environmental perception, infant status monitoring, and human-computer interaction, and lack effective information sharing and coordination mechanisms among the three, making it difficult to make unified control decisions when facing complex scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal information fusion decision-making method for infant care, which solves the problems in the prior art where environmental perception, infant status monitoring and voice interaction are independent, lack fusion decision-making and priority coordination mechanisms, and are affected by shaking interference, occlusion loss and behavior confusion in infant posture recognition. It achieves deep fusion of multi-source information, collaborative control in complex scenarios and active safety protection.

[0006] The technical solution adopted in this invention is a multimodal information fusion decision-making method for infant and toddler care, comprising the following steps: Step S1: By deploying various modules in the perception layer of the stroller, environmental images, baby posture images, sound, and foot pedal force data are collected and monitored synchronously, and the data streams of each sensor are globally timestamped to provide raw input for subsequent algorithm processing; Step S2: Preprocess the collected multimodal data, and perform environmental perception module processing, infant state perception module processing, and speech intent module processing based on the deep learning and signal processing algorithms deployed in the decision layer; Step S3: Construct a three-layer fusion architecture that includes an adaptive weighted fusion model, a three-level hierarchical decision-making system, and priority coordination rules. Deeply fuse the outputs of the environmental perception, infant state perception, and voice intent modules to output unified control commands. Step S4: After the decision-making layer generates the control command vector, it enters the actuator action and closed-loop feedback stage. The actuator performs physical actions according to the control command and returns to the initial perception and monitoring stage to form a closed-loop monitoring cycle.

[0007] Furthermore, in step S2, The environmental perception module processing includes: based on the YOLO algorithm, performing model lightweighting, feature enhancement and anti-interference processing, trajectory prediction and active obstacle avoidance processing; The infant state perception module processing includes: anti-shaking pose estimation based on spatiotemporal graph convolution and adaptive adjacency matrix based on the spatiotemporal graph convolutional network ST-GCN algorithm; occlusion key point compensation based on heatmap guidance and Kalman filtering; and behavioral intention recognition based on multimodal feature fusion. The speech intent module processing includes: in the audio preprocessing stage, pre-emphasizing the acquired audio signal, dividing the pre-emphasized signal into short-time frames, and windowing each frame using a Hamming window; using a dual-threshold endpoint detection algorithm based on short-time energy and short-time zero-crossing rate to filter out silent segments and retain valid speech segments; in the feature extraction stage, extracting Mel-frequency cepstral coefficients (MFCCs) as acoustic representations, extracting MFCC features from valid speech frames and concatenating them into a temporal feature vector; in the model inference stage, analyzing the crying intent using a lightweight CNN speaker model, and recognizing speech commands using a natural language understanding model; and in the post-processing and output stage, outputting the crying intent analysis results and speech command recognition results.

[0008] Furthermore, in the environmental perception module processing, The model lightweighting process specifically includes: compressing the model using channel-level structured pruning techniques based on batch normalization layer scaling factors; constructing a sparse training loss function by introducing an L1 regularization term; and implementing differentiated pruning strategies according to the characteristics of different network layers to fine-tune the pruned model to restore model accuracy. The feature enhancement and anti-interference processing specifically includes: adding a 4x downsampling layer, composed of upsampling of shallow feature maps and fusion of deep feature maps, to the standard YOLO downsampling detection layer to achieve fine-grained feature extraction; using a bidirectional feature pyramid network (BiFPN) for multi-scale feature fusion; removing single-input nodes, adding cross-scale connections, and employing a weighted fusion method in the BiFPN; generating attention maps through channel attention and spatial attention mechanisms, and then adding the channel attention map and spatial attention map element-wise and obtaining the total attention map through a sigmoid activation function; The trajectory prediction and active obstacle avoidance processing specifically includes: introducing a Kalman filter and setting a discrete time step. Control of time Define the state vector. ;in, For planar position, For speed, Let T be the acceleration and T be the transpose; define the observation vector. Includes only the target's position components A uniform acceleration motion model is adopted, and a state transition matrix and an observation matrix are constructed based on the state vector and the observation vector. The trajectory of the dynamic target is predicted through prediction and update iteration, and deceleration or attitude adjustment commands are triggered in advance according to the prediction results to achieve active obstacle avoidance.

[0009] Furthermore, in the model lightweighting process, the sparse training loss function is:

[0010] in, This represents the total loss function value during the sparse training phase. Input data; For the corresponding real tags; These are the weight parameters to be trained in the network model; This represents the loss during normal network training. For sparsification parameters; Indicates traversing the network The set of scaling factors corresponding to all channels in the dataset; This is the L1 regularization penalty function applied to the scaling factor; For scaling factor The applied L1 regularization sparsity penalty term; In the feature enhancement and anti-interference processing, the output of the weighted fusion method is:

[0011] in, The output value after fusion; , The first , The fusion weight coefficients corresponding to each input feature map; For the first One input feature map; It is the minimum value; The formula for calculating channel attention is:

[0012] in, Channel attention map; Input feature map; For batch normalization operation; To Perform global average pooling; , The weights and biases of the fully connected layer in the first layer of a multilayer perceptron (MLP); The weights and biases of the fully connected layer 2 of the MLP; The formula for calculating spatial attention is:

[0013] in, Spatial attention map; For activation functions; This represents a 7×7 convolution operation. Indicates a splicing operation; Indicates to Perform max pooling at the channel dimension.

[0014] Furthermore, in the infant state perception module processing, The anti-shaking pose estimation based on spatiotemporal graph convolution and adaptive adjacency matrix specifically includes: continuously acquiring multiple frames of images, extracting human key points to construct a spatiotemporal graph structure, wherein the edges of the spatiotemporal graph include spatial edges and temporal edges; constructing and defining an adjacency matrix based on the spatiotemporal graph structure, and introducing a learnable adaptive adjacency matrix, and obtaining the output feature tensor through graph convolution operation; The occlusion key point compensation based on heatmap guidance and Kalman filtering specifically includes: determining the degree of occlusion by calculating the confidence level of the heatmap; implementing three-level graded compensation based on the confidence level of the heatmap; when it is determined to be partial occlusion, the compensation mechanism is activated, geometric derivation is performed in combination with human kinematic constraints to determine the position of the candidate area of ​​the wrist, and the Kalman filter is introduced to predict the motion trajectory of the occluded wrist. The final key point position is obtained by weighted fusion of the geometrically compensated position and the Kalman predicted position. The behavior intent recognition based on multimodal feature fusion: constructing multimodal feature vectors. ;in, Based on the key geometric features, the torso tilt angle, the distance between the wrist and the fence, the eye closure ratio, and the angles of each joint were extracted. To identify dense optical flow features, the Farneback algorithm is used to calculate the optical flow of adjacent frames and statistically analyze the motion amplitude histogram. For depth features, monocular depth estimation is used to obtain depth values ​​for the torso center, wrists, ankles, and fences; For temporal features, an LSTM network is used to capture the behavioral temporal patterns of the first 10 frames; based on the multimodal feature vectors, a discrete state space is constructed including sleep, crying, normal sitting posture, attempting to stand, and calm states. The state transition determination is completed through a fully connected neural network classifier.

[0015] Furthermore, in the anti-shake attitude estimation based on spatiotemporal graph convolution and adaptive adjacency matrix, the graph convolution operation is as follows:

[0016] in, , These are the output and input feature tensors, respectively. For the convolution branch index used for multi-scale feature extraction; This is an element-wise dot product operation; To sum the outputs of multiple convolutional branches; The convolutional kernel weight matrix is ​​a learnable matrix. For predefined adjacency matrices; for adaptive adjacency matrices During the training phase, the shaking noise pattern is actively learned by superimposing synthetic jitter, and it remains unchanged during the inference phase without superimposing synthetic jitter. The confidence level of the heat map The calculation formula is:

[0017] in, Indicates the first The response intensity of key points at their locations in the image; the operation is used to extract the maximum response value from the heatmap; The three-level compensation is as follows: when When there is no obstruction, the peak value of the heatmap is used directly; when When it is considered partial occlusion, the compensation mechanism is activated; when If the location is considered severely obstructed, then the position predicted by the Kalman filter is used directly. When partial occlusion is determined, human kinematic constraints are considered, based on forearm length. elbow joint angle elbow joint position Shoulder joint position and upper arm length Through rotation matrix Derivation of candidate wrist positions: The position of the wrist relative to the elbow is... Upper arm vector Forearm vector The candidate area for the wrist is... ; In the Kalman filter prediction and weighted fusion... The state vector of the Kalman filter is Includes current position coordinates and the corresponding speed of motion The observation vector is T stands for transpose; The final formula for calculating the key point location is:

[0018] Among them, the geometric compensation position is derived based on human kinematic constraints; The location is predicted based on the Kalman filter algorithm; This refers to the final key point location after the weighted fusion of the two. In the behavior intent recognition based on multimodal feature fusion, the fully connected neural network classifier sequentially includes: an input layer, a first fully connected layer, a Dropout regularization layer, a second fully connected layer, and an output layer; wherein, the first and second fully connected layers are both activated by the ReLU activation function; the dropout rate of the Dropout regularization layer is 0.3; and the output layer is activated by the Softmax activation function to output the normalized probability of each state.

[0019] Furthermore, in the voice intent module processing, In the audio preprocessing stage, the pre-emphasis operation is implemented using a first-order difference equation, and the calculation formula is as follows: ;in, This is the pre-emphasized signal sequence; The original signal sequence; The previous sampling point of the original audio signal; the pre-emphasis coefficient is set to 0.97; Hamming windowing is used to window each frame of the signal, and the calculation formula is as follows: ;in, A sequence of Hamming window functions; This is the index of the sampling point within the current frame; The frame length; the Hamming window function With the original signal frame Perform point-by-point multiplication to obtain the windowed signal sequence. ; The short-time energy is defined as: To distinguish the energy difference between unvoiced and voiced sounds and background noise; define the short-time zero-crossing rate. : To help distinguish between unvoiced sounds and noise; among them, The square of the windowed signal sequence; It is a symbolic function; This refers to the previous sampling point in the windowed signal sequence; Mel frequency The calculation formula is: Next, calculate the logarithmic energy of each filter bank output; finally, perform discrete cosine transform on the logarithmic energy to obtain the MFCC coefficients of the preset dimension; concatenate the feature vectors of the current frame with the feature vectors of the five frames before and after it to form a temporal feature vector as the model input; if the current frame is located at the beginning or end of a speech segment, then the boundary frame is processed by mirror filling. In the model inference stage, regarding the analysis of crying intent, the CNN voiceprint model sequentially includes: three identical one-dimensional convolutional layers, each followed by a batch normalization layer, a ReLU activation function, and a max pooling layer; then a flattening layer, a Dropout layer, a first fully connected layer, and a second fully connected layer; the second fully connected layer is the output layer, applying a Softmax activation function to output the probability distribution of three types of intents: hunger, drowsiness, and discomfort; wherein, the one-dimensional convolutional layer has a kernel size of 3, a stride of 1, and padding of 1; the max pooling layer has a kernel size of 2 and a stride of 2; the dropout rate of the Dropout layer is 0.3; and the first fully connected layer applies a ReLU activation function. In terms of voice command recognition, the natural language understanding model converts speech into text based on a pre-trained Chinese speech recognition engine, and then maps the text into structured commands through a lightweight intent classifier. In the post-processing and output stage, regarding the analysis of crying intent, the model output results are as follows: ;in, The representative model predicted the first Specific categories of the intention behind a baby's cry; The model indicates that the sample belongs to an intent. The probability value; The index variable is used; the intent with the highest confidence level exceeding 0.6 is selected as the final output; if no intent exceeds this threshold, it is determined that there is no valid crying intent; in terms of voice command recognition, the model output is... .

[0020] Furthermore, step S3 specifically includes: An adaptive weighted fusion mechanism is designed, with weights dynamically adjusted according to the real-time scenario. Based on the stroller's real-time speed feedback, the baby's status recognition results, and the voice command detection results, the current scenario type is dynamically determined and the corresponding weight switching is triggered. The weight switching is smoothed by using a first-order low-pass filter. Introducing hazard levels Its calculation formula is ;in, , , , These are the weighting coefficients. The risk coefficient for the target category is customized and calibrated based on the characteristics of the stroller scenario. For the target distance, For relative velocity, To predict collision probability; hazard level The calculation is triggered synchronously with the camera frame rate, and the state update of the Kalman filter is synchronized with the target detection to output the latest predicted collision probability. ;set up The corresponding degradation response will only be executed after the value remains unchanged or decreases for 3 consecutive frames, and the tracking record of the target will be cleared and no longer included in subsequent calculations when the same target is not detected for 10 consecutive frames. Based on the calculated risk level The value is to classify environmental response into three levels, and to formulate three core integrated decision-making rules based on the hierarchical decision-making. After the above multi-layered decision-making logic, the fusion decision module finally outputs a unified seven-dimensional control command vector, the expression of which is: ; where, instruction vector Directly drives the underlying actuators; Indicates the motor speed; Indicates the steering angle; Represents the backrest angle; Indicates speaker volume; Indicates the content to be played; Indicates system mode; Indicates the current task priority; Based on adaptive weighted fusion, a cross-modal attention mechanism is introduced; a multi-head cross-modal attention mechanism is adopted to realize the interaction between visual features and speech features. This mechanism includes three stages: modal feature extraction, cross-modal attention interaction, and multi-modal fusion output, in order to generate a modal fusion feature vector.

[0021] Furthermore, the weight switching formula is as follows: ;in, For the current moment The actual weight vector used; This is the weight vector corresponding to the target scene; For smoothing coefficients; The cutoff frequency; To control the cycle; The environmental response is divided into three levels: when R ≥ 0.7, it is judged as Level 1 danger; when 0.4 ≤ R < 0.7, it is judged as Level 2 warning; when 0.2 ≤ R < 0.4, it is judged as Level 3 alert; when R < 0.2, it is judged as safe. The three core fusion decision rules are as follows: The first rule is safety priority. When the danger level reaches level one (R≥0.7), the system will unconditionally perform an emergency stop and suspend all other tasks. The second rule is state priority. In the infant state assessment, the priority order is as follows: attempting to stand, crying, sleeping, and normal. Higher priority states interrupt the execution of tasks in lower priority states. The third rule is composite scenario coordination. When environmental danger and changes in infant state coexist, the system will make decisions according to a preset coordination matrix: Under level one danger, the system will unconditionally stop regardless of the infant's state; under level two danger, if the infant attempts to stand, the system will prioritize infant safety and then obstacle avoidance; if the infant cries, the system will handle deceleration and soothing in parallel; under level three danger, if the infant is sleeping, the system will decelerate and adjust the backrest to provide white noise. In the modal feature extraction stage, features are extracted from the visual environment, the visual infant, and the speech data respectively: environmental visual features are extracted using the backbone network of the improved YOLO; infant visual features are extracted based on the spatiotemporal feature layer of the improved ST-GCN; speech features are extracted using a lightweight CNN; the environmental visual features and the infant visual features are concatenated to obtain joint visual features. In the cross-modal attention interaction phase, a multi-head cross-modal attention mechanism is employed, using speech features... As the query vector Q, combined with visual features As key vector K and value vector V, the association weights between speech and vision are calculated: through a linear transformation matrix. , The original features are mapped to the attention space, then the attention score is calculated and normalized. The attention function expression is as follows: ;in, For attention function; It is a normalized exponential function; Key vector The transpose of the matrix; This is the scaling factor; An 8-head attention mechanism is employed, with each attention head computing independently. The 8 attention heads utilize a multi-head division of labor: heads 1 and 2 are responsible for extracting cross-modal correlation features between the type of cry evoked by speech and facial expressions; heads 3 and 4 are responsible for extracting dynamic correlation features between cry intensity and upper limb movement amplitude; heads 5 and 6 are responsible for extracting semantic alignment features between speech commands and environmental targets; and heads 7 and 8 are responsible for extracting temporal correlation features between cry sequence and posture change trends. The outputs of the 8 attention heads are concatenated along the feature dimension and then processed by an output projection matrix. The transformation formula is: ;in, For multi-head attention computation function; This is a tensor splicing operation; For attention; The multimodal fusion output stage will convert the original speech features It is residually connected to the output of the attention layer and then passed through... Normalization yields fusion features The calculation formula is: The fused features are input into a two-layer fully connected network for dimensionality reduction to generate a modality fusion feature vector.

[0022] Furthermore, step S4 specifically includes: The actuators include wheel-side motors, backrest motors, and speakers, which work together according to control commands: the wheel-side motors execute speed and steering commands, the backrest motors execute backrest angle commands, and the speakers execute volume and playback content commands; in different operating scenarios, during emergency braking, the wheel-side motors decelerate and the speakers sound an alarm; during deceleration and calming, the motors decelerate and the speakers play calming voice messages; during deceleration with white noise, the motors slightly decelerate, the speakers play white noise, and the backrest motors slowly level the backrest. Both the wheel-side motor and the backrest motor use PID controllers for closed-loop control. The PID output expression is: ;in, Output control quantity to the controller; Indicates the current time The error value is the expected value. Compared with actual feedback value The error between; For a historic moment The error value; , , These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. During the turning process, the backrest is linked according to the turning angle and the baby's state: when the baby is awake, the backrest maintains the current angle; when the baby is asleep and the turning angle is greater than 10 degrees, if the backrest angle is greater than 30 degrees, the angle remains unchanged; if the backrest angle is less than 15 degrees, the backrest is slightly adjusted 5 degrees in the opposite direction of the turn; after the turn is straightened, the backrest smoothly returns to the angle before the turn at a speed of 5 degrees every 5 seconds. The end-to-end total response time is controlled within the preset time. If any link times out, the decision of the previous cycle is used. If the detection fails for 3 consecutive cycles, the security protection is triggered. The safe recovery system employs a dual protection mechanism of hardware watchdog and software monitoring: the hardware watchdog cuts off the enable pin of the wheel-side motor driver and triggers the brakes if it does not receive a watchdog signal within a preset time; the software monitoring uses a single frame image processing cycle as a monitoring cycle, and forces deceleration to a stop if the detection fails for 3 consecutive cycles; the wheel-side motor driver has built-in overcurrent protection and overtemperature protection. After the actuator completes its action, it returns to the initial sensing and monitoring phase to form a closed-loop monitoring cycle, and uses the execution result of the previous cycle as the PID feedback input for the next cycle.

[0023] The beneficial effects of this invention are: 1. This invention constructs a multi-source information fusion decision-making framework, deeply fusing the outputs of three sub-modules: environmental perception, infant state monitoring, and voice interaction. It also introduces a cross-modal attention mechanism to achieve deep interaction and alignment of visual and speech features, solving the problem of independent operation and lack of collaboration among functional modules in existing technologies. Based on this, priority rules and a collaborative decision-making mechanism for complex scenarios are established, enabling unified control command output when multi-source information conflicts occur. The accuracy of multi-modal fusion is significantly improved compared to conventional fusion methods in existing technologies.

[0024] 2. This invention represents a leap from "passive obstacle avoidance" to "active cognitive protection." Through improved YOLO and ST-GCN algorithms, it effectively overcomes technical challenges such as dynamic swaying interference during stroller movement, partial upper limb occlusion, and confusion regarding behavioral intent. Specifically, the adaptive adjacency matrix and occlusion compensation mechanism significantly improve the accuracy of pose estimation and the success rate of wrist keypoint detection, while the multimodal feature fusion state machine greatly reduces the false alarm rate of attempts to stand. The fast end-to-end response speed significantly enhances the safety of infant and toddler care.

[0025] 3. This invention, through cry intent analysis and an active interaction mechanism, achieves intelligent recognition and proactive soothing or reminders for three states of an infant: hunger, drowsiness, and discomfort. It also supports multi-turn dialogue, completing an emotional upgrade of companionship from "passive response" to "proactive care." Simultaneously, combined with an intelligent following control method based on Bluetooth signal strength, it achieves stable synchronous start / stop and turning, significantly reducing the physical burden on caregivers.

[0026] 4. This invention tightly couples perception, decision-making, interaction and execution by integrating a decision-making center, enabling it to handle complex scenarios such as an infant attempting to stand while in motion or an infant encountering a bumpy road while asleep. It automatically performs collaborative operations such as braking, deceleration, backrest leveling, and playing white noise. Through deep integration and collaborative decision-making in complex scenarios, the system generates significant collaborative gains, fully demonstrating the overall innovation and practical value of the method. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the overall workflow of the present invention.

[0029] Figure 2 This is a flowchart of the multimodal intelligent human-computer interaction technology of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] like Figures 1-2 As shown, this embodiment provides a multimodal information fusion decision-making method for infant and toddler care; to implement this method, This invention constructs a four-layer integrated architecture of "perception-decision-interaction-execution": the perception layer integrates environmental image, infant posture image, sound and physical signal acquisition modules, and simultaneously acquires environmental image, infant posture, sound and foot pedal force data; the decision layer receives multimodal data, runs environmental perception, infant state perception, voice intent understanding and multi-source information fusion algorithms, and outputs unified control commands; the interaction layer realizes voice feedback and command reception; the execution layer drives the wheel-side motion mechanism and the backrest posture adjustment mechanism.

[0032] In this embodiment, the perception layer includes a first camera (located at the front of the stroller, dedicated to capturing images of the surrounding environment), a second camera (located on the sunshade of the stroller, dedicated to capturing images of the baby's posture), and a microphone array (located on the stroller frame, used for pinpoint sound pickup, capturing ambient sound, and voice commands); physical signal acquisition is achieved based on pressure sensors on the foot pedals of the caregiver's standing area located at the bottom of the stroller; the decision layer uses a Raspberry Pi as its core, deploying improved YOLO, ST-GCN algorithms, and a fusion model, respectively used to process image data and fuse multi-source information; the interaction layer enables voice interaction through a speaker; the execution layer uses an Arduino as a controller to drive the motor to complete corresponding actions; the above hardware is only an exemplary carrier.

[0033] Based on the above system, this embodiment of a multimodal information fusion decision-making method for infant and toddler care includes the following steps: Step S1: Multimodal data acquisition and monitoring: This step continuously monitors the environment and the baby's state through various acquisition modules in the perception layer, and simultaneously acquires multimodal data to provide raw input for subsequent algorithm processing. The first camera, as the environmental image acquisition module, is installed at the front of the stroller to capture images of the surrounding environment. Its resolution is set to 640×480, and the frame rate is no less than 25 FPS to ensure clear capture of environmental targets. The second camera, as the baby posture image acquisition module, is installed inside the stroller's sunshade to capture images of the baby's posture. Its parameters are consistent with the first camera, namely a resolution of 640×480 and a frame rate of no less than 25 FPS, to accurately reproduce the details of the baby's movements. The microphone array... The sound acquisition module is fixed in a pre-set mounting position on the stroller frame crossbeam. It is used to collect ambient sound and voice commands, with a sampling rate of 16kHz and a quantization bit depth of 16bit. This configuration balances the richness of ambient sound and the recognizability of voice commands. The pressure sensor, as a foot pedal force data acquisition module, is integrated into the foot pedal of the caregiver's standing area at the bottom of the stroller. It is used to sense the caregiver's standing status and weight pressure in real time, providing a status trigger signal for the intelligent following and synchronous start-stop driven by the wheel-side motor. To ensure the accuracy of subsequent multi-source information fusion, the system adopts a global timestamp marking mechanism to perform logical-level time synchronization of the data streams collected by each sensor.

[0034] Step S2: Parallel algorithm processing: This step first performs necessary preprocessing on the collected multimodal data, such as image denoising, audio denoising, and normalization. Then, it performs targeted processing through deep learning and signal processing algorithms deployed in the decision layer. Specifically, it is divided into three sub-processes: environmental perception module processing, infant state perception module processing, and voice intent module processing. The three are executed in parallel to improve efficiency.

[0035] S21: Environmental perception module processing: To address the specific characteristics of scenarios involving strollers traveling at speeds ≤5km / h, low field of vision, and the need to identify small obstacles, the decision-making layer implemented four customized configurations for the YOLO algorithm to achieve efficient and accurate environmental perception.

[0036] (1) Lightweight model: To address the performance bottleneck of edge computing devices (such as Raspberry Pi) with limited computing power in the stroller scenario when processing complex models, a channel-level structured pruning technique based on batch normalization (BN) layer scaling factor is adopted to compress the model. The process is as follows: Scaling factor of BN layer The output scale of each channel can be directly adjusted, and its size reflects the adaptive importance of the corresponding channel in the network training process. The smaller the weight, the lower the contribution of the channel to subsequent tasks; to force the weights of unimportant channels to approach zero, during the training phase... The L1 regularization term is introduced into the loss function to construct a sparse training loss function:

[0037] in, This represents the total loss function value during the sparse training phase. Input data; For the corresponding real tags; These are the weight parameters in the network model that need to be automatically learned through training, and their specific values ​​are dynamically determined during the training process. This represents the loss during normal network training. The sparsity parameter is set to a value of in this embodiment. , used to adjust the sparsification intensity; Indicates traversing the network The set of scaling factors corresponding to all channels in the dataset; This is the L1 regularization penalty function applied to the scaling factor; For scaling factor The applied L1 regularization sparsity penalty term.

[0038] After sparse training is complete, read the scaling factor of the BN layer. and remove Redundant channels (threshold); simultaneously, considering the differentiated needs between layers in the stroller scenario, differentiated pruning strategies are implemented according to the characteristics of different network layers: shallow feature extraction layers retain more channels with a pruning rate of 20%-30% to ensure basic feature extraction capabilities; mid-layer feature fusion layers have information redundancy and can be significantly pruned with a pruning rate of 40%-50%; the detection head layer is sensitive to accuracy and is only slightly pruned with a pruning rate of 10%-20%; finally, the global pruning rate is controlled at 40%-60% to achieve a simplified model structure; pruning will lead to a loss of model accuracy, so the pruned model needs to be fine-tuned during training: in this embodiment, 20-30 epochs are used for training; the initial learning rate is set to 0.001; the learning rate strategy adopts a cosine annealing decay strategy, that is, the learning rate decreases with the epochs in a cosine function to smoothly transition to the optimal solution and restore the detection accuracy lost due to pruning.

[0039] The above-mentioned sparse training, pruning and fine-tuning process can be iterated repeatedly, that is, sparse training, pruning and fine-tuning are performed in a loop. Through multiple rounds of progressive optimization, the final compression effect is gradually improved.

[0040] (2) Feature enhancement and anti-interference: To further address complex interferences in the driving environment, such as pedestrian occlusion, lighting changes, and cluttered backgrounds, this solution introduces multi-scale feature fusion and attention mechanisms to improve model robustness.

[0041] To address the issue of missed detection of small targets due to low field of view, a 4x downsampling layer with fine-grained feature extraction capability is added to the standard YOLO's 8x, 16x, and 32x downsampling detection layers. This layer is obtained by fusing shallow feature maps with deep feature maps, resulting in an output feature map size that is 1 / 4 of the original image. Taking a 640×640 image as an example, the feature map size of this layer is 160×160, specifically designed for detecting small targets smaller than 20×20 pixels, such as baby bottles on the ground or small pets in the distance. This is because large-scale shallow feature maps contain a wealth of detailed information, which is crucial for the classification and localization of small targets. By fusing shallow and deep feature maps, the detection layer can simultaneously obtain shallow spatial location information and deep semantic information, thus becoming more sensitive to even smaller targets.

[0042] In terms of network architecture design, a Bidirectional Feature Pyramid Network (BiFPN) is used for multi-scale feature fusion to replace the traditional FPN. BiFPN removes nodes with only one input edge and adds cross-scale connections, i.e., skip connections between input and output nodes at the same scale. It fuses more features without increasing computational cost, and a weighted fusion method is used to allow the network to learn the importance of each input feature. Specifically, a fast normalized fusion method is used, the expression of which is:

[0043] In this embodiment, the weighting coefficients in the above formula The importance of the input feature map, upsampled feature map, and downsampled feature map, respectively, is... , , To ensure the non-negativity of the fusion weights, the fusion output is calculated... Previously, ReLU activation functions were applied to each weight to constrain it to satisfy... Among them, each fusion weight is a learnable parameter, with an initial value of 1, and is automatically adjusted through gradient backpropagation during training. For the first One input feature map, The value is set to a minimum to prevent numerical instability. Compared to the fusion method based on Softmax, the method of this invention avoids the additional Softmax computation overhead and has higher inference efficiency.

[0044] In terms of attention mechanisms, channel attention allows the model to focus on what target it is paying attention to; the channel attention module generates channel vectors by performing global average pooling on the feature maps. ;in, The vector represents the number of channels; it performs soft encoding on the global information of each channel, i.e., compressing the spatial dimension while preserving the global features of the channel dimension; subsequently, channel attention is estimated from the channel vector using a multilayer perceptron (MLP) with hidden layers; to balance computational efficiency and expressive power, a compression ratio is introduced. To reduce the amount of parameter computation, the formula for calculating channel attention is:

[0045] in, Channel attention map; Input feature map; To Perform global average pooling; , The weights and biases of the fully connected layer 1 of the MLP; The weights and biases of the fully connected layers in the second layer of the MLP; This is a batch normalization operation.

[0046] Spatial attention focuses the model on specific locations, allowing the network to learn which spatial locations in the feature map are more critical. The spatial attention module performs max pooling and average pooling on the feature map (both with 1 channel), concatenates the results, and then generates a spatial attention map using a 7×7 convolutional kernel. Finally, it outputs spatial weights through a sigmoid activation function. The formula for calculating spatial attention is:

[0047] in, Spatial attention map; This represents a 7×7 convolution operation. Indicates a splicing operation; Indicates to Perform max pooling at the channel dimension.

[0048] The total attention map is an element-wise sum of channel attention and spatial attention, and gradient flow is promoted through residual learning. The calculation formula is as follows:

[0049] in, For activation functions; Channel attention map; Spatial attention map; , Before summing the outputs, adjust their dimensions to the same size; this improved CBAM module promotes gradient propagation through additive attention fusion, enabling the model to focus on dangerous targets such as pedestrians and vehicles and suppress irrelevant background interference.

[0050] (3) Trajectory prediction and active obstacle avoidance: Based on the above configuration that ensures accurate vision and clear identification, a Kalman filter is introduced to predict the trajectory of dynamic targets such as pedestrians and vehicles, realizing a leap from passive response to active prediction. The Kalman filter is a recursive optimal estimation algorithm for estimating the state of a linear dynamic system. It is suitable for processing noisy sensor data such as visual inspection and can predict future positions and suppress noise interference in target tracking.

[0051] In this embodiment, since the stroller itself does not exert any active driving force, the system has no external control input, i.e., discrete time step. Control of time The state vector is defined as follows: , including planar position ,speed and acceleration ; Observation vector Includes only the target's position components The uniformly accelerated motion model is adopted. The discretized system model includes a state equation describing the recursive relationship of the target state and a measurement equation describing the relationship between sensor observations and the target state, as shown in the following formulas: Equations of state:

[0052] Measurement equation:

[0053] in, for The state vector at any given time; for The observation vector at time; for The control variables of the system at all times; This is the state transition matrix; For the control matrix, since In actual calculations, the control matrix has no effect; For measurement matrix; The process noise is in the form of Gaussian white noise, and its covariance is... This describes the uncertainty of the target's motion; The measurement noise is in the form of Gaussian white noise, with a covariance of This describes the sensor detection error.

[0054] Combining uniform acceleration model and sampling interval (0.1 seconds in this embodiment), state transition matrix for:

[0055] This embodiment only observes the position. Observation matrix for:

[0056] By predicting and iterating, accurate trajectory prediction of the future position of a dynamic target is achieved, and deceleration or attitude adjustment commands are triggered in advance based on the prediction results to achieve active obstacle avoidance. The five core steps of Kalman filtering are as follows: First, the posterior state estimate from the previous time step is used. and state transition matrix Perform state prediction to obtain the prior state estimate at the current time. At the same time, the error covariance matrix of the previous time step is used. and Predicting the current prior error covariance And superimposed process noise covariance The uncertainty in the prediction is quantified; then the Kalman gain is calculated. By balancing prior error covariance Observation matrix and observation noise covariance To determine the weight ratio between predicted and observed values; then, the observation vector at the current time is used. and Kalman gain By correcting the prior state estimate, we obtain the posterior state estimate at the current time step. That is, the optimal position / velocity / acceleration results after fusing prediction and measurement; finally, based on the Kalman gain... Update the error covariance matrix to the posterior error covariance. This reflects the accuracy of the current estimate and provides a basis for the next iteration; among which, the process noise covariance Observation noise covariance Based on actual system debugging, the prediction time domain is set to three time points: 0.5 seconds, 1.0 seconds, and 1.5 seconds, corresponding to mild deceleration warning, moderate deceleration detour planning, and emergency braking trigger, respectively. By predicting the movement trend of dangerous targets in advance, active obstacle avoidance is achieved. Among them, the detour command changes the center of gravity of the vehicle body by adjusting the posture of the stroller back or by linking the steering mechanism to achieve trajectory deviation, ensuring that dangerous targets are in the safe avoidance zone.

[0057] The above configurations, layer by layer and deeply synergistic, jointly construct a high-performance environmental perception system adapted to stroller scenarios: channel pruning technology breaks through the computing power bottleneck, providing real-time assurance for understanding complex scenes; fine-grained feature extraction and multi-scale feature fusion compensate for the defects of missing small targets and improve the completeness of perception; hybrid attention mechanism enhances the ability to extract key features in complex backgrounds; Kalman filtering gives the system the ability to predict trajectory forward, realizing the leap from passive response to active cognition; the deep integration of the above configurations has enabled the YOLO algorithm to evolve from a general object detector into a high-efficiency environmental perception engine customized for stroller scenarios.

[0058] S22: Infant state awareness module processing: To address the three major challenges in the dynamic driving scenarios of strollers—swaying interference, upper limb occlusion, and behavior confusion—the decision layer implements three systematic configurations for the Spatiotemporal Graph Convolutional Network (ST-GCN) algorithm to improve the accuracy of pose estimation and behavior recognition.

[0059] (1) Anti-shaking attitude estimation based on spatiotemporal graph convolution and adaptive adjacency matrix: First, to address the issue of attitude misjudgment caused by random swaying between 0.5 and 20 Hz during driving, the system continuously acquires 5 to 7 frames of images, corresponding to durations of 0.2 to 0.28 seconds, and extracts nine key human body points, including shoulders, elbows, and wrists, to construct a spatiotemporal graph structure. Among them, nodes Represents key nodes in each frame, with one key node per frame; edges It includes two types of connections: spatial edges Connecting different key points within the same frame, such as from the left shoulder to the left elbow and then to the left wrist, reflects the physical connections of the human skeleton; time edge Connecting the same keypoint in adjacent frames, denoted as ;in, Indicates the first The index in the frame is The key point, The frame number, The keypoint index reflects the trajectory of keypoint changes over time; based on the above connection relationships, a predefined adjacency matrix is ​​constructed. Its elements Represents a node With nodes There are connecting edges. This indicates no connection, allowing features to propagate along the skeleton edges in the spatial dimension and along the frame sequence in the temporal dimension, thus achieving synchronous feature extraction in both the spatiotemporal dimensions.

[0060] To enhance the model's adaptability to dynamic environments, a learnable adaptive adjacency matrix is ​​introduced. The graph convolution operation is as follows:

[0061] in, The output feature tensor has the following shape: ; The input feature tensor has the shape of ; The time frame number is 5-7 frames in this embodiment; The number of input feature channels; To determine the number of nodes, this embodiment selects 9 key points on the human body; The number of output feature channels; For the convolution branch index used for multi-scale feature extraction; This is an element-wise dot product operation; To sum the outputs of multiple convolutional branches; The learnable convolutional kernel weight matrix has the following shape: This is used to perform channel transformation on the input features; The predefined adjacency matrix is ​​a fixed matrix based on the physical connections of the human skeleton, with the following shape: ; It is an adaptive adjacency matrix with dimension . , which are learnable parameters, are initialized to an all-zero matrix in the early stages of training. They are used to gradually learn the potential connections between nodes during training, thereby achieving adaptive modeling of data dependencies.

[0062] To further address random swaying interference during stroller movement, an adaptive adjacency matrix is ​​used. During the training phase, jitter noise patterns are actively learned by superimposing and synthesizing jitter. The data augmentation strategy samples independently on each frame to simulate the random spatiotemporal variation characteristics during driving. Translation uses uniform distribution. Pixels are sampled independently in both the horizontal and vertical directions, and frame by frame; rotation is achieved using a uniform distribution. The entire image frame is rotated uniformly and each frame is rotated independently; scaling is achieved using a uniform distribution. The entire frame of the image is scaled uniformly and independently for each frame; the dithering enhancement ratio is set to 1:1, meaning all training samples are enhanced; during the inference phase, By keeping the jitter constant and not adding it to the composite jitter, the shaking noise pattern in the real scene is automatically suppressed.

[0063] (2) Occlusion key point compensation based on heatmap guidance and Kalman filtering: After initially eliminating driving sway interference and obtaining accurate posture estimation, a heatmap-guided occlusion compensation strategy is adopted to address the problem of key point loss caused by upper limbs being obscured by the basket or toys.

[0064] The strategy first calculates the confidence level of the heatmap. To determine the degree of occlusion; among them, Indicates the first A key point in the image location The response intensity at the point ranges from 0 to 1. The peak position represents the coordinates of the key point, and the peak value reflects the confidence level of the detection. The operation is used to extract the maximum response value from the heatmap; based on this, a three-level compensation is implemented: when When there is no obstruction, the peak value of the heatmap is used directly; when When it is considered partial occlusion, the compensation mechanism is activated; when If the occlusion is severe, the position predicted by the Kalman filter will be used directly. The above thresholds can be determined through offline experimental calibration based on the actual hardware imaging quality and scene lighting conditions.

[0065] In this embodiment, the thresholds of 0.7 and 0.3 are determined based on experimental verification results and can also be adaptively adjusted according to actual application needs. The specific verification process is as follows: A total of 2,000 image samples with different degrees of occlusion in the stroller scene were collected. Based on this, a correlation curve between the confidence threshold and the accuracy of key point localization was plotted, and the balance point of accuracy-recall was selected as the candidate threshold. The experimental results show that when the confidence threshold is 0.7, the localization accuracy of unoccluded samples can reach more than 95%, and the false detection rate is less than 3%. When the threshold is 0.3, the recall rate of severely occluded samples is increased to more than 65%, and the false compensation rate is controlled within 8%. After adopting this threshold setting, the overall detection success rate is significantly improved from about 60% to more than 85%.

[0066] In the specific compensation mechanism, geometric derivation is performed using human kinematic constraints to determine the candidate wrist position: based on the prior knowledge that the ratio of the human upper limb length to the elbow is approximately 1:1, and the physiological limitation that the elbow joint angle is between 60° and 120°, the position of the wrist relative to the elbow is derived; in the unobstructed state, the position of the wrist relative to the elbow can be represented by polar coordinates. ;in, Forearm length is given; once upper arm length is known, the ratio is determined at 1:1. Given the elbow joint angle; and the shoulder joint position. and elbow joint position Calculate the upper arm vector Its direction is from the shoulder to the elbow, and its length is the length of the upper arm; with a fixed elbow joint angle Then, the direction of the wrist relative to the elbow can be obtained by rotating the upper arm vector around the elbow joint; let the rotation matrix be... Rotate the upper arm to the forearm direction; the direction after rotation is the forearm orientation. From this, the candidate wrist region can be determined. ;in, , Forearm vector, The upper arm length is defined in this embodiment. This approximation is based on the average physiological characteristic that the upper and lower limb bones develop in similar proportions during infancy, and is applicable to occlusion compensation in most cases.

[0067] To further improve positioning accuracy under occlusion conditions, a Kalman filter was introduced to predict the motion trajectory of the occluded wrist; the state vector used by this filter is... Includes current position coordinates and the corresponding speed of motion The observation vector is The final position output is obtained by weighted fusion of the geometrically compensated position and the Kalman predicted position, calculated using the following formula: ;in, The geometrically compensated position is derived based on human kinematic constraints; The location is predicted based on the Kalman filter algorithm; The final key point location is obtained by weighted fusion of the two; the geometric compensation weight is set to 0.6 and the Kalman prediction weight is set to 0.4. This ratio is set based on experience to emphasize the strong prior knowledge provided by geometric constraints; through the comprehensive application of the above compensation mechanism, the success rate of wrist key point detection can be increased from about 60% to more than 85%.

[0068] (3) Complex behavioral intent recognition based on multimodal feature fusion: Based on the stable detection of posture key points, this invention further tackles the high-level semantic problem of behavioral intent confusion. Specifically, it addresses the high similarity in geometric features between normal physiological behaviors such as bending over to pick up toys and stretching and actions such as attempting to stand up. For example, the torso tilt angles of both may overlap between 25° and 40°. Traditional single geometric rules are difficult to effectively distinguish between active movement and passive swaying, and relying solely on optical flow analysis is also difficult to eliminate interference from complex spatial positions. Previously, the false alarm rate caused by this type of confusion was as high as 40%.

[0069] To overcome this challenge, this invention constructs a multimodal feature vector. It is formed by the fusion of four types of complementary information, and its expression is: Among them, the basic pose determination depends on the geometric features of key points. Its dimensions are eight-dimensional, and this feature comprehensively extracts the torso tilt angle, the distance between the wrist and the fence, the eye closure ratio, and the angles of each joint; to distinguish between active movement and passive swaying, a dense optical flow feature is introduced. Its dimensions are 16-dimensional. The Farneback algorithm is used to calculate the optical flow of adjacent frames and statistically analyze the motion amplitude histogram, dividing the frame into 16 intervals to capture dynamic information. To address spatial location interference, depth features are extracted. Its dimension is 4-dimensional. Monocular depth estimation technology is used to obtain the depth values ​​of the torso center, wrist, ankle, and fence to accurately determine the relative positional relationship with the fence. Finally, to solve the problem of temporal pattern recognition, temporal features are introduced. Its dimensions are 32, and it uses an LSTM network to capture the behavioral temporal patterns of the first 10 frames, thereby giving the model the ability to reason on a time series basis.

[0070] Based on the aforementioned multimodal feature vectors, this invention constructs a discrete state space containing five states. The set includes sleep, crying, normal sitting posture, attempting to stand, and calm. State transition determination is performed by a lightweight classifier. This classifier adopts a fully connected neural network structure with an input layer dimension of 60. The input layer is mapped to 64 dimensions through the first fully connected layer and a ReLU activation function is applied. Then, a Dropout regularization layer with a dropout rate of 0.3 is passed to prevent overfitting. Next, the 64 dimensions are reduced to 32 dimensions through the second fully connected layer, still using the ReLU activation function. Finally, the output is passed through a fully connected layer with a 5-dimensional output and a Softmax activation function is applied to output the normalized probability of each state.

[0071] In terms of training set construction, based on the publicly available infant pose dataset, approximately 5,000 real stroller scene images covering different age groups, lighting conditions, and occlusion levels were collected and manually annotated with nine key points and corresponding behavioral state labels. The data augmentation step adopted the same synthetic jitter strategy as in step S22 (1), specifically including translation. Pixels, rotation and scaling .

[0072] The model was trained using the Adam optimizer, with an initial learning rate set to [value missing]. The learning rate is reduced to 0.1 times its original value every 20 epochs, the batch size is set to 32, the maximum number of training epochs is 100, and an early stopping strategy is introduced; if the loss on the validation set does not decrease for 10 consecutive epochs, training is terminated; the cross-entropy loss function is used to ensure classification accuracy.

[0073] The three configurations mentioned above work in a progressive and synergistic manner: the first layer's adaptive adjacency matrix eliminates the interference of driving swaying on basic detection, providing stable key point input for subsequent processing; the second layer's occlusion compensation mechanism, based on the first layer, compensates for the loss of key points caused by local occlusion, improving detection completeness; the third layer's multimodal state machine, on the premise that the first two layers ensure the accuracy and completeness of key points, integrates multi-dimensional features to solve behavioral semantic confusion; together, these three elements transform ST-GCN from a general pose estimation model into a state monitoring engine specifically customized for the dynamic scenes of strollers.

[0074] S23: Voice intent module processing: This step addresses two types of input: environmental sound and voice commands. It performs cry intent analysis and voice command recognition respectively to understand user needs and the infant's state. The entire processing flow is divided into four stages: audio preprocessing, feature extraction, model inference, and post-processing and output, as detailed below: (1) Audio preprocessing stage: The original audio signal captured by the microphone has a sampling rate of 16kHz and a quantization bit depth of 16bit. It first undergoes pre-emphasis processing to compensate for the attenuation of high-frequency components during transmission, thereby enhancing high-frequency details. The pre-emphasis operation is implemented using a first-order difference equation, the formula of which is: ;in, This is the pre-emphasized signal sequence; The original signal sequence; The previous sampling point of the original audio signal is set to 0.97. Then, the pre-emphasis coefficient is set to 0.97. Subsequently, the pre-emphasized signal is divided into short frames with a frame length of 32ms, corresponding to 512 sampling points. The frame shift is set to 16ms, corresponding to 256 sampling points, to ensure 50% overlap between adjacent frames and to ensure inter-frame continuity.

[0075] To improve the spectral leakage suppression effect, a Hamming window is used to window each frame of the signal, and its expression is as follows:

[0076] in, A sequence of Hamming window functions; This is the index of the sampling point within the current frame; The frame length; the Hamming window function With the original signal frame By performing point-by-point multiplication, the windowed signal sequence can be obtained.

[0077] Next, endpoint detection is performed; this invention employs a dual-threshold endpoint detection algorithm (VAD) based on short-time energy and short-time zero-crossing rate to effectively remove silent segments and background noise segments, extracting valid speech segments; specifically, short-time energy... Defined as: It is mainly used to distinguish the energy difference between unvoiced and voiced sounds and background noise; short-time zero-crossing rate Defined as: This algorithm helps distinguish between unvoiced sounds and noise; after processing, silent segments are filtered out, and the remaining valid speech segments then proceed to the next stage; among them, The square of the windowed signal sequence; This is a sign function; it takes the value 1 when the independent variable is greater than or equal to 0, and -1 when it is less than 0. This is the previous sampling point in the windowed signal sequence.

[0078] (2) Feature extraction stage: This invention extracts Mel-frequency cepstral coefficients (MFCCs) as an acoustic representation, and extracts 40-dimensional MFCC features from each frame of signal retained after endpoint detection; the extraction process is as follows: First, perform a Fast Fourier Transform (FFT) on the windowed signal to obtain its spectral distribution. Next, the linear spectrum is mapped to the Mel frequency scale using a triangular filter bank; the conversion from linear frequency to Mel frequency is calculated using the following formula, which converts the linear frequency... Converted to Mel frequencies that conform to human hearing characteristics Then, the logarithmic energy of each filter bank's output is calculated; finally, the discrete cosine transform (DCT) is performed on the logarithmic energy to obtain the 40-dimensional MFCC coefficients, which are the MFCC feature vectors with a dimension of 40 for each frame of signal output. To capture temporal context information, the feature vectors of the current frame and the five frames before and after it (a total of 11 frames) are concatenated to form a temporal feature vector with a dimension of 40 × 11 = 440. As input to the model; if the current frame is at the beginning or end of a speech segment, the boundary frame is processed by mirroring.

[0079] (3) Model reasoning stage: The model inference stage comprises two parallel processing branches: cry intent analysis and voice command recognition. For cry intent analysis, a lightweight CNN voiceprint model is used for recognition, and its network structure is as follows: The input feature tensor has a shape equal to the number of time frames T and a feature dimension of 440; where T depends on the duration of the effective speech segment, typically between 30 and 200 frames. The first layer is a one-dimensional convolutional layer with a kernel size of 3, a stride of 1, and padding of 1, resulting in 128 output channels and an output tensor shape of T×128. Subsequently, a batch normalization layer, a ReLU activation function, and a max pooling layer are sequentially applied, with a pooling kernel size of 2 and a stride of 2, changing the output tensor shape to T / 2×128. The second layer is a one-dimensional convolutional layer with a kernel size of 3, a stride of 1, and padding of 1, resulting in 256 output channels and an output shape of T / 2×256. The next layer is applied... Batch normalization, ReLU, and max pooling are applied, resulting in an output shape of T / 4×256. The third layer is a one-dimensional convolutional layer with 512 output channels and an output shape of T / 4×512. After batch normalization, ReLU, and max pooling, the output shape becomes T / 8×512. Subsequently, a flattening layer converts the multidimensional features into a one-dimensional vector. After passing through a Dropout layer with a dropout rate of 0.3, it is connected to a first fully connected layer with 1024 neurons and a ReLU activation function is applied. Finally, a second fully connected layer with an output dimension of 3 is applied and a Softmax activation function is applied to output the probability distribution of three intentions: hunger, drowsiness, and discomfort.

[0080] In this embodiment, the model has approximately 83,000 parameters, an inference latency of less than 20ms, and is tested on a Raspberry Pi 4B platform. The model is custom-trained to analyze the frequency domain characteristics of infant cries, including key indicators such as fundamental frequency and harmonic distribution. The training data includes an open-source infant cry dataset and self-collected data, with a total sample size of approximately 5,000 records, covering different age groups (0-12 months), various crying causes (hunger, drowsiness, discomfort), and different environmental noise levels, with a signal-to-noise ratio ranging from -5dB to 20dB. The training configuration uses the Adam optimizer, with a learning rate set to... The batch size was set to 32, the maximum number of training rounds was set to 50, and an early stopping strategy was adopted.

[0081] In terms of voice command recognition, a natural language understanding model is adopted to support the recognition of commands such as play or stop, increase or decrease volume, switch songs, follow or pause. The model is based on a pre-trained Chinese speech recognition engine, such as Whisper-tiny or a self-trained model, to convert speech into text, and then uses a lightweight intent classifier to map the text into structured commands.

[0082] (4) Post-processing and output stage: The specific workflow for the post-processing and output stages is as follows: Regarding the analysis of crying intent, the model output is as follows:

[0083] in, The representative model predicted the first Specific categories of the intention behind a baby's cry; The model indicates that the sample belongs to an intent. The probability value; This is an index variable used to iterate through all possible intent categories; the intent with the highest confidence level exceeding 0.6 will be selected as the final output; if no intent exceeds this threshold, it will be determined that there is no valid crying intent.

[0084] In terms of voice command recognition, the model output is as follows: For example, when it is recognized that the user wants to play a certain song, the output format is: .

[0085] The specific linkage rules between the above recognition results and the aforementioned environmental perception module and infant state perception module are as follows: When the crying intent analysis module identifies hunger with a confidence level greater than 0.6, the system triggers a hunger reminder signal. Upon receiving this signal, the fusion decision module reminds the caregiver to feed the infant via a speaker. If drowsiness is identified with a confidence level greater than 0.6, a soothing request signal is triggered, and the system automatically plays a lullaby and adjusts the backrest. If discomfort is identified with a confidence level greater than 0.6, a discomfort reminder signal is triggered, reminding the caregiver to check the infant's condition via a speaker. For voice command recognition results, when control commands such as play / stop are identified with a confidence level greater than 0.7, the system triggers a command execution signal, directly driving the corresponding actuator to perform the action. The specific execution logic of this linkage rule will be explained in detail in the subsequent step S3, fusion decision.

[0086] Step S3: Multi-source information fusion decision: This step aims to deeply integrate the outputs of the three sub-modules—environmental perception, infant state perception, and voice intent—to output a unified control command. Addressing the complex driving scenarios of strollers, this invention designs a three-layer fusion architecture that includes an adaptive weighted fusion model, a three-level hierarchical decision-making system, and priority coordination rules.

[0087] The three-layer mechanism has a close collaborative relationship: the first layer, the adaptive weighting mechanism, dynamically adjusts weights based on scenarios such as driving, parking, and interaction to ensure the correctness of the decision focus in different scenarios; the second layer, the hazard level calculation and three-level classification mechanism, quantifies multi-source information into a unified R value, enabling differentiated responses and avoiding overreaction or underreaction; the third layer, the priority rules and collaborative matrix, makes decisions according to the priority matrix when multiple tasks conflict, ensuring that critical tasks are not interrupted. The three layers work together to upgrade the fusion decision-making process from a fixed weighted summation to an intelligent decision-making system with scenario adaptation, hierarchical response, and priority coordination. This effectively handles complex scenarios such as an infant attempting to stand while driving or encountering bumpy roads while sleeping, realizing the transformation from multi-source heterogeneous data to precise control commands.

[0088] S31: Overall Design of Converged Architecture Traditional fixed-weight fusion strategies cannot adapt to dynamically changing scenario requirements. For example, when driving at high speed, the weight of environmental perception should be significantly increased to ensure safety, while in a parking monitoring state, the weight of infant state perception should dominate. To solve this problem, this invention configures an adaptive weighted fusion mechanism, in which the weights are dynamically adjusted according to the real-time scenario. The specific scenario classification and weight configuration are shown in Table 1.

[0089] Table 1 Scene Classification and Adaptive Weight Configuration

[0090] The system of this invention dynamically determines the current scene type based on real-time speed feedback from the wheel-side motors, infant status recognition results, and voice command detection results, and triggers corresponding weight switching. The specific switching mechanism is as follows: the system uses the detection of valid voice command input as the highest criterion. Once this condition is met, regardless of the current driving state, it immediately triggers a switch to interactive mode, which has the highest priority to ensure timely execution of caregiver commands. In the absence of voice command interaction, the system classifies the scene based on the speed feedback from the wheel-side motors, with a speed threshold of 0.8 m / s serving as the boundary between full-speed driving and low-speed cruising. The threshold is set based on typical stroller travel scenarios: below 0.8m / s corresponds to low-speed pushing or cruising in congested areas or complex road surfaces, and above 0.8m / s corresponds to normal driving on open roads; full-speed driving is set as the second priority, triggered when the wheel-side motor speed is greater than 0.8m / s and there is no voice command interaction; low-speed cruising is set as the third priority, triggered when the wheel-side motor speed is between 0 and 0.8m / s and there is no voice command interaction; parking monitoring is set as the fourth priority, triggered when the wheel-side motor is completely still for more than 2 seconds to avoid frequent switching due to temporary parking.

[0091] To avoid abrupt changes in control commands due to sudden weight shifts, which could affect the infant's riding comfort, a first-order low-pass filter is used for smooth transition during weight switching. The calculation formula is as follows: ;in, For the current moment The actual weight vector used; The weight vectors corresponding to the target scene are shown in Table 1. This is the smoothing coefficient; the cutoff frequency is here. Set to 0.5Hz; control cycle Set to 0.1s; the smoothing coefficient was calculated. The weight will take approximately 63% of the target value to transition from the current value. It takes approximately seconds to transition to 95%. This transition speed ensures both smooth switching and avoids being too sluggish in responding to changes in the infant's state.

[0092] S32: Hazard Level Calculation: To implement differentiated response strategies, the decision-making body introduced the concept of hazard levels; hazard levels The calculation is based on four dimensions: target category, distance, relative speed, and predicted collision probability. The formula is as follows:

[0093] in, , , , These are weighting coefficients, with specific values ​​of 0.35, 0.25, 0.25, and 0.15. Target category hazard coefficient; target distance The value range is set from 0.5 m to 20 m. This parameter follows the principle that the closer the distance, the greater the danger, and its reciprocal is used in the calculation; relative velocity The range is -5 m / s to 5 m / s, with a positive value indicating that the target is approaching; predicting the collision probability. The value ranges from 0 to 1, and this data is provided by the output of the Kalman filter algorithm.

[0094] To solve the problem of variables in the formula , , , The issue of inconsistent dimensions can be addressed by specifically designing the value ranges to ensure that the contributions of each variable are within the same order of magnitude. Therefore, no additional normalization is required, and the variables can be directly substituted into the formula for calculation. Among these, the target category hazard coefficient... The coefficient is dimensionless and has been compressed to the range of 0.20–0.95 by expert calibration; distance from the reciprocal term. With distance The value decreases from 20 meters to 0.5 meters, and its value monotonically increases from 0.05 to 2.0; relative speed The value range is limited to -5 to 5 meters per second; predicted collision probability The output of the Kalman filter algorithm is naturally limited to the range of 0 to 1. The contribution of each of the above variables is uniformly scaled by its corresponding weight coefficient, without the need for additional normalization.

[0095] Hazard level The calculation is triggered synchronously with the camera frame rate; since the camera acquires images at a rate of no less than 25 frames per second, the system immediately triggers the hazard level calculation after each frame completes YOLO target detection, ensuring that the perception results are promptly converted into decision instructions; the calculation cycle is approximately once per frame, or about 40 milliseconds, and the hazard level of the current frame is output in real time. The Kalman filter's state update is synchronized with target detection, also updating the target trajectory state 25 times per second, and outputting the latest predicted collision probability. .

[0096] Based on this, to address the decision jitter that may be caused by false detections in a single frame, the system is set to... The corresponding degradation response is only executed after the value remains unchanged or decreases for 3 consecutive frames. At the same time, in order to avoid frequent interruptions caused by the temporary disappearance of the target, when the same target is not detected for 10 consecutive frames, the system clears the tracking record of the target and no longer includes it in the subsequent calculation. The above synchronization design ensures the timeliness and consistency of data between the perception layer and the decision layer, so that the hazard level calculation can reflect environmental changes in real time and provide reliable input for subsequent classification decision.

[0097] Regarding the target category hazard coefficient in the above formula This invention incorporates customized calibrations tailored to the specific characteristics of stroller scenarios, with the detailed values ​​shown in Table 2. Table 2 Target Category Hazard Coefficient Table

[0098] S33: Three-level hierarchical decision-making: Based on the calculated hazard level R value, the system divides the environmental response into three levels to achieve refined control; the threshold and deceleration settings are based on braking experiments calibrated in a low-speed stroller scenario: deceleration 2.5 m / s². 2 For comfortable braking, the upper limit is 1.0 m / s. 2 For comfortable deceleration, the upper limit is 0.3 m / s. 2 For micro-adjustment.

[0099] When R ≥ 0.7, the system determines it to be a Level 1 hazard and executes emergency braking with a deceleration of 2.5 m / s².2 Simultaneously, a high-decibel alarm is issued. This level corresponds to scenarios where the infant's safety is directly threatened, such as rapidly approaching large vehicles, strangers, or stairwells; the braking distance is approximately 0.2 m at an initial speed of 1 m / s and approximately 0.8 m at an initial speed of 2 m / s.

[0100] When 0.4 ≤ R < 0.7, the system issues a Level 2 warning and performs a comfortable deceleration of 1.0 m / s². 2 Simultaneously, the system sounds its horn to warn the user and attempts to detour. This level corresponds to potential threats such as pedestrians or animals within 1 meter of the target. The braking distance is approximately 0.5 m at an initial speed of 1 m / s and approximately 2.0 m at an initial speed of 2 m / s. In the Level 2 warning response, the system calculates an avoidance path based on the target's location while decelerating. The avoidance maneuver is subject to several restrictions to ensure safety and comfort: lateral acceleration does not exceed 0.5 m / s². 2 The maximum steering angle is 30 degrees, and the steering angular velocity does not exceed 30° / s. Steering and avoidance are only performed when the target distance is not less than 1.5 m and the lateral space is not less than 0.5 m. If the target distance is less than 1.5 m or the lateral space is insufficient, the vehicle will only decelerate and stop without steering. If deceleration and stopping cannot avoid a collision during this process, the vehicle will automatically escalate to emergency braking. The avoidance direction is preferentially chosen to be the side away from the center line of the road. If there is sufficient space on both sides, the vehicle will preferentially avoid to the left to facilitate observation by the accompanying personnel, and the avoidance direction will be indicated through the loudspeaker. When there are multiple targets at the same time, the vehicle will preferentially avoid the target with the closest distance and the highest R value. If the R values ​​of multiple targets are close, the vehicle will choose to pass in the middle direction. If there is insufficient clearance, the vehicle will decelerate and wait.

[0101] When 0.2 ≤ R < 0.4, the system issues a Level 3 warning and performs a slight deceleration of 0.3 m / s². 2 Alternatively, it may only provide voice prompts, corresponding to warning scenarios involving ordinary obstacles or uneven road conditions.

[0102] When R < 0.2, the system determines that it is safe and drives normally without applying brakes.

[0103] S34: Fusion Decision Rules Based on hierarchical decision-making, in order to ensure the continuity of critical tasks when there are conflicts of multi-source information, this invention formulates three core fusion decision-making rules.

[0104] The first rule is safety first; when the danger level reaches level one, i.e., R ≥ 0.7, the system will unconditionally perform an emergency stop and suspend all other tasks. This rule is designed based on the principle that life safety is paramount.

[0105] The second rule is state priority; in infant state assessment, the priority order is as follows: attempting to stand, crying, sleeping, and normal. Higher priority states can interrupt the execution of tasks in lower priority states; for example, if an attempt to stand is detected while playing a lullaby, the system will immediately stop the music and issue a voice warning.

[0106] The third rule is collaborative operation in complex scenarios. When environmental hazards and changes in the infant's state coexist, the system makes decisions according to a preset collaborative matrix: under Level 1 hazard, the system will stop unconditionally regardless of the infant's state; under Level 2 hazard, if the infant tries to stand up, the system will prioritize infant safety and then obstacle avoidance; if the infant cries, the system will handle deceleration and soothing in parallel; under Level 3 hazard, if the infant is asleep, the system will decelerate and adjust the backrest to provide white noise.

[0107] When a high-priority infant attempts to stand simultaneously with an environmental hazard, the system follows an arbitration procedure, with the principle that life safety is the highest priority, taking precedence over all other infant states. The highest priority corresponds to Level 1 hazard with an R ≥ 0.7, in which case unconditional emergency braking is required, with a deceleration of 2.5 m / s². 2 The braking process takes approximately 0.4 to 0.8 seconds. During this time, the infant's condition is not addressed to avoid startling them simultaneously with the sudden braking and voice alarm. Once the vehicle has come to a complete stop, if the infant is still attempting to stand, then a voice warning and backrest adjustment will be implemented. For the second highest priority scenario (R value between 0.4 and R < 0.7), which corresponds to a secondary hazard where the infant is attempting to stand, a comfortable deceleration of 1.0 m / s² is initiated. 2 The braking time is about 1 to 2 seconds. Ensure the vehicle is stationary before handling the baby to avoid aggravating the baby's posture imbalance due to sudden braking. The normal priority corresponds to environmental safety, i.e., R < 0.2 plus the baby trying to stand up. At this time, the baby's status should be handled immediately, and voice warnings and backrest adjustments should be performed.

[0108] Rule 2 does not conflict with the setting of a high infant status weight of 0.6 in the previous parking monitoring state. The two belong to different decision-making levels and together form a complete decision-making chain. The first layer is the weight allocation layer, which determines which type of information the system pays more attention to. For example, an infant status weight of 0.6 in the parking monitoring scenario means that the system will allocate more attention to the infant status perception module. The second layer is the priority arbitration layer, which determines which task to perform first when multiple tasks need to be executed at the same time. When a high-priority infant status occurs, the system will respond to the high-priority task first, regardless of the current weight allocation. Taking the infant attempting to stand up in the parking monitoring scenario as an example, the infant status weight is 0.6, which allows the system to quickly capture the status. Then, under the guidance of Rule 2, it immediately interrupts all current tasks and prioritizes the execution of voice warnings and deceleration braking.

[0109] S35: Output unified control commands: After the above multi-layered decision-making logic, the fusion decision module finally outputs a unified seven-dimensional control command vector, the expression of which is:

[0110] Instruction vector The specific definitions and control ranges of each parameter for directly driving the underlying actuator are explained below: in, The motor speed command is in floating-point format, with a range of -2.5 to 2.5 m / s. Negative values ​​indicate backward movement. This command is sent to the motor driver via serial communication in the form of a 16-bit signed integer, with a resolution of 0.001 m / s and a communication cycle of 20 ms. In practical applications, this command has a linear mapping relationship with the PWM duty cycle of the wheel-side motor. Specifically, a duty cycle of 50% corresponds to a stop state of 0 m / s, a duty cycle of 75% corresponds to a maximum forward speed of 2.5 m / s, and a duty cycle of 25% corresponds to a maximum reverse speed of -2.5 m / s.

[0111] in, The steering angle command is in floating-point format, with a value range of -30° to 30°. A negative value indicates a left turn. The command is sent to the steering servo driver via serial communication in the form of a 16-bit signed integer, with a resolution of 0.1 degrees and a communication cycle of 20ms. In practical applications, the command is linearly mapped to the PWM duty cycle of the steering servo. Specifically, a duty cycle of 7.5% corresponds to a maximum left turn angle of -30 degrees, a duty cycle of 10% corresponds to a straight-ahead state of 0 degrees, and a duty cycle of 12.5% ​​corresponds to a maximum right turn angle of 30 degrees.

[0112] in, This represents the backrest angle. The backrest angle command uses a floating-point format, with a value range of 0 to 45 degrees, corresponding to a 0-degree sitting posture and a 45-degree reclining posture, respectively. This command is sent to the backrest motor driver via serial communication in 16-bit unsigned integer form, with a resolution of 0.5 degrees and a communication cycle of 50ms. In practical applications, this command has a linear mapping relationship with the PWM duty cycle of the backrest motor. Specifically, a 5% duty cycle corresponds to a 0-degree sitting posture (holding torque), and a 15% duty cycle corresponds to a 45-degree reclining posture (maximum driving torque). Intermediate angles are calculated using linear interpolation. The mapping formula is: D represents the PWM duty cycle. The current backrest angle command value (in degrees) is used, and closed-loop feedback is achieved through the motor encoder to ensure angle positioning accuracy.

[0113] in, This command indicates the speaker volume. The speaker volume command uses a floating-point format with a value range of 0 to 100 and supports linear adjustment. This command is sent to the audio module via serial communication in the form of an 8-bit unsigned integer with a resolution of 0.5. In practical applications, this command has a logarithmic mapping relationship with the gain coefficient of the audio module to conform to the characteristics of human hearing. Specifically, a value of 0 corresponds to the mute state, and a value of 100 corresponds to the maximum volume of 0dB.

[0114] in, This command specifies the content to be played. The content to be played command is an enumeration type, and the specific values ​​include 0 for no playback, 1 for an alarm sound (triggered during emergency braking), 2 for a lullaby (triggered when soothing the baby), 3 for a voice prompt (outputted via speech synthesis, such as "Baby, please sit still" or "There is an obstacle in front of you"), and 4 for white noise (triggered when helping the baby sleep). This command is sent in the form of an 8-bit unsigned integer, and a single content to be played or playback can be paused in each frame.

[0115] in, This indicates the system mode. The system mode command is an enumeration type, with specific values ​​including: 0 for manual mode (fully controlled by the caregiver), 1 for automatic mode (determined by the fusion decision module), 2 for follow mode (determined by Bluetooth signal strength calculation), and 3 for monitoring mode (prioritizing responses to changes in the infant's state). This command is sent in 8-bit unsigned integer form. In manual mode... , The decision is determined by remote control input; in automatic mode, it is determined by the fusion decision module; in follow mode, it is determined by Bluetooth signal strength calculation; and in monitoring mode, it prioritizes responding to changes in the baby's state.

[0116] in, This indicates the current task priority. The task priority instruction is an 8-bit unsigned integer, ranging from 0 to 3, and is mainly used for task interruption and resumption. 0 represents the highest priority, corresponding to a level one emergency braking task; 1 represents a high priority, corresponding to a task where an infant attempts to stand; 2 represents a medium priority, corresponding to a level two warning deceleration task; and 3 represents the lowest priority, corresponding to a normal driving task. During execution, the actuator can determine whether it is permissible to be interrupted by a higher-priority task based on the priority flag.

[0117] S36: Cross-modal attention fusion mechanism: To further enhance the deep interaction capabilities of visual and speech features in complex scenarios, this invention introduces a cross-modal attention mechanism based on adaptive weighted fusion to address issues such as independent feature extraction, temporal asynchrony, and noise contamination inherent in simple decision-level weighted summation. This invention employs a multi-head cross-modal attention mechanism to achieve deep interaction between visual and speech features, which consists of three stages: The first stage is modal feature extraction: feature extraction is performed on the visual environment, visual infant, and speech data respectively. Environmental visual features Infant visual features were extracted using a modified YOLO backbone network with 512 dimensions. Spatiotemporal feature extraction based on an improved ST-GCN layer, with a dimension of 256; speech features Visual features were extracted using a lightweight CNN with 128 dimensions; the environmental visual features were then concatenated with the infant's visual features to obtain joint visual features. It has 768 dimensions.

[0118] The second stage is cross-modal attention interaction: a multi-head cross-modal attention mechanism is adopted, based on speech features. As a query vector (denoted as Q), combined with visual features As key vectors (K) and value vectors (V), the association weights between speech and vision are calculated; first, the original features are mapped to the attention space through three independent linear transformation matrices: ;in, , , All are learnable matrices; The dimension of the query and key vector; Let be the dimension of the value vector; then, calculate the attention score and normalize it: ;in, For attention function; It is a normalized exponential function; Key vector The transpose of the matrix; The scaling factor is used; an 8-head attention mechanism is employed, with each attention head computed independently. The output dimension of each head is The eight attention heads employ a multi-head division of labor mechanism, specifically configured as follows: Heads 1 and 2 are responsible for extracting cross-modal correlation features between the type of crying evoked by speech and facial expressions; Heads 3 and 4 are responsible for extracting dynamic correlation features between crying intensity and upper limb movement amplitude; Heads 5 and 6 are responsible for extracting semantic alignment features between speech commands and environmental targets; and Heads 7 and 8 are responsible for extracting temporal correlation features between crying sequence and posture change trends. The outputs of the eight attention heads are concatenated along the feature dimension and then projected onto an output matrix. Transformation:

[0119] in, For multi-head attention computation function; This is a tensor splicing operation; the spliced ​​dimension is 8 × 64 = 512. , , with speech features The dimensions must be kept consistent to meet the dimension matching requirements of residual connections.

[0120] The third stage is multimodal fusion output: combining the original speech features The output of the attention layer is residually connected and then normalized using LayerNorm. Subsequently, the fused features are input into a two-layer fully connected network for dimensionality reduction, with the dimensionality changing from 256 dimensions to 128 dimensions and then to 64 dimensions, ultimately generating a compact multimodal fused feature vector. It has 64 dimensions.

[0121] The cross-modal attention fusion mechanism and the adaptive weighted fusion in step S31 are in a sequential and reinforcing relationship: the adaptive weighted fusion in step S31 acts on the decision layer, dynamically adjusting the weight ratio of each module according to the scene type; the cross-modal attention fusion mechanism of this invention acts on the feature layer, processing speech features before S31. With visual features Perform deep interaction and alignment to generate fused features. ; Subsequently, the original three independent features are replaced as one of the inputs for the weighted fusion of the decision layer in step S31. When the weighted sum is performed, it is regarded as a whole and together with other information such as the infant state features, it constitutes a complete decision input. The two complement each other and improve the multimodal fusion decision link.

[0122] The cross-modal attention module requires end-to-end joint training. The training dataset uses a self-collected multimodal dataset of stroller scenes, containing approximately 10,000 sets of synchronously acquired front camera images, top camera images, and microphone audio triplet sets. Each set of data is labeled with environmental object category, infant behavior state, and voice intent category. The loss function is a multi-task joint loss. ;in, For target detection loss (CIoU loss); The loss is classified as the infant state (cross-entropy loss). The loss is used for speech intent classification (cross-entropy loss). , , These are the weighting coefficients in the multi-task joint loss function, with values ​​of 0.5, 0.3, and 0.2 respectively.

[0123] The training strategy involves first training each single-modal model separately until convergence, then loading pre-trained weights for end-to-end joint training, with a learning rate set to [value missing]. The training rounds were 30, and the batch size was 16. Considering the limited computing power of edge devices such as Raspberry Pi, the following optimizations were made during deployment: 8-bit integer quantization was performed on the trained model, which improved the inference speed by about 2 times and kept the accuracy loss within 1%; sparse attention mechanism was used to retain only the top 50% of the attention scores for calculation, reducing the amount of computation by about half; the NEON instruction set of Raspberry Pi 4B was used to accelerate matrix operations; the measured end-to-end inference latency increased by about 15~20ms, but the overall latency could still be controlled within 150ms, meeting the real-time requirements.

[0124] Step S4: Actuator Action and Closed-Loop Feedback: After the decision-making level generates the control command vector, it enters the stage of actuator action and closed-loop feedback. The actuator performs physical actions according to the control commands and returns to the initial perception and monitoring stage to form a closed-loop monitoring cycle, so as to ensure that all commands are accurately implemented and maintain the continuous and stable operation of the system. This stage integrates the action execution logic of the actuator and the closed-loop feedback mechanism of the system.

[0125] S41: Actions and Coordination of Actuators: The actuators perform corresponding physical actions based on the received control commands, achieving a high degree of coordination. Specifically, the wheel-side motors execute speed and steering angle commands with a response time of less than 20 milliseconds, providing speed and steering control during execution. The backrest motor executes backrest angle commands with a response time of less than 50 milliseconds and forms a linkage mechanism with the wheel-side motors. The speaker executes volume and playback content commands with a response time of less than 10 milliseconds, ensuring high synchronization between motor deceleration and audible warnings through multi-threaded concurrency mechanisms or hardware cascading triggering. The coordination logic of each mechanism differs under different operating scenarios: in emergency braking scenarios, the wheel-side motors execute deceleration commands while the speaker simultaneously emits an alarm sound; in deceleration and soothing scenarios, the motor decelerates while the speaker plays a preset soothing voice message; and in deceleration combined with white noise scenarios, the motor performs a slight deceleration action, the speaker plays white noise, and the backrest motor performs a slow leveling action to jointly create a comfortable riding environment.

[0126] S42: PID closed-loop control parameter configuration: To ensure execution accuracy, both the wheel-side motor and the backrest motor employ a PID proportional-integral-derivative controller to achieve closed-loop speed and position control; the output expression of the PID controller is: ;in, Output control quantity to the controller; Indicates the current time The error value is the expected value. Compared with actual feedback value The error between them, i.e. ; For a historic moment The error value; , , These are the proportional coefficient, integral coefficient, and derivative coefficient, respectively. The specific configurations of these parameters are as follows: Both the speed loop and steering angle loop of the wheel-side motors use PID control, with a control cycle set to 20 milliseconds. The proportional coefficient of the speed loop is set to 1.50, the integral coefficient to 0.20, and the derivative coefficient to 0.05; the proportional coefficient of the steering angle loop is set to 2.00, the integral coefficient to 0.10, and the derivative coefficient to 0.08; the position loop control cycle of the backrest motor is 50 milliseconds, and the PID coefficients are set to 0.80, 0.05, and 0.02. Through these parameter configurations, the system ensures that the speed loop overshoot is controlled within 5%, and the steady-state error is within ±0.05 meters per second; the position loop overshoot is controlled within 3 degrees, and the steady-state error is within ±1 degree.

[0127] S43: Steering linkage logic: During the turning process, to ensure the stability and comfort of the infant, the system executes linkage logic based on the turning angle and the infant's current state. When the infant is awake, the backrest does not make any additional adjustments during the turning process, maintaining the current angle to avoid disturbing the infant; when the infant is asleep, if the turning angle is greater than 10 degrees, the backrest linkage mechanism is triggered; if the infant is in a lying position, i.e., the backrest angle is greater than 30 degrees, the backrest angle remains unchanged, as the lying position already provides sufficient lateral support; if the infant is in a sitting position, i.e., the backrest angle is less than 15 degrees, the backrest is slightly adjusted 5 degrees in the opposite direction of the turn (e.g., to the rear and outward) to increase lateral support and help the infant resist centrifugal force during the turn; after the turning angle returns to the center, the backrest automatically returns to the angle before the turn, with a recovery speed of 5 degrees every 5 seconds. The recovery process is smooth and completes full recovery within the set time.

[0128] S44: Time Budgeting and Timeout Handling To ensure the total end-to-end response time is kept within 150 milliseconds, the system precisely allocates time budgets for each functional module: image acquisition and transmission no more than 15 milliseconds, YOLO object detection inference no more than 30 milliseconds, ST-GCN pose estimation and speech intent recognition no more than 20 milliseconds each, cross-modal attention fusion no more than 15 milliseconds, fusion decision calculation no more than 10 milliseconds, instruction issuance and execution no more than 20 milliseconds, and other system overhead reserved no more than 20 milliseconds. When a certain step times out, the system will use the valid decision result from the previous cycle. If detection fails for three consecutive cycles, a safety protection mechanism is triggered. This safety protection mechanism is a software-layer perceptual decision failure handling mechanism, which is different from the system-level reset protection mechanism of the hardware watchdog in S45.

[0129] S45: Security Recovery Mechanism The safety recovery mechanism employs a dual-protection architecture of hardware watchdog and software monitoring. The system monitors the main control chip's watchdog signal via an independent hardware timer. If no watchdog signal is received for more than 2000 milliseconds, the watchdog timer overflows, triggering a hardware reset signal that directly cuts off the enable pin of the wheel-side motor driver. The wheel-side motor enters a free-glide stop state, preventing uncontrolled acceleration. Simultaneously, an emergency brake relay, independent of the main control chip, is triggered and closes, driving the braking device to bring the stroller to a stop. This relay is directly driven by the watchdog reset signal and does not rely on software execution, ensuring safety even if the main control chip crashes or the software malfunctions. Even in the event of a runaway vehicle, it can still achieve a safe stop. In addition, the wheel-side motor driver has built-in overcurrent and overtemperature protection. The current threshold is set to 5 amps, and protection is triggered if the current exceeds 1 second. The temperature threshold is set to 85 degrees Celsius, and protection is triggered if the temperature exceeds 5 seconds. When triggered, the output power is automatically reduced to 50%. At the software monitoring level, the system uses a single frame image processing cycle as a monitoring cycle, which lasts about 40 milliseconds. If three consecutive monitoring cycles (approximately 120 milliseconds in total) fail to detect, i.e., there is no output for target detection or the attitude estimation confidence is less than 0.3, the system will force deceleration to a stop regardless of the current danger level.

[0130] S46: Closed-loop monitoring cycle: To ensure the real-time and continuous operation of the system, after the actuator completes its action, the system automatically returns to the initial sensing and monitoring phase, thus forming a continuous closed-loop monitoring cycle. During the cyclic feedback process, the system ensures real-time performance through an end-to-end full-link response mechanism, ensuring that the overall response time is controlled within 150 milliseconds. Simultaneously, the system uses the execution results of the previous cycle, such as the actual motor speed and the actual backrest angle, as feedback input to the PID controller for the next cycle, ensuring the continuity of the operating state. This hardware and software collaborative safety recovery mechanism comprehensively protects the safety of the equipment and its occupants.

[0131] Example 1: Full-speed driving scenario (primarily obstacle avoidance): When the stroller is moving normally on the park road and the baby is in a normal sitting position, the system, based on an adaptive weighted fusion mechanism, increases the fusion weight of the environmental perception module to 0.6, while decreasing the weight of the baby's status module to 0.2, and keeping the weight of the voice module at 0.2. Under this weight configuration, the system prioritizes obstacle avoidance, focusing on executing the obstacle avoidance logic output by the environmental perception algorithm. When an obstacle is detected ahead, the system prioritizes deceleration or detour operations based on the level of danger to ensure driving safety.

[0132] The infant's normal sitting posture is determined in real time by the multimodal state machine in step S22. The state space of this state machine includes five states: sleeping, crying, normal sitting posture, attempting to stand, and calm. The criteria for determining a normal sitting posture include four indicators: the infant's trunk tilt angle is less than 10 degrees; the wrists are inside the crib rail and more than 15 cm from the top edge of the rail; the eye closure percentage is less than 60%; and the upper limb movement range is within the normal range of motion rather than vigorous waving. When all four indicators are met simultaneously, the state machine outputs the current state as a normal sitting posture, with a confidence level typically higher than 0.85. When the infant is in an abnormal sitting posture, such as bending over or lying on their side, the state machine outputs the corresponding other states, and the system adjusts the fusion weight configuration accordingly.

[0133] The deceleration value is dynamically set according to the hazard level: if the obstacle hazard level is level two (0.4 ≤ R < 0.7), the deceleration rate is 1.0 m / s². 2 The deceleration; if it is level three, i.e., 0.2 ≤ R < 0.4, then the deceleration is 0.3 m / s². 2 The deceleration is controlled by the target. The turning angle range for obstacle avoidance is from -30 degrees to 30 degrees, dynamically calculated based on the target position: if the obstacle is to the left, the system will detour to the right; if it is to the right, the system will detour to the left. The obstacle avoidance safety distance is set so that turning is allowed when the target distance is greater than or equal to 1.5 meters; otherwise, only deceleration and braking are performed without turning. If the lateral space on both sides is insufficient (less than 0.5 meters), the system will only decelerate and will not attempt to detour.

[0134] When the environmental perception module outputs a danger signal and the infant status module simultaneously outputs an abnormal status, the system makes a decision according to the following arbitration logic. First, a hierarchical judgment is performed: If the danger level reaches level one (R ≥ 0.7), regardless of the infant's status, an emergency stop is unconditionally executed, and the infant's status is addressed only after the vehicle has come to a complete stop. If the danger level is level two (0.4 ≤ R < 0.7), a priority judgment is then performed: If the infant is attempting to stand, a comfortable deceleration stop is first executed, with a deceleration rate of 1.0 m / s². 2 Wait until the vehicle comes to a complete stop before dealing with the infant; if the infant is crying, decelerate and play a soothing voice simultaneously. If the danger level is level three (0.2 ≤ R < 0.4), prioritize addressing the infant's condition: if the infant is attempting to stand, first perform a slight deceleration and issue a voice warning, with a deceleration rate of 0.3 m / s². 2 If the infant is crying, soothing voice prompts will be played first, and environmental obstacle avoidance will only involve slight deceleration or voice prompts. If the environment is safe (R < 0.2), the infant's state will be given full priority, and the system will execute the corresponding actions according to the scenario weights configured in Table 1.

[0135] Example 2: Parking monitoring scenario (primarily infant care): When the stroller is parked next to a park bench, the parent is taking a short break, and the baby is asleep, the system increases the fusion weight of the baby status module to 0.6, decreases the weight of the environment perception module to 0.2, and keeps the weight of the voice module at 0.2. Under this weight configuration, the system prioritizes baby care, adjusting the backrest angle to a flat position and playing white noise at a very low volume through the speaker to help the baby sleep. The wheel motors enter standby mode, ensuring the baby rests in a quiet and comfortable environment.

[0136] The infant's sleep state is determined in real time by the multimodal state machine in step S22, specifically based on the following four quantitative indicators: the eye closure ratio is calculated by the ratio of the vertical axis to the horizontal axis of the key points of the eyes through the upper camera; if the closure ratio is greater than or equal to 80% for 5 consecutive minutes, the infant is considered to be in a sleep state; the trunk tilt angle is usually less than 5 degrees in a sleep state, and the trunk presents a relaxed, backward posture; the upper limb movement amplitude is statistically analyzed using dense optical flow features, and the optical flow amplitude is lower than a set threshold in a sleep state; if the above three conditions are maintained for more than 5 minutes without interruption, the state machine outputs a sleep state with a confidence level usually higher than 0.90. When all four indicators are met simultaneously, the system confirms that the infant is in a sleep state.

[0137] The backrest is adjusted to a 45-degree reclining position, which is the maximum adjustment angle of the backrest motor relative to the sitting position (0 degrees). This reclining position setting is based on infant ergonomics: the 45-degree angle provides sufficient support to prevent the baby from slipping, while avoiding an increased risk of spitting up due to a completely flat position. The backrest adjusts smoothly from a sitting to a reclining position at a rate of 10 degrees every 5 seconds, with a total adjustment time of approximately 20 seconds, avoiding sudden changes that might startle the baby. The very low volume corresponds to the speaker volume command. The value is 10; among which, The value ranges from 0 to 100. At this volume, the sound pressure level of white noise at the baby's ear is about 45 to 50 decibels, which is equivalent to the volume of a soft whisper. It can effectively mask sudden environmental noise without stimulating the baby's hearing.

[0138] When the baby wakes up or the environment changes, the system automatically switches modes in real time by monitoring three trigger signals through a fusion decision module: signals indicating changes in the baby's state, changes in the environment, and interactions with the caregiver. When the baby changes from sleeping to crying or attempting to stand, the system immediately exits the parking monitoring mode, triggers proactive soothing or safety protection actions, and alerts the caregiver via speaker. When the parent wakes the system with voice commands such as "keep going" or "play music," the system immediately exits the parking monitoring mode and enters interactive mode or full-speed driving mode. Simultaneously, the system monitors the environmental status: if the current camera detects someone approaching the stroller within 1 meter, or the front camera detects environmental changes such as a pet approaching with a danger level exceeding level three (R ≥ 0.4), the system proactively exits parking monitoring, issues a voice reminder, or executes a warning action. All these switching processes are completed within 100 milliseconds, ensuring timely response.

[0139] Example 3: Composite Scenario (Core Collaboration): As the stroller travels at approximately 1 meter per second, the infant awakens from a sleeping state and attempts to stand. Simultaneously, the front camera detects a pedestrian 10 meters ahead. At this point, the environmental perception module outputs a pedestrian distance of 10 meters and a danger level of 0.3 (corresponding to a level 3 warning), while the infant status module outputs an attempt to stand with a confidence level of 0.85. The fusion decision module, based on priority rules, determines that the infant's dangerous posture has a higher priority than the mild environmental hazard. Therefore, the final decision is: first, decelerate to a stop and issue a voice warning; then, proceed with the obstacle avoidance task only after the infant returns to a sitting position. Details are as follows: Hazard level The calculation formula is:

[0140] in, , , , These are weighting coefficients, with specific values ​​of 0.35, 0.25, 0.25, and 0.15. Risk factor for the target category; Distance to target; It is the relative velocity; To predict the probability of collision.

[0141] Based on the currently monitored pedestrian data, the specific values ​​are substituted into the above formula for calculation: Pedestrian corresponding risk factor. =0.30; distance It is 10 meters, therefore the reciprocal term The relative speed is 0.10; the stroller moves forward at 1 m / s, the pedestrian is stationary, and the relative speed is... The value is 1; assuming there is currently no collision risk, predict the collision probability. Let's take 0.05. Substituting the weights and data above, we get: After rounding, we get... It lies within the interval 0.2≤R<0.4, corresponding to a level 3 prompt.

[0142] However, since the infant's state was an attempt to stand, the system further refined the decision priority through a cross-modal attention mechanism. When the crying and the attempt to stand posture occurred simultaneously, cross-modal feature fusion increased the confidence of the infant's dangerous state from 0.85 in the visual branch to 0.92. This revised high confidence served as input to Rule 2, driving the system to increase the braking priority and execute a decision to prioritize infant safety before obstacle avoidance.

[0143] At the action execution level, the system employs a first-level braking strategy. The deceleration is set at 1.0 m / s². 2 This is 0.3 m / s higher than the standard level three warning. 2 Braking distance is calculated as follows: The braking distance is much less than the distance to a pedestrian 10 meters ahead, ensuring a safe stop.

[0144] Simultaneously, the speaker plays a voice warning: "Baby, please sit still, the car is about to stop." The voice uses a gentle female voice, lasts for 2 seconds, and is set to 60 at a volume to ensure clear audibility in ambient noise without startling the baby. Once the multimodal state machine detects that the baby has returned to a sitting position (i.e., the trunk tilt angle is less than 10 degrees) for more than 2 seconds, the system prompts "Baby, you're sitting still now, let's continue," resumes driving, and reassesses the pedestrian avoidance task ahead.

[0145] To verify the effectiveness of this invention in multimodal information fusion decision-making, the three core modules were independently and jointly verified. The environment perception module was tested using the COCO2017 validation set and a self-built stroller scene test set, with mAP (including mAP50 and mAP50-95) as the evaluation metric. The model parameter count, computational cost (FLOPs), and inference speed (FPS) were also statistically analyzed. The infant state perception module was tested using a self-built infant pose dataset (containing 5000 stroller scene images, labeled with 9 key points), with key point detection accuracy (PCK@0.1) and state classification accuracy as the evaluation metrics. The overall system response time was measured on a Raspberry Pi 4B, and end-to-end latency was statistically analyzed.

[0146] The experimental training configuration is as follows: all experiments were conducted on the Ubuntu 20.04 operating system, with an NVIDIA RTX 3060 GPU (12GB VRAM), an Intel Core i7-12700K CPU, and PyTorch 1.12.0 and CUDA 11.6 as the software environment. The training epochs were set to 300, with an initial learning rate of 0.01, a stepped decay strategy (decaying to 1 / 10 of the current learning rate in epochs 100 and 200 respectively), a batch size of 32, a momentum parameter of 0.9, and a weight decay coefficient of 0.0005.

[0147] To assess the detection accuracy and lightweight performance of the YOLO environment awareness module, the following methods were selected for comparison: the original YOLOv8 model, the pruned YOLOv8 model (without added detection layers and attention), the mainstream lightweight object detection models MobileNet-SSD, and EfficientDet-Lite. Evaluation metrics included mAP, model parameter count (Params), computational cost (FLOPs), and inference speed (FPS).

[0148] To assess the accuracy of infant pose estimation, the following methods were selected for comparison: the original ST-GCN (fixed adjacency matrix), OpenPose (bottom-up multi-person pose estimation), and MediaPipe BlazePose (lightweight pose estimation). The evaluation metrics were keypoint detection accuracy (PCK@0.1) and state recognition accuracy under shaking conditions.

[0149] To verify the effectiveness of each improved module of this invention, an ablation experiment was designed to progressively verify the contributions of the following four improvements: channel pruning and fine-grained detection layer; BiFPN multi-scale feature fusion and CAM / SAM attention; adaptive adjacency matrix anti-shaking and heatmap occlusion compensation; and cross-modal attention fusion mechanism.

[0150] (1) Ablation experiment of environmental perception module: To verify the effectiveness of the various customized improvements proposed in this invention for the stroller scenario, the following ablation experiment was designed: On the COCO validation set and the self-built stroller scenario test set, with the original YOLOv8 model as the baseline, the improved modules of this invention were gradually added, and the results are shown in Table 3.

[0151] Table 3 Ablation Experiment Results of Environmental Sensing Module

[0152] As shown in Table 3, on the COCO validation set, the baseline YOLOv8 achieved a mAP of 69.7%, with 43.6M parameters and an inference speed of 28.7 FPS. After pruning, the model parameters decreased by 58.3%, FLOPs decreased by 58.7%, and inference speed increased by 48.4%, while mAP only decreased by 1.6 percentage points, demonstrating that pruning can significantly improve inference speed without significantly sacrificing accuracy. Adding a 4x downsampling fine-grained detection layer further restored the model's mAP to 69.4%, an improvement of 1.3 percentage points compared to the pruning, indicating that the fine-grained detection layer effectively compensated for the missed detection of small targets. Further introducing BiFPN multi-scale feature fusion and CAM / SAM attention mechanisms resulted in an mAP of 71.2%, an improvement of 1.5 percentage points compared to the baseline, demonstrating that attention mechanisms and multi-scale fusion can effectively address occlusion and complex background interference. The final combination of pruning, fine-grained processing, and attention achieved a mAP of 71.8%, an improvement of 3.7 percentage points compared to the pruning result. The number of parameters was only 43.1% of the baseline, and the FPS reached 40.3, an improvement of 40.4% compared to the baseline. These results fully verify the synergistic effectiveness of the various customized improvements of this invention in terms of lightweight design and accuracy enhancement.

[0153] (2) Ablation experiment of infant state perception module: To verify the effectiveness of the ST-GCN improvement strategy proposed in this invention for the dynamic driving scenario of strollers, an ablation experiment was conducted on a self-built infant pose dataset, and the results are shown in Table 4.

[0154] Table 4. Results of the ablation experiment on the infant state perception module

[0155] Table 4 shows that in a swaying environment, the baseline ST-GCN's pose estimation accuracy is only 68.2%, the keypoint detection success rate under occlusion is only 57.3%, and the false alarm rate for attempting to stand is as high as 38.5%. After adding the adaptive adjacency matrix anti-sway module, the accuracy in a swaying environment improves to 83.1%, while the false alarm rate decreases to 22.7%, indicating that the adaptive adjacency matrix effectively suppresses sway noise. Further introducing heatmap-guided occlusion compensation increases the detection success rate under occlusion from 58.1% to 83.7%, verifying the compensation effect of geometric constraints and Kalman filter prediction. Finally, after fusing the multimodal state machine, the false alarm rate for attempting to stand is reduced to 8.6%, a decrease of 29.9 percentage points compared to the baseline. These results demonstrate that the three improvements complement each other and progress step by step, jointly achieving robust pose estimation and behavior recognition in dynamic stroller scenarios.

[0156] (3) Cross-modal attention fusion ablation experiment: To verify the effectiveness of the multi-head cross-modal attention mechanism in multimodal feature fusion, a comparative experiment was conducted on the constructed multimodal test set, and the results are shown in Table 5.

[0157] Table 5 Results of cross-modal attention fusion ablation experiments

[0158] Table 5 shows that on a complex scene test set including crying sounds, postures, and environmental noise, the weighted summation accuracy of the decision layer was 78.2%, and the robustness was 72.5%. After introducing cross-modal attention, the multimodal fusion accuracy improved from 78.2% to 86.5%, an improvement of 8.3 percentage points compared to the weighted summation of the decision layer; the robustness in complex scenes improved from 72.5% to 81.3%, an improvement of 8.8 percentage points. Compared to single-head attention, 8-head attention significantly improved both accuracy and robustness, with an increase in inference latency of 15.2 ms, meeting the real-time operation requirements of edge devices. This verifies that the cross-modal attention mechanism can effectively capture the correlation information between different modalities and improve multi-source fusion perception capabilities.

[0159] In addition to ablation experiments on the core algorithm modules, this invention also conducted field tests to verify the overall end-to-end system performance. Experimental results show that the end-to-end system constructed in this invention exhibits excellent performance in terms of lightweight design, real-time performance, and multimodal fusion decision-making, meeting the needs of stroller edge device deployment and complex scenario applications. Regarding target detection accuracy, the system achieves an mAP of 71.8% on the COCO validation set, a pose estimation accuracy of 83.8% in a swaying environment using a self-built infant pose dataset, a wrist detection success rate of 84.2% under simulated occlusion, and a false alarm rate of 8.6% for attempted standing behavior. In terms of real-time performance and lightweight design, the system achieves an end-to-end response time of less than 150ms and an inference speed of 40.3 FPS on the Raspberry Pi 4B platform; simultaneously, the model parameter count is optimized to 18.8M, a 57% reduction compared to the baseline model. These data fully validate the effectiveness of this invention in improving the intelligence level of infant and toddler care.

[0160] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0161] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A multimodal information fusion decision-making method for infant and toddler care, characterized in that, Includes the following steps: Step S1: By deploying various modules in the perception layer of the stroller, environmental images, baby posture images, sound, and foot pedal force data are collected and monitored synchronously, and the data streams of each sensor are globally timestamped to provide raw input for subsequent algorithm processing; Step S2: Preprocess the collected multimodal data, and perform environmental perception module processing, infant state perception module processing, and speech intent module processing based on the deep learning and signal processing algorithms deployed in the decision layer; Step S3: Construct a three-layer fusion architecture that includes an adaptive weighted fusion model, a three-level hierarchical decision-making system, and priority coordination rules. Deeply fuse the outputs of the environmental perception, infant state perception, and voice intent modules to output unified control commands. Step S4: After the decision-making layer generates the control command vector, it enters the actuator action and closed-loop feedback stage. The actuator performs physical actions according to the control command and returns to the initial perception and monitoring stage to form a closed-loop monitoring cycle.

2. The multimodal information fusion decision-making method for infant and toddler care according to claim 1, characterized in that, In step S2, The environmental perception module processing includes: based on the YOLO algorithm, performing model lightweighting, feature enhancement and anti-interference processing, trajectory prediction and active obstacle avoidance processing; The infant state perception module processing includes: anti-shaking pose estimation based on spatiotemporal graph convolution and adaptive adjacency matrix based on the spatiotemporal graph convolutional network ST-GCN algorithm; occlusion key point compensation based on heatmap guidance and Kalman filtering; and behavioral intention recognition based on multimodal feature fusion. The speech intent module processing includes: in the audio preprocessing stage, pre-emphasizing the acquired audio signal, dividing the pre-emphasized signal into short-time frames, and windowing each frame using a Hamming window; using a dual-threshold endpoint detection algorithm based on short-time energy and short-time zero-crossing rate to filter out silent segments and retain valid speech segments; in the feature extraction stage, extracting Mel-frequency cepstral coefficients (MFCCs) as acoustic representations, extracting MFCC features from valid speech frames and concatenating them into a temporal feature vector; in the model inference stage, analyzing the crying intent using a lightweight CNN speaker model, and recognizing speech commands using a natural language understanding model; and in the post-processing and output stage, outputting the crying intent analysis results and speech command recognition results.

3. The multimodal information fusion decision-making method for infant and toddler care according to claim 2, characterized in that, In the environmental perception module processing... The model lightweighting process specifically includes: compressing the model using channel-level structured pruning techniques based on batch normalization layer scaling factors; constructing a sparse training loss function by introducing an L1 regularization term; and implementing differentiated pruning strategies according to the characteristics of different network layers to fine-tune the pruned model to restore model accuracy. The feature enhancement and anti-interference processing specifically includes: adding a 4x downsampling layer, composed of upsampling of shallow feature maps and fusion of deep feature maps, to the standard YOLO downsampling detection layer to achieve fine-grained feature extraction; using a bidirectional feature pyramid network (BiFPN) for multi-scale feature fusion; removing single-input nodes, adding cross-scale connections, and employing a weighted fusion method in the BiFPN; generating attention maps through channel attention and spatial attention mechanisms, and then adding the channel attention map and spatial attention map element-wise and obtaining the total attention map through a sigmoid activation function; The trajectory prediction and active obstacle avoidance processing specifically includes: introducing a Kalman filter and setting a discrete time step. Control of time Define the state vector. ;in, For planar position, For speed, Let T be the acceleration and T be the transpose; define the observation vector. Includes only the target's position components A uniform acceleration motion model is adopted, and a state transition matrix and an observation matrix are constructed based on the state vector and the observation vector. The trajectory of the dynamic target is predicted through prediction and update iteration, and deceleration or attitude adjustment commands are triggered in advance according to the prediction results to achieve active obstacle avoidance.

4. The multimodal information fusion decision-making method for infant and toddler care according to claim 3, characterized in that, In the lightweighting process of the model, the sparse training loss function is: ； in, This represents the total loss function value during the sparse training phase. Input data; For the corresponding real tags; These are the weight parameters to be trained in the network model; This represents the loss during normal network training. For sparsification parameters; Indicates traversing the network The set of scaling factors corresponding to all channels in the dataset; This is the L1 regularization penalty function applied to the scaling factor; For scaling factor The applied L1 regularization sparsity penalty term; In the feature enhancement and anti-interference processing, the output of the weighted fusion method is: ; in, The output value after fusion; , The first , The fusion weight coefficients corresponding to each input feature map; For the first One input feature map; It is the minimum value; The formula for calculating channel attention is: ； in, Channel attention map; Input feature map; For batch normalization operation; To Perform global average pooling; , The weights and biases of the fully connected layer in the first layer of a multilayer perceptron (MLP); The weights and biases of the fully connected layer 2 of the MLP; The formula for calculating spatial attention is: ； in, Spatial attention map; For activation functions; This represents a 7×7 convolution operation. Indicates a splicing operation; Indicates to Perform max pooling at the channel dimension.

5. A multimodal information fusion decision-making method for infant and toddler care according to claim 2, characterized in that, In the process of the infant state perception module. The anti-shaking pose estimation based on spatiotemporal graph convolution and adaptive adjacency matrix specifically includes: continuously acquiring multiple frames of images, extracting human key points to construct a spatiotemporal graph structure, wherein the edges of the spatiotemporal graph include spatial edges and temporal edges; constructing and defining an adjacency matrix based on the spatiotemporal graph structure, and introducing a learnable adaptive adjacency matrix, and obtaining the output feature tensor through graph convolution operation; The occlusion key point compensation based on heatmap guidance and Kalman filtering specifically includes: determining the degree of occlusion by calculating the confidence level of the heatmap; implementing three-level graded compensation based on the confidence level of the heatmap; when it is determined to be partial occlusion, the compensation mechanism is activated, geometric derivation is performed in combination with human kinematic constraints to determine the position of the candidate area of ​​the wrist, and the Kalman filter is introduced to predict the motion trajectory of the occluded wrist. The final key point position is obtained by weighted fusion of the geometrically compensated position and the Kalman predicted position. The behavior intent recognition based on multimodal feature fusion: constructing multimodal feature vectors. ;in, Based on the key geometric features, the torso tilt angle, the distance between the wrist and the fence, the eye closure ratio, and the angles of each joint were extracted. To identify dense optical flow features, the Farneback algorithm is used to calculate the optical flow of adjacent frames and statistically analyze the motion amplitude histogram. For depth features, monocular depth estimation is used to obtain depth values ​​for the torso center, wrists, ankles, and fences; For temporal features, an LSTM network is used to capture the behavioral temporal patterns of the first 10 frames; based on the multimodal feature vectors, a discrete state space is constructed including sleep, crying, normal sitting posture, attempting to stand, and calm states. The state transition determination is completed through a fully connected neural network classifier.

6. A multimodal information fusion decision-making method for infant and toddler care according to claim 5, characterized in that, In the anti-shake attitude estimation based on spatiotemporal graph convolution and adaptive adjacency matrix, the graph convolution operation is as follows: ； in, , These are the output and input feature tensors, respectively. For the convolution branch index used for multi-scale feature extraction; This is an element-wise dot product operation; To sum the outputs of multiple convolutional branches; The convolutional kernel weight matrix is ​​a learnable matrix. For predefined adjacency matrices; for adaptive adjacency matrices During the training phase, the shaking noise pattern is actively learned by superimposing synthetic jitter, and it remains unchanged during the inference phase without superimposing synthetic jitter. The confidence level of the heatmap The calculation formula is: ； in, Indicates the first A key point in the image location The response intensity at that location; The operation is used to extract the maximum response value from the heatmap; The three-level compensation is as follows: when When there is no obstruction, the peak value of the heatmap is used directly; when When it is considered partial occlusion, the compensation mechanism is activated; when If the location is considered severely obstructed, then the position predicted by the Kalman filter is used directly. When partial occlusion is determined, human kinematic constraints are considered, based on forearm length. elbow joint angle elbow joint position Shoulder joint position and upper arm length Through rotation matrix Derivation of candidate wrist positions: The position of the wrist relative to the elbow is... Upper arm vector Forearm vector The candidate area for the wrist is... ; In the Kalman filter prediction and weighted fusion... The state vector of the Kalman filter is Includes current position coordinates and the corresponding speed of motion The observation vector is T stands for transpose; The final formula for calculating the key point location is: ； in, The geometrically compensated position is derived based on human kinematic constraints; The location is predicted based on the Kalman filter algorithm; This refers to the final key point location after the weighted fusion of the two. In the behavior intent recognition based on multimodal feature fusion, the fully connected neural network classifier sequentially includes: an input layer, a first fully connected layer, a Dropout regularization layer, a second fully connected layer, and an output layer; wherein, the first and second fully connected layers are both activated by the ReLU activation function; the dropout rate of the Dropout regularization layer is 0.3; and the output layer is activated by the Softmax activation function to output the normalized probability of each state.

7. A multimodal information fusion decision-making method for infant and toddler care according to claim 2, characterized in that, In the voice intent module processing In the audio preprocessing stage, the pre-emphasis operation is implemented using a first-order difference equation, and the calculation formula is as follows: ;in, This is the pre-emphasized signal sequence; The original signal sequence; The previous sampling point of the original audio signal; the pre-emphasis coefficient is set to 0.97; Hamming windowing is used to window each frame of the signal, and the calculation formula is as follows: ;in, A sequence of Hamming window functions; This is the index of the sampling point within the current frame; The frame length; the Hamming window function With the original signal frame Perform point-by-point multiplication to obtain the windowed signal sequence. ; Define short-time energy for: To distinguish the energy difference between unvoiced and voiced sounds and background noise; define the short-time zero-crossing rate. : To help distinguish between unvoiced sounds and noise; among them, The square of the windowed signal sequence; It is a symbolic function; This refers to the previous sampling point in the windowed signal sequence; Mel frequency The calculation formula is: Next, calculate the logarithmic energy of each filter bank output; finally, perform discrete cosine transform on the logarithmic energy to obtain the MFCC coefficients of the preset dimension; concatenate the feature vectors of the current frame with the feature vectors of the five frames before and after it to form a temporal feature vector as the model input; if the current frame is located at the beginning or end of a speech segment, then the boundary frame is processed by mirror filling. In the model inference stage, regarding the analysis of crying intent, the CNN voiceprint model sequentially includes: three identical one-dimensional convolutional layers, each followed by a batch normalization layer, a ReLU activation function, and a max pooling layer; then a flattening layer, a Dropout layer, a first fully connected layer, and a second fully connected layer; the second fully connected layer is the output layer, applying a Softmax activation function to output the probability distribution of three types of intents: hunger, drowsiness, and discomfort; wherein, the one-dimensional convolutional layer has a kernel size of 3, a stride of 1, and padding of 1; the max pooling layer has a kernel size of 2 and a stride of 2; the dropout rate of the Dropout layer is 0.3; and the first fully connected layer applies a ReLU activation function. In terms of voice command recognition, the natural language understanding model converts speech into text based on a pre-trained Chinese speech recognition engine, and then maps the text into structured commands through a lightweight intent classifier. In the post-processing and output stage, regarding the analysis of crying intent, the model output results are as follows: ;in, The representative model predicted the first Specific categories of the intention behind a baby's cry; The model indicates that the sample belongs to an intent. The probability value; The index variable is used; the intent with the highest confidence level exceeding 0.6 is selected as the final output; if no intent exceeds this threshold, it is determined that there is no valid crying intent; in terms of voice command recognition, the model output is... .

8. A multimodal information fusion decision-making method for infant and toddler care according to claim 1, characterized in that, Step S3 specifically includes: An adaptive weighted fusion mechanism is designed, with weights dynamically adjusted according to the real-time scenario. Based on the stroller's real-time speed feedback, the baby's status recognition results, and the voice command detection results, the current scenario type is dynamically determined and the corresponding weight switching is triggered. The weight switching is smoothed by using a first-order low-pass filter. Introducing hazard levels Its calculation formula is ;in, , , , These are the weighting coefficients. The risk coefficient for the target category is customized and calibrated based on the characteristics of the stroller scenario. For the target distance, For relative velocity, To predict collision probability; hazard level The calculation is triggered synchronously with the camera frame rate, and the state update of the Kalman filter is synchronized with the target detection to output the latest predicted collision probability. ;set up The corresponding degradation response will only be executed after the value remains unchanged or decreases for 3 consecutive frames, and the tracking record of the target will be cleared and no longer included in subsequent calculations when the same target is not detected for 10 consecutive frames. Based on the calculated risk level The value is to classify environmental response into three levels, and to formulate three core integrated decision-making rules based on the hierarchical decision-making. After the above multi-layered decision-making logic, the fusion decision module finally outputs a unified seven-dimensional control command vector, the expression of which is: Among them, instruction vector Directly drives the underlying actuators; Indicates the motor speed; Indicates the steering angle; Represents the backrest angle; Indicates speaker volume; Indicates the content to be played; Indicates system mode; Indicates the current task priority; Based on adaptive weighted fusion, a cross-modal attention mechanism is introduced; a multi-head cross-modal attention mechanism is adopted to realize the interaction between visual features and speech features. This mechanism includes three stages: modal feature extraction, cross-modal attention interaction, and multi-modal fusion output, in order to generate a modal fusion feature vector.

9. A multimodal information fusion decision-making method for infant and toddler care according to claim 8, characterized in that, The weight switching formula is as follows: ;in, For the current moment The actual weight vector used; This is the weight vector corresponding to the target scene; For smoothing coefficients; The cutoff frequency; To control the cycle; The environmental response is divided into three levels: when R ≥ 0.7, it is judged as Level 1 danger; when 0.4 ≤ R < 0.7, it is judged as Level 2 warning; when 0.2 ≤ R < 0.4, it is judged as Level 3 alert; when R < 0.2, it is judged as safe. The three core fusion decision rules are as follows: The first rule is safety priority. When the danger level reaches level one (R≥0.7), the system will unconditionally perform an emergency stop and suspend all other tasks. The second rule is state priority. In the infant state assessment, the priority order is as follows: attempting to stand, crying, sleeping, and normal. Higher priority states interrupt the execution of tasks in lower priority states. The third rule is composite scenario coordination. When environmental danger and changes in infant state coexist, the system will make decisions according to a preset coordination matrix: Under level one danger, the system will unconditionally stop regardless of the infant's state; under level two danger, if the infant attempts to stand, the system will prioritize infant safety and then obstacle avoidance; if the infant cries, the system will handle deceleration and soothing in parallel; under level three danger, if the infant is sleeping, the system will decelerate and adjust the backrest to provide white noise. In the modal feature extraction stage, features are extracted from the visual environment, the visual infant, and the speech data respectively: environmental visual features are extracted using the backbone network of the improved YOLO; infant visual features are extracted based on the spatiotemporal feature layer of the improved ST-GCN; speech features are extracted using a lightweight CNN; the environmental visual features and the infant visual features are concatenated to obtain joint visual features. In the cross-modal attention interaction phase, a multi-head cross-modal attention mechanism is employed, using speech features... As the query vector Q, combined with visual features As key vector K and value vector V, the association weights between speech and vision are calculated: through a linear transformation matrix. , , The original features are mapped to the attention space, then the attention score is calculated and normalized. The attention function expression is as follows: ;in, For attention function; It is a normalized exponential function; Key vector The transpose of the matrix; This is the scaling factor; An 8-head attention mechanism is employed, with each attention head computing independently. The 8 attention heads utilize a multi-head division of labor: heads 1 and 2 are responsible for extracting cross-modal correlation features between the type of cry evoked by speech and facial expressions; heads 3 and 4 are responsible for extracting dynamic correlation features between cry intensity and upper limb movement amplitude; heads 5 and 6 are responsible for extracting semantic alignment features between speech commands and environmental targets; and heads 7 and 8 are responsible for extracting temporal correlation features between cry sequence and posture change trends. The outputs of the 8 attention heads are concatenated along the feature dimension and then processed by an output projection matrix. The transformation formula is: ;in, For multi-head attention computation function; This is a tensor splicing operation; For attention; The multimodal fusion output stage will convert the original speech features It is residually connected to the output of the attention layer and then passed through... Normalization yields fusion features The calculation formula is: The fused features are input into a two-layer fully connected network for dimensionality reduction to generate a modality fusion feature vector.

10. A multimodal information fusion decision-making method for infant and toddler care according to claim 1, characterized in that, Step S4 specifically includes: The actuators include wheel-side motors, backrest motors, and speakers, which work together according to control commands: the wheel-side motors execute speed and steering commands, the backrest motors execute backrest angle commands, and the speakers execute volume and playback content commands; in different operating scenarios, during emergency braking, the wheel-side motors decelerate and the speakers sound an alarm; during deceleration and calming, the motors decelerate and the speakers play calming voice messages; during deceleration with white noise, the motors slightly decelerate, the speakers play white noise, and the backrest motors slowly level the backrest. Both the wheel-side motor and the backrest motor use PID controllers for closed-loop control. The PID output expression is: ;in, Output control quantity to the controller; Indicates the current time The error value is the expected value. Compared with actual feedback value The error between; For a historic moment The error value; , , These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. During the turning process, the backrest is linked according to the turning angle and the baby's state: when the baby is awake, the backrest maintains the current angle; when the baby is asleep and the turning angle is greater than 10 degrees, if the backrest angle is greater than 30 degrees, the angle remains unchanged; if the backrest angle is less than 15 degrees, the backrest is slightly adjusted 5 degrees in the opposite direction of the turn; after the turn is straightened, the backrest smoothly returns to the angle before the turn at a speed of 5 degrees every 5 seconds. The end-to-end total response time is controlled within the preset time. If any link times out, the decision of the previous cycle is used. If the detection fails for 3 consecutive cycles, the security protection is triggered. The safe recovery system employs a dual protection mechanism of hardware watchdog and software monitoring: the hardware watchdog cuts off the enable pin of the wheel-side motor driver and triggers the brakes if it does not receive a watchdog signal within a preset time; the software monitoring uses a single frame image processing cycle as a monitoring cycle, and forces deceleration to a stop if the detection fails for 3 consecutive cycles; the wheel-side motor driver has built-in overcurrent protection and overtemperature protection. After the actuator completes its action, it returns to the initial sensing and monitoring phase to form a closed-loop monitoring cycle, and uses the execution result of the previous cycle as the PID feedback input for the next cycle.

Images