A baby facial expression recognition and emotional state analysis method and system

Abstract

The present application relates to the technical field of infant care, and in particular to a method and system for infant facial expression recognition and emotional state analysis, which synchronously collects infant facial video and wearable physiological parameters to obtain multi-modal monitoring data; uses physiological data to match expression timing feature fluctuation, quantifies full-face muscle movement differences, and further estimates emotional abnormality risk; combines timing feature and muscle difference data to output emotional state classification, and generates targeted emotional intervention strategies according to the classification results and risk data matching; constructs an automatic analysis model based on the above classification data and intervention strategies, and deploys it to a cloud platform to perform full-process analysis. This method overcomes single-modal interference through multi-source fusion of vision and physiology, realizes closed-loop management from dynamic expression capture, risk warning to intelligent intervention, and significantly improves the accuracy and real-time monitoring capability of infant emotional recognition.

Description

Technical Field

[0001] This invention relates to the field of infant and toddler care technology, and in particular to a method and system for recognizing facial expressions and analyzing emotional states in infants and toddlers. Background Technology

[0002] With the rapid development of artificial intelligence and intelligent sensing technologies, non-contact, automated identification of infants' and toddlers' emotional states has gradually become an important research direction in the fields of smart healthcare, intelligent care, and early developmental assessment. Because infants and toddlers' language expression abilities are not yet fully developed, they mainly convey emotional information through facial expressions, body movements, and physiological changes. Traditional manual observation methods suffer from problems such as strong subjectivity, poor real-time performance, and difficulty in continuous monitoring, making it difficult to meet the needs of families and childcare institutions for 24 / 7, precise monitoring of infants' and toddlers' emotional health.

[0003] In recent years, computer vision-based facial expression recognition technology has been widely used in adult emotion analysis, but it still faces many challenges in infants and young children. Infants and young children have unique facial structure proportions, large and irregular expression variations, and their skin is highly reflective and their head movements are frequent, leading to insufficient robustness of traditional expression recognition algorithms. Furthermore, relying solely on visual information is easily affected by factors such as lighting and occlusion, making it difficult to accurately distinguish the true emotional state behind similar expressions such as crying, irritability, and pain. Existing research attempts to combine physiological signals (such as heart rate and skin conductance response) to improve the accuracy of emotion recognition, but most systems lack modeling of the correlation between the dynamic temporal characteristics of facial expressions and physiological responses, failing to fully explore the synergistic patterns of multimodal data in the process of emotion evolution. At the same time, risk warning mechanisms for abnormal emotions in infants and young children (such as persistent anxiety and stress responses) are still imperfect, lacking a closed-loop analysis process from expression recognition to intervention recommendations.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method and system for recognizing facial expressions and analyzing emotional states in infants and young children. This aims to solve the technical problems of existing infant and young child emotion recognition technologies, such as weak anti-interference ability of a single visual modality, lack of deep correlation modeling of the dynamic temporal sequence of facial expressions and physiological signals, and failure to form a closed-loop analysis process from accurate recognition to abnormal risk warning and intervention suggestions.

[0006] To achieve the above objectives, the present invention provides a method for facial expression recognition and emotional state analysis in infants and young children, the method comprising: Step S1: Collect facial expressions of infants and young children under monitoring conditions using an image acquisition device to obtain facial expression videos of infants and young children; based on the facial expression videos of infants and young children, monitor the emotion-related physiological parameters of infants and young children under monitoring conditions using wearable physiological sensors to obtain emotion-related physiological monitoring data of infants and young children. Step S2: Match the temporal feature fluctuations of facial expressions and movements based on the physiological monitoring data related to infant emotions to obtain temporal feature fluctuation data; quantify the differences in facial muscle movements in different areas of the face based on the temporal feature fluctuation data to obtain facial muscle movement difference data; estimate the risk of abnormal emotions in infants and young children based on the facial muscle movement difference data to obtain abnormal emotional risk data in infants and young children. Step S3: Match the emotional state level output based on the temporal feature fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state classification data; match infant and toddler emotional intervention strategies based on the emotional state classification data and infant and toddler emotional abnormality risk data to obtain infant and toddler emotional intervention and regulation data. Step S4: Based on the emotional state classification data and infant emotional intervention and regulation data, construct an automatic analysis model for infant emotions to obtain an automatic analysis model for infant emotional states. Deploy the automatic analysis model for infant emotional states to a cloud platform to execute an emotional state analysis method based on infant facial expression recognition.

[0007] Optionally, step S1 includes the following steps: Step S11: Deploy image acquisition devices and wearable physiological sensors in daily monitoring scenarios for infants and young children; Step S12: Capture the facial expressions of the infants under monitoring conditions using an image acquisition device to obtain a video of the infants' facial expressions; Step S13: Perform time-series frame-segmentation processing on the infant's facial expression video to obtain continuous frames of the infant's facial expression; Step S14: Based on continuous frame segmentation of infants' facial expressions, and through wearable physiological sensors, monitor the emotion-related physiological parameters of infants under monitoring conditions to obtain infant emotion-related physiological monitoring data.

[0008] Optionally, step S2 includes the following steps: Step S21: Extract facial expression and motion feature trajectories from the infant's facial expression video to obtain the facial expression and motion feature trajectories; Step S22: Based on the physiological monitoring data related to infant emotions, perform temporal feature fluctuation matching on the facial expression and movement feature trajectory to obtain facial expression temporal feature fluctuation data; Step S23: Perform facial muscle dynamic activity analysis based on facial expression temporal feature fluctuation data and facial expression action feature trajectory to obtain facial muscle dynamic activity data; Step S24: Quantify the differences in facial muscle movement in different areas of the face by analyzing the dynamic activity data of facial muscles to obtain facial muscle movement difference data. Step S25: Estimate the risk of infant and toddler emotional abnormalities based on facial muscle movement difference data and expression temporal characteristic fluctuation data to obtain infant and toddler emotional abnormality risk data.

[0009] Optionally, step S23 includes the following steps: Step S231: Estimate the facial size of infants and young children from the facial expression video to obtain facial size estimation data; Step S232: Based on the temporal feature fluctuation data of facial expressions and the facial size estimation data of infants and young children, the range of motion of different facial muscle groups of infants and young children is deduced to obtain the range of muscle motion of different facial muscle groups of infants and young children. Step S233: Based on the estimated facial size data of infants and young children, perform a correlation coefficient analysis on the range of muscle movement amplitude to obtain the correlation coefficient of size-related movement intensity. Step S234: Perform facial expression key point pose recognition on the facial expression action feature trajectory to obtain facial expression pose key point recognition data; Step S235: Based on the facial expression and posture key point recognition data, extrapolate the distribution of muscle activation intensity among different facial expressions and postures of infants and young children in the range of muscle movement amplitude to obtain facial muscle activation intensity distribution data. Step S236: Perform facial muscle dynamic activity analysis based on the size-related motion intensity correlation coefficient and facial muscle activation intensity distribution data to obtain facial muscle dynamic activity data.

[0010] Optionally, step S3 includes the following steps: Step S31: Match the emotional state level output based on the temporal feature fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state classification data; Step S32: Normalize the emotional state classification data to obtain normalized emotional state classification data. Step S33: Based on the normalized data of emotional state classification, the data of differences in facial muscle movement, and the data on the risk of abnormal emotions in infants and young children, match the intervention strategies for infant and young children's emotions to obtain data on the regulation and control of infant and young children's emotions.

[0011] Optionally, step S31 includes the following steps: Step S311: Plot the temporal fluctuation curve of the facial expression temporal feature data to obtain the temporal fluctuation curve of the facial expression feature intensity; Step S312: Calculate the instantaneous intensity rise smoothness of the facial expression feature intensity temporal fluctuation curve to obtain the instantaneous intensity rise smoothness of the facial expression feature. Step S313: Based on the smoothness of the instantaneous intensity rise of facial expression features, perform multi-peak region pattern recognition on the temporal fluctuation curve of facial expression feature intensity to obtain the multi-peak region pattern of facial expression feature intensity; Step S314: Perform dynamic matching of emotion levels based on the multi-peak region pattern of facial expression feature intensity to obtain dynamic emotion classification data; Step S315: Perform facial expression feature weight matching based on facial muscle movement difference data, and perform multi-feature weighted optimization on the emotional dynamic grading data to obtain multi-feature weighted emotional grading data; Step S316: Match the emotional state level output based on the emotional dynamic grading data and the multi-feature weighted emotional grading data to obtain emotional state grading data.

[0012] Optionally, step S4 includes the following steps: Step S41: Construct a labeled training sample set based on the emotional state classification data and infant emotional intervention and regulation data, and train and iterate the initial convolutional neural network emotion analysis model to obtain the trained automatic analysis model of infant emotional state. Step S42: Perform lightweight pruning and compression on the trained automatic analysis model of infant and toddler emotional state to adapt to the requirements of edge-cloud collaborative deployment and obtain a deployable lightweight automatic analysis model of infant and toddler emotional state. Step S43: Send the lightweight infant and toddler emotional state automatic analysis model to the infant and toddler cloud care platform and connect it with the real-time infant and toddler care system of the family or institution to run the emotional state analysis method based on infant and toddler facial expression recognition.

[0013] Furthermore, to achieve the above objectives, the present invention also provides an infant facial expression recognition and emotion state analysis system, the system comprising: The multi-source acquisition module is used to capture facial expressions of infants and young children under monitoring conditions through image acquisition devices to obtain facial expression videos of infants and young children; based on the facial expression videos of infants and young children, and through wearable physiological sensors, to monitor the emotion-related physiological parameters of infants and young children under monitoring conditions to obtain emotion-related physiological monitoring data of infants and young children. The feature quantization module is used to match the temporal feature fluctuations of facial expressions based on physiological monitoring data related to infant emotions, and obtain the temporal feature fluctuation data of facial expressions; based on the temporal feature fluctuation data of facial expressions, it quantifies the differences in facial muscle movements in different areas of the face, and obtains the facial muscle movement difference data; based on the facial muscle movement difference data, it estimates the risk of abnormal emotions in infants and young children, and obtains the risk data of abnormal emotions in infants and young children. The graded intervention module is used to match the emotional state level output based on the temporal characteristic fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state graded data; and to match the emotional state graded data and the risk data of abnormal emotions in infants and young children to obtain emotional intervention and regulation data for infants and young children. The cloud deployment module is used to build an automatic analysis model of infant emotions based on emotional state classification data and infant emotion intervention and regulation data. The resulting automatic analysis model of infant emotions is then deployed to the cloud platform to execute an emotional state analysis method based on infant facial expression recognition.

[0014] In addition, to achieve the above objectives, the present invention also provides an infant facial expression recognition and emotion state analysis device, the device comprising: a memory, a processor, and an infant facial expression recognition and emotion state analysis program stored in the memory and executable on the processor, the infant facial expression recognition and emotion state analysis program being configured to implement the steps of the infant facial expression recognition and emotion state analysis method as described in any one of the above descriptions.

[0015] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an infant facial expression recognition and emotion state analysis program, wherein when the infant facial expression recognition and emotion state analysis program is executed by a processor, it implements the steps of the infant facial expression recognition and emotion state analysis method as described above.

[0016] This invention provides a method for facial expression recognition and emotional state analysis in infants and toddlers. This method overcomes the limitations of traditional methods that rely solely on visual images, innovatively combining facial expression videos with wearable physiological sensor data (such as heart rate and skin conductance). Through cross-validation of visual features and physiological parameters, it effectively overcomes misidentification problems caused by changes in lighting, facial occlusion, or vigorous head movements in infants and toddlers, significantly improving the accuracy and anti-interference ability of emotional state recognition in complex environments. By performing temporal feature fluctuation matching of facial expressions and quantifying the differences in muscle movement across different areas of the face, this method not only focuses on static expressions but also captures the dynamic evolution of emotions. This fine-grained analysis can distinguish between instantaneous reactions and persistent emotions, accurately capturing subtle facial muscle changes, thus more realistically reflecting the inner emotional state of infants and toddlers and avoiding superficial judgments. Furthermore, this method goes beyond emotion classification and introduces an emotional abnormality risk estimation mechanism, enabling early identification of potential risks such as anxiety, pain, or stress. Based on this, the system can automatically generate targeted emotion intervention strategies, realizing closed-loop management from passive monitoring to proactive intervention. This provides scientific decision support for parents and caregivers, and helps in the early detection and timely intervention of infant and toddler mental health. By deploying the constructed automatic analysis model to the cloud platform, this method possesses powerful data processing capabilities and scalability. This enables the system to support multi-terminal access, achieving remote, real-time, and 24 / 7 monitoring of a large number of infants and toddlers, reducing local hardware costs, and facilitating its application in various scenarios such as homes, childcare institutions, and hospitals, thereby improving the efficiency of socialized infant and toddler care. Addressing the unique facial structure and irregular expression patterns of infants and toddlers, this method establishes a dedicated analysis model adapted to their physiological characteristics through specific muscle movement difference quantification and risk estimation algorithms. This solves the problem of poor applicability of general adult expression recognition algorithms to infants and toddlers, making it more targeted and practical. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the infant facial expression recognition and emotional state analysis method of the present invention; Figure 2 This is a structural block diagram of an embodiment of the infant facial expression recognition and emotion state analysis system of the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Reference Figure 1 , Figure 1This is a flowchart illustrating an embodiment of the infant facial expression recognition and emotion state analysis method of the present invention, which presents an embodiment of the infant facial expression recognition and emotion state analysis method of the present invention.

[0021] In one embodiment, the infant facial expression recognition and emotion state analysis method includes: Step S1: Facial expressions of infants under monitoring are captured using an image acquisition device to obtain facial expression videos. Based on these videos, wearable physiological sensors are used to monitor emotion-related physiological parameters of the infants under monitoring, resulting in emotion-related physiological monitoring data.

[0022] The image acquisition device is an imaging apparatus used to continuously capture visual information of an infant's face, acquiring videos of the infant's facial expressions that include dynamic changes in facial expressions. In this embodiment, the image acquisition device can capture video streams of the infant's face using an optical lens and an image sensor. Further, the image acquisition device may include, but is not limited to, one or more of visible light cameras, near-infrared cameras, and depth cameras. The infant's facial expression video can be digital video data recording a sequence of facial expressions and movements over time, which can be used as visual modal input to extract temporal features of facial expressions and muscle movement patterns. Exemplarily, the infant's facial expression video is continuously captured by the image acquisition device under monitoring conditions. In an exemplary embodiment, the infant's facial expression video is aligned with wearable physiological sensor data in the temporal dimension, jointly participating in emotional state modeling.

[0023] Wearable physiological sensors can be miniature sensing devices attached to the surface of an infant's body to continuously collect emotion-related physiological signals. They can provide emotion-related physiological monitoring data for cross-validation of visual recognition results. In one specific embodiment, the wearable physiological sensor measures parameters such as heart rate and skin conductance in real time through photoplethysmography (PPG), electrode contact, etc. Further, the wearable physiological sensor can include, but is not limited to, heart rate sensors, skin conductance sensors, and body temperature sensors. The infant's emotion-related physiological monitoring data can be a time series of physiological signals reflecting the infant's emotional state, collected by the wearable physiological sensor. This data can be used as a non-visual modal input to assist in judging the authenticity of emotions and improve anti-interference capabilities. In this embodiment, the infant's emotion-related physiological monitoring data is continuously sampled by the wearable physiological sensor within a time window synchronized with facial video. For example, the infant's emotion-related physiological monitoring data and the infant's facial expression video are used together for temporal feature matching of facial expressions.

[0024] Facial expressions of infants under monitoring are captured using image acquisition devices to obtain facial expression videos. This can be achieved by continuously recording the infant's facial area while the infant is in a natural, active, or resting state. Furthermore, this operation can be achieved by using a high-definition camera mounted on a fixed bracket for frontal view capture, or by adaptive tracking capture using a mobile smart terminal (such as a baby monitor), thereby obtaining raw visual data containing the complete dynamic process of facial expressions. Based on the infant's facial expression video, emotional-related physiological parameters of the infant under monitoring are monitored using wearable physiological sensors to obtain emotional-related physiological monitoring data. This can be achieved by simultaneously activating the wearable physiological sensors to record physiological signals such as heart rate and skin conductance during the same time period as video capture. For example, this operation can be achieved by using the Bluetooth Low Energy (BLE) protocol to transmit sensor data to the central processing unit in real time and align it with the video frame timestamps, or by locally caching physiological and video data and then performing precise alignment using a post-processing time synchronization algorithm. This establishes a foundation for time synchronization between visual and physiological signals, providing a data alignment prerequisite for multimodal fusion.

[0025] Step S2: Match the temporal characteristic fluctuations of facial expressions based on the physiological monitoring data related to infant emotions to obtain temporal characteristic fluctuation data. Quantify the differences in facial muscle movements in different areas of the face based on the temporal characteristic fluctuation data to obtain facial muscle movement difference data. Estimate the risk of abnormal emotions in infants based on the facial muscle movement difference data to obtain abnormal emotional risk data for infants.

[0026] Among them, the temporal feature fluctuation data of facial expressions can be a feature sequence describing the dynamic changes of infants' facial expressions over time. It can be used to capture the emotional evolution process and distinguish between instantaneous reactions and continuous emotional states. In this embodiment, the temporal feature fluctuation data of facial expressions is based on the timestamps of infants' emotion-related physiological monitoring data, and the visual features of key frames in the facial expression video are aligned and fluctuation modeled. Furthermore, the temporal feature fluctuation data of facial expressions can include, but is not limited to, temporal fluctuation features of the perioral region, temporal fluctuation features of the perioral region, and temporal fluctuation features of the eyebrow region. Facial muscle movement difference data can be structured data that quantifies the differences in the amplitude and pattern of facial muscle movements in different areas of the face. It can be used to reflect the unique irregular facial expression patterns of infants and improve the sensitivity to subtle emotional changes. In an exemplary embodiment, the facial muscle movement difference data is based on the temporal feature fluctuation data of facial expressions, and the differences in muscle movement intensity in each region are calculated through facial action unit (AU) analysis or optical flow method. For example, the facial muscle movement difference data can include, but is not limited to, upper face-lower face movement asymmetry, left and right half face movement differences, activation gradients of the core expression area and the edge area.

[0027] Infant and toddler emotional abnormality risk data can represent the probability or level of an infant's current emotional state deviating from the normal range, indicating potential risks such as anxiety, pain, or stress. This data can be used to support early warning of emotional abnormalities and provide a basis for intervention. In one specific embodiment, infant and toddler emotional abnormality risk data is input into a preset risk estimation algorithm model based on facial muscle movement difference data. Furthermore, infant and toddler emotional abnormality risk data and emotional state grading data jointly participate in matching infant and toddler emotional intervention strategies. Based on infant and toddler emotion-related physiological monitoring data, temporal feature fluctuations of facial expressions are matched to obtain temporal feature fluctuation data. This can be achieved by using significant changes in physiological signals (such as a sudden increase in heart rate) as anchor points, locating corresponding time periods in facial expression videos, and extracting visual feature fluctuation patterns. For example, this operation can be achieved by using a dynamic time warping (DTW) algorithm to non-linearly align physiological signals and facial expression feature sequences, or by constructing a two-stream neural network to encode physiological and visual temporal sequences separately, and then performing cross-modal matching through an attention mechanism. This allows for the modeling of the temporal correlation between visual expressions and internal emotional responses.

[0028] Based on the temporal fluctuation data of facial expressions, the differences in facial muscle movement across different regions are quantified to obtain facial muscle movement difference data. This can be achieved by dividing the face into multiple anatomical regions and calculating the temporal differences in the amplitude, speed, and coordination of movement in each region. In a specific embodiment, this operation can be achieved by calculating the differences in vertex displacement of each region based on a 3D face reconstruction model, or by segmenting facial regions using a dense optical flow field and statistically analyzing the differences in local motion vector distribution. This allows for the capture of atypical facial expression patterns in infants and young children due to their unique facial structure. The risk of abnormal emotions in infants and young children is then estimated based on the facial muscle movement difference data. This can be achieved by inputting the facial muscle movement difference data into a pre-trained risk estimation algorithm, which outputs an abnormal emotion risk score or category. Furthermore, this operation can be achieved by using a rule-based expert system to set a threshold for muscle movement asymmetry to trigger risk warnings, or by using a deep learning classifier to train a risk prediction model with historical labeled data. This enables the early identification of potential negative emotions such as anxiety and pain.

[0029] Step S3: Match emotional state level outputs based on facial expression temporal characteristic fluctuation data and facial muscle movement difference data to obtain emotional state classification data. Based on the emotional state classification data and infant emotional abnormality risk data, match infant emotional intervention strategies to obtain infant emotional intervention and regulation data.

[0030] The emotional state grading data can be the result data of classifying the infant's current emotional state according to preset categories or intensity levels. This data can be used to provide a structured description of the emotional state, facilitating the generation of subsequent intervention strategies. In this embodiment, the emotional state grading data is obtained by matching facial expression temporal feature fluctuation data and facial muscle movement difference data to an emotion classification model. For example, the emotional state grading data may include, but is not limited to, levels of pleasant emotions, levels of uncomfortable emotions, and levels of neutral / calm emotions. The infant's emotional intervention and regulation data can be a set of intervention suggestions or execution instructions automatically generated for specific emotional states and risk levels. This can be used to achieve closed-loop management from emotion recognition to proactive intervention. In an exemplary embodiment, the infant's emotional intervention and regulation data matches the optimal intervention plan in a preset strategy library based on the emotional state grading data and the infant's emotional abnormality risk data. Furthermore, the infant's emotional intervention and regulation data may include, but is not limited to, soothing voice playback instructions, ambient light adjustment suggestions, and caregiver reminders.

[0031] Emotional state level output matching is performed based on facial expression temporal feature fluctuation data and facial muscle movement difference data to obtain emotional state classification data. This can be achieved by fusing the two types of features and mapping them to a predefined emotional state level system. For example, this operation can be performed by concatenating features and inputting them into a Softmax classification layer to output a probability distribution, or by using a multi-task learning framework to simultaneously optimize emotion classification and risk estimation objectives, thereby outputting structured and interpretable emotion classification results. Based on the emotional state classification data and infant emotional abnormality risk data, infant emotional intervention strategy matching is performed to obtain infant emotional intervention and regulation data. This can be achieved by querying a pre-defined intervention strategy knowledge base and selecting the optimal intervention measure based on the current emotional level and risk level. In a specific embodiment, this operation can be implemented by matching intervention strategies based on a decision tree rule engine, or by using a reinforcement learning agent to dynamically optimize strategy selection based on historical intervention effects, thereby generating executable intervention suggestions and forming a closed-loop management system.

[0032] Step S4: Based on the emotional state classification data and infant emotional intervention and regulation data, construct an automatic analysis model for infant emotions to obtain an automatic analysis model for infant emotional states. Deploy the automatic analysis model for infant emotional states to a cloud platform to execute an emotional state analysis method based on infant facial expression recognition.

[0033] The automatic analysis model for infant and toddler emotional states can be a machine learning model that integrates multimodal inputs and possesses capabilities for emotion recognition, risk assessment, and intervention recommendation. It can be used to automate and intelligently analyze the emotional states of infants and toddlers. In this embodiment, the automatic analysis model is built through end-to-end training or modular integration based on emotional state grading data and infant and toddler emotional intervention and regulation data. Furthermore, after being deployed on a cloud platform, the automatic analysis model receives video and physiological data streams from terminals and outputs analysis results. The cloud platform can be a distributed network infrastructure providing remote computing, storage, and service scheduling capabilities, which can support concurrent access from multiple terminals, reduce local hardware dependence, and achieve 24 / 7 remote monitoring. For example, the cloud platform accesses via the internet and invokes the automatic analysis model for infant and toddler emotional states deployed on it to process user requests. In a specific embodiment, the cloud platform can include, but is not limited to, a public cloud service platform, a private cloud server cluster, or an edge-cloud collaborative computing architecture.

[0034] An automatic emotion analysis model for infants and toddlers is constructed based on emotional state grading data and infant emotion intervention and regulation data. This model integrates emotion recognition, risk assessment, and intervention recommendation modules, training an end-to-end or multi-stage joint model. Furthermore, this operation can be achieved through modular design, training each sub-model separately and then cascading them together, or by constructing a unified Transformer architecture to directly output intervention suggestions from multimodal inputs, thus forming a dedicated emotion analysis system adapted to the infant and toddler population. The automatic emotion analysis model is deployed to a cloud platform to execute emotion state analysis methods based on infant facial expression recognition. This can be achieved by encapsulating the trained model as an API service, deploying it on a cloud server, and providing open interfaces for terminal calls. In an exemplary embodiment, this operation can be achieved by deploying model microservices using containerization technology (such as Docker + Kubernetes), or by combining edge computing nodes for initial processing, uploading only key data to the cloud for precise calculation, thereby enabling high-concurrency, low-latency remote emotion analysis services.

[0035] Taking home-based smart care as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be achieved by installing a smart monitor with a camera in the nursery and having the infant wear a flexible heart rate-skin electric field bracelet. The system collects facial video and physiological data in real time and uploads it to a cloud platform. The infant's emotional state automatic analysis model on the cloud platform processes the data and identifies persistent irritability caused by colic (manifested as high-frequency twitching of the perioral muscles and a continuously elevated heart rate). The system determines this as a high-risk pain state and automatically generates intervention suggestions: playing white noise, adjusting the tightness of the swaddling clothes, and pushing reminders to the parents' mobile phones. Parents can then soothe the infant in a timely manner to prevent the emotional state from worsening.

[0036] For example, in a scenario of group monitoring in a childcare facility, the infant facial expression recognition and emotional state analysis method of this embodiment can be implemented by deploying multiple cameras and wearable physiological monitoring vests in the childcare classroom, allowing the system to monitor multiple infants simultaneously. When a child experiences a brief awakening during naptime accompanied by severe eyebrow contraction and a sudden increase in skin conductance, the model identifies it as a transient stress response with low risk, and only records it in the log; while another child continues to cry and exhibits highly asymmetrical facial muscle movements, combined with an abnormally high heart rate, the system determines it to be a high-risk anxiety state, immediately sends a priority alert to the caregiver's terminal, suggests one-on-one comforting, and records developmental behavioral abnormalities.

[0037] In one embodiment, step S1 includes the following steps: Step S11: Deploy image acquisition devices and wearable physiological sensors in daily monitoring scenarios for infants and young children; The daily monitoring scenarios for infants and toddlers can be typical environments where infants and toddlers are naturally active or resting in homes, childcare facilities, or medical institutions. These environments can serve as the physical spaces for deploying image acquisition devices and wearable physiological sensors, ensuring that data collection closely resembles real-life care situations. In one exemplary embodiment, the daily monitoring scenarios for infants and toddlers can include, but are not limited to, one or more of the following: a home bedroom environment, a childcare classroom environment, or a pediatric clinic environment. Deploying image acquisition devices and wearable physiological sensors in these scenarios can involve installing the image acquisition devices at appropriate viewing angles within the infant's activity area and equipping the infant with suitable wearable physiological sensors. Furthermore, the deployment of image acquisition devices and wearable physiological sensors in daily monitoring scenarios can be achieved through seamless deployment using wall-mounted or crib-integrated cameras combined with flexible fabric sensor vests, or through temporary and rapid deployment using a combination of a mobile terminal bracket and a wristband-style physiological bracelet. This allows for the establishment of a multimodal data acquisition environment that closely resembles real life, ensuring the validity of the data ecosystem.

[0038] Step S12: Capture the facial expressions of the infants under monitoring conditions using an image acquisition device to obtain a video of the infants' facial expressions; Step S13: Perform time-series frame-segmentation processing on the infant's facial expression video to obtain continuous frames of the infant's facial expression; The continuous frame segmentation of infant facial expressions can be achieved by dividing an infant facial expression video into a structured sequence of single-frame images in chronological order. This sequence can provide a basic processing unit for temporal alignment in subsequent temporal feature fluctuation matching and muscle motion quantification. For example, continuous frame segmentation of infant facial expressions can be performed frame-by-frame by a video decoder at a fixed frame rate (e.g., 30fps). In a specific embodiment, continuous frame segmentation of infant facial expressions can include, but is not limited to, one or more of key expression frame sequences, uniformly sampled frame sequences, and event-triggered frame sequences. Temporal frame segmentation of infant facial expression videos to obtain continuous frames can be achieved by decoding the acquired video stream into independent image frames in chronological order while maintaining the original temporal relationship. Furthermore, temporal frame segmentation of infant facial expression videos to obtain continuous frames can be achieved by using uniform sampling at fixed frame intervals (e.g., one frame every 33ms) or by dynamically selecting high-variable frames and key frames based on motion detection to form a sparse-dense hybrid sequence. This transforms continuous dynamic information into structured data that can be analyzed frame-by-frame, supporting subsequent temporal modeling.

[0039] Step S14: Based on continuous frame segmentation of infants' facial expressions, and through wearable physiological sensors, monitor the emotion-related physiological parameters of infants under monitoring conditions to obtain infant emotion-related physiological monitoring data.

[0040] The emotion-related physiological parameters can be measurable biosignal indicators that are physiologically correlated with emotional states. These parameters can be used as non-visual modal inputs to verify and supplement the authenticity of emotions reflected in facial expressions. In this embodiment, emotion-related physiological parameters may include, but are not limited to, one or more of heart rate variability, skin conductance response level, and respiratory rate. Based on continuous frame-by-frame analysis of infant facial expressions, and by monitoring the emotion-related physiological parameters of infants under monitoring conditions using wearable physiological sensors, emotion-related physiological monitoring data for infants is obtained. This data can be collected by the wearable physiological sensors within a time window synchronized with the continuous frame-by-frame analysis, capturing physiological signals at corresponding moments. Furthermore, this operation can be achieved by using hardware synchronization signals (such as GPIO triggers) to ensure strict alignment between video frames and physiological sampling points, or by using software timestamp calibration algorithms to perform post-alignment between asynchronously acquired video frames and physiological data. This allows for precise alignment of visual frames and physiological parameters in the temporal dimension, laying the foundation for cross-modal correlation modeling.

[0041] Taking home nighttime care as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be achieved by installing a wide-angle infrared camera on the ceiling of the nursery and having the infant wear low-power flexible skin conductance-heart rate sensing pajamas. The system automatically starts at night, acquiring infrared video containing subtle facial expression changes; the video is framed into a continuous image sequence in real time, while the physiological sensor records the signal at a sampling rate of 100Hz. When the infant briefly frowns due to a nightmare (frames 125–130) accompanied by a sudden increase in skin conductance, the system confirms that the expression is consistent with autonomic nerve activation through frame-physiological synchronization alignment, avoiding misinterpreting rolling over and occlusion as an emotional event.

[0042] In one embodiment, step S2 includes the following steps: Step S21: Extract facial expression and motion feature trajectories from the infant's facial expression video to obtain the facial expression and motion feature trajectories; The facial expression and movement feature trajectory can be structured data describing the movement paths of facial key points or action units in an infant's face over time. This data can provide a foundation for dynamic modeling input for subsequent temporal fluctuation matching and muscle activity analysis. In this embodiment, the facial expression and movement feature trajectory can be extracted by detecting facial key points or recognizing action units in an infant's facial expression video, extracting the spatial coordinate sequence of each key region over time. For example, the facial expression and movement feature trajectory may include, but is not limited to, one or more of the following: periorbital key point trajectory, corner of the mouth movement trajectory, and brow bone displacement trajectory. Extracting the facial expression and movement feature trajectory from an infant's facial expression video can be achieved by detecting facial key points or action units frame by frame in the video and recording their spatial position changes over time to form a trajectory sequence. Furthermore, this operation can be implemented by using a 3D face reconstruction model to track the three-dimensional displacement trajectory of key points, or by using a 2D face alignment network to output a sequence of 68 or 106 key points as the trajectory basis. This transforms the original video into a quantifiable dynamic expression representation, preserving the spatial-temporal information during the emotional evolution process.

[0043] Step S22: Based on the physiological monitoring data related to infant emotions, perform temporal feature fluctuation matching on the facial expression and movement feature trajectory to obtain facial expression temporal feature fluctuation data; Based on physiological monitoring data related to infant emotions, the temporal fluctuations of facial expression movement feature trajectories are matched to obtain temporal fluctuation data of facial expressions. This can be achieved by using physiological signals (such as heart rate mutation points and skin conductance peaks) as references to locate corresponding time periods in the facial expression movement feature trajectory and extracting visual fluctuation features. In an exemplary embodiment, this operation can be performed by using sliding window correlation analysis to match physiological and trajectory fluctuation cycles, or by constructing a dual-stream LSTM network to encode physiological and trajectory sequences separately and achieve matching through a cross-attention mechanism. This establishes a cross-modal temporal correlation between physiological responses and facial expression dynamics, enhancing the ability to distinguish real emotional events.

[0044] Step S23: Perform facial muscle dynamic activity analysis based on facial expression temporal feature fluctuation data and facial expression action feature trajectory to obtain facial muscle dynamic activity data; Facial muscle dynamic activity data can be a temporal indicator that quantifies the activation intensity and duration of muscles in different facial regions during emotional expression. It can be used to reveal fine patterns of facial muscle participation under different emotional states, supporting the analysis of micro-expressions and atypical expressions. In one specific embodiment, facial muscle dynamic activity data can be calculated based on facial expression movement feature trajectories and temporal fluctuation data, using parameters such as amplitude, velocity, and acceleration to determine the activity level of local muscle groups. For example, facial muscle dynamic activity data can include, but is not limited to, one or more of the following: instantaneous burst-type activity, sustained low-amplitude activity, and synergistic-antagonistic activity. Facial muscle dynamic activity analysis based on temporal fluctuation data and facial expression movement feature trajectories yields facial muscle dynamic activity data, which can be obtained by fusing temporal fluctuation intensity and trajectory motion parameters (such as displacement and velocity) to calculate the activation level of muscles in each facial region per unit time. Furthermore, this operation can be achieved by generating an activity heatmap by weighted fusion of optical flow vector amplitude and temporal fluctuation amplitude, or by estimating muscle contraction rate and fatigue based on the first / second derivative of the action unit intensity curve. This allows for the extension from static muscle movement differences to dynamic activity process modeling, thereby improving sensitivity to transient or weak emotional signals.

[0045] Step S24: Quantify the differences in facial muscle movement in different areas of the face by analyzing the dynamic activity data of facial muscles to obtain facial muscle movement difference data. The dynamic activity data of facial muscles is quantified to measure the differences in facial muscle movement across different regions of the face. This results in facial muscle movement difference data, which can be obtained by comparing the dynamic activity values ​​of multiple anatomical regions of the face (such as upper / lower face, left / right half of face) and calculating their asymmetry, gradient, or covariance differences. In an exemplary embodiment, this operation can be achieved by calculating the ratio of activity levels at the left and right corners of the mouth as a criterion for pain expression, or by constructing a regional activity covariance matrix and extracting difference patterns through principal component analysis. This can capture the asymmetry and local dominance features of facial expressions in infants and young children due to their immature neurological development, thereby improving the specificity of emotion recognition.

[0046] Step S25: Estimate the risk of infant and toddler emotional abnormalities based on facial muscle movement difference data and expression temporal characteristic fluctuation data to obtain infant and toddler emotional abnormality risk data.

[0047] Based on facial muscle movement difference data and expression temporal feature fluctuation data, the risk of abnormal emotions in infants and young children is estimated, resulting in abnormal emotion risk data. This can be achieved by jointly inputting muscle movement differences and expression-physiological temporal co-features into a risk assessment model, outputting the abnormal emotion risk level. Furthermore, this operation can be implemented by designing a multi-input fusion network to process the difference features and temporal fluctuation features separately before concatenating them for classification, or by using graph neural networks to model the activity relationship between facial regions and combining it with physiological synchronicity for risk inference. This allows for the differentiation of emotional states that appear similar but have different underlying mechanisms (such as crying vs. pain), and the identification of potential risks such as persistent stress.

[0048] Taking pain monitoring in the neonatal ward as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be as follows: when a premature infant undergoes heel prick blood sampling, the system captures facial video through a high-definition camera, while simultaneously monitoring heart rate and skin conductance through a flexible patch. The system first extracts the movement trajectory of key points such as downward pressure of the brow and tightness of the lips; then it finds that the heart rate rises sharply at the moment of blood sampling, and simultaneously matches the high-frequency short fluctuations in the brow trajectory; combined with the fact that the dynamic activity of the orbicularis oculi muscle and perioral muscle group is significantly higher than other areas during this period, and is asymmetrical from left to right, the system determines that it is acute pain rather than ordinary irritability, triggers a high-risk alarm, and suggests that the nurse implement non-nutritive sucking intervention.

[0049] For example, in a home-based nighttime care scenario, the infant facial expression recognition and emotional state analysis method of this embodiment can be used when an infant suddenly cries at night, and the parents check on a smart monitor. The system analyzes the facial expression video, extracting the trajectory of rapid twitching of the corners of the mouth and eyelid tremors; synchronous physiological data shows no significant increase in skin conductance but a slight increase in heart rate; facial muscle dynamic activity analysis shows that only the perioral area is active while the perioral area is calm, and the two sides are symmetrical; combined with historical data, it is determined to be physiological night terrors rather than pathological discomfort, and the system generates a low-risk prompt and suggests gentle patting and soothing rather than immediate medication.

[0050] In one embodiment, step S23 includes the following steps: Step S231: Estimate the facial size of infants and young children from the facial expression video to obtain facial size estimation data; The infant facial size estimation data can be structured parameters reflecting the actual physical scale of an individual infant's face, used to characterize the relative size and proportion of different facial regions. In an exemplary embodiment, the infant facial size estimation data can be explained in context by performing geometric modeling or keypoint spacing analysis on the face in an infant's facial expression video to calculate scale indicators such as inter-eye distance, nose height, and face width. Furthermore, the infant facial size estimation data can provide a benchmark for the physical normalization of muscle movement amplitude, eliminating scale biases caused by age, shooting distance, or head posture. Exemplarily, the infant facial size estimation data can include, but is not limited to, one or more of the following: horizontal facial scale parameters, vertical facial scale parameters, and local region scale ratios. Estimating the infant facial size from an infant's facial expression video to obtain infant facial size estimation data can be achieved by detecting facial keypoints in video frames, calculating key anatomical distances (such as interpupillary distance and nasolabial distance), and estimating the overall and local scale of the face accordingly. In one specific embodiment, this operation can be achieved by obtaining the true scale through back-projection of key points based on a 3D face reconstruction model, or by using known reference objects (such as pacifiers or auricles) for relative scale calibration. This can achieve the technical effect of obtaining facial physical scale parameters that are suitable for the individual's developmental stage and providing a normalized benchmark for subsequent motion quantification.

[0051] Step S232: Based on the temporal feature fluctuation data of facial expressions and the facial size estimation data of infants and young children, the range of motion of different facial muscle groups of infants and young children is deduced to obtain the range of muscle motion of different facial muscle groups of infants and young children. The range of muscle movement amplitude for different facial expression muscle groups in infants and young children can be the upper and lower limits of the range of motion exhibited by each anatomically defined facial expression muscle group (such as the zygomaticus major, corrugator supercilii, and orbicularis oculi) under a specific facial expression state. In this embodiment, the range of muscle movement amplitude for different facial expression muscle groups in infants and young children can be derived by combining displacement information based on temporal characteristic fluctuation data of facial expressions with physical unit conversion of facial size estimation data. Furthermore, the range of muscle movement amplitude for different facial expression muscle groups in infants and young children can be a range of muscle activity with physiological significance by converting pixel-level movement, supporting comparability analysis across individuals and age groups. For example, the range of muscle movement amplitude for different facial expression muscle groups in infants and young children can include, but is not limited to, one or more of the following: the range of movement amplitude for upper facial muscles, the range of movement amplitude for lower facial muscles, and the range of movement amplitude for perioral muscles.

[0052] Based on temporal feature fluctuation data of facial expressions and facial size estimation data of infants and young children, the range of motion of different facial muscle groups in infants and young children is extrapolated. This can be achieved by converting pixel displacements in the temporal feature fluctuation data of facial expressions into physical units (such as millimeters) and dividing the range of motion by muscle group. Furthermore, this operation can be achieved by mapping keypoint displacements to muscle attachment points using a facial mesh deformation model to estimate the amplitude, or by using regional optical flow integrals combined with scale parameters to calculate local muscle contraction. This allows for physiological scale adaptation of muscle activity, making the range of motion comparable across individuals.

[0053] Step S233: Based on the estimated facial size data of infants and young children, perform a correlation coefficient analysis on the range of muscle movement amplitude to obtain the correlation coefficient of size-related movement intensity. The size-related motion intensity correlation coefficient can be a normalized intensity index describing the relationship between muscle movement amplitude and actual facial size. In an exemplary embodiment, the size-related motion intensity correlation coefficient can be described in context as using estimated facial size data of infants and young children as a benchmark, standardizing the muscle movement amplitude range, and calculating the relative motion intensity per unit scale. Furthermore, the size-related motion intensity correlation coefficient can eliminate misjudgments of motion intensity caused by individual differences in facial size or changes in camera distance, thereby improving the robustness of emotion recognition. Exemplarily, the size-related motion intensity correlation coefficient can include, but is not limited to, one or more of the following: local region normalization coefficient, global scale correction factor, dynamic proportional adjustment weight, etc. Analyzing the motion intensity correlation coefficient of the muscle movement amplitude range based on estimated facial size data of infants and young children to obtain the size-related motion intensity correlation coefficient can be achieved by normalizing the muscle movement amplitude with facial size as the denominator, generating a dimensionless relative intensity coefficient. In one specific embodiment, this operation can be achieved by independently calculating the local normalization coefficient for each facial region, or by constructing a scale-intensity mapping function to dynamically adjust the normalization strategy for different age groups. This can eliminate measurement biases caused by shooting distance, head orientation, or individual size, thereby improving the consistency of intensity assessment.

[0054] Step S234: Perform facial expression key point pose recognition on the facial expression action feature trajectory to obtain facial expression pose key point recognition data; The facial expression and posture key point recognition data can be a set of spatial configurations of key points representing specific facial expressions and postures (such as crying, laughing, and surprise) identified from facial expression action feature trajectories. In this embodiment, the facial expression and posture key point recognition data can be used to cluster or template match facial expression action feature trajectories in conjunction with contextual explanations to extract key point combinations with typical posture semantics and their spatial relationships. Furthermore, the facial expression and posture key point recognition data can serve as a basis for establishing a mapping between facial expressions and postures and muscle activation patterns, supporting differential analysis by posture category. For example, the facial expression and posture key point recognition data can include, but is not limited to, one or more of the following: crying posture key point configuration, smiling posture key point configuration, and painful frowning posture key point configuration.

[0055] Facial expression pose recognition is performed on the trajectory of facial expression features to obtain facial pose key point recognition data. This can be achieved by semantic clustering of the facial expression feature trajectory or matching it with a preset pose template to identify the typical facial expression pose. Furthermore, this operation can be implemented by using a Hidden Markov Model to segment the pose state of the trajectory sequence, or by constructing a pose embedding space through contrastive learning to achieve unsupervised pose recognition. This allows for the discretization of continuous trajectories into pose categories with emotional semantics, supporting differentiated modeling based on pose.

[0056] Step S235: Based on the facial expression and posture key point recognition data, extrapolate the distribution of muscle activation intensity among different facial expressions and postures of infants and young children in the range of muscle movement amplitude to obtain facial muscle activation intensity distribution data. The facial muscle activation intensity distribution data can be the relative intensity and spatial distribution pattern of different facial muscle groups activated under a specific facial expression posture. In an exemplary embodiment, the facial muscle activation intensity distribution data can be combined with contextual explanations based on facial expression posture key point identification data and muscle movement amplitude ranges to infer the degree of participation of each muscle group under that posture. Furthermore, the facial muscle activation intensity distribution data can reveal the muscle synergy and antagonism mechanisms in emotional expression, supporting fine-grained analysis of complex or mixed emotions. For example, the facial muscle activation intensity distribution data can include, but is not limited to, one or more of the following: dominant activation area distribution, auxiliary synergy area distribution, and inhibitory low-activity area distribution.

[0057] Based on facial expression and posture key point recognition data, the distribution of muscle activation intensity across different facial expressions and postures in infants and toddlers is extrapolated within the range of muscle movement amplitude. This yields facial expression muscle activation intensity distribution data, which can be obtained by combining the recognized facial expression and posture with the corresponding time period's muscle movement amplitude range to infer the activation dominance and synergistic relationship of each muscle group. In a specific embodiment, this operation can be achieved by mapping postures to standard muscle group activation patterns based on an action unit knowledge base, or by using graph neural networks to model the dependencies between key points to infer the potential muscle drive intensity. This achieves the technical effect of establishing a posture-muscle activation mapping and revealing the functional anatomy mechanisms under different emotional expressions.

[0058] Step S236: Perform facial muscle dynamic activity analysis based on the size-related motion intensity correlation coefficient and facial muscle activation intensity distribution data to obtain facial muscle dynamic activity data.

[0059] Facial muscle dynamic activity analysis is performed based on size-related motion intensity correlation coefficients and facial muscle activation intensity distribution data to obtain facial muscle dynamic activity data. This can be achieved by fusing normalized motion intensity coefficients with posture-related activation distributions and weighting them to generate a spatiotemporally consistent muscle activity sequence. Furthermore, this operation can be achieved by dynamically fusing scale coefficients and posture distribution weights using an attention mechanism, or by constructing a dual-branch network to separately process intensity normalization and posture semantics before feature fusion. This allows for the output of high-fidelity, physiologically meaningful dynamic activity data, supporting fine-grained emotion recognition and risk assessment.

[0060] Taking premature infant pain assessment as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be used to monitor a premature infant with a corrected age of 1 month in the NICU. First, the facial size is estimated to be only 70% of that of a full-term infant through facial video. During blood collection, the system identifies a "painful frowning" posture, with key points showing significant downward pressure between the eyebrows. Combining the range of muscle movement in this posture, it is found that the corrugator supercilii muscle movement reaches 90% of the individual's maximum range, and after size normalization, the intensity coefficient is much higher than that of full-term infants in the same posture. At the same time, the activation distribution shows that the orbicularis oculi and orbicularis oris muscles are synchronously highly activated. Based on this, the system judges it as severe pain and triggers a high-level intervention recommendation.

[0061] For example, in a family developmental behavior observation scenario, the infant facial expression recognition and emotional state analysis method of this embodiment can be used to identify a 6-month-old infant smiling when looking in a mirror. The system identifies the "social smile" posture, with key points showing that the corners of the mouth are outward but there is no obvious contraction around the eyes; facial size estimation shows that the face width is in the median for its age; the range of muscle movement amplitude shows that the zygomaticus major muscle is active but the orbicularis oculi muscle is almost inactive; after size normalization, the activation intensity distribution shows a typical "Duxiang smile absence" pattern, which the system records as a non-emotional smile, indicating to parents that this behavior is in the imitation stage and does not require intervention, but can be used as an indicator for tracking social development.

[0062] In one embodiment, step S3 includes the following steps: Step S31: Match the emotional state level output based on the temporal feature fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state classification data.

[0063] The emotional state rating data can be a quantitative output reflecting the current intensity or category of an infant's emotion, used to characterize the level of their emotion. In an exemplary embodiment, this data is generated by fusing time-series features of dynamic facial expression changes with differential information on local facial muscle activation patterns, serving as the basis for subsequent standardization and intervention decisions.

[0064] Step S32: Normalize the emotional state classification data to obtain normalized emotional state classification data.

[0065] Normalization can be a mathematical operation that converts data with different dimensions or value ranges into a unified scale range, thereby improving the compatibility and stability of multimodal features in joint decision-making. In this embodiment, normalization can be performed on the input data using a preset normalization algorithm, either linearly or nonlinearly. Furthermore, the normalized emotional state classification data can be a standardized result that maps the original emotional state classification data to a unified numerical range through mathematical transformation, eliminating inconsistencies in dimensions caused by individual differences, devices, or environments, and providing standardized input for multi-source data fusion. For example, the normalized emotional state classification data can include, but is not limited to, one or more of the following: Min-Max normalization results, probability distribution normalization results, and quantile normalization results. In a specific embodiment, normalizing the emotional state classification data to obtain normalized emotional state classification data can be achieved by inputting the emotional state classification data into a normalization module and outputting standardized data within a preset unified range (e.g., [0, 1]). Furthermore, this operation can be achieved by using Min-Max normalization to map the emotion level to the 0-1 interval, or by using the Softmax function to convert the discrete emotion category scores into a probability distribution, thereby eliminating the scaling bias in the emotion level output caused by differences in the model output layer structure, training sample distribution, or individual baselines.

[0066] Step S33: Based on the normalized data of emotional state classification, the data of differences in facial muscle movement, and the data on the risk of abnormal emotions in infants and young children, match the intervention strategies for infant and young children's emotions to obtain data on the regulation and control of infant and young children's emotions.

[0067] Among them, facial muscle movement difference data can be a feature quantity describing the asymmetry or dynamic changes in muscle activation on the left and right sides or different areas of an infant's face, used to capture subtle facial expression abnormalities. Infant emotional abnormality risk data can be a potential emotional health risk score based on physiological indicators, historical emotional records, or clinical assessments, used to identify the presence of high-risk emotional states. In an exemplary embodiment, the above three types of data together constitute a multi-dimensional decision-making basis. Further, based on emotional state grading normalization data, facial muscle movement difference data, and infant emotional abnormality risk data, infant emotional intervention strategies are matched to obtain infant emotional intervention regulation data. This can be achieved by using the three types of data as joint inputs to retrieve or generate the optimal intervention plan from a preset intervention strategy knowledge base. For example, this operation can be implemented by constructing a multi-input decision tree, triggering different intervention actions based on a combination of normalized emotional level thresholds, muscle asymmetry indicators, and risk scores, or by using a weighted fusion mechanism to weight the three types of data and input them into a strategy recommendation neural network to output intervention regulation instructions, thereby achieving refined and personalized intervention decisions that integrate emotional intensity, facial expression dynamics, and health risks.

[0068] Taking home nighttime care as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be as follows: An infant experiences brief crying during sleep at night, and the system collects emotional state rating data as "moderate irritability" (original score 0.72). However, because the infant's daily emotional baseline is relatively high, directly matching an intervention strategy might lead to overreaction. After normalization, the normalized emotional state rating data drops to 0.45 (reflecting a slight fluctuation relative to their normal state). Combined with facial muscle movement difference data showing only slight twitching around the mouth and no eyebrow tension, and the emotional abnormality risk data being low, the system ultimately matches a "delayed response + maintaining a quiet environment" strategy to avoid unnecessary awakening interventions.

[0069] For example, in the context of NICU monitoring of premature infants, the infant facial expression recognition and emotional state analysis method in this embodiment can be used when a premature infant exhibits persistent frowning and elevated heart rate after nursing procedures. The original emotional state rating is "high discomfort" (0.89), which remains high after normalization (0.82). Facial muscle movement difference data reveals significant asymmetrical activation of the left and right sides of the face, and emotional abnormality risk data indicates a moderate to high risk of pain. The system integrates these three pieces of information to match composite intervention and control data of "immediately pausing the procedure + playing soothing music + notifying the nurse to assess pain," achieving a precise clinical response.

[0070] In one embodiment, step S31 includes the following steps: Step S311: Plot the temporal fluctuation curve of the facial expression temporal feature data to obtain the temporal fluctuation curve of the facial expression feature intensity; The temporal fluctuation curve of facial expression feature intensity can be a continuous curve reflecting the changing trend of facial expression feature activation intensity, plotted sequentially over time from the temporal fluctuation data. It can provide a quantifiable time-series representation basis for emotional dynamic modeling and support subsequent temporal pattern analysis. In this embodiment, the temporal fluctuation curve of facial expression feature intensity can perform serialized visualization or numerical representation of the feature intensity values ​​at each time point in the temporal fluctuation data. For example, the temporal fluctuation curve of facial expression feature intensity can include, but is not limited to, one or more of the following: periorbital AU intensity temporal curve, oral AU intensity temporal curve, and full-face comprehensive intensity temporal curve.

[0071] Plotting the temporal fluctuation curves of facial expression features yields the facial expression feature intensity temporal fluctuation curves. This can be achieved by sorting the facial expression temporal fluctuation data by timestamp and extracting its intensity dimension to form a continuous function or a discrete sequence. Furthermore, this operation can be implemented by plotting a temporal curve for the intensity of each action unit (AU) individually, or by plotting a single temporal curve using the L2 norm of the comprehensive feature vector of the entire face as the overall intensity. This transforms abstract temporal features into analyzable intensity-time relationship curves, supporting subsequent dynamic modeling.

[0072] Step S312: Calculate the instantaneous intensity rise smoothness of the facial expression feature intensity temporal fluctuation curve to obtain the instantaneous intensity rise smoothness of the facial expression feature. The instantaneous intensity rise smoothness of facial features can be a temporal indicator measuring the stability of the rate of change of facial feature intensity during the rising phase. It can be used to distinguish between sudden emotional responses (such as fright) and gradual emotional accumulation (such as irritability), improving the ability to discern the mechanism of emotion initiation. In an exemplary embodiment, the instantaneous intensity rise smoothness of facial features can be calculated based on the temporal fluctuation curve of facial feature intensity, by calculating the local variance or slope change rate of the first or second derivative during the intensity rising interval. Calculating the instantaneous intensity rise smoothness of the temporal fluctuation curve of facial feature intensity yields the instantaneous intensity rise smoothness, which can be a stability indicator of the local rate of change calculated in the rising segment of the curve (the monotonically increasing intensity interval). Furthermore, this operation can be achieved by using a sliding window to calculate the standard deviation of the intensity increment between adjacent frames as a smoothness indicator, or by fitting a local cubic spline curve and integrating its absolute second derivative to measure smoothness, thereby quantifying the speed of emotion initiation and helping to distinguish between stimulus type and emotion nature.

[0073] Step S313: Based on the smoothness of the instantaneous intensity rise of facial expression features, perform multi-peak region pattern recognition on the temporal fluctuation curve of facial expression feature intensity to obtain the multi-peak region pattern of facial expression feature intensity; The multi-peak region pattern of facial expression intensity can be identified from the temporal fluctuation curve of facial expression intensity, containing emotional expression segments with multiple local maxima and their structured descriptions. This can be used to capture common emotional repetitions and interruptions in infants and young children, avoiding the simplification of complex emotions into a single state. In one specific embodiment, the multi-peak region pattern of facial expression intensity can be combined with the instantaneous intensity rise smoothness of facial expressions as a threshold condition to detect multi-peak regions on the temporal curve that satisfy a specific rise-fall pattern. The multi-peak region pattern of facial expression intensity is obtained by performing feature intensity multi-peak region pattern recognition on the temporal fluctuation curve of facial expression intensity based on the instantaneous intensity rise smoothness. This can be achieved by identifying continuous time periods on the temporal curve that satisfy a multi-peak structure, using the instantaneous intensity rise smoothness as a constraint. Furthermore, this operation can be implemented by setting a smoothness threshold, allowing new peaks to be identified as independent emotional events only when the smoothness of the rising segment is below the threshold, or by using a Hidden Markov Model (HMM) to model the multi-peak state transition and combining it with smoothness as the observation probability input. This can accurately locate the unique multiple fluctuating emotional expression segments of infants and young children, avoiding misjudging intermittent crying as a single emotional event.

[0074] Step S314: Perform dynamic matching of emotion levels based on the multi-peak region pattern of facial expression feature intensity to obtain dynamic emotion classification data; The dynamic emotion grading data can be a fine-grained grading sequence reflecting the evolution of emotions over time, obtained by mapping multi-peak region patterns based on facial expression feature intensity. This allows for continuous characterization of the emotion's evolutionary trajectory, rather than simply outputting static snapshot-style classification results. In this embodiment, the dynamic emotion grading data can input multi-peak region patterns into a dynamic emotion grading matching model, outputting an emotion category or intensity grading sequence for the corresponding time period. For example, the dynamic emotion grading data can include, but is not limited to, grading sequences for the rising phase of emotions, the plateau phase of emotions, and the falling phase of emotions. Dynamic emotion grading data is obtained by dynamically matching emotion grading based on multi-peak region patterns of facial expression feature intensity. This can be achieved by matching the identified multi-peak region patterns with a predefined dynamic emotion template library, outputting an emotion grading sequence for the corresponding time period. Furthermore, this operation can be implemented by using Dynamic Time Warping (DTW) to align the multi-peak patterns with standard emotion templates before classification, or by training a temporal classification network (such as TCN or Transformer) to directly predict the emotion grading sequence from the multi-peak regions. This allows for fine-grained annotation of the entire emotion evolution process, supporting dynamic intervention timing judgment.

[0075] Step S315: Perform facial expression feature weight matching based on facial muscle movement difference data, and perform multi-feature weighted optimization on the emotional dynamic grading data to obtain multi-feature weighted emotional grading data; The multi-feature weighted emotion grading data can be a weighted emotion level result obtained by optimizing the dynamic emotion grading data after incorporating regional weights assigned by facial muscle movement difference data. This can be used to enhance the physiological rationality and individual adaptability of emotion grading, making the classification results more closely match the unique facial expression patterns of infants and young children. In an exemplary embodiment, the multi-feature weighted emotion grading data can determine the contribution weight of different expression regions to the current emotion type based on facial muscle movement difference data, and perform weighted averaging or attention redistribution on the dynamic emotion grading data. Exemplarily, the multi-feature weighted emotion grading data may include, but is not limited to, regional attention weighting results, anatomical weight fusion results, and dynamic confidence modulation results.

[0076] Facial muscle movement difference data is used to match the weights of facial expression features, and multi-feature weighted optimization is applied to the dynamic emotion grading data to obtain multi-feature weighted emotion grading data. This can be achieved by dynamically adjusting the weights of different facial expression features in the emotion grading based on the activation level of each region in the facial muscle movement difference data, and recalculating the final emotion level. Furthermore, this operation can be implemented by constructing a region weight mapping table (e.g., increasing the weight of "pain" emotions when there are large differences in the glabella muscles), or by using an attention mechanism to use muscle movement difference data as the time step in the dynamically weighted emotion grading data, thereby making the emotion grading results more consistent with the facial physiological characteristics of infants and young children and improving sensitivity to subtle differences.

[0077] Step S316: Match the emotional state level output based on the emotional dynamic grading data and the multi-feature weighted emotional grading data to obtain emotional state grading data.

[0078] Emotional state level output is matched based on dynamic emotion grading data and multi-feature weighted emotion grading data to obtain emotion state grading data. This can be achieved by fusing the original dynamic grading and weighted optimization results, and generating the final emotion state grading through voting, weighted averaging, or gating mechanisms. Furthermore, this operation can be achieved by weighted fusion of the two (with weights dynamically adjusted by the current signal-to-noise ratio or occlusion level), or by using an ensemble learning strategy to input both grading results as base model inputs to the meta-classifier to output the final grading. This approach can balance the temporal continuity of emotion evolution with the physiological rationality of multi-region features, improving output stability and accuracy.

[0079] Taking neonatal pain assessment as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be used to record the facial expressions of premature infants when they undergo heel prick blood sampling. The temporal fluctuation curve of facial expression feature intensity shows three obvious peaks, with an interval of about 8 seconds; the calculation of the instantaneous intensity rise smoothness shows that each rise is extremely steep (low smoothness), which is consistent with the characteristics of acute pain; the multi-peak area pattern recognition confirms "repetitive severe pain response"; the emotional dynamic grading data output is a continuous "severe pain" level; at the same time, the facial muscle movement difference data shows that the movement in the glabella area is significantly stronger than that around the mouth. Based on this, the system increases the AU weight of the glabella in the facial expression feature weight matching. After multi-feature weighted optimization, the final emotional state grading data is stably output as "high-risk pain", triggering the immediate intervention process.

[0080] For example, in the scenario of monitoring irritability at home, the facial expression recognition and emotional state analysis method for infants and young children in this embodiment can be as follows: a 6-month-old infant experiences persistent restlessness in the evening, with the temporal fluctuation curve of facial feature intensity showing a slow rise followed by multi-peak oscillations; the instantaneous intensity rise has a high smoothness, indicating that it is not a sudden stimulus; the multi-peak region pattern is identified as "progressive irritability with intermittent relief"; the emotional dynamic grading data reflects a gradual increase from moderate discomfort to high irritability and then a decline; facial muscle movement difference data shows that the corners of the mouth are mainly trembling and the eyebrow movement is weak, and the system reduces the weight of pain-related factors and increases the weight of anxiety in the weight matching; the multi-feature weighted emotional grading data emphasizes perioral dynamics; the final emotional state grading data output is "moderate to high anxiety irritability", suggesting playing white noise and reducing visual stimulation, rather than medical intervention.

[0081] In one embodiment, step S4 includes the following steps: Step S41: Construct a labeled training sample set based on the emotional state classification data and infant emotional intervention and regulation data, and train and iterate the initial convolutional neural network emotion analysis model to obtain the trained automatic analysis model of infant emotional state. The initial convolutional neural network (CNN) sentiment analysis model can be an untrained or only pre-trained CNN architecture used for analyzing the emotional states of infants and toddlers. It can serve as the starting point for model training, receiving multimodal inputs and outputting emotion-related predictions. In this embodiment, the initial CNN sentiment analysis model can be structurally adapted based on general image classification or facial expression recognition networks (such as ResNet or EfficientNet), retaining convolutional layer designs suitable for facial feature extraction in infants and toddlers. For example, the initial CNN sentiment analysis model can include, but is not limited to, one or more of the following: 2D-CNN-based sentiment analysis models, spatiotemporal 3D-CNN sentiment modeling networks, and multi-branch dual-stream CNN architectures.

[0082] The labeled training sample set can be a labeled supervised learning dataset jointly composed of emotion state classification data and infant emotion intervention and regulation data. This dataset can be used to support end-to-end training of the model, enabling it to simultaneously learn emotion classification, risk assessment, and intervention recommendation capabilities. In one exemplary embodiment, the labeled training sample set can use emotion state classification as the primary label and infant emotion intervention and regulation data as the secondary label or task objective, binding them to corresponding multimodal inputs (facial expression video clips, physiological signals) to form sample pairs. Furthermore, the labeled training sample set can be used to iteratively train and optimize the initial convolutional neural network emotion analysis model. The trained automatic infant emotion state analysis model can be a neural network model that has converged after sufficient training on the labeled training sample set and possesses complete emotion analysis capabilities. It can be used to achieve high-precision infant emotion state recognition and risk assessment, but may be unsuitable for direct deployment on resource-constrained devices due to its large number of parameters. In a specific embodiment, the trained automatic infant emotion state analysis model can iteratively update parameters on the labeled training sample set through backpropagation and optimization algorithms until performance stabilizes. Furthermore, the trained automatic infant emotion state analysis model can be used as input for lightweight pruning and compression processing.

[0083] A labeled training sample set is constructed based on emotional state grading data and infant emotional intervention and regulation data. This can be achieved by using emotional state grading as the primary task label and infant emotional intervention and regulation data as the secondary task label, structurally binding them with corresponding multimodal input data (facial expression video clips, physiological signals). Further, this operation can be implemented by using a multi-task learning format to encode emotional levels and intervention strategies into joint label vectors, or by constructing graph-structured samples, with emotional states as nodes and intervention strategies as edges, forming a semantically related training set. This allows the model to learn clinically interpretable emotion-behavior mapping relationships. The initial convolutional neural network emotion analysis model is then trained and iteratively optimized to obtain a trained automatic infant emotional state analysis model. This can be achieved by using the labeled training sample set to perform end-to-end training on the initial model, jointly optimizing the emotion classification accuracy and intervention strategy matching degree through a loss function. Further, this operation can be achieved by employing a curriculum learning strategy, first training the emotion classification and then introducing intervention tasks for fine-tuning, or by using adversarial training to enhance the model's robustness to interferences such as lighting and occlusion. This results in a customized emotion understanding model adapted to the infant population, embedding risk assessment and intervention logic.

[0084] Step S42: Perform lightweight pruning and compression on the trained automatic analysis model of infant and toddler emotional state to adapt to the requirements of edge-cloud collaborative deployment and obtain a deployable lightweight automatic analysis model of infant and toddler emotional state. Lightweight pruning and compression processing can be a technical process for simplifying the structure and optimizing the parameters of a neural network model. This can reduce the model's computational complexity and storage footprint, adapting to the deployment requirements of edge devices. In one specific embodiment, lightweight pruning and compression processing can remove redundant computational units through methods such as channel pruning, weight sparsity, knowledge distillation, or quantization. For example, lightweight pruning and compression processing can include, but is not limited to, one or more of structured channel pruning, unstructured weight sparsity, and mixed-precision quantization. The deployable lightweight automatic infant emotion state analysis model can be an optimized emotion analysis model that has undergone lightweight pruning and compression processing and meets the requirements for edge-cloud collaborative deployment. It can be used to support real-time initial emotion screening on edge terminals and collaboratively complete complex analysis tasks with the cloud. In this embodiment, the deployable lightweight automatic infant emotion state analysis model can significantly reduce the number of parameters and inference latency while retaining key emotion discrimination capabilities. Furthermore, the deployable lightweight automatic infant emotion state analysis model can be sent to an infant cloud care platform and interface with a real-time infant care system.

[0085] A lightweight pruning and compression process is applied to the trained automatic emotional state analysis model for infants and toddlers to adapt to edge-cloud collaborative deployment requirements, resulting in a deployable lightweight automatic emotional state analysis model for infants and toddlers. This can involve analyzing the sensitivity of each layer of the model, removing convolutional channels or neurons with minor impact on the output, and performing parameter quantization or knowledge distillation. Furthermore, this operation can be achieved by performing structured pruning after evaluating the channel importance based on gradient magnitude, or by using a teacher-student framework where a large model guides a smaller network to learn its output distribution. This significantly reduces computational overhead while maintaining core emotion discrimination capabilities, supporting real-time inference on edge devices.

[0086] Step S43: Send the lightweight infant and toddler emotional state automatic analysis model to the infant and toddler cloud care platform and connect it with the real-time infant and toddler care system of the family or institution to run the emotional state analysis method based on infant and toddler facial expression recognition.

[0087] The infant and toddler cloud care platform can be a cloud service platform specifically designed for infant and toddler care scenarios, integrating emotion analysis, data management, and remote interaction functions. It can serve as a deployment platform for lightweight models, supporting concurrent emotion analysis requests from multiple users and terminals. In an exemplary embodiment, the infant and toddler cloud care platform can be built on cloud computing infrastructure, providing API interfaces, user management, model scheduling, and data storage services. The real-time infant and toddler care system in a home or institution can be a hardware and software integrated system deployed locally in the home or childcare institution for continuous monitoring of the infant's status. It can collect raw data and receive emotion analysis results returned by the cloud platform, executing intervention suggestions or alarms. In this embodiment, the real-time infant and toddler care system in a home or institution can include a camera, wearable sensors, a local processing unit, and a user interface, maintaining data synchronization with the cloud platform. Furthermore, the real-time infant and toddler care system in a home or institution can interface with the infant and toddler cloud care platform to run an emotion state analysis method based on a lightweight model.

[0088] A lightweight, automated infant and toddler emotional state analysis model is sent to an infant and toddler cloud care platform and integrated with the real-time infant and toddler care system in homes or institutions. This allows for the execution of an emotional state analysis method based on infant facial expression recognition. The model can be packaged as a standardized service module, integrated into the cloud platform via API or SDK, and configured with a data communication protocol for the local care system. Furthermore, this operation can deploy the model to a Kubernetes cluster on the cloud platform using a container image, providing a RESTful API for the care system to call. Alternatively, the lightweight model can be simultaneously distributed to an edge gateway for initial local analysis, with only key events uploaded to the cloud. This enables seamless deployment of the model from development to practical application, supporting large-scale, multi-scenario deployment.

[0089] Taking intelligent monitoring in a hospital's neonatal ward as an example, the infant facial expression recognition and emotional state analysis method in this embodiment can be implemented in the NICU by systematically collecting facial videos and electrocardiogram-dermal conductance signals of premature infants to construct training samples labeled "pain level 3 + requiring non-pharmacological soothing." After initial training on this sample set, the CNN model can accurately identify the pain state represented by the combination of subtle frowning and heart rate fluctuations. Subsequently, the model is compressed to 30% of its original size through channel pruning and deployed on the hospital's private cloud care platform. The intelligent bedside terminal in the ward loads this lightweight model to analyze the infant's emotions in real time. When high-risk pain is detected, it automatically adjusts the incubator lighting and notifies the nurses, achieving low-latency closed-loop intervention.

[0090] For example, in a family-community collaborative care scenario, the infant facial expression recognition and emotional state analysis method in this embodiment can be implemented by parents collecting data through a home smart camera and flexible wristband. The local device runs a lightweight model to perform real-time preliminary screening of emotions (such as distinguishing between normal crying and abnormal irritability). If a high-risk state is determined, the encrypted feature data is uploaded to the community infant cloud care platform, where a complete model in the cloud verifies the data and generates professional intervention suggestions (such as recommending medical treatment or playing specific soothing audio). These suggestions are simultaneously pushed to the family APP and the community pediatrician's terminal, forming a family-community collaborative intervention link.

[0091] In addition, refer to Figure 2 To achieve the above objectives, the present invention also provides an infant facial expression recognition and emotion state analysis system, the system comprising: The multi-source acquisition module 10 is used to acquire facial expressions of infants and young children under monitoring conditions through image acquisition devices to obtain facial expression videos of infants and young children; based on the facial expression videos of infants and young children, and through wearable physiological sensors, to monitor the emotion-related physiological parameters of infants and young children under monitoring conditions to obtain emotion-related physiological monitoring data of infants and young children. The feature quantization module 20 is used to match the temporal feature fluctuations of facial expressions based on the physiological monitoring data related to infant emotions, and obtain the temporal feature fluctuation data of facial expressions; to quantify the differences in facial muscle movements in different areas of the face based on the temporal feature fluctuation data of facial expressions, and obtain the facial muscle movement difference data; and to estimate the risk of abnormal emotions in infants and young children based on the facial muscle movement difference data, and obtain the risk data of abnormal emotions in infants and young children. The graded intervention module 30 is used to match the emotional state level output based on the temporal characteristic fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state graded data; and to match the emotional intervention strategies for infants and young children based on the emotional state graded data and the risk data of abnormal emotions in infants and young children to obtain emotional intervention and regulation data for infants and young children. The cloud deployment module 40 is used to construct an automatic analysis model of infant emotions based on emotional state classification data and infant emotion intervention and regulation data, obtain an automatic analysis model of infant emotions, and deploy the automatic analysis model of infant emotions to the cloud platform to execute the emotional state analysis method based on infant facial expression recognition.

[0092] Other embodiments or specific implementations of the infant facial expression recognition and emotion state analysis system of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0093] In addition, to achieve the above objectives, the present invention also provides an infant facial expression recognition and emotion state analysis device, the device comprising: a memory, a processor, and an infant facial expression recognition and emotion state analysis program stored in the memory and executable on the processor, the infant facial expression recognition and emotion state analysis program being configured to implement the steps of the infant facial expression recognition and emotion state analysis method as described in any one of the above descriptions.

[0094] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an infant facial expression recognition and emotion state analysis program, wherein when the infant facial expression recognition and emotion state analysis program is executed by a processor, it implements the steps of the infant facial expression recognition and emotion state analysis method as described above.

[0095] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for recognizing facial expressions and analyzing emotional states in infants and young children, characterized in that, The method includes: Step S1: Collect facial expressions of infants and young children under monitoring conditions using an image acquisition device to obtain facial expression videos of infants and young children; based on the facial expression videos of infants and young children, monitor the emotion-related physiological parameters of infants and young children under monitoring conditions using wearable physiological sensors to obtain emotion-related physiological monitoring data of infants and young children. Step S2: Match the temporal feature fluctuations of facial expressions and movements based on the physiological monitoring data related to infant emotions to obtain temporal feature fluctuation data; quantify the differences in facial muscle movements in different areas of the face based on the temporal feature fluctuation data to obtain facial muscle movement difference data; estimate the risk of abnormal emotions in infants and young children based on the facial muscle movement difference data to obtain abnormal emotional risk data in infants and young children. Step S3: Match the emotional state level output based on the facial expression time sequence feature fluctuation data and facial muscle movement difference data to obtain emotional state classification data; match infant and toddler emotional intervention strategies based on emotional state classification data and infant and toddler emotional abnormality risk data to obtain infant and toddler emotional intervention and regulation data. Step S4: Based on the emotional state classification data and infant emotional intervention and regulation data, construct an automatic analysis model for infant emotions to obtain an automatic analysis model for infant emotional states. Deploy the automatic analysis model for infant emotional states to a cloud platform to execute an emotional state analysis method based on infant facial expression recognition.

2. The method for recognizing facial expressions and analyzing emotional states in infants and young children as described in claim 1, characterized in that, Step S1 includes the following steps: Step S11: Deploy image acquisition devices and wearable physiological sensors in daily monitoring scenarios for infants and young children; Step S12: Capture the facial expressions of the infants under monitoring conditions using an image acquisition device to obtain a video of the infants' facial expressions; Step S13: Perform time-series frame-segmentation processing on the infant's facial expression video to obtain continuous frames of the infant's facial expression; Step S14: Based on continuous frame segmentation of infants' facial expressions, and through wearable physiological sensors, monitor the emotion-related physiological parameters of infants under monitoring conditions to obtain infant emotion-related physiological monitoring data.

3. The method for infant facial expression recognition and emotional state analysis as described in claim 1, characterized in that, Step S2 includes the following steps: Step S21: Extract facial expression and motion feature trajectories from the infant's facial expression video to obtain the facial expression and motion feature trajectories; Step S22: Based on the physiological monitoring data related to infant emotions, perform temporal feature fluctuation matching on the facial expression and movement feature trajectory to obtain facial expression temporal feature fluctuation data; Step S23: Perform facial muscle dynamic activity analysis based on facial expression temporal feature fluctuation data and facial expression action feature trajectory to obtain facial muscle dynamic activity data; Step S24: Quantify the differences in facial muscle movement in different areas of the face by analyzing the dynamic activity data of facial muscles to obtain facial muscle movement difference data. Step S25: Estimate the risk of infant and toddler emotional abnormalities based on facial muscle movement difference data and expression temporal characteristic fluctuation data to obtain infant and toddler emotional abnormality risk data.

4. The method for recognizing facial expressions and analyzing emotional states in infants and young children as described in claim 3, characterized in that, Step S23 includes the following steps: Step S231: Estimate the facial size of infants and young children from the facial expression video to obtain facial size estimation data; Step S232: Based on the temporal feature fluctuation data of facial expressions and the facial size estimation data of infants and young children, the range of motion of different facial muscle groups of infants and young children is deduced to obtain the range of muscle motion of different facial muscle groups of infants and young children. Step S233: Based on the estimated facial size data of infants and young children, perform a correlation coefficient analysis on the range of muscle movement amplitude to obtain the correlation coefficient of size-related movement intensity. Step S234: Perform facial expression key point pose recognition on the facial expression action feature trajectory to obtain facial expression pose key point recognition data; Step S235: Based on the facial expression and posture key point recognition data, extrapolate the distribution of muscle activation intensity among different facial expressions and postures of infants and young children in the range of muscle movement amplitude to obtain facial muscle activation intensity distribution data. Step S236: Perform facial muscle dynamic activity analysis based on the size-related motion intensity correlation coefficient and facial muscle activation intensity distribution data to obtain facial muscle dynamic activity data.

5. The method for recognizing facial expressions and analyzing emotional states in infants and young children as described in claim 1, characterized in that, Step S3 includes the following steps: Step S31: Match the emotional state level output based on the temporal feature fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state classification data; Step S32: Normalize the emotional state classification data to obtain normalized emotional state classification data. Step S33: Based on the normalized data of emotional state classification, the data of differences in facial muscle movement, and the data on the risk of abnormal emotions in infants and young children, match the intervention strategies for infant and young children's emotions to obtain data on the regulation and control of infant and young children's emotions.

6. The method for infant facial expression recognition and emotional state analysis as described in claim 5, characterized in that, Step S31 includes the following steps: Step S311: Plot the temporal fluctuation curve of the facial expression temporal feature data to obtain the temporal fluctuation curve of the facial expression feature intensity; Step S312: Calculate the instantaneous intensity rise smoothness of the facial expression feature intensity temporal fluctuation curve to obtain the instantaneous intensity rise smoothness of the facial expression feature. Step S313: Based on the smoothness of the instantaneous intensity rise of facial expression features, perform multi-peak region pattern recognition on the temporal fluctuation curve of facial expression feature intensity to obtain the multi-peak region pattern of facial expression feature intensity; Step S314: Perform dynamic matching of emotion levels based on the multi-peak region pattern of facial expression feature intensity to obtain dynamic emotion classification data; Step S315: Perform facial expression feature weight matching based on facial muscle movement difference data, and perform multi-feature weighted optimization on the emotional dynamic grading data to obtain multi-feature weighted emotional grading data; Step S316: Match the emotional state level output based on the emotional dynamic grading data and the multi-feature weighted emotional grading data to obtain emotional state grading data.

7. The method for infant facial expression recognition and emotional state analysis as described in claim 1, characterized in that, Step S4 includes the following steps: Step S41: Construct a labeled training sample set based on the emotional state classification data and infant emotional intervention and regulation data, and train and iterate the initial convolutional neural network emotion analysis model to obtain the trained automatic analysis model of infant emotional state. Step S42: Perform lightweight pruning and compression on the trained automatic analysis model of infant and toddler emotional state to adapt to the requirements of edge-cloud collaborative deployment and obtain a deployable lightweight automatic analysis model of infant and toddler emotional state. Step S43: Send the lightweight infant and toddler emotional state automatic analysis model to the infant and toddler cloud care platform and connect it with the real-time infant and toddler care system of the family or institution to run the emotional state analysis method based on infant and toddler facial expression recognition.

8. A facial expression recognition and emotion state analysis system for infants and young children, characterized in that, The system includes: The multi-source acquisition module is used to capture facial expressions of infants and young children under monitoring conditions through image acquisition devices to obtain facial expression videos of infants and young children; based on the facial expression videos of infants and young children, and through wearable physiological sensors, to monitor the emotion-related physiological parameters of infants and young children under monitoring conditions to obtain emotion-related physiological monitoring data of infants and young children. The feature quantization module is used to match the temporal feature fluctuations of facial expressions based on physiological monitoring data related to infant emotions, and obtain temporal feature fluctuation data of facial expressions; based on the temporal feature fluctuation data of facial expressions, it quantifies the differences in facial muscle movements in different areas of the face, and obtains facial muscle movement difference data; based on the facial muscle movement difference data, it estimates the risk of abnormal emotions in infants and young children, and obtains risk data of abnormal emotions in infants and young children. The graded intervention module is used to match the emotional state level output based on the temporal characteristic fluctuation data of facial expressions and the difference data of facial muscle movements to obtain emotional state graded data; and to match the emotional state graded data and the risk data of abnormal emotions in infants and young children to obtain emotional intervention and regulation data for infants and young children. The cloud deployment module is used to build an automatic analysis model of infant emotions based on emotional state classification data and infant emotion intervention and regulation data. The resulting automatic analysis model of infant emotions is then deployed to the cloud platform to execute an emotional state analysis method based on infant facial expression recognition.

9. A device for recognizing facial expressions and analyzing emotional states in infants and young children, characterized in that, The device includes: a memory, a processor, and an infant facial expression recognition and emotion state analysis program stored in the memory and executable on the processor, the infant facial expression recognition and emotion state analysis program being configured to implement the steps of the infant facial expression recognition and emotion state analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an infant facial expression recognition and emotion state analysis program, which, when executed by a processor, implements the steps of the infant facial expression recognition and emotion state analysis method as described in any one of claims 1 to 7.

Images