Intelligent infant care system based on parent identity cloning and multi-modal emotion recognition

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an intelligent infant care system based on parent identity cloning and multi-modal emotion recognition, and the system comprises an identity cloning module which is used for generating a three-dimensional face model and a sound cloning body of a parent, and constructing an accompanying body; the emotion recognition module is used for recognizing the state of the infant; the interaction module is used for calling an accompanying body after the state of the infant is recognized, and performing emotional response interaction with the sound and / or facial image of parents; and the data storage module is used for performing on-chain encryption storage on the growth data and the interaction records of the infants by utilizing an SM4 encryption algorithm and a block chain technology. The infant emotion accompanying and real-time nursing system realizes double breakthrough of infant emotion accompanying and real-time nursing, and ensures data privacy security at the same time.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an intelligent infant care system based on parent identity cloning and multimodal emotion recognition. Background Technology

[0002] With societal development and the prevalence of dual-income families, the quality of early care for infants aged 0-2 years in childcare institutions has increasingly become a focus of social attention. Infants in this age group are in a critical period of emotional attachment and the formation of a sense of security; their core needs extend far beyond physical care, encompassing timely and appropriate emotional responses. This "responsive care" is crucial for children's mental health, cognitive development, and socio-emotional growth. However, existing technological solutions in childcare institutions and home settings have significant shortcomings in meeting this higher-level need.

[0003] Currently, most smart care technologies on the market remain at the level of "security monitoring." A common approach is a physiological parameter monitoring system based on cameras or wearable sensors. Its function focuses on monitoring infants' rolling over, crying, or abnormal vital signs, and sending alarm notifications to caregivers or parents after triggering thresholds. These systems suffer from a significant "emotional deficiency": they can only passively and belatedly report "what happened," unable to understand "why" the infant is crying, let alone provide any substantial emotional comfort in the immediate future. Caregivers often take several seconds or even longer to arrive at the scene after receiving an alarm, missing the golden window for emotional support, leading to prolonged separation anxiety and increased nighttime awakenings in infants. Furthermore, existing technologies face serious challenges in data processing and privacy protection. Large amounts of infant video, audio, and behavioral data are collected and stored on centralized servers, containing highly sensitive biometric information. Traditional storage methods pose risks of data leakage and misuse, making parents hesitant to entrust their children's data to institutions, which greatly hinders the widespread adoption and application of intelligent care systems.

[0004] Therefore, it is necessary to provide an intelligent infant care system based on parental identity cloning and multimodal emotion recognition to address the current challenges in intelligent care technology, which mainly focuses on security monitoring, lacks emotional understanding and timely comforting functions, and faces challenges in data processing and privacy protection. Summary of the Invention

[0005] In view of this, the present invention proposes an intelligent infant care system based on parent identity cloning and multimodal emotion recognition, aiming to solve the problems that current intelligent care technologies mainly focus on security monitoring, lack emotional understanding and timely comforting functions, and face challenges in data processing and privacy protection.

[0006] This invention proposes an intelligent infant care system based on parental identity cloning and multimodal emotion recognition, comprising: The identity cloning module is used to generate a 3D facial model and voice clone of the parents by taking at least one parent's facial image and voice sample as input, thus constructing a companion entity; The emotion recognition module is used to process the micro-expression optical flow information, the MFCC features of crying, and the motion information detected by millimeter-wave radar in infants and young children through pre-fusion, and to use the edge lightweight Transformer model to classify needs and identify the state of infants and young children. The interaction module is used to invoke the companion after recognizing the infant's state, and to provide emotional response interaction with the parents' voice and / or facial image; The data storage module is used to encrypt and store infants' growth data and interaction records on the blockchain using the SM4 encryption algorithm and blockchain technology.

[0007] Furthermore, the identity cloning module is used to generate a 3D facial model and a voice clone of the parents from at least one input parent facial image and voice sample. When constructing the companion entity, it includes: Based on at least one parent's facial image, a lightweight 3D deformation model is used to reconstruct the facial geometry and texture on the terminal device, generating a real-time driven 3D facial model of the parents. At the same time, voice cloning technology is used to extract speaker features by analyzing the parents' voice samples and synthesize voices with the parents' timbre characteristics by combining them with a vocoder on the terminal side. The generation of the 3D facial model and the voice cloning process are processed in parallel on the terminal side.

[0008] Furthermore, when the emotion recognition module processes the micro-expression optical flow information, the MFCC features of crying sounds, and the motion information detected by millimeter-wave radar in an infant using a pre-fusion method, it includes: The facial motion unit intensity features of the micro-expression optical flow information, the fundamental frequency and formant parameter features of the crying MFCC features, and the body motion frequency and amplitude features of the movement information are extracted and spliced ​​in the early stage to form a unified multi-dimensional temporal feature sequence.

[0009] Furthermore, the early splicing includes: Feature synchronization and alignment: Based on a unified timestamp, the micro-expression optical flow information, the MFCC features of crying sounds, and the motion information detected by millimeter-wave radar are temporally aligned, and features within the same time period are extracted using a common sampling time window as the unit. Feature standardization involves Z-score standardization of the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features extracted from micro-expression optical flow information, crying sound MFCC features, and motion information detected by millimeter-wave radar, so that all features are on the same order of magnitude. Vector concatenation and sequence construction involves concatenating the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features that have undergone Z-score normalization along the feature dimension to form a fused feature vector. The fused feature vectors of multiple consecutive time windows are then arranged in chronological order to form the multidimensional temporal feature sequence.

[0010] Furthermore, the emotion recognition module is used to perform demand classification using a lightweight Transformer model on the edge. When recognizing the state of infants and young children, it includes: The multidimensional temporal feature sequence is input into the edge lightweight Transformer model, and each fused feature vector in the multidimensional temporal feature sequence is mapped to a high-dimensional representation through the edge lightweight Transformer model embedding layer; By utilizing the multi-head self-attention mechanism in the Transformer encoder, the correlation weights between feature representations at different time steps within a multidimensional temporal feature sequence are calculated to capture the long-term dependencies of cross-modal features. After the output of the self-attention mechanism is processed by the feedforward neural network, the feature representation of the last time step of the multi-dimensional temporal feature sequence is taken as the global context representation. Finally, the global context representation is mapped to a preset demand classification category using a softmax classifier, and the corresponding infant state recognition result is output.

[0011] Furthermore, the demand categorization includes crying, turning over, and anxiety.

[0012] Furthermore, when the interaction module detects the infant's state and invokes the companion to provide an emotional response interaction using the parents' voice and / or facial image, it includes: Based on the demand classification output by the emotion recognition module, the corresponding emotional response content is matched from the preset response strategy library.

[0013] Furthermore, when matching corresponding emotional response content from a preset response strategy library based on the demand classification output by the emotion recognition module, the process includes: Establish a multi-level response strategy library corresponding to different status levels in the aforementioned demand classification categories; Based on the identified specific status level, the system automatically matches and triggers the corresponding preset response strategy; Each response strategy defines a specific type of combination of vocal response content and facial expression performed by the companion, matching the state level; the vocal response content includes soothing statements, songs, or stories, and the facial expression includes facial movements consistent with the emotional tone of the response content.

[0014] Furthermore, when the data storage module is used to perform on-chain encrypted storage of infants' growth data and interaction records using the SM4 encryption algorithm and blockchain technology, it includes: The SM4 encryption algorithm is used to encrypt infant growth data and interaction records, including timestamps, demand classification categories, infant status and interaction content, to generate ciphertext data. The encrypted data and its corresponding data hash value are uploaded to the blockchain network for distributed storage, wherein the data hash value is recorded in the block as a unique identifier of the interaction record; By managing the storage process through smart contracts deployed on the blockchain and setting access control rules, authorized institutions are only authorized to query aggregated and anonymized data metrics.

[0015] Furthermore, the data metrics include the occurrence rate of demand categorization and the success rate of interaction, and the generation and querying of all data metrics are completed on-chain by calling smart contracts.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: First, in terms of infant and toddler psychological and emotional health, this system realizes a paradigm shift from passive care to proactive emotional companionship. Traditional care devices mostly remain at the level of physiological signal monitoring and abnormal alarms, exhibiting a serious lack of emotional engagement. This system, through an identity cloning module, constructs a companion entity composed of the parents' faces and voices, providing the child with a highly realistic, familiar, and reassuring emotional anchor. More importantly, the emotion recognition module employs multimodal pre-fusion technology based on micro-expressions, cries, and body movements, combined with a lightweight Transformer model on the edge, enabling the system to understand the child's non-verbal needs like a parent, accurately identifying their state of unease, anxiety, or desire for attention. This deep understanding, combined with the interactive module's proactive and personalized emotional responses by calling upon the companion entity in a very short time, can meet the emotional needs of infants and toddlers. Second, this system also ensures the immediacy of care and the security of privacy. The golden window for emotional soothing is fleeting; prolonged delays can exacerbate infant and toddler anxiety. This invention places the multimodal recognition and decision-making process on the edge and uses a lightweight model, enabling the entire closed loop from perception to response to be completed in milliseconds. This instantaneous response capability is the technological cornerstone of effective soothing. Simultaneously, the data storage module employs the SM4 encryption algorithm and blockchain technology to encrypt and store sensitive growth data and interaction records on the blockchain, ensuring that all infant biometric data, behavioral data, and family privacy data are available but not visible. This allows institutions to conduct macro-level care quality analysis while fundamentally eliminating the risk of data leakage and misuse. Finally, this invention provides childcare institutions with quantifiable, high-quality care tools, enabling them to scientifically evaluate and optimize their care processes while reducing the repetitive workload of caregivers during nighttime and other times. In summary, this invention achieves a dual breakthrough in infant emotional companionship and instant care while ensuring data privacy and security. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a functional block diagram of an intelligent infant care system based on parental identity cloning and multimodal emotion recognition, provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] In some embodiments of this application, see Figure 1 As shown, this embodiment provides an intelligent infant care system based on parental identity cloning and multimodal emotion recognition, including: The identity cloning module is used to generate a 3D facial model and voice clone of the parents by taking at least one parent's facial image and voice sample as input, thus constructing a companion entity; The emotion recognition module is used to process the micro-expression optical flow information, the MFCC features of crying, and the motion information detected by millimeter-wave radar in infants and young children through pre-fusion, and to use the edge lightweight Transformer model to classify needs and identify the state of infants and young children. The interaction module is used to invoke the companion after recognizing the infant's state, and to provide emotional response interaction with the parents' voice and / or facial image; The data storage module is used to encrypt and store infants' growth data and interaction records on the blockchain using the SM4 encryption algorithm and blockchain technology.

[0020] Understandably, firstly, in terms of infant and toddler psychological and emotional health, this system represents a paradigm shift from passive care to proactive emotional companionship. Traditional care devices mostly focus on physiological signal monitoring and abnormal alarms, exhibiting a significant lack of emotional engagement. This system, through its identity cloning module, constructs a companion entity composed of the parents' faces and voices, providing the child with a highly realistic, familiar, and reassuring emotional anchor. More importantly, the emotion recognition module employs multimodal pre-fusion technology based on micro-expressions, cries, and body movements, combined with a lightweight Transformer model on the edge, enabling the system to understand the child's nonverbal needs like a parent, accurately identifying their unease, anxiety, or desire for attention. This deep understanding, combined with the interactive module's rapid activation of the companion entity for proactive and personalized emotional responses, satisfies the emotional needs of infants and toddlers. Secondly, this system also ensures the immediacy of care and the security of privacy. The golden window for emotional comfort is fleeting; prolonged delays can exacerbate an infant's anxiety. This invention places the multimodal recognition and decision-making process on the edge and uses a lightweight model, enabling the entire closed loop from perception to response to be completed in milliseconds. This instantaneous response capability is the technological cornerstone of effective soothing. Simultaneously, the data storage module employs the SM4 encryption algorithm and blockchain technology to encrypt and store sensitive growth data and interaction records on the blockchain, ensuring that all infant biometric data, behavioral data, and family privacy data are available but not visible. This allows institutions to conduct macro-level care quality analysis while fundamentally eliminating the risk of data leakage and misuse. Finally, this invention provides childcare institutions with quantifiable, high-quality care tools, enabling them to scientifically evaluate and optimize their care processes while reducing the repetitive workload of caregivers during nighttime and other times. In summary, this invention achieves a dual breakthrough in infant emotional companionship and instant care while ensuring data privacy and security.

[0021] In one specific implementation of this application, the identity cloning module is first activated, acquiring a frontal facial image of the parents and a clean audio sample of approximately one minute via a terminal device. The system utilizes a lightweight 3D deformation model to directly reconstruct the facial geometry and texture on the local terminal, generating a real-time 3D facial model of the parents. Simultaneously, few-shot speech cloning technology is employed to extract speaker features, and a neural vocoder optimized on the terminal side synthesizes speech with the parents' vocal characteristics. The entire process is completed in parallel on the terminal within 5 minutes, constructing a complete digital companion. When the infant exhibits emotional fluctuations, the emotion recognition module activates: the camera continuously captures facial micro-expressions and calculates optical flow features, the microphone acquires audio signals and extracts the MFCC features and fundamental frequency parameters of the cry, and the millimeter-wave radar accurately senses the frequency and amplitude of body movements. These three heterogeneous modal feature data are timestamped and then early-stage stitched together at the feature layer to form a unified multi-dimensional temporal feature sequence, which is then input into a lightweight Transformer model deployed on the terminal side. This model calculates the correlation weights between different modal features using a self-attention mechanism, completing demand classification within 200 milliseconds and outputting state recognition results such as anxiety and crying. Immediately afterward, the interaction module matches a corresponding solution from a pre-set response strategy library based on the recognition results, driving the cloned parent 3D model to display the corresponding expression (e.g., a gentle smile when anxious), and simultaneously playing matching soothing phrases or humming lullabies using the cloned parent's voice. The entire perception-decision-response closed loop is completed within 800 milliseconds. Finally, the data storage module encrypts the timestamp, event type, and response content of this interaction using the SM4 national cryptographic algorithm and uploads its hash value to the blockchain network for storage. Authorized childcare institutions can query the aggregated and desensitized indicators (such as anxiety incidence and soothing success rate) through smart contracts, achieving secure management of data that is usable but not visible, thus completing a full intelligent care cycle.

[0022] In some embodiments of this application, the identity cloning module is used to generate a three-dimensional facial model and a voice clone of the parents from at least one input parent facial image and voice sample. When constructing a companion entity, it includes: Based on at least one parent's facial image, a lightweight 3D deformation model is used to reconstruct the facial geometry and texture on the terminal device, generating a real-time driven 3D facial model of the parents. At the same time, voice cloning technology is used to extract speaker features by analyzing the parents' voice samples and synthesize voices with the parents' timbre characteristics by combining them with a vocoder on the terminal side. The generation of the 3D facial model and the voice cloning process are processed in parallel on the terminal side.

[0023] In some embodiments of this application, when the emotion recognition module processes the micro-expression optical flow information of infants, the MFCC features of crying sounds, and the motion information detected by millimeter-wave radar using a pre-fusion method, it includes: The facial motion unit intensity features of the micro-expression optical flow information, the fundamental frequency and formant parameter features of the crying MFCC features, and the body motion frequency and amplitude features of the movement information are extracted and spliced ​​in the early stage to form a unified multi-dimensional temporal feature sequence.

[0024] In some embodiments of this application, the early splicing includes: Feature synchronization and alignment: Based on a unified timestamp, the micro-expression optical flow information, the MFCC features of crying sounds, and the motion information detected by millimeter-wave radar are temporally aligned, and features within the same time period are extracted using a common sampling time window as the unit. Feature standardization involves Z-score standardization of the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features extracted from micro-expression optical flow information, crying sound MFCC features, and motion information detected by millimeter-wave radar, so that all features are on the same order of magnitude. Vector concatenation and sequence construction involves concatenating the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features that have undergone Z-score normalization along the feature dimension to form a fused feature vector. The fused feature vectors of multiple consecutive time windows are then arranged in chronological order to form the multidimensional temporal feature sequence.

[0025] In some embodiments of this application, the emotion recognition module is used to perform demand classification using a lightweight Transformer model on the edge. When recognizing the state of infants and toddlers, it includes: The multidimensional temporal feature sequence is input into the edge lightweight Transformer model, and each fused feature vector in the multidimensional temporal feature sequence is mapped to a high-dimensional representation through the edge lightweight Transformer model embedding layer; By utilizing the multi-head self-attention mechanism in the Transformer encoder, the correlation weights between feature representations at different time steps within a multidimensional temporal feature sequence are calculated to capture the long-term dependencies of cross-modal features. After the output of the self-attention mechanism is processed by the feedforward neural network, the feature representation of the last time step of the multi-dimensional temporal feature sequence is taken as the global context representation. Finally, the global context representation is mapped to a preset demand classification category using a softmax classifier, and the corresponding infant state recognition result is output.

[0026] In some embodiments of this application, the demand classification categories include crying, turning over, and anxiety.

[0027] Understandably, the combination of multimodal pre-fusion and a lightweight Transformer model on the device enables accurate and real-time identification of infant states. Employing a feature-level early stitching technique, micro-expression optical flow, cry MFCC, and millimeter-wave motion information are deeply fused after standardization, breaking through the information barriers of traditional single-modal or decision-level fusion. This allows the model to capture the intrinsic relationships between different modalities at the underlying feature level. This pre-fusion strategy, combined with the Transformer's multi-head self-attention mechanism, enhances the model's understanding of long-term cross-modal dependencies, enabling more accurate analysis of complex infant behavioral patterns. Simultaneously, the entire identification process is completed on the device, ensuring real-time performance (response time controlled within milliseconds) while preventing sensitive data from leaving the device, balancing identification efficiency and privacy. The final output of rating results such as "crying, rolling over, and anxiety" provides a reliable basis for subsequent precise intervention, transforming the system from simple behavior monitoring into a true understanding of infant needs.

[0028] In practical deployment, the system is implemented using a combination of front-end hardware and edge computing units. High-speed cameras with 1080P resolution, a high-sensitivity microphone array, and a 60GHz millimeter-wave radar sensor are deployed around the cribs in the childcare facility, all synchronized at millisecond levels via a unified clock source. The system uses a 500ms sampling window to collect real-time facial video, environmental audio, and body movement signals from infants. The video stream is fed into a lightweight OpenCV optical flow algorithm to extract motion unit intensities in areas such as the brow and corners of the mouth; the audio stream undergoes pre-emphasis and frame segmentation to extract 13-dimensional MFCC features and their first-order differences, and calculates the fundamental frequency and formant parameters; the millimeter-wave signal is analyzed using FFT to extract body movement frequency and amplitude features. All features are Z-score normalized and then stitched together on the FPGA chip of the edge computing unit to form a 384-dimensional fused feature vector. Feature sequences from 20 consecutive time windows are fed into an 8-layer lightweight Transformer model based on TensorFlow Lite. This model calculates cross-modal weights using four attention heads, and finally, a softmax classifier outputs the state recognition result. The entire inference process takes no more than 200 milliseconds. The recognition result is pushed to the interaction module via the MQTT protocol, while the hash value of the original data is uploaded in real time to a data storage platform based on the Fabric consortium blockchain.

[0029] In some embodiments of this application, when the interaction module is used to invoke the companion after recognizing the infant's state, and to provide an emotional response interaction using the parents' voice and / or facial image, the following is included: Based on the demand classification output by the emotion recognition module, the corresponding emotional response content is matched from the preset response strategy library.

[0030] In some embodiments of this application, the step of matching corresponding emotional response content from a preset response strategy library based on the demand classification category output by the emotion recognition module includes: Establish a multi-level response strategy library corresponding to different status levels in the aforementioned demand classification categories; Based on the identified specific status level, the system automatically matches and triggers the corresponding preset response strategy; Each response strategy defines a specific type of combination of vocal response content and facial expression performed by the companion, matching the state level; the vocal response content includes soothing statements, songs, or stories, and the facial expression includes facial movements consistent with the emotional tone of the response content.

[0031] Understandably, by establishing a hierarchical, multimodal response strategy library, precise emotional responses to infants' needs are achieved. Differentiated combinations of sound content and facial expressions are preset for different levels of states such as crying, anxiety, and rolling over, ensuring the system's targeted and effective responses. This hierarchical response mode breaks through the single, mechanical interaction method of traditional care devices, providing just the right emotional intervention based on the infant's actual state—neither insufficient response leading to ineffective soothing nor excessive response causing interference. Particularly noteworthy is the collaborative output of sound responses and facial expressions in this invention, achieving dual-channel emotional transmission through visual and auditory means using cloned parent images. This multimodal interaction significantly enhances the realism and trustworthiness of soothing. Furthermore, the structured design of the preset strategy library ensures both the immediacy of the system's response and reserves space for personalized expansion, making the entire interaction process both scientifically standardized and emotionally warm, truly achieving a leap from functional response to emotional connection.

[0032] In practical applications, once the edge computing unit identifies the infant's state, the system immediately initiates a tiered response process. On the intelligent companion robot in childcare facilities, a pre-defined three-tiered response strategy library is stored locally in memory as a JSON configuration file: The first tier, for crying, triggers a strong soothing mode, calling a pre-recorded parent's voice saying "Baby, don't cry, Mommy's here," accompanied by a slightly furrowed brow and a concerned expression; the second tier, for anxious states, activates a soothing mode, playing a cloned version of the parent's humming of "Twinkle Twinkle Little Star," while simultaneously driving the 3D facial model to display a gentle smile with the corners of the mouth raised 15 degrees; the third tier handles rolling over, using a reminder mode, outputting "Roll over slowly" in a calm tone, accompanied by a blinking, attentive expression. After receiving the recognition result, the system completes strategy matching within 100 milliseconds, renders the parent's 3D expression in real time, and simultaneously outputs the corresponding voice.

[0033] In some embodiments of this application, when the data storage module is used to perform on-chain encrypted storage of infant growth data and interaction records using the SM4 encryption algorithm and blockchain technology, it includes: The SM4 encryption algorithm is used to encrypt infant growth data and interaction records, including timestamps, demand classification categories, infant status and interaction content, to generate ciphertext data. The encrypted data and its corresponding data hash value are uploaded to the blockchain network for distributed storage, wherein the data hash value is recorded in the block as a unique identifier of the interaction record; By managing the storage process through smart contracts deployed on the blockchain and setting access control rules, authorized institutions are only authorized to query aggregated and anonymized data metrics.

[0034] In some embodiments of this application, the data metrics include the occurrence rate of demand categorization and the success rate of interaction, and the generation and querying of all data metrics are completed on-chain by calling smart contracts.

[0035] Understandably, this invention constructs a secure and trustworthy data governance system by integrating national cryptographic algorithms and blockchain technology. It employs the SM4 encryption algorithm to locally encrypt sensitive information such as timestamps and status categories, ensuring data privacy from the source and achieving "original data not leaving the domain." Distributing the ciphertext hash values ​​on the blockchain leverages its immutability to establish a trusted data storage chain; any interaction record, once on the chain, possesses traceable and auditable legal validity. Of particular note is the access control mechanism managed by smart contracts, which achieves a compliant usage model of "data usable but not visible." Authorized institutions can only obtain aggregated statistical indicators (such as the occurrence rate of various demands and the success rate of interactions) through contract calls, satisfying the institutions' need for quantitative assessment of care quality while fundamentally eliminating the risk of original privacy data leakage. This technical solution provides full lifecycle security protection for sensitive data of infants and young children.

[0036] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0038] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0039] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An intelligent infant care system based on parental identity cloning and multimodal emotion recognition, characterized in that, include: The identity cloning module is used to generate a 3D facial model and voice clone of the parents by taking at least one parent's facial image and voice sample as input, thus constructing a companion entity; The emotion recognition module is used to process the micro-expression optical flow information, the MFCC features of crying, and the motion information detected by millimeter-wave radar in infants and young children through pre-fusion, and to use the edge lightweight Transformer model to classify needs and identify the state of infants and young children. The interaction module is used to invoke the companion after recognizing the infant's state, and to provide emotional response interaction with the parents' voice and / or facial image; The data storage module is used to encrypt and store infants' growth data and interaction records on the blockchain using the SM4 encryption algorithm and blockchain technology.

2. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 1, characterized in that, The identity cloning module is used to generate a 3D facial model and a voice clone of the parents from at least one input parent facial image and voice sample. When constructing the companion entity, it includes: Based on at least one parent's facial image, a lightweight 3D deformation model is used to reconstruct the facial geometry and texture on the terminal device, generating a real-time driven 3D facial model of the parents. At the same time, voice cloning technology is used to extract speaker features by analyzing the parents' voice samples and synthesize voices with the parents' timbre characteristics by combining them with a vocoder on the terminal side. The generation of the 3D facial model and the voice cloning process are processed in parallel on the terminal side.

3. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 1, characterized in that, The emotion recognition module is used to process micro-expression optical flow information, MFCC features of crying sounds, and motion information detected by millimeter-wave radar in infants and young children through pre-fusion, including: The facial motion unit intensity features of the micro-expression optical flow information, the fundamental frequency and formant parameter features of the crying MFCC features, and the body motion frequency and amplitude features of the movement information are extracted and spliced ​​in the early stage to form a unified multi-dimensional temporal feature sequence.

4. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 3, characterized in that, The early splicing includes: Feature synchronization and alignment: Based on a unified timestamp, the micro-expression optical flow information, the MFCC features of crying sounds, and the motion information detected by millimeter-wave radar are temporally aligned, and features within the same time period are extracted using a common sampling time window as the unit. Feature standardization involves Z-score standardization of the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features extracted from micro-expression optical flow information, crying sound MFCC features, and motion information detected by millimeter-wave radar, so that all features are on the same order of magnitude. Vector concatenation and sequence construction involves concatenating the feature vectors of facial motion unit intensity features, fundamental frequency and formant parameter features, and body motion frequency and amplitude features that have undergone Z-score normalization along the feature dimension to form a fused feature vector. The fused feature vectors of multiple consecutive time windows are then arranged in chronological order to form the multidimensional temporal feature sequence.

5. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 3, characterized in that, The emotion recognition module is used to perform demand classification using a lightweight Transformer model on the edge. When recognizing the state of infants and young children, it includes: The multidimensional temporal feature sequence is input into the edge lightweight Transformer model, and each fused feature vector in the multidimensional temporal feature sequence is mapped to a high-dimensional representation through the edge lightweight Transformer model embedding layer; By utilizing the multi-head self-attention mechanism in the Transformer encoder, the correlation weights between feature representations at different time steps within a multidimensional temporal feature sequence are calculated to capture the long-term dependencies of cross-modal features. After the output of the self-attention mechanism is processed by the feedforward neural network, the feature representation of the last time step of the multi-dimensional temporal feature sequence is taken as the global context representation. Finally, the global context representation is mapped to a preset demand classification category using a softmax classifier, and the corresponding infant state recognition result is output.

6. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 5, characterized in that, The demand categories include crying, turning over, and anxiety.

7. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 5, characterized in that, The interaction module, upon recognizing the infant's state, invokes the companion to provide an emotional response interaction using the parents' voices and / or facial images, including: Based on the demand classification output by the emotion recognition module, the corresponding emotional response content is matched from the preset response strategy library.

8. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 7, characterized in that, When matching corresponding emotional response content from a preset response strategy library based on the demand classification output by the emotion recognition module, the process includes: Establish a multi-level response strategy library corresponding to different status levels in the aforementioned demand classification categories; Based on the identified specific status level, the system automatically matches and triggers the corresponding preset response strategy; Each response strategy defines a specific type of combination of vocal response content and facial expression performed by the companion, matching the state level; the vocal response content includes soothing statements, songs, or stories, and the facial expression includes facial movements consistent with the emotional tone of the response content.

9. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 1, characterized in that, The data storage module is used to perform on-chain encrypted storage of infants' growth data and interaction records using the SM4 encryption algorithm and blockchain technology, including: The SM4 encryption algorithm is used to encrypt infant growth data and interaction records, including timestamps, demand classification categories, infant status and interaction content, to generate ciphertext data. The encrypted data and its corresponding data hash value are uploaded to the blockchain network for distributed storage, wherein the data hash value is recorded in the block as a unique identifier of the interaction record; By managing the storage process through smart contracts deployed on the blockchain and setting access control rules, authorized institutions are only authorized to query aggregated and anonymized data metrics.

10. The intelligent infant care system based on parental identity cloning and multimodal emotion recognition according to claim 9, characterized in that, The data metrics include the occurrence rate of demand categorization and the success rate of interaction, and the generation and querying of all data metrics are completed on-chain by calling smart contracts.

Images