Human-machine interaction control method and system for monitoring a scene

Abstract

The present application provides a human-computer interaction control method and system for monitoring a scene, the human-computer interaction control method comprising: establishing a parent digital human database according to the collected multi-modal data of the parent; obtaining the current emotional category of the baby according to the real-time state information by using an emotion recognition model; automatically generating a corresponding pacification strategy according to the current emotional category of the baby, generating a parent digital human video corresponding to the current pacification strategy according to the parent digital human database, and implementing the pacification strategy; monitoring the state change of the baby during the implementation of the pacification strategy, generating a pacification effect score according to the state change of the baby, and judging whether to automatically adjust the pacification strategy according to the pacification effect score; and the system comprises modules corresponding to the method steps.

Description

Technical Field

[0001] This invention proposes a human-computer interaction control method and system for monitoring scenarios, belonging to the field of human-computer interaction technology. Background Technology

[0002] With the rapid iteration of AI-powered affective computing, multimodal perception, and human-computer interaction technologies, intelligent childcare and home-based infant and toddler care have become core development directions in the fields of smart homes and intelligent childcare. Currently, with the continuous changes in family structures and a significant increase in the proportion of dual-income families, the supply of professional childcare resources is insufficient. Many families face situations where infants and toddlers are left unattended, and parents are unable to provide constant care. When infants and toddlers are unattended or lack care, they are prone to negative emotions such as crying, anxiety, and restlessness. If these emotions are not promptly and effectively soothed, it will not only affect their emotional state and sleep quality but also, in the long term, adversely impact their emotional cognition, sense of security, and psychological health development. Using AI technology to achieve automatic recognition and intelligent soothing of infants and toddlers' emotions has become a key technological path to address the needs of modern, refined childcare and compensate for the shortcomings of human care.

[0003] The currently disclosed or known related technologies mainly include the following categories: (1) Infant monitoring devices. Such as infant monitoring cameras and sleep monitoring devices, which are mainly used to remotely view the status of infants, but usually only have video monitoring and basic alarm functions, and lack the ability to actively soothe infants.

[0004] (2) Voice-based soothing devices. For example, white noise players, voice toys, and fixed voice companion robots usually soothe the baby by playing preset music, nursery rhymes, or recordings, but they cannot dynamically adjust the content according to the baby's real-time state.

[0005] (3) Intelligent companion robots. Some intelligent robots have the ability to interact with voice, recognize faces or provide simple emotional feedback, but most are designed for children’s education or family companionship scenarios and have not been optimized for the emotional characteristics of infants in their early stages.

[0006] (4) Digital human and voice cloning technology. In recent years, AI-based digital human generation and voice cloning technology has emerged, which can simulate the image and voice of a specific person. However, it is mainly used in live streaming, customer service, short video or virtual anchor scenarios, and lacks a closed-loop control mechanism that is combined with infant emotional soothing.

[0007] (5) Emotion recognition system. Some existing studies can recognize users' emotions through sound or images, but most of them are only used in adult interaction scenarios and usually only perform single-modal analysis, making it difficult to accurately identify the complex emotional states of infants.

[0008] Existing technologies generally suffer from the following shortcomings: they cannot simulate the real voices and images of parents; they lack the ability to dynamically soothe infants based on their real-time state; their emotion recognition dimensions are limited and their accuracy is limited; at the same time, they lack the ability to protect family privacy data locally; their soothing content is fixed and has a low degree of personalization; and they cannot effectively reduce the frequency of parental intervention. Summary of the Invention

[0009] This invention provides a human-computer interaction control method and system for monitoring scenarios, in order to solve the aforementioned technical problems existing in the prior art. The technical solution adopted is as follows: A human-computer interaction control method for monitoring a scene, the human-computer interaction control method comprising: A digital human database of parents will be established based on the collected multimodal data of parents. Real-time data collection of the baby's status information; and use an emotion recognition model to determine the baby's current emotion category based on the real-time status information. The system automatically generates a corresponding soothing strategy based on the infant's current emotional category, generates a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implements the soothing strategy. During the implementation of soothing strategies, monitor changes in the infant's state, generate a soothing effectiveness score based on these changes, and determine whether to automatically adjust the soothing strategy based on the soothing effectiveness score.

[0010] Furthermore, based on the collected multimodal data of parents, a digital human database of parents is established, including: Multimodal data of parents is collected through mobile terminals or the robot itself. The multimodal data includes voice samples, facial images, video clips, and facial expression data. The voice samples preferably include calm voice, soothing voice, voice at different speeds, and voice data under different emotional states. The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data. The preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization to improve the stability of subsequent model training and inference. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

[0011] Furthermore, real-time status information of the infant is collected, and an emotion recognition model is used to obtain the infant's current emotion category based on the real-time status information, including: The system collects real-time status information of the infant through a sensing module, which includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequences, audio spectrum and action time sequence, etc.; and the fixed time window is preferably 2 seconds. The state data packet is input into the emotion recognition model, and visual features, audio features, and action features are extracted through the emotion recognition model. The visual features include the infant's facial state and action features. The audio features are obtained by performing a short-time Fourier transform on the crying signal to obtain the crying spectrum and extract the fundamental frequency, intensity, rhythm changes, and high-frequency energy distribution. The action features include the frequency of limb swinging, rolling over movements, and body twisting amplitude. The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0012] Furthermore, a corresponding soothing strategy is automatically generated based on the infant's current emotional category, and a parent digital human video corresponding to the current soothing strategy is generated based on the parent digital human database. The soothing strategy is then implemented, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The digital human driving model is retrieved, and the human feature parameter data in the parent digital human database is retrieved using the digital human driving model. The human feature parameter data is combined with the current soothing strategy to generate the image, target voice, facial expression parameters and mouth shape driving parameters of the parents corresponding to the soothing measurement. Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

[0013] Furthermore, during the implementation of soothing strategies, the infant's state changes are monitored, a soothing effectiveness score is generated based on these changes, and the soothing effectiveness score is used to determine whether to automatically adjust the soothing strategy, including: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

[0014] A human-computer interaction control system for monitoring a scene, the human-computer interaction control system comprising: The Parents Digital Human Database Building Module is used to build a parents digital human database based on the collected multimodal data of parents. The emotion recognition root class module is used to collect the baby's real-time state information and use the emotion recognition model to obtain the baby's current emotion category based on the real-time state information. The automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implement the soothing strategy. The automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of soothing strategies, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score.

[0015] Furthermore, the parent digital human database establishment module is used to establish a parent digital human database based on the collected multimodal data of the parents, including: Multimodal data of parents is collected through mobile terminals or the robot itself. The multimodal data includes voice samples, facial images, video clips, and facial expression data. The voice samples preferably include calm voice, soothing voice, voice at different speeds, and voice data under different emotional states. The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data. The preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization to improve the stability of subsequent model training and inference. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

[0016] Furthermore, the emotion recognition root class module is used to collect the infant's real-time state information and use the emotion recognition model to obtain the infant's current emotion category based on the real-time state information, including: The system collects real-time status information of the infant through a sensing module, which includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequences, audio spectrum and action time sequence, etc.; and the fixed time window is preferably 2 seconds. The state data packet is input into the emotion recognition model, and visual features, audio features, and action features are extracted through the emotion recognition model. The visual features include the infant's facial state and action features. The audio features are obtained by performing a short-time Fourier transform on the crying signal to obtain the crying spectrum and extract the fundamental frequency, intensity, rhythm changes, and high-frequency energy distribution. The action features include the frequency of limb swinging, rolling over movements, and body twisting amplitude. The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0017] Furthermore, the automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implement the soothing strategy, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The digital human driving model is retrieved, and the human feature parameter data in the parent digital human database is retrieved using the digital human driving model. The human feature parameter data is combined with the current soothing strategy to generate the image, target voice, facial expression parameters and mouth shape driving parameters of the parents corresponding to the soothing measurement. Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

[0018] Furthermore, the automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of the soothing strategy, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score, including: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

[0019] Beneficial effects of this invention: This invention proposes a human-computer interaction control method and system for monitoring scenarios, addressing the lack of realistic emotional interaction in traditional soothing devices. Through voice cloning and digital human video generation technology, this invention achieves realistic reconstruction of parents' voices, expressions, and images, providing infants with an interactive experience closer to real parental companionship. Simultaneously, it collects the infant's status in real time through cameras, microphones, and sensors, and analyzes the infant's current emotions based on a multimodal emotion recognition model, dynamically adjusting soothing content and interaction methods. Furthermore, by comprehensively utilizing and jointly analyzing multiple information sources such as sound, movement, and facial expressions, it can more accurately identify the infant's emotional state compared to single-sound detection methods. In addition, it can dynamically generate different voice, video, light, and movement feedback based on different emotional states of the infant, improving the soothing effect and the level of interactive intelligence. This invention also supports localized voice processing and some AI inference processes, reducing the frequency of sensitive voice data being uploaded to the cloud, thereby reducing the risk of privacy leaks. By constructing a closed-loop interaction mechanism from perception, analysis, soothing to feedback, it enables autonomous soothing by the robot, reducing the frequency of continuous manual intervention by parents and improving childcare convenience. Attached Figure Description

[0020] Figure 1 The flowchart is as follows: Figure 2 This is a schematic diagram illustrating the working principle of the method described in this invention. Figure 3 This is a system block diagram of the system described in this invention. Detailed Implementation

[0021] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0022] This embodiment proposes a human-computer interaction control method for monitoring scenarios, such as... Figure 1 and Figure 2 As shown, the human-computer interaction control method includes: A digital human database of parents will be established based on the collected multimodal data of parents. Real-time data collection of the baby's status information; and use an emotion recognition model to determine the baby's current emotion category based on the real-time status information. The system automatically generates a corresponding soothing strategy based on the infant's current emotional category, generates a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implements the soothing strategy. During the implementation of soothing strategies, monitor changes in the infant's state, generate a soothing effectiveness score based on these changes, and determine whether to automatically adjust the soothing strategy based on the soothing effectiveness score.

[0023] The technical solution described in this embodiment is mainly applied to home-based infant care scenarios. It uses a robot terminal to collect real-time infant status information and combines this with parent digital human generation technology and emotion recognition technology to achieve an intelligent and personalized dynamic soothing process. The entire method mainly includes steps such as parent information collection, infant status perception, multimodal emotion recognition, soothing strategy generation, real-time digital human generation, soothing feedback output, and closed-loop effect evaluation.

[0024] Meanwhile, combining the above technical solutions, the human-computer interaction control method proposed in this embodiment mainly includes multimodal emotion recognition, real-time generation of parent digital humans, and closed-loop soothing feedback. Among them, the multimodal emotion recognition step improves the accuracy of infant emotion analysis by integrating visual, audio, and motion information; the real-time digital human generation step realizes the dynamic and realistic reconstruction of the parents' voices and images; and the closed-loop soothing feedback step realizes a dynamic interactive mechanism that automatically adjusts the soothing content according to the infant's real-time state.

[0025] Compared with existing baby comforting devices, intelligent companion robots, and digital human interaction methods, the above-described technical solution in this embodiment has more intelligent, personalized, and emotional interaction capabilities, which can effectively improve the baby comforting effect and reduce the frequency of continuous parental intervention.

[0026] First, the technical solution described above in this embodiment achieves a realistic reconstruction of the parents' voices, expressions, and images through voice cloning and digital human video generation technology. Compared to traditional white noise devices, fixed voice players, or ordinary companion robots, this invention can generate dynamic interactive content with parental characteristics, allowing infants to obtain an emotional experience closer to that of real parental companionship during the soothing process, thereby improving the infant's emotional stability and the success rate of soothing.

[0027] Secondly, the technical solution described in this embodiment introduces a multimodal emotion recognition mechanism, which can comprehensively analyze various information such as the infant's cries, facial expressions, body movements, and sleep behavior. Compared with existing solutions that rely solely on single sound detection, this approach can more accurately identify the infant's current emotional state. Through multimodal joint analysis, this invention can effectively reduce the false positive rate and improve the targeting and adaptability of soothing strategies.

[0028] Furthermore, the technical solution described in this embodiment establishes a dynamic closed-loop control mechanism encompassing state perception, emotion analysis, soothing feedback, effect evaluation, and strategy updating. Compared to existing devices that can only play fixed content unidirectionally, this invention can dynamically adjust the voice content, digital human expressions, lighting effects, and robot movements based on the infant's real-time state, thereby achieving adaptive soothing. This closed-loop mechanism continuously optimizes the soothing process and improves the system's intelligence level.

[0029] One embodiment of the present invention involves establishing a digital human database of parents based on the collected multimodal data of parents, including: Multimodal data of parents is collected through mobile terminals or the robot itself. The multimodal data includes voice samples, facial images, video clips, and facial expression data. The voice samples preferably include calm voice, soothing voice, voice at different speeds, and voice data under different emotional states. The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data. The preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization to improve the stability of subsequent model training and inference. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

[0030] In the initialization phase of this system, the companion robot can collect multimodal data related to the guardian's parents through a connected mobile terminal device or the robot itself. Specifically, the collected multimodal data includes parents' voice samples, facial images, video clips, and various facial expressions and body movements. To ensure the comprehensiveness and adaptability of the data samples, the collected voice samples cover calm speech, personalized soothing speech, speech at different speeds, and speech data under different emotional states, fully covering various voice interaction needs in subsequent infant and toddler soothing scenarios.

[0031] After acquiring the raw multimodal data, standardized preprocessing operations are performed on various types of raw data. These operations include audio denoising, face alignment correction, image normalization, precise audio segmentation, and data feature standardization. These preprocessing operations effectively remove invalid interference information from the raw data, unify data format and feature dimensions, and significantly improve the accuracy and operational stability of subsequent model training, feature extraction, and real-time inference processes.

[0032] After preprocessing the multimodal data, a dedicated digital parent database is constructed based on the purified and standardized feature data. This database stores core feature parameters associated with the parents, such as facial feature vectors, voice feature vectors, lip-shape driving parameters, and expression mapping parameters. This provides complete and reliable data information for subsequent analysis of infants' emotional states, matching corresponding soothing strategies, and generating appropriate digital parent comfort videos in real time.

[0033] One embodiment of the present invention involves real-time acquisition of an infant's real-time state information and using an emotion recognition model to obtain the infant's current emotion category based on the real-time state information, including: The system collects real-time status information of the infant through a sensing module, which includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequences, audio spectrum and action time sequence, etc.; and the fixed time window is preferably 2 seconds. The state data packet is input into the emotion recognition model, and visual features, audio features, and action features are extracted through the emotion recognition model. The visual features include the infant's facial state and action features. The audio features are obtained by performing a short-time Fourier transform on the crying signal to obtain the crying spectrum and extract the fundamental frequency, intensity, rhythm changes, and high-frequency energy distribution. The action features include the frequency of limb swinging, rolling over movements, and body twisting amplitude. The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0034] In this embodiment, the system continuously collects real-time status information of the infant through a sensing module during robot operation. The sensing module includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The RGB camera is used to collect facial images and body movement information of the infant; the infrared camera is used for nighttime status monitoring; the microphone array is used to collect crying sounds and ambient noise; and the motion sensor is used to analyze behaviors such as rolling over and limb movements. The system caches the collected data according to a fixed time window, for example, forming a status data packet every two seconds. The data packet includes image frame sequences, audio spectrum, and action time sequences.

[0035] Subsequently, the system inputs the collected multimodal data into the emotion recognition model for analysis. The system first uses a convolutional neural network to extract facial state and movement features of the infant, including mouth opening degree, eye state, head movements, and body posture. Simultaneously, the system performs a short-time Fourier transform on the crying signal to obtain the crying spectral features. The spectral effect corresponding to the crying signal is obtained using the following formula: ; Where x[n] represents the original crying signal, w[n] represents the window function, τ represents the time window, ω represents the frequency, and X(τ,ω) represents the corresponding spectrum result. Further characteristic parameters such as the fundamental frequency of the crying, crying intensity, rhythm changes, and high-frequency energy distribution are extracted. Simultaneously, the system extracts behavioral features such as limb swing frequency, rolling over movements, and body twisting amplitude through a motion timing analysis model.

[0036] To improve the accuracy of infant emotion recognition, the system employs a multimodal fusion mechanism to jointly encode visual features, audio features, and action features. This joint encoding is achieved using the following formula: ; in, Indicates visual characteristics, Indicates audio features, The values ​​represent action features, and α, β, and γ represent the fusion weights of the corresponding modalities. This represents the integrated emotional characteristics after fusion. Based on the fused characteristics, the corresponding infant emotional category is output, including states such as hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0037] By collaboratively collecting multi-dimensional real-time status information of infants through multiple types of sensing modules, it can comprehensively and accurately capture the infant's facial, limb, crying, and environmental states, avoiding the problems of partial information collection and missing features that exist in single sensing methods, and ensuring the integrity and authenticity of the original status data. By setting a fixed time window to cache and package real-time status information, the temporal data structure can be standardized, the model input standard can be unified, and the recognition deviation caused by the disorder of real-time data stream and temporal sequence can be effectively avoided, improving the stability of subsequent feature extraction and emotion judgment.

[0038] This embodiment's technical solution specifically extracts differentiated modal features from three categories: visual, audio, and motor. By performing spectral transformation and multi-dimensional parameter analysis on infant crying signals, combined with quantified limb movement features and facial visual features, it achieves refined feature mining of subtle emotional representations in infants. Simultaneously, by integrating multi-dimensional feature information through a multi-modal fusion mechanism, it overcomes the limitations of single-modal recognition, effectively improving the accuracy and robustness of infant emotion recognition. It can accurately distinguish between subcategories of emotions such as hunger, drowsiness, fright, anxiety, ordinary crying, and emotional stability. This provides reliable and high-precision emotional data support for subsequent adaptation and matching of precise parental digital human soothing strategies and the realization of closed-loop intelligent soothing control, thus improving the overall intelligence and precision of human-computer interaction in infant care.

[0039] In one embodiment of the present invention, a corresponding soothing strategy is automatically generated based on the infant's current emotional category, and a parent digital human video corresponding to the current soothing strategy is generated based on a parent digital human database, and the soothing strategy is implemented, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The digital human driving model is retrieved, and the human feature parameter data in the parent digital human database is retrieved using the digital human driving model. The human feature parameter data is combined with the current soothing strategy to generate the image, target voice, facial expression parameters and mouth shape driving parameters of the parents corresponding to the soothing measurement. Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

[0040] After obtaining the emotion recognition results, the above-described technical solution in this embodiment automatically generates a corresponding soothing strategy. The soothing strategy includes parameters such as speech content, speech rate, volume, digital human facial expressions, light color, and robot movement amplitude. For example, when the infant is identified as being frightened, a slower speech rate, gentle facial expressions, low-brightness warm light, and small movements are preferred for soothing; when the infant is identified as being anxious and crying, the frequency of voice interaction and the intensity of the digital human's movement response are appropriately increased. The soothing strategy can be selected based on preset strategy mapping rules, or the soothing effect can be continuously optimized through a reinforcement learning model.

[0041] After generating the soothing strategy, the above-mentioned technical solution in this embodiment further utilizes a digital human-driven model to generate real-time digital human videos of the parents. Inputting images of the parents, target voice, facial expression parameters, and lip-syncing parameters, the digital human video generation model generates dynamic video content with synchronized lip movements, facial actions, eye movements, and head movements, which is then played in real-time on the robot's display screen. Simultaneously, the robot's speaker outputs soothing voice messages, the lighting module outputs corresponding ambient lighting, and the robot's motion module performs slight swaying or moving movements to enhance the soothing effect on the infant.

[0042] The technical solution described in this embodiment adaptively matches the corresponding soothing strategy based on the infant's precise emotional category, achieving a precise adaptation between the soothing mode and the infant's emotional state. This abandons the traditional fixed and singular soothing execution mode, effectively improving the targeting and adaptability of soothing intervention. By finely configuring multi-dimensional soothing parameters such as voice content, speech rate and volume, digital human facial expressions, lighting color, and robot movement amplitude, a multi-dimensional and three-dimensional intelligent soothing system is constructed, increasing the dimensions of soothing interaction. This allows for simultaneous soothing intervention of the infant's emotions from multiple levels of vision, hearing, and touch.

[0043] Meanwhile, the aforementioned technical solution utilizes a digital human-driven model to access parents' unique feature parameters, combined with adapted soothing strategies to generate highly synchronized digital human videos of parents in real time, featuring mouth movements, facial expressions, eye movements, and head gestures. This highly replicates the parents' realistic expressions and voice characteristics, aligning with the infant's natural parent-child attachment, effectively eliminating the unfamiliarity of conventional virtual soothing interactions, and significantly enhancing the interactivity and emotional soothing effect. Furthermore, through the coordinated execution of soothing strategies via multiple modules including robot display, sound, lighting, and movement, the integrated implementation of soothing strategies is achieved, ensuring the synchronicity and integrity of the soothing interaction, and maximizing the overall effectiveness and level of intelligent interaction in smart soothing.

[0044] One embodiment of the present invention involves monitoring changes in the infant's state during the implementation of a soothing strategy, generating a soothing effectiveness score based on these changes, and determining whether to automatically adjust the soothing strategy based on the soothing effectiveness score. This includes: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

[0045] During the soothing process, the system continuously monitors changes in the infant's state and evaluates the soothing effect in real time. The system calculates a soothing effect score based on changes in crying, stability of movement, and changes in emotional state. This soothing effect score is obtained using the following formula: ; Where C represents the degree of crying reduction, M represents the degree of movement stability, E represents the degree of emotional stability, λ1, λ2, and λ3 represent the corresponding weight parameters, and S represents the final soothing effect score. When the score is lower than a preset threshold, the system automatically adjusts the soothing strategy, including modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm, thus forming a dynamic closed-loop control process of state perception, emotion analysis, soothing feedback, effect evaluation, and strategy updating.

[0046] The technical solution described in this embodiment monitors changes in the infant's crying, movement stability, and emotional state in real time during the execution of the soothing strategy. It can dynamically capture subtle fluctuations in the infant's state during soothing intervention, achieving real-time perception and data tracking of the entire soothing process. This avoids the drawbacks of traditional fixed soothing models that cannot reflect the infant's real-time state. By quantifying soothing effect scores through multi-dimensional state change indicators, it achieves standardized, digital, and objective evaluation of soothing effects, eliminating traditional subjective and vague effect judgment methods and effectively improving the accuracy and standardization of soothing effect assessment.

[0047] Meanwhile, by comparing the soothing effect score with preset thresholds, the system achieves automated identification and adaptive adjustment of soothing strategies. For scenarios where the soothing effect is unsatisfactory, core soothing parameters such as voice content, digital human facial expressions, lighting parameters, and movement rhythm can be dynamically optimized. A closed-loop intelligent control mechanism has been constructed, from emotion recognition, strategy execution, effect monitoring to strategy iteration. This allows the soothing strategy to be dynamically iterated and optimized according to the infant's real-time emotional recovery progress, continuously improving the soothing adaptability and intervention effectiveness, and further enhancing the overall system's intelligent adaptive capability and scenario adaptability.

[0048] This invention proposes a human-computer interaction control system for monitoring scenarios, such as... Figure 3 As shown, the human-computer interaction control system includes: The Parents Digital Human Database Building Module is used to build a parents digital human database based on the collected multimodal data of parents. The emotion recognition root class module is used to collect the baby's real-time state information and use the emotion recognition model to obtain the baby's current emotion category based on the real-time state information. The automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implement the soothing strategy. The automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of soothing strategies, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score.

[0049] The above-mentioned technical solution in this embodiment is a software system for implementing a human-computer interaction control method. It includes execution modules corresponding to the steps of the human-computer interaction control method, namely, a parent digital human database establishment module, an emotion recognition root class module, an automatic generation module for soothing strategies, and an automatic adjustment module for strategies.

[0050] The human-computer interaction control system for monitoring scenarios proposed in this embodiment corresponds to an infant soothing robot with a bedside or desktop structure design. The overall structure includes a display module, camera module, microphone module, speaker module, AI processing module, wireless communication module, light feedback module, mobile chassis module, and power supply module. The robot's main shell preferably adopts a rounded corner design to reduce the risk of infant contact, and the shell material is preferably made of lightweight and environmentally friendly material with a certain degree of cushioning performance.

[0051] The robot features a display module on its front for playing digital avatar videos of the parents. The display module preferably uses an LCD or OLED screen and has a low blue light mode to reduce visual stimulation for the infant at night. Above the display module is a camera module, which includes an RGB camera and an infrared camera. The RGB camera is used for daytime image acquisition, while the infrared camera is used for nighttime status monitoring. A microphone module is located on the front of the robot, preferably using a microphone array structure to improve the quality of cry sound capture and environmental noise reduction.

[0052] The robot is equipped with an AI processing module to run emotion recognition models, digital human generation models, and soothing control models. The AI ​​processing module preferably uses a GPU, NPU, or edge AI chip to meet real-time inference requirements. A wireless communication module is used to enable data communication between the robot and mobile terminals and cloud servers, preferably supporting WiFi or Bluetooth communication methods.

[0053] The robot features a mobile chassis module at its base, comprising drive wheels, omnidirectional wheels, and a motor-driven structure, enabling it to move at low speeds near the crib. The robot's movement speed is preferably controlled within a safe range to avoid the risk of collision with the infant. A lighting feedback module is located at the robot's edge area to output ambient lighting of different colors and brightness to enhance the soothing effect.

[0054] Therefore, the key structures corresponding to the aforementioned physical robot include a multimodal perception structure, a local AI processing structure, a digital human interaction display structure, and a low-speed safe movement structure. Among these, the multimodal perception structure enhances the ability to capture the infant's state, the local AI processing structure reduces the risk of privacy leaks, the digital human interaction display structure enables dynamic and realistic display of the parent's digital human, and the low-speed safe movement structure enhances the robot's companionship and interaction capabilities.

[0055] Meanwhile, the system's control circuitry mainly includes a main control chip, an AI acceleration chip, a camera interface circuit, a microphone acquisition circuit, an audio output circuit, a motor drive circuit, a wireless communication circuit, and a power management circuit. The main control chip coordinates the operation of the robot's various functional modules, while the AI ​​acceleration chip performs AI inference tasks such as emotion recognition, voice cloning, and digital human generation.

[0056] The camera interface circuit is used to connect the RGB camera and the infrared camera and complete the image data transmission; the microphone acquisition circuit is used to complete the acquisition of crying signals and audio amplification; the audio output circuit is used to drive the speaker to play soothing voices; the motor drive circuit is used to control the movement of the robot chassis; the wireless communication circuit is used to realize data communication between the robot and external terminals; and the power management circuit is responsible for powering each module and controlling low power consumption.

[0057] The AI ​​edge inference circuit is a key circuit structure in this invention. This circuit can perform voice analysis, emotion recognition, and some digital human generation tasks locally, thereby reducing the frequency of voice data uploading to the cloud and lowering the risk of family privacy leaks. The system also features a low-power wake-up circuit that continuously listens for crying signals while the robot is in standby mode. When abnormal crying is detected, it automatically wakes up the main system to reduce overall power consumption.

[0058] This embodiment involves multi-device interaction between a robot, a parent's mobile terminal, and a cloud server. The robot is responsible for collecting the infant's status, performing local AI inference, and providing comforting output; the mobile terminal is used for remote configuration and remote companionship; and the cloud server is used for model updates and handling highly complex generation tasks. The system preferably adopts a local-first processing mechanism, that is, the extraction of parents' voice features and some emotion recognition are completed primarily in the local AI module, and cloud resources are only called when necessary, thereby reducing the risk of privacy leakage and improving the system's real-time response capability.

[0059] The human-computer interaction control system for monitoring scenarios proposed in this embodiment, along with its corresponding physical robot and circuit system, mainly addresses the shortcomings of existing baby comforting devices, intelligent companion robots, and digital human-computer interaction systems, solving the following technical problems: (1) Solving the problem of the lack of real emotional interaction in existing baby soothing devices. Existing white noise devices, voice toys and traditional companion robots usually use fixed audio or pre-recorded content, which cannot simulate the real voice, expression and interactive behavior of parents, resulting in limited soothing effect. This invention realizes the realistic generation of parents' voice and image through voice cloning and digital human video generation technology, thereby improving the realism and intimacy of baby emotional soothing.

[0060] (2) Solving the problem that existing devices cannot dynamically adjust soothing strategies according to the baby's real-time state. Traditional devices usually use fixed playback logic and cannot provide dynamic feedback based on the baby's current crying level, movement state, or emotional changes. This invention collects the baby's sound, movement, and image information in real time and analyzes the baby's state based on a multimodal emotion recognition model, thereby dynamically adjusting the soothing content, interaction method, and feedback intensity.

[0061] (3) Addressing the problem that existing emotion recognition technologies are applicable to adult scenarios but not to infant scenarios. Most existing emotion recognition technologies are designed for analyzing adult speech, facial expressions, or text emotions, making it difficult to accurately identify the complex and non-verbal emotional states of infants. This invention achieves multimodal emotion analysis for infant scenarios by integrating infant crying features, action behavior features, facial state features, and sleep behavior features, thereby improving the accuracy of emotion recognition.

[0062] (4) Solving the problem of the lack of dynamic closed-loop interaction capabilities in existing digital human systems. Most existing digital human systems only perform one-way content playback and cannot adjust interactive behavior according to real-time user feedback. This invention establishes a closed-loop reassurance mechanism of "perception-analysis-feedback-re-perception" to realize a dynamic and adaptive reassurance process.

[0063] (5) Addressing the issue of easy leakage of family privacy data. Existing cloud-based AI systems typically require continuous uploading of voice and video data, posing a risk of privacy leakage. This invention supports localized voice processing and edge AI inference, enabling parents' voice data and some of the baby's status data to be processed locally first, thereby reducing the risk of sensitive data being leaked.

[0064] (6) Solving the problem of frequent parental intervention. Existing devices have a low level of intelligence and cannot continuously and autonomously complete soothing tasks, so parents still need to intervene frequently. This invention realizes the robot's autonomous emotional soothing and dynamic interaction through an intelligent closed-loop soothing mechanism, thereby reducing the frequency of repeated parental intervention and improving the convenience and efficiency of childcare.

[0065] The technical solution proposed in this embodiment has good real-time performance and interactivity. Through local AI inference and edge computing architecture, the system can complete emotion recognition and digital human generation under low latency conditions, avoiding the interaction delay problem caused by cloud communication, thereby improving real-time response capability and user experience.

[0066] Meanwhile, regarding privacy protection, a local-first processing mechanism is adopted to localize the processing of parents' voice characteristics, infant status data, and some AI inference processes, reducing the frequency of sensitive data being uploaded to the cloud. Compared to solutions that rely entirely on cloud-based AI, this invention can effectively reduce the risk of family privacy leaks and improve system security and user acceptance.

[0067] In addition, the robot in the above technical solution has mobility and multi-device collaboration capabilities, and can actively approach the baby when the baby is crying and perform dynamic soothing. It also supports remote access from parents' mobile terminals to achieve remote companionship and remote interaction, further improving the system's practicality and adaptability to application scenarios.

[0068] In summary, this invention not only enhances the intelligence and emotional intelligence of infant soothing devices, but also realizes several key functions such as multimodal emotion analysis, real-time digital human generation, adaptive closed-loop soothing, and local privacy protection, demonstrating high practical application value and promising market prospects.

[0069] In one embodiment of the present invention, the parent digital human database establishment module is used to establish a parent digital human database based on the collected multimodal data of parents, including: Multimodal data of parents is collected through mobile terminals or the robot itself. The multimodal data includes voice samples, facial images, video clips, and facial expression data. The voice samples preferably include calm voice, soothing voice, voice at different speeds, and voice data under different emotional states. The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data. The preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization to improve the stability of subsequent model training and inference. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

[0070] In the initialization phase of this system, the companion robot can collect multimodal data related to the guardian's parents through a connected mobile terminal device or the robot itself. Specifically, the collected multimodal data includes parents' voice samples, facial images, video clips, and various facial expressions and body movements. To ensure the comprehensiveness and adaptability of the data samples, the collected voice samples cover calm speech, personalized soothing speech, speech at different speeds, and speech data under different emotional states, fully covering various voice interaction needs in subsequent infant and toddler soothing scenarios.

[0071] After completing the acquisition of raw multimodal data, standardized preprocessing operations are performed on various types of raw data, including audio denoising, face alignment correction, image normalization, precise audio segmentation, and data feature standardization. These preprocessing operations effectively remove invalid interference information from the raw data, unify data format and feature dimensions, and significantly improve the accuracy and operational stability of subsequent model training, feature extraction, and real-time inference processes. Following the multimodal data preprocessing, a dedicated digital parent database is constructed based on the purified standardized feature data. This database stores core feature parameters associated with parents, such as facial feature vectors, timbre feature vectors, lip-shape driving parameters, and expression mapping parameters. This provides complete and reliable data information for subsequently combining infants' emotional states, matching corresponding soothing strategies, and generating appropriate digital parent soothing videos in real time.

[0072] In one embodiment of the present invention, the emotion recognition root class module is used to collect real-time state information of the infant and obtain the infant's current emotion category based on the real-time state information using an emotion recognition model, including: The system collects real-time status information of the infant through a sensing module, which includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequences, audio spectrum and action time sequence, etc.; and the fixed time window is preferably 2 seconds. The state data packet is input into an emotion recognition model, which extracts visual, audio, and motion features. The visual features include the infant's facial expressions and motion characteristics. Audio features are obtained by performing a short-time Fourier transform on the crying signal to obtain the crying spectrum and extracting the fundamental frequency, intensity, rhythm variations, and high-frequency energy distribution. Motion features include the frequency of limb movements, rolling over, and body twisting amplitude. The emotion recognition model is primarily used to analyze the infant's current emotional state. During the training phase, the input to the emotion recognition model includes an infant image, a crying spectrum, and a motion time series; the output is the corresponding infant emotion category. The model employs a multi-branch structure, including a CNN visual feature extraction branch, a Transformer temporal analysis branch, an audio encoding branch, and a multimodal fusion module. The visual branch extracts facial and motion features, the audio branch analyzes the crying spectrum features, the Transformer branch analyzes time-series behavioral changes, and the multimodal fusion module jointly encodes features from different modalities.

[0073] The training data primarily includes infant crying data, infant behavior videos, and infant sleep videos, all of which need to be pre-labeled with emotion categories. Supervised learning is employed during model training, using a cross-entropy loss function to optimize model parameters and improve emotion recognition accuracy.

[0074] The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0075] In this embodiment, the system continuously collects real-time status information of the infant through a sensing module during robot operation. The sensing module includes devices such as an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The RGB camera is used to collect facial images and body movement information of the infant; the infrared camera is used for nighttime status monitoring; the microphone array is used to collect crying sounds and ambient noise; and the motion sensor is used to analyze behaviors such as rolling over and limb movements. The system caches the collected data according to a fixed time window, for example, forming a status data packet every two seconds. The data packet includes image frame sequences, audio spectrum, and action time sequences.

[0076] Subsequently, the system inputs the collected multimodal data into the emotion recognition model for analysis. The system first uses a convolutional neural network to extract facial state and movement features of the infant, including mouth opening degree, eye state, head movements, and body posture. Simultaneously, the system performs a short-time Fourier transform on the crying signal to obtain the crying spectral features. The spectral effect corresponding to the crying signal is obtained using the following formula: ; Where x[n] represents the original crying signal, w[n] represents the window function, τ represents the time window, ω represents the frequency, and X(τ,ω) represents the corresponding spectrum result. Further characteristic parameters such as the fundamental frequency of the crying, crying intensity, rhythm changes, and high-frequency energy distribution are extracted. Simultaneously, the system extracts behavioral features such as limb swing frequency, rolling over movements, and body twisting amplitude through a motion timing analysis model.

[0077] To improve the accuracy of infant emotion recognition, the system employs a multimodal fusion mechanism to jointly encode visual features, audio features, and action features. This joint encoding is achieved using the following formula: ; in, Indicates visual characteristics, Indicates audio features, The values ​​represent action features, and α, β, and γ represent the fusion weights of the corresponding modalities. This represents the integrated emotional characteristics after fusion. Based on the fused characteristics, the corresponding infant emotional category is output, including states such as hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

[0078] By collaboratively collecting multi-dimensional real-time status information of infants through multiple types of sensing modules, it can comprehensively and accurately capture the infant's facial, limb, crying, and environmental states, avoiding the problems of partial information collection and missing features that exist in single sensing methods, and ensuring the integrity and authenticity of the original status data. By setting a fixed time window to cache and package real-time status information, the temporal data structure can be standardized, the model input standard can be unified, and the recognition deviation caused by the disorder of real-time data stream and temporal sequence can be effectively avoided, improving the stability of subsequent feature extraction and emotion judgment.

[0079] This embodiment's technical solution specifically extracts differentiated modal features from three categories: visual, audio, and motor. By performing spectral transformation and multi-dimensional parameter analysis on infant crying signals, combined with quantified limb movement features and facial visual features, it achieves refined feature mining of subtle emotional representations in infants. Simultaneously, by integrating multi-dimensional feature information through a multi-modal fusion mechanism, it overcomes the limitations of single-modal recognition, effectively improving the accuracy and robustness of infant emotion recognition. It can accurately distinguish between subcategories of emotions such as hunger, drowsiness, fright, anxiety, ordinary crying, and emotional stability. This provides reliable and high-precision emotional data support for subsequent adaptation and matching of precise parental digital human soothing strategies and the realization of closed-loop intelligent soothing control, thus improving the overall intelligence and precision of human-computer interaction in infant care.

[0080] In one embodiment of the present invention, the automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on a parent digital human database, and implement the soothing strategy, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The system retrieves a digital human-driven model, which uses this model to access character feature parameters from a parent digital human database. This data, combined with the current soothing strategy, generates images of the parents, target speech, facial expressions, and lip-sync parameters corresponding to the soothing measurement. Specifically, the digital human-driven model's input includes parent images and target speech, and its output is a dynamic video with lip-sync. The model dynamically generates the digital human video through lip-sync, facial expression, and head motion generation modules. A speech cloning model learns corresponding timbre features from parent speech samples to generate personalized soothing speech. Compared to existing technologies, this model is optimized specifically for infant scenarios, achieving multimodal fusion of visual, audio, and motion information, and constructing a dynamic closed-loop soothing mechanism based on emotional feedback. Furthermore, it supports localized speech processing and edge AI inference, ensuring real-time performance while reducing the risk of family privacy data leakage.

[0081] Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

[0082] After obtaining the emotion recognition results, the above-described technical solution in this embodiment automatically generates a corresponding soothing strategy. The soothing strategy includes parameters such as speech content, speech rate, volume, digital human facial expressions, light color, and robot movement amplitude. For example, when the infant is identified as being frightened, a slower speech rate, gentle facial expressions, low-brightness warm light, and small movements are preferred for soothing; when the infant is identified as being anxious and crying, the frequency of voice interaction and the intensity of the digital human's movement response are appropriately increased. The soothing strategy can be selected based on preset strategy mapping rules, or the soothing effect can be continuously optimized through a reinforcement learning model.

[0083] After generating the soothing strategy, the above-mentioned technical solution in this embodiment further utilizes a digital human-driven model to generate real-time digital human videos of the parents. Inputting images of the parents, target voice, facial expression parameters, and lip-syncing parameters, the digital human video generation model generates dynamic video content with synchronized lip movements, facial actions, eye movements, and head movements, which is then played in real-time on the robot's display screen. Simultaneously, the robot's speaker outputs soothing voice messages, the lighting module outputs corresponding ambient lighting, and the robot's motion module performs slight swaying or moving movements to enhance the soothing effect on the infant.

[0084] The technical solution described in this embodiment adaptively matches the corresponding soothing strategy based on the infant's precise emotional category, achieving a precise adaptation between the soothing mode and the infant's emotional state. This abandons the traditional fixed and singular soothing execution mode, effectively improving the targeting and adaptability of soothing intervention. By finely configuring multi-dimensional soothing parameters such as voice content, speech rate and volume, digital human facial expressions, lighting color, and robot movement amplitude, a multi-dimensional and three-dimensional intelligent soothing system is constructed, increasing the dimensions of soothing interaction. This allows for simultaneous soothing intervention of the infant's emotions from multiple levels of vision, hearing, and touch.

[0085] Meanwhile, the aforementioned technical solution utilizes a digital human-driven model to access parents' unique feature parameters, combined with adapted soothing strategies to generate highly synchronized digital human videos of parents in real time, featuring mouth movements, facial expressions, eye movements, and head gestures. This highly replicates the parents' realistic expressions and voice characteristics, aligning with the infant's natural parent-child attachment, effectively eliminating the unfamiliarity of conventional virtual soothing interactions, and significantly enhancing the interactivity and emotional soothing effect. Furthermore, through the coordinated execution of soothing strategies via multiple modules including robot display, sound, lighting, and movement, the integrated implementation of soothing strategies is achieved, ensuring the synchronicity and integrity of the soothing interaction, and maximizing the overall effectiveness and level of intelligent interaction in smart soothing.

[0086] In one embodiment of the present invention, the automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of a soothing strategy, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score, including: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

[0087] During the soothing process, the system continuously monitors changes in the infant's state and evaluates the soothing effect in real time. The system calculates a soothing effect score based on changes in crying, stability of movement, and changes in emotional state. This soothing effect score is obtained using the following formula: ; Where C represents the degree of crying reduction, M represents the degree of movement stability, E represents the degree of emotional stability, λ1, λ2, and λ3 represent the corresponding weight parameters, and S represents the final soothing effect score. When the score is lower than a preset threshold, the system automatically adjusts the soothing strategy, including modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm, thus forming a dynamic closed-loop control process of state perception, emotion analysis, soothing feedback, effect evaluation, and strategy updating.

[0088] The technical solution described in this embodiment monitors changes in the infant's crying, movement stability, and emotional state in real time during the execution of the soothing strategy. It can dynamically capture subtle fluctuations in the infant's state during soothing intervention, achieving real-time perception and data tracking of the entire soothing process. This avoids the drawbacks of traditional fixed soothing models that cannot reflect the infant's real-time state. By quantifying soothing effect scores through multi-dimensional state change indicators, it achieves standardized, digital, and objective evaluation of soothing effects, eliminating traditional subjective and vague effect judgment methods and effectively improving the accuracy and standardization of soothing effect assessment.

[0089] Meanwhile, by comparing the soothing effect score with preset thresholds, the system achieves automated identification and adaptive adjustment of soothing strategies. For scenarios where the soothing effect is unsatisfactory, core soothing parameters such as voice content, digital human facial expressions, lighting parameters, and movement rhythm can be dynamically optimized. A closed-loop intelligent control mechanism has been constructed, from emotion recognition, strategy execution, effect monitoring to strategy iteration. This allows the soothing strategy to be dynamically iterated and optimized according to the infant's real-time emotional recovery progress, continuously improving the soothing adaptability and intervention effectiveness, and further enhancing the overall system's intelligent adaptive capability and scenario adaptability.

[0090] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A human-computer interaction control method for monitoring a scene, characterized in that, The human-computer interaction control method includes: A digital human database of parents will be established based on the collected multimodal data of parents. Real-time data collection of the baby's status information; using an emotion recognition model to determine the baby's current emotion category based on the real-time status information. The system automatically generates a corresponding soothing strategy based on the infant's current emotional category, generates a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implements the soothing strategy. During the implementation of soothing strategies, monitor changes in the infant's state, generate a soothing effectiveness score based on these changes, and determine whether to automatically adjust the soothing strategy based on the soothing effectiveness score.

2. The human-computer interaction control method according to claim 1, characterized in that, Based on the collected multimodal data of parents, a digital human database of parents is established, including: Multimodal data of parents is collected through mobile terminals or the robot itself, wherein the multimodal data includes voice samples, facial images, video clips and facial expression data; The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data, wherein the preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

3. The human-computer interaction control method according to claim 1, characterized in that, Real-time data collection of the infant's status information; using an emotion recognition model to determine the infant's current emotion category based on this real-time status information, including: The system collects real-time status information of the infant through a sensing module, which includes an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequence, audio spectrum and action time sequence; The state data packet is input into the emotion recognition model, and visual features, audio features, and action features are extracted by the emotion recognition model. The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

4. The human-computer interaction control method according to claim 1, characterized in that, Based on the infant's current emotional category, a corresponding soothing strategy is automatically generated, and a corresponding parent digital human video is generated based on the parent digital human database. The soothing strategy is then implemented, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The digital human driving model is retrieved, and the human feature parameter data in the parent digital human database is retrieved using the digital human driving model. The human feature parameter data is combined with the current soothing strategy to generate the image, target voice, facial expression parameters and mouth shape driving parameters of the parents corresponding to the soothing measurement. Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

5. The human-computer interaction control method according to claim 1, characterized in that, During the implementation of soothing strategies, monitor changes in the infant's state, generate a soothing effectiveness score based on these changes, and determine whether to automatically adjust the soothing strategy based on the score. This includes: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

6. A human-computer interaction control system for monitoring a scene, characterized in that, The human-computer interaction control system includes: The Parents Digital Human Database Building Module is used to build a parents digital human database based on the collected multimodal data of parents. The emotion recognition root class module is used to collect the baby's real-time state information and use the emotion recognition model to obtain the baby's current emotion category based on the real-time state information. The automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implement the soothing strategy. The automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of soothing strategies, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score.

7. The human-computer interaction control system according to claim 6, characterized in that, The parent digital human database establishment module is used to establish a parent digital human database based on the collected multimodal data of parents, including: Multimodal data of parents is collected through mobile terminals or the robot itself, wherein the multimodal data includes voice samples, facial images, video clips and facial expression data; The multimodal data of the parents is preprocessed to obtain preprocessed multimodal data, wherein the preprocessing includes audio denoising, face alignment, image normalization, audio segmentation, and feature standardization. Feature extraction is performed on the preprocessed multimodal data to obtain the human characteristic parameter data corresponding to the parents. The human characteristic parameter data includes facial feature vector, timbre feature vector, mouth shape driving parameters and expression mapping parameters. A digital database of parents is established based on the characteristic parameters of the parents.

8. The human-computer interaction control system according to claim 6, characterized in that, The emotion recognition root class module is used to collect the infant's real-time state information and use the emotion recognition model to obtain the infant's current emotion category based on the real-time state information, including: The system collects real-time status information of the infant through a sensing module, which includes an RGB camera, an infrared camera, a microphone array, a motion sensor, and a distance sensor. The real-time status information of the infant includes facial images, body movement information, cries, environmental sounds of the infant's environment, rolling over, and limb movements. The real-time status information of the infants collected is cached according to a preset fixed time window to form a status data packet; wherein, the status data packet contains image frame sequence, audio spectrum and action time sequence; The state data packet is input into the emotion recognition model, and visual features, audio features, and action features are extracted by the emotion recognition model. The emotion recognition model uses a multimodal fusion mechanism to fuse visual features, audio features, and motion features to output the infant's emotion category, which includes hunger, drowsiness, fright, anxiety, normal crying, and emotional stability.

9. The human-computer interaction control system according to claim 6, characterized in that, The automatic soothing strategy generation module is used to automatically generate a corresponding soothing strategy based on the infant's current emotional category, generate a parent digital human video corresponding to the current soothing strategy based on the parent digital human database, and implement the soothing strategy, including: The system automatically generates a corresponding soothing strategy based on the infant's current emotional category; wherein, the soothing strategy includes voice content, voice speed, volume, digital human facial expressions, light color, and robot movement amplitude; The digital human driving model is retrieved, and the human feature parameter data in the parent digital human database is retrieved using the digital human driving model. The human feature parameter data is combined with the current soothing strategy to generate the image, target voice, facial expression parameters and mouth shape driving parameters of the parents corresponding to the soothing measurement. Using the images of the parents, target speech, facial expression parameters, and lip-syncing parameters, digital human videos of the parents are generated in real time, featuring lip-syncing, facial movements, eye movements, and head movements. During the implementation of the soothing strategy, the video of the parents' digital human is played on the robot's display screen. At the same time, soothing voice is output through the robot's speaker, corresponding ambient lighting is output through the robot's lighting module, and slight swaying or moving movements are performed through the robot's motion module.

10. The human-computer interaction control system according to claim 6, characterized in that, The automatic strategy adjustment module is used to monitor changes in the infant's state during the implementation of the soothing strategy, generate a soothing effect score based on the changes in the infant's state, and determine whether to automatically adjust the soothing strategy based on the soothing effect score, including: During the implementation of the soothing strategy, the changes in the infant's state are monitored in real time, including changes in the degree of crying, the stability of movement, and changes in emotional state. A soothing effect score is generated based on the degree of change in the crying sound, the stability of the movement, and the change in the emotional state. The generated soothing effect score is compared with a preset score threshold. When the generated soothing effect score is lower than the preset score threshold, the soothing strategy is automatically adjusted. The adjustment of the soothing strategy includes modifying the voice content, adjusting the digital human's facial expressions, changing the lighting parameters, and reconfiguring the movement rhythm.

Images