Intelligent robot toy interaction control method and system based on multi-modal perception

Abstract

The application relates to the technical field of artificial intelligence and robot interaction, and discloses an intelligent robot toy interaction control method and system based on multi-modal perception, which comprises the following steps: preprocessing multi-source original data streams to obtain a data set; extracting and fusing multi-modal features to obtain a feature vector; using a long short-term memory network and an attention mechanism to perform state inference and obtain a comprehensive state vector containing a sentiment label and a behavior probability; searching for facial display driving parameters, using a convolutional neural network to combine ambient light and user distance for fine tuning, and obtaining a basic expression parameter set; combining a motion state to perform action planning and smoothing processing to obtain joint motion trajectory parameters; performing time synchronization, redundancy elimination and bit width mapping on the expression parameters and the trajectory parameters to obtain a coherent control sequence; and executing the sequence and correcting deviations according to sensor feedback. The application can realize high-precision cooperation of robot expressions and actions, and improve the naturalness and real-time performance of interaction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and robot interaction technology, and in particular to an intelligent robot toy interaction control method and system based on multimodal perception. Background Technology

[0002] Currently, in the field of intelligent robot toy R&D, human-computer interaction technology is key to realizing the core value of products, and its development directly determines the naturalness and consistency of robot-child interaction. The optimization and innovation of human-computer interaction technology has become a core direction for industry technological upgrading.

[0003] In one existing technology, the interaction scheme of intelligent robot toys achieves interaction through a preset fixed program or a single voice signal response. Specifically, it collects children's voice commands through a microphone, matches them with a preset command library through a built-in program, and triggers corresponding preset actions or voice responses to complete the basic human-computer interaction process. However, for the core technical requirements of multi-source information fusion and intent prediction, this type of interaction method can only process single voice signals in practical applications. It cannot simultaneously receive multi-dimensional perception data from cameras, touch sensors, etc., and lacks the ability to collaboratively interpret multi-source heterogeneous information. It is difficult to associate children's expressions, actions, and voice signals to form a holistic judgment, and there is no real-time intent prediction mechanism, resulting in a lag in interaction response and an inability to achieve effective fusion of multimodal information and dynamic intent recognition.

[0004] In summary, existing technologies cannot achieve real-time fusion and collaborative interpretation of multi-source heterogeneous sensory information, making it difficult for intelligent robot toys to accurately capture children's interactive information and respond in a timely manner. Summary of the Invention

[0005] This invention provides an intelligent robot toy interactive control method and system based on multimodal perception, so as to realize the real-time fusion and collaborative interpretation of multi-source heterogeneous perception information.

[0006] In a first aspect, to address the aforementioned technical problems, this invention provides an intelligent robot toy interaction control method based on multimodal perception, comprising: Collect raw data streams from multiple sources and preprocess the raw data streams from multiple sources to obtain a multi-source data set; Multimodal features are extracted from the multi-source dataset, and the multimodal features are simultaneously fused to obtain a multimodal feature vector; The multimodal feature vector is input into a pre-trained long short-term memory network to obtain an emotion hidden representation vector. An attention mechanism is used to dynamically assign weights to the emotion hidden representation vector to obtain a comprehensive state vector containing emotion state labels and behavior probability distributions. Based on the emotional state label, facial display driving parameters are retrieved from a preset emotional mapping library. A pre-trained convolutional neural network model is used to calculate fine-tuning coefficients by combining real-time ambient lighting and user distance. The facial display driving parameters are then adaptively fine-tuned to obtain a set of basic expression parameters. The robot's current motion state is combined with the comprehensive state vector to perform motion planning processing to obtain the joint motion trajectory, and the joint motion trajectory is then subjected to motion smoothing processing to obtain the joint motion trajectory parameters; The basic facial expression parameter set and the joint motion trajectory parameters are synchronized in time to obtain a synchronization control command. Redundancy elimination is performed on the synchronization control command to obtain an optimized control command. Bit-width mapping is performed on the optimized control command to obtain a coherent control sequence. The sequential control sequence is executed to drive the facial display screen and the servo motors of each joint, and sensor feedback data is collected and deviation correction is performed to obtain the final interactive effect.

[0007] Secondly, the present invention provides an intelligent robot toy interaction control system based on multimodal perception, comprising: The data preprocessing module is used to collect multi-source raw data streams and preprocess the multi-source raw data streams to obtain a multi-source data set; The feature synchronization fusion module is used to extract multimodal features from the multi-source dataset and perform synchronous fusion processing on the multimodal features to obtain a multimodal feature vector. The comprehensive state modeling module is used to input the multimodal feature vector into a pre-trained long short-term memory network to obtain the sentiment hidden representation vector, and to use an attention mechanism to dynamically assign weights to the sentiment hidden representation vector to obtain a comprehensive state vector containing sentiment state labels and behavioral probability distributions. The facial expression parameter adaptive module is used to retrieve facial display driving parameters from a preset emotion mapping library based on the emotion state label, calculate fine-tuning coefficients using a pre-trained convolutional neural network model combined with real-time acquired ambient lighting and user distance, and adaptively fine-tune the facial display driving parameters to obtain a basic set of facial expression parameters. The motion planning and smoothing module is used to perform motion planning processing on the comprehensive state vector in combination with the robot's current motion state to obtain joint motion trajectories, and to perform motion smoothing processing on the joint motion trajectories to obtain joint motion trajectory parameters. The instruction optimization and integration module is used to synchronize the basic facial expression parameter set with the joint motion trajectory parameters in time to obtain a synchronous control instruction, perform a redundancy elimination operation on the synchronous control instruction to obtain an optimized control instruction, and perform bit-width mapping processing on the optimized control instruction to obtain a coherent control sequence. The drive feedback correction module is used to execute the coherent control sequence to drive the face display screen and the servo motors of each joint, collect sensor feedback data and perform deviation correction processing to obtain the final interactive effect.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention acquires multi-source raw data streams of images, audio, pressure, and acceleration simultaneously, and after targeted preprocessing, completes timestamp integration and feature fusion to obtain multimodal feature vectors. This captures more comprehensive interactive features of children than existing single speech signals, such as the coordinated state of facial expressions and speech intonation. This overcomes the shortcomings of the one-sided information acquisition in existing technologies, significantly improves the completeness of interactive information representation, provides a precise data foundation for subsequent emotion recognition and intent judgment, and thus improves the initial accuracy of interactive control.

[0009] (2) This invention inputs multimodal feature vectors into a long short-term memory network for temporal analysis, and dynamically allocates weights by combining an attention mechanism to generate a comprehensive state vector containing emotional state labels and behavioral probability distributions. This processing method makes up for the shortcomings of existing technologies in capturing children's dynamic emotions and implicit intentions, significantly enhances the accuracy of interaction state judgment, and enables robots to accurately meet children's needs, avoiding rigid and disconnected interaction behaviors.

[0010] (3) This invention retrieves facial display driving parameters based on emotion state tags, calculates fine-tuning coefficients by combining real-time ambient lighting and user distance, and optimizes joint movement trajectories to achieve adaptive matching of expressions and movements. This solves the problems of incompatible interaction response and scene, and stiff movements in existing technologies, significantly improving the naturalness of expression display and the smoothness of joint movements, and enhancing the vividness of human-computer interaction.

[0011] (4) This invention synchronously optimizes facial expression and joint motion parameters to generate a coherent control sequence, and combines sensor feedback data with a PID algorithm to correct deviations in real time, forming a closed-loop control system. This effectively eliminates the limitations of redundant interactive instructions and large execution deviations in existing technologies, significantly improves the coherence and reliability of interactive control, and fundamentally solves the core pain point of delayed interactive response. Attached Figure Description

[0012] Figure 1 This is a schematic flowchart of the intelligent robot toy interaction control method based on multimodal perception provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the intelligent robot toy interactive control system based on multimodal perception provided in the second embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Reference Figure 1 The first embodiment of the present invention provides an intelligent robot toy interaction control method based on multimodal perception, comprising the following steps: S11, Collect multiple source raw data streams and preprocess the multiple source raw data streams to obtain a multi-source data set; S12, extract multimodal features from the multi-source data set, and perform synchronous fusion processing on the multimodal features to obtain a multimodal feature vector; S13, input the multimodal feature vector into a pre-trained long short-term memory network to obtain an emotion hidden representation vector, and use an attention mechanism to dynamically assign weights to the emotion hidden representation vector to obtain a comprehensive state vector containing emotion state labels and behavior probability distributions; S14, retrieve facial display driving parameters from a preset emotion mapping library based on the emotion state label, calculate fine-tuning coefficients using a pre-trained convolutional neural network model combined with real-time acquired ambient light and user distance, and adaptively fine-tune the facial display driving parameters to obtain a set of basic expression parameters. S15, Combine the robot's current motion state with the comprehensive state vector to perform motion planning processing to obtain the joint motion trajectory, and perform motion smoothing processing on the joint motion trajectory to obtain the joint motion trajectory parameters; S16, synchronize the basic facial expression parameter set with the joint motion trajectory parameters in time to obtain a synchronization control command, perform redundancy elimination operation on the synchronization control command to obtain an optimized control command, and perform bit-width mapping processing on the optimized control command to obtain a coherent control sequence. S17, execute the coherent control sequence to drive the face display screen and each joint servo motor to operate, collect sensor feedback data and perform deviation correction processing to obtain the final interactive effect.

[0015] In step S11, multiple source raw data streams are acquired, and the multiple source raw data streams are preprocessed to obtain a multi-source data set, including: S111, synchronously acquires multi-source raw data streams including image data, audio data, pressure data, and acceleration data; S112, the image data is used to extract features using a pre-trained facial key point regression model to obtain facial key point coordinates, and the image data is subjected to adaptive histogram equalization to obtain structured image data. S113, Perform a short-time Fourier transform on the audio data to obtain the power spectral density, and perform noise reduction processing on the audio data based on the power spectral density to obtain an effective sound segment; S114, the pressure data and the acceleration data are smoothed and filtered respectively to obtain filtered pressure data and filtered acceleration data, and the facial key point coordinates, the structured image data, the effective sound segments, the filtered pressure data and the filtered acceleration data are timestamped and integrated to obtain a multi-source data set.

[0016] In step S111, a multi-source raw data stream containing image data, audio data, pressure data, and acceleration data is synchronously acquired. Specifically, hardware-level synchronous triggering is performed through a high-resolution camera, microphone array, touch sensor, and inertial measurement unit. A unified trigger signal is used to control the acquisition of each module, ensuring that the multi-channel data are consistent at the starting point of time. Image data is acquired at a fixed sampling frequency of 30 frames per second, which is a common real-time visual acquisition standard for intelligent interactive devices, ensuring smooth expression and motion tracking. Its physical resolution is set to 1920x1080 pixels, which is a standard configuration for high-definition image acquisition and can meet the needs of fine extraction of facial key points. The sampling rate of audio data is set to 44.1kHz, and the bit depth is 16 bits. This combination is a common standard configuration in the field of voice interaction, which can fully cover the frequency range of children's voices and reduce quantization noise. The sampling frequencies of pressure data and acceleration data are configured to 100Hz and 200Hz, respectively. 100Hz is a commonly used sampling frequency for touch interaction, which can stably reflect pressure changes, and 200Hz is a commonly used sampling frequency for motion detection, which can fully capture the details of rapid limb movements. Each hardware acquisition channel is controlled by a unified system clock source, with a clock synchronization error of less than 1 millisecond, reducing hardware time difference in the initial acquisition phase. It should be noted that while simultaneously acquiring image data, audio data, pressure data, and acceleration data, ambient light intensity data and user distance data are also acquired simultaneously. Ambient light intensity is obtained through a photosensor, and user distance is obtained through a depth camera or ultrasonic sensor.

[0017] In step S112, the facial keypoint regression model is constructed using a supervised descent method based on cascaded regression trees. During offline training, the model uses a standardized face dataset, such as the 300-W dataset, containing tens of thousands of images covering different ethnicities, lighting conditions, and occlusion angles. During training, shape index features are used for initial feature point localization. By minimizing the exponential loss function between the predicted shape and the manually labeled shape, a gradient boosting algorithm is used to iteratively generate multiple regression trees layer by layer. Each tree is responsible for predicting the residual of the previous layer. Preferably, the depth of the regression trees is set to 5 layers, and the number of cascaded layers is set to 10 layers. This parameter combination was determined experimentally based on a balance between model complexity and inference speed, aiming to ensure millisecond-level real-time inference on the embedded processor of the smart toy.

[0018] In practical applications, a detection algorithm based on a Haar feature cascade classifier is first used to perform preliminary face region localization. During the search process, the detection window size is set to 24x24 pixels, and the scaling factor is configured to 1.1. This setting is based on the general detection performance evaluation results of OpenCV and can effectively cover facial targets at different depths while taking into account computational overhead. After locking the face target region, the cropped local image is input into the pre-trained facial keypoint regression model to extract the coordinates of 68 standardized facial keypoints containing contours, facial features, and expression details.

[0019] Furthermore, to address the issue of insufficient image contrast caused by uneven ambient lighting or high background complexity, a contrast-limited adaptive histogram equalization algorithm is applied. During processing, the contrast limit coefficient is set to 2.0, and the local computation grid size is set to 8x8. These values ​​are based on general optimization experience in image processing for facial texture enhancement, and can significantly improve the gradient characteristics of facial edges without introducing significant noise amplification. Finally, the extracted facial keypoint coordinates are used as feature vectors and encapsulated with the equalized image data to construct structured image data. This structured data, by decoupling geometric features from pixel features, improves the robustness of subsequent emotion recognition logic to environmental changes.

[0020] In step S113, a short-time Fourier transform is performed on the audio data to obtain the power spectral density. Based on this power spectral density, noise reduction processing is applied to the audio data to obtain effective sound segments. Specifically, a Hanning window is used to perform frame-by-frame windowing on the original audio stream. This window type is a common window function for speech signal processing, effectively reducing spectral leakage. The time window length is set to 1024 sampling points, corresponding to a duration of approximately 23.2 milliseconds, which conforms to the common analysis window length for the short-time stationary characteristics of speech signals. A 50% frame overlap rate is configured, a standard setting for speech spectrum analysis that ensures spectral continuity and reduces the picket fence effect. The time-domain signal is projected to the frequency domain using a short-time Fourier transform, and the power spectral density at each frequency point is calculated. Subsequently, a Wiener filtering-based noise reduction algorithm is used to estimate the power spectrum of non-stationary noise based on the prior distribution, and a noise reduction gain lower limit threshold is set to -60dB. This threshold is a common setting for speech noise reduction, effectively filtering out steady-state environmental noise while preserving clear speech components. This operation recovers the speech signal from noisy frequencies by minimizing the mean square error criterion, improving the signal-to-noise ratio of the output signal by approximately 15 dB. This improvement is verified through standard environmental noise testing. By calculating the energy distribution and zero-crossing rate of each frame, segments with energy higher than the average of silent segments and zero-crossing rates within the normal speech range are identified as valid sound segments and extracted.

[0021] In step S114, the pressure data and acceleration data are smoothed and filtered to obtain filtered pressure data and filtered acceleration data, respectively. The facial key point coordinates, structured image data, effective sound segments, filtered pressure data, and filtered acceleration data are then timestamped and integrated to obtain a multi-source data set. Specifically, considering the thermal noise and external electromagnetic interference experienced by the sensor during static or dynamic interaction, a Kalman filter algorithm with linear minimum mean square error estimation is used for time-series prediction and correction. Regarding parameter configuration, based on the hardware error characteristics of consumer-grade sensors and test results from actual toy interaction environments, the process noise covariance is set to 0.01, and the measurement noise covariance is set to 0.1. These two sets of parameters are common configurations for filtering consumer-grade haptic and motion sensors. Real-time smoothing of pressure and acceleration values ​​is achieved through iterative calculation of the prediction residual and Kalman gain. The variance of the processed pressure data decreases from the original 0.5 to 0.2, and the variance of the acceleration data decreases from 0.8 to 0.3. This change is verified through actual acquisition and filtering comparison tests, significantly enhancing the statistical stability of the signal.

[0022] Furthermore, for the heterogeneous data processed above, time alignment is performed based on the global system timestamp. Specifically, using the sampling time of the image frame as a reference, linear interpolation is used to uniformly map the asynchronously acquired audio, pressure, and acceleration data to the image time axis. Linear interpolation is a common method for multi-sensor time alignment, controlling the synchronization error of multi-source data to within 5 milliseconds. This error range meets the synchronization requirements of multimodal real-time interaction. Finally, facial keypoint coordinates, structured images, effective sound segments, filtered pressure data, and filtered acceleration data are encapsulated in the form of multi-dimensional tensors to construct a multi-source data set. It is worth noting that this step, by establishing strong spatiotemporal correlations between data, provides a reliable underlying input for subsequent multimodal feature fusion and real-time emotion intent prediction.

[0023] In step S12, multimodal features are extracted from the multi-source dataset, and the multimodal features are simultaneously fused to obtain a multimodal feature vector, including: S121, extract the facial key point coordinates, the structured image data, the effective sound segments, the filter pressure data, and the filter acceleration data from the multi-source data set; S122, calculate the distance feature by Euclidean distance of the facial key point coordinates, perform pose estimation processing on the structured image data to obtain the pose feature vector, and perform feature aggregation processing on the distance feature and the pose feature vector to obtain the image feature vector. S123, Mel frequency cepstral coefficient extraction is performed on the effective sound segment to obtain the audio feature vector; S124, perform time-domain statistical feature extraction processing on the pressure data and the acceleration data respectively to obtain pressure feature vector and acceleration feature vector, and perform vector concatenation processing on the pressure feature vector and the acceleration feature vector to obtain motion feature vector; S125, the image feature vector, the audio feature vector and the motion feature vector are time-stamp aligned using a linear interpolation algorithm to obtain a synchronization feature set, and the synchronization feature set is vector-joined to obtain a multimodal feature vector.

[0024] In step S121, the facial keypoint coordinates, structured image data, effective sound segments, filter pressure data, and filter acceleration data are extracted from the multi-source dataset. Specifically, the preprocessed modal data is unpacked and reconstructed by accessing a preset data buffer or memory object. This process ensures that subsequent feature extraction steps can be performed based on a synchronized and denoised high-quality data source, providing a consistent input foundation for cross-modal correlation analysis.

[0025] In step S122, Euclidean distance is calculated to obtain distance features from the facial keypoint coordinates. Pose estimation is performed on the structured image data to obtain a pose feature vector. Finally, feature aggregation is performed on the distance features and the pose feature vector to obtain an image feature vector. Specifically, for the structured image data, a pose estimation algorithm based on a deep convolutional neural network is used to extract human topological features. The pre-trained model corresponding to the pose estimation algorithm is trained offline using a large-scale human pose dataset. At the model architecture level, the model adopts a two-branch, multi-stage cascaded architecture. The first branch uses a deep convolutional neural network to extract global image features, generating image feature maps through continuous residual modules and transposed convolutional layers. The second branch contains two parallel prediction subnetworks, used to generate confidence maps for each joint of the human body and partial affinity fields representing the direction of limb connections, respectively.

[0026] During training, a supervised learning strategy was employed to iteratively optimize the model parameters. First, a confidence map loss function was defined to calculate the mean square error between the model-predicted joint positions and the manually labeled true positions, guiding the model to learn the spatial distribution characteristics of the joints. Simultaneously, a partial affinity field loss function was defined to calculate the cosine similarity loss of the limb vector field in terms of direction and amplitude, improving the model's robustness in reconstructing human topology under complex occlusion environments.

[0027] Furthermore, at the convergence control level of model training, an optimizer based on stochastic gradient descent is employed, and a learning rate decay mechanism is introduced. The convergence criterion for the training process is set as follows: when the mean accuracy on the validation set fluctuates less than 0.001 over 10 consecutive periods, or when the gradient norm of the global loss function is lower than a preset 1e-4 threshold, the model is considered to have reached convergence. In addition, to prevent overfitting, a dropout method and a weight decay term are introduced during training. Preferably, data augmentation is performed on samples with different lighting intensities and shooting angles to ensure that the model has a strong generalization ability to the interactive environment of intelligent toys after convergence.

[0028] In practical applications, the trained convolutional neural network is used to generate multi-scale feature maps from the input structured image data. Through the model's prediction branch, confidence maps for each joint position are first generated, and anatomical relationships between joints are established by combining some affinity fields, thereby identifying and outputting the coordinates of 18 key nodes, including the trunk and limbs. By calculating the vector direction formed by the coordinates of each node, the trunk tilt angle and limb extension features reflecting the user's body language are analyzed.

[0029] It is worth noting that comparative experimental analysis shows a high statistical correlation between the trend of facial expression changes and the amplitude of body posture swaying. Finally, the distance features calculated from facial key points and the posture feature vector extracted by the convolutional neural network are aggregated to construct a 128-dimensional image feature vector. This aggregation process, by spatially reconstructing local micro-expression features and macro-body posture features, ensures that the image feature vector can comprehensively and robustly characterize the user's emotional expression intentions.

[0030] In step S123, the effective sound segments are processed by Mel frequency cepstral coefficient extraction to obtain audio feature vectors. Specifically, after pre-emphasis, framing, and windowing of the effective sound segments, Mel filter banks are used to simulate the auditory characteristics of the human ear. Preferably, the number of Mel filter banks is set to 26. Within the frequency range of human auditory perception, frequency perception has a non-linear relationship with physical frequency. By using 26 filters in a non-uniform distribution at the Mel scale, sufficient frequency domain resolution can be provided to cover the key speech formant regions, while balancing the computational load of feature extraction with the degree of information preservation, ensuring extremely high feature expressiveness in the low-frequency band where the speech signal energy is concentrated.

[0031] Subsequently, a discrete cosine transform is performed on the logarithmic energy spectrum obtained through the filter bank. Preferably, the first 13 cepstral coefficients reflecting the shape of the vocal tract are extracted. In the cepstral domain, the first 13 low-frequency components of the speech signal concentrate most of the envelope information of the signal energy, which can accurately characterize the physical shape characteristics of the vocal tract and the state of the vocal organs, while the high-frequency components usually correspond to the details of the excitation source or background noise. By extracting the first 13 coefficients, efficient dimensionality reduction of features can be achieved and the correlation between coefficients can be eliminated, thereby improving the robustness of the subsequent intent prediction model to different speakers and environmental interference.

[0032] Furthermore, by combining the fundamental frequency value estimated from the audio signal with the signal energy characteristics, a 40-dimensional audio feature vector is finally encapsulated. This dimensional configuration can effectively capture abrupt changes in frequency domain energy distribution caused by user emotional excitement or specific contexts, providing robust acoustic evidence for subsequent emotion classification.

[0033] In step S124, time-domain statistical feature extraction is performed on the pressure data and the acceleration data to obtain pressure feature vectors and acceleration feature vectors, respectively. Then, the pressure feature vectors and acceleration feature vectors are concatenated to obtain motion feature vectors. Specifically, statistical indicators of the pressure data, including the mean pressure, standard deviation, and peak pressure, are calculated within a sliding window; simultaneously, statistical indicators of the acceleration data, including the mean acceleration, variance, and rate of change, are calculated. These statistical indicators effectively reflect the intensity of tactile interaction and the frequency of limb movement. By concatenating these indicators into a 20-dimensional feature vector, the aim is to accurately characterize the coordinated changes in touch and motion.

[0034] In step S125, a synchronous feature set is obtained by aligning the image feature vector, audio feature vector, and motion feature vector using a linear interpolation algorithm to achieve timestamp alignment. The synchronous feature set is then vector-concatenated to obtain a multimodal feature vector. Specifically, considering the differences in computational time of different modal feature extraction algorithms, a linear interpolation algorithm is used to resample the asynchronously generated feature vectors, strictly controlling the timestamp error to within 3 milliseconds to construct the synchronous feature set. Subsequently, the image, audio, and motion feature vectors are vector-concatenated in a preset order to generate a 188-dimensional comprehensive multimodal feature vector. Before generating the final vector, principal component analysis is preferably used to orthogonally transform the concatenated high-dimensional vector. By extracting principal components with a cumulative contribution rate exceeding 95%, linear redundancy between features is eliminated while preserving core semantic information, reducing the overlap of features to approximately 5%. This dimensionality reduction operation not only reduces the input complexity of the subsequent intent prediction model but also improves the model's inference response speed in real-time interactive scenarios.

[0035] In step S13, the multimodal feature vector is input into a pre-trained long short-term memory network to obtain an emotion hidden representation vector. An attention mechanism is then used to dynamically assign weights to the emotion hidden representation vector, resulting in a comprehensive state vector containing emotion state labels and behavioral probability distributions, including: S131, the multimodal feature vector is input into a pre-trained long short-term memory network for temporal dependency analysis to obtain the sentiment hidden representation vector, and the sentiment hidden representation vector is normalized exponentially mapped to obtain the sentiment state label. S132, use an attention mechanism to calculate the weights of the emotion hidden representation vector to obtain dynamic weight coefficients, and then perform weighted integration on the emotion hidden representation vector according to the dynamic weight coefficients to obtain a weighted feature vector; S133, the weighted feature vector is linearly transformed using a fully connected layer to obtain the behavior probability distribution, and a comprehensive state vector is constructed by combining the emotion state label and the behavior probability distribution as feature vectors.

[0036] In step S131, the multimodal feature vector is input into a pre-trained long short-term memory network for temporal dependency analysis to obtain an emotion hidden representation vector, and the emotion hidden representation vector is normalized exponentially mapped to obtain an emotion state label.

[0037] Specifically, the Long Short-Term Memory (LSTM) network effectively captures the long-term evolution of multimodal features over time by introducing control mechanisms such as forget gates, input gates, and output gates. The LTM network model is trained offline using a labeled sentiment time-series dataset. At the model architecture level, the number of hidden layer units is set to 64, a scale that ensures nonlinear fitting capability while avoiding excessive computational overhead on the embedded side. During training, a backpropagation algorithm is used for gradient updates, and the model is guided to learn the temporal correlations between features by minimizing the cross-entropy loss function. The convergence criterion is set as follows: the loss value on the validation set no longer decreases for 15 consecutive cycles, or the accuracy reaches 85% or higher.

[0038] In practical applications, a 188-dimensional integrated multimodal feature vector is input into the model, and the sliding time window is set to the past 10 frames of data, with a sampling interval of 100 milliseconds per frame. This window length is based on the fact that typical human facial expressions and body language interactions usually last about one second. Covering the data with 10 frames ensures the completeness of emotional semantic capture, while the 100-millisecond sampling precision effectively filters out high-frequency physical noise and retains significant interaction features. A pre-trained Long Short-Term Memory (LSTM) network is used to perform recursive calculations on the data sequence within the window. Pearson correlation analysis of the historical feature vectors yields an autocorrelation coefficient of approximately 0.72 over time. This coefficient reflects the high temporal coherence of the multimodal features, demonstrating that emotional states have a stable evolutionary trend, thus ensuring the high predictive reliability of the hidden representation vector output by the LSM network. Subsequently, a normalized exponential mapping is applied to the output 32-dimensional emotional hidden representation vector, converting the numerical values ​​in the feature space into confidence probabilities representing each emotional category, and selecting the category with the highest probability value as the emotional state label.

[0039] In step S132, an attention mechanism is used to calculate the weights of the emotion hidden representation vector to obtain dynamic weight coefficients, and the emotion hidden representation vector is weighted and integrated according to the dynamic weight coefficients to obtain a weighted feature vector.

[0040] Specifically, the attention mechanism is used to dynamically identify the importance of different modal features in complex interactive scenarios. By calculating the energy distribution of the sentiment hidden representation vector across different feature dimensions, corresponding dynamic weight coefficients are generated. This process aims to allocate resources based on the real-time contribution of features to the sentiment classification task. For example, in environments with low audio signal-to-noise ratios, the attention mechanism automatically reduces the weight coefficients corresponding to audio features and correspondingly increases the weight proportion of image or tactile features.

[0041] By performing element-wise multiplication of the obtained dynamic weight coefficients with the original sentiment hidden representation vector and then performing linear weighted summation, a weighted feature vector is obtained through weighted integration. This adaptive feature reconstruction strategy can significantly improve the model's robustness to non-steady-state environments. Actual operation tests show that, through dynamic adjustment of the attention mechanism, the system's response speed to sudden changes in user sentiment can be improved by about 15%, effectively solving the recognition delay problem that occurs when traditional fixed-weight fusion methods switch scenes.

[0042] In step S133, the weighted feature vector is linearly transformed using a fully connected layer to obtain a behavior probability distribution, and a comprehensive state vector is constructed by combining the emotion state label and the behavior probability distribution as feature vectors.

[0043] Specifically, the fully connected layer, as a computational architecture for realizing cross-dimensional mapping of the feature space, consists of an input layer, a weight matrix containing learning parameters, and a non-linear activation function. During construction, the input dimension of the fully connected layer matches the 32-dimensional space of the weighted feature vector, while the output dimension is set to 5 dimensions according to a preset set of interactive behaviors.

[0044] The training process of the fully connected layer is performed synchronously with the preceding Long Short-Term Memory (LSTM) network during the offline phase. During iterative training, the weight parameters of the fully connected layer are updated in reverse using a stochastic gradient descent algorithm combined with backpropagation, by calculating the cross-entropy loss between the predicted probability distribution and the manually labeled real-world behavior. The objective function of the training aims to achieve model convergence by minimizing the distance between the predicted distribution and the expected distribution.

[0045] It is worth noting that after the model was trained, its inference performance was evaluated using an independent validation set containing a large number of real interaction trajectory samples. Specifically, by comparing the predicted behavior probability sequence with the actual physical action ground truth, the mean absolute error between the two was calculated. Statistical analysis using two hundred sets of real children's interaction validation samples showed that the average error rate of behavior prediction remained stable within 8.2%. This evaluation result verifies the model's generalization ability when handling nonlinear intent mapping, ensuring the reliability and stability of the prediction results in real-world interactive environments.

[0046] Finally, a feature vector construction operation is performed, spatially mapping and vector concatenating the discrete label information representing the emotion judgment result with the behavior probability distribution representing the movement trend. It's worth noting that the generated comprehensive state vector is set to 5 dimensions. The essence of this technique lies in linearly projecting high-dimensional emotional semantic features onto a low-dimensional interaction decision space. Within this space, each dimension represents the confidence level of action execution under the current specific emotional constraint. This dimensionality reduction approach not only integrates the determinism of emotion labels into behavior prediction but also ensures that the final 5-dimensional comprehensive state vector can provide a temporally forward-looking driving index for subsequent facial animation rendering and joint trajectory planning of the robot toy with minimal data overhead.

[0047] Specifically, a mapping table is pre-established from sentiment labels to five-dimensional modulation coefficients, with each sentiment label corresponding to a set of five modulation coefficients; the five-dimensional vector of the behavioral probability distribution is multiplied element-wise with the corresponding five modulation coefficients to obtain a five-dimensional comprehensive state vector, and the value of each dimension of this vector is the confidence level of performing the corresponding action under the corresponding sentiment constraint.

[0048] In step S14, facial display driving parameters are retrieved from a preset emotion mapping library based on the emotion state label. A pre-trained convolutional neural network model is used to calculate fine-tuning coefficients based on real-time ambient lighting and user distance. The facial display driving parameters are then adaptively fine-tuned to obtain a set of basic expression parameters, including: S141, Based on the emotional state tags, the initial facial display driving parameters are obtained by key-value indexing of the preset emotional mapping library; S142, acquire real-time ambient lighting and user data, and use a pre-trained convolutional neural network model to perform nonlinear feature mapping on the ambient lighting and the distance to the user to obtain fine-tuning coefficients; S143, The initial facial display driving parameters are numerically corrected according to the fine-tuning coefficient to obtain the target driving parameters; S144, the target driving parameters are verified for inter-frame smoothness using a preset rendering engine to obtain a set of basic expression parameters.

[0049] In step S141, the initial facial display driving parameters are obtained by indexing the preset emotion mapping library according to the emotion state label.

[0050] Specifically, the emotion mapping library is constructed through statistical analysis of a large amount of real facial expression data. During the construction phase, facial image samples covering different emotion categories are collected, and principal component analysis is used to extract key motion features of facial muscle groups, transforming them into structured driving parameters. The emotion mapping library uses emotion state labels as keys and driving values ​​of key facial features as corresponding values. For example, for the "happy" label, the initial facial display driving parameters obtained through key-value indexing include feature values ​​such as eyebrow raising angle, mouth corner upturn, and eye squinting ratio. It is worth noting that through cross-validation with no fewer than one thousand sets of real children's expression samples, the matching accuracy between the parameters in the mapping library and the emotion labels reaches 92.5%, ensuring the accuracy of the initial semantic expression of the emotion.

[0051] In step S142, real-time ambient lighting and user data are acquired, and a pre-trained convolutional neural network model is used to perform nonlinear feature mapping on the ambient lighting and the distance to the user to obtain fine-tuning coefficients.

[0052] Specifically, the convolutional neural network model employs a deep learning architecture combining a multi-layer perceptual structure with a one-dimensional convolution operator. During construction, the input layer is designed as a dual-channel vector space, carrying the ambient light intensity acquired by a photosensor and the user distance measured by a depth camera or ultrasonic sensor, respectively. To enable the model to understand the physical meaning under different dimensions, feature normalization based on standard deviation is performed on the input data before it enters the convolutional layer.

[0053] The model is trained offline using a visual preference dataset simulating complex lighting environments and multi-scale viewing distance variations. The model architecture comprises multiple sequentially connected feature extraction layers, utilizing convolutional kernels to mine spatial features of environmental parameters. Linear rectified units, such as ReLU, are used to introduce non-linear activation characteristics, simulating the non-linear compensation behavior of the human visual system in perceiving facial expressions under different environmental constraints. During training iterations, the mean squared error function is used as the loss function to evaluate prediction accuracy, and backpropagation is employed to adjust the weight parameters of each neuron. Convergence is determined when the validation set loss function curve enters a stable period and the gradient norm decreases to below 1e-5, ensuring the model can establish a robust mapping relationship from environmental perturbations to visual optimization coefficients.

[0054] In practical applications, real-time collected illumination data and user distance data are input into the trained model. Through layer-by-layer feature mapping of the hidden layers within the model, the original physical quantities are transformed into deep semantic features that can characterize visual compensation needs. Finally, the output layer produces a multi-dimensional fine-tuning coefficient vector through a fully connected linear transformation. This fine-tuning coefficient vector contains gain correction values ​​for facial display contrast, saturation, and key motion amplitude. Experimental tests show that using the fine-tuning coefficients generated by this model for real-time correction can improve the naturalness score of the generated facial animation under different complex backgrounds by approximately 18.3%, while controlling the end-to-end inference response latency to within 50 milliseconds, meeting the stringent timeliness requirements of real-time interaction in smart toys.

[0055] In step S143, the initial facial display driving parameters are numerically corrected according to the fine-tuning coefficient to obtain the target driving parameters.

[0056] Specifically, adaptive compensation of the parameters is achieved by performing element-wise weighted operations on the fine-tuning coefficients and the initial facial display driving parameters. This numerical correction process not only considers the baseline of emotional expression but also incorporates the constraints of the current physical environment. For example, based on the fine-tuning coefficients, incremental angle compensation is performed on the eyebrow raising angle, and linear scaling is applied to the mouth corner raising amplitude, thereby generating target driving parameters that can adapt to specific lighting and viewing distance conditions. This dynamic fine-tuning mechanism ensures that the robot's expressions maintain visual consistency and vividness in different interaction scenarios.

[0057] In step S144, the target driving parameters are verified for inter-frame smoothness using a preset rendering engine to obtain a set of basic expression parameters.

[0058] Specifically, the target driving parameters are input into a rendering engine built based on a graphics library interface to perform virtual rendering simulation. The preset rendering engine is built using an open-source embedded graphics library, pre-configured with a display resolution corresponding to the physical pixel size of the facial display, and employs a preset double-buffering rendering mechanism to eliminate screen tearing. During verification, the inter-frame smoothness of the facial animation is evaluated by calculating the rate of change of driving parameters between consecutive frames. The judgment criterion is set at an inter-frame jitter rate of less than 0.05, meaning the second derivative of parameters between adjacent frames must be within a preset stable threshold range to eliminate visual flickering or abruptness caused by parameter mutations. This threshold is determined through visual smoothness testing of continuous facial animations. In actual verification using no fewer than two hundred sets of children's interactive scenarios, when the inter-frame jitter rate is below this value, the human eye cannot perceive abrupt changes or stuttering in the facial animation, ensuring a natural and smooth effect for the facial expression display. If the verification passes, the current parameters are encapsulated into a basic facial expression parameter set. In addition, the system introduces a closed-loop optimization mechanism based on user feedback, which fine-tunes the parameter mapping logic based on user satisfaction evaluations in historical interactions, thereby ensuring that facial expression details remain natural and smooth over long periods of operation.

[0059] Specifically, the inter-frame jitter rate threshold was determined by creating multiple sets of jitter rate gradient animations, which were then viewed by twenty evaluators under uniform lighting conditions. Each evaluator recorded the lowest jitter rate at which they could perceive stuttering, and the average of all evaluators' thresholds was taken and rounded down. Verification using two hundred sets of interactive scenarios with children showed that when the jitter rate was below 0.05, children did not experience any stuttering.

[0060] In step S15, the comprehensive state vector is processed for motion planning based on the robot's current motion state to obtain joint motion trajectories. The joint motion trajectories are then processed for motion smoothing to obtain joint motion trajectory parameters, including: S151, Obtain the robot's current position information and joint angle information as the current motion state; S152, the target pose is mapped according to the comprehensive state vector to obtain the desired motion target, and the initial value of the joint motion trajectory is obtained by solving the spatial path in combination with the current motion state. S153, the initial value of the joint motion trajectory is optimized by continuous constraint using a quadratic programming algorithm to obtain a smooth joint motion trajectory, and the smooth joint motion trajectory is sampled and quantized to obtain the joint motion trajectory parameters.

[0061] In step S151, the robot's current position and joint angle information are acquired as its current motion state. Specifically, the current physical parameters of each degree of freedom of the robot are acquired in real time by reading encoder data set at each active joint and feedback from the inertial measurement unit. The current motion state includes the real-time absolute angles of key nodes such as the right shoulder joint and right elbow joint, as well as the coordinate pose of the robot body in the workspace. These data constitute the initial boundary conditions for dynamic calculations, ensuring the continuity of subsequent motion planning.

[0062] In step S152, the target pose is mapped based on the comprehensive state vector to obtain the desired motion target, and the initial value of the joint motion trajectory is obtained by solving the spatial path in conjunction with the current motion state. Specifically, the spatial path solution adopts a modeling technique based on dynamic motion primitives or periodic motion templates. The behavior with the highest probability distribution in the comprehensive state vector is used as the decision basis, and the high-dimensional interaction intention is mapped to the desired motion target in the workspace. For periodic actions, a preset sinusoidal motion function template is invoked. This template is constructed based on the biomechanical characteristics of periodic limb movements, conforms to the movement law of natural human swinging, and is a common method for describing periodic movements in the field of robot interaction. The urgency of the behavior is converted into the motion frequency, and the motion amplitude requirement is converted into the amplitude parameters of the corresponding joints.

[0063] During the solution process, the current motion state is used as the initial phase for alignment, and the target command sequence for each joint servo in the next action cycle is calculated. For example, for a waving motion, the angle evolution curves of the shoulder and elbow joints on a specific time axis are calculated through inverse kinematics, generating a discrete angle sequence with a time resolution of 20 milliseconds as the initial value of the joint motion trajectory. This time resolution is determined through real-time interactive response testing, which can reduce the computational load of the embedded platform while ensuring the smoothness of the action. It is worth noting that this initial value generation process, by transforming the static pose target into a dynamic sequence that changes continuously over time, initially establishes the spatiotemporal correspondence between intent and action.

[0064] In step S153, the initial value of the joint motion trajectory is optimized by continuous constraint using a quadratic programming algorithm to obtain a smooth joint motion trajectory. The smooth joint motion trajectory is then sampled and quantized to obtain the joint motion trajectory parameters.

[0065] Specifically, the quadratic programming algorithm is used to address the smoothness issue during action transitions. By constructing a cost function containing objective and constraint terms, continuous hard constraints on joint displacement, angular velocity, and angular acceleration are introduced while minimizing trajectory deviation. During optimization, the transition region between adjacent action cycles is defined as the optimization interval, and a fifth-degree polynomial is used as the basis function for trajectory reconstruction, ensuring that the angular acceleration at both the starting and ending points is zero.

[0066] Furthermore, the algorithm iteratively solves for the optimal coefficient matrix that satisfies the above constraints, eliminating potential velocity abrupt changes in the initial trajectory value, thereby obtaining a smooth joint motion trajectory that is continuous and without jumps on the second derivative. Subsequently, the smooth joint motion trajectory is sampled at equal intervals and numerically quantized according to the resolution of the servo controller, encapsulating it into joint motion trajectory parameters containing precise timestamps and target angle values. Through this smooth transition processing, the joint motion can achieve a smooth evolution of acceleration within a 200-millisecond switching time. This switching duration was determined through a child interaction compliance test, effectively avoiding the impact vibration of the hardware actuator during high-speed operation and the abruptness of the visual switching.

[0067] In step S16, the basic facial expression parameter set and the joint motion trajectory parameters are synchronized in time to obtain a synchronization control command. Redundancy elimination is performed on the synchronization control command to obtain an optimized control command. Bit-width mapping is then performed on the optimized control command to obtain a coherent control sequence, including: S161, The basic facial expression parameter set is timestamped using a linear interpolation algorithm to obtain a synchronous control command that is aligned with the joint motion trajectory parameters; S162, use a clustering algorithm to identify redundant synchronous control commands, and eliminate redundant synchronous control commands based on the identification results to obtain optimized control commands; S163, the optimized control instruction is mapped to a numerical range according to the communication protocol of the target hardware to obtain a bit-width adaptation instruction, and the bit-width adaptation instruction is timestamped to obtain a coherent control sequence.

[0068] In step S161, the basic facial expression parameter set and the joint motion trajectory parameters are mapped to a unified time axis to obtain a synchronized control command. Specifically, to ensure a high degree of visual coordination between the robot toy's facial expressions and body movements, a global high-precision clock is used as a reference to align the asynchronously generated visual rendering parameters and dynamic motion parameters in time. During processing, a linear extrapolation or nearest neighbor matching algorithm is used to compress the relative offset of the two parameter sequences to within 2 milliseconds. This synchronization accuracy is determined through a children's interactive visual coordination test. Within this accuracy range, the human eye cannot perceive the temporal deviation between facial expressions and movements, effectively eliminating the perceptual deviation between audiovisual feedback and limb movement, thus forming a synchronized control command with unified spatiotemporal attributes.

[0069] In step S162, a clustering algorithm is used to identify redundant synchronous control commands, and the synchronous control commands are eliminated based on the identification results to obtain optimized control commands.

[0070] Specifically, the clustering algorithm employs a dynamic clustering architecture based on Euclidean distance to identify highly similar motion feature clusters within a sequence of instructions. During the construction process, the joint angle vectors and facial expression driving parameters in each frame of synchronization control instructions are mapped to a high-dimensional feature space. The clustering algorithm iteratively calculates the spatial distance between sample points and cluster centers, grouping consecutive instruction frames with similar trends and minimal amplitude deviations into the same feature cluster.

[0071] It is worth noting that the cluster radius threshold is set based on a comprehensive mapping between the minimum mechanical resolution of the hardware actuator and the human visual persistence limit. Specifically, by obtaining the minimum stable step angle of the servo controller under load (this parameter is determined by the resolution index specified in the servo hardware specification manual), and combining it with the user's sensitivity threshold to facial animation flickering in the interactive experiment, the parameter is optimized using a grid search method in an offline simulation environment to determine the value that maximizes the ratio of command compression rate to action continuity score as the cluster radius.

[0072] In practical applications, when the Euclidean distance between adjacent instruction frames in the feature space is less than a certain threshold, the clustering algorithm identifies these frames as redundant instructions. This clustering-based identification mechanism can accurately distinguish between intentional stillness and meaningless jitter caused by sensor noise. For the identified redundant instruction clusters, the system retains only the starting keyframe of the cluster or calculates its mean center as the representative instruction, and performs a removal process on the remaining duplicates within the cluster.

[0073] Furthermore, through this clustering optimization process, an instruction compression rate of approximately 25% can be achieved without sacrificing the continuity of actions. This optimization strategy significantly reduces the instantaneous bandwidth pressure on the communication bus, effectively avoids communication blockage caused by excessively dense instructions, and ensures that control signals can overcome hardware channel throughput limitations to achieve real-time transmission during multi-degree-of-freedom collaborative control.

[0074] In step S17, the coherent control sequence is executed to drive the facial display screen and each joint servo motor, sensor feedback data is collected and deviation correction processing is performed to obtain the final interactive effect, including: S171, synchronously distribute the face display driving command and the joint servo command in the continuous control sequence, and drive the face display screen and each joint servo to execute synchronously. S172, Real-time acquisition of encoder data and current data of each joint servo motor as sensor feedback data; S173, using a PID control algorithm to calculate the deviation between the joint servo command and the sensor feedback data to obtain compensation parameters, and then using the compensation parameters to correct the joint servo command in real time to obtain the final interactive effect.

[0075] In step S171, the facial display drive command and the joint servo command in the continuous control sequence are synchronously distributed to drive the facial display screen and each joint servo to execute synchronously.

[0076] Specifically, the underlying bus controller breaks down the composite instruction stream into visual rendering instructions and motion control instructions. For facial displays, pixel-level rendering is performed based on the expression intensity parameters in the instructions. For example, when expressing a "surprised" expression, a linear smooth interpolation algorithm is used to calculate the brightness intensity increment of each frame within a preset 0.5-second transition period, ensuring that the pixel grayscale value increases uniformly from the initial state to the target quantization value, eliminating screen flicker. This transition period is set based on the persistence of vision characteristics of the human eye, ensuring smooth and abrupt changes in expression.

[0077] Synchronously, each joint servo receives the corresponding angle target command and plans the execution path using a trapezoidal acceleration curve. During the planning process, the initial acceleration is set to 30 degrees per second. 2 The constant speed phase speed is set to 100 degrees / second. The above parameters were determined by a combination of servo hardware performance testing and child interaction safety testing. By controlling the smooth rise and fall of acceleration, the total displacement error of a single action is suppressed to within 0.2 degrees.

[0078] In step S172, encoder data and current data of each joint servo motor are collected in real time as sensor feedback data. Specifically, to construct a closed-loop monitoring system, high-speed data transmission is performed with a sampling period of 10 milliseconds during actuator operation. This period balances feedback real-time performance and processor load, meeting the requirements of real-time closed-loop control. The actual rotation angle of the joint is obtained by reading the high-resolution incremental encoder built into the servo motor, while the current sampling circuit monitors current fluctuations during execution. The current data can indirectly reflect the torque output state of the servo motor when overcoming friction or external load, providing a physical basis for subsequent evaluation of action completion and hardware health status.

[0079] In step S173, a PID control algorithm is used to calculate the deviation between the joint servo command and the sensor feedback data to obtain compensation parameters, and the joint servo command is corrected in real time according to the compensation parameters to obtain the final interactive effect.

[0080] Specifically, the PID control algorithm employs a positional control structure, achieving dynamic deviation convergence through a linear combination of proportional, integral, and derivative dimensions. During operation, the difference between the target joint angle and the actual feedback angle is used as the deviation input, and the instantaneous deviation between the commanded target value and the actual value fed back by the sensor is calculated in real time. The proportional unit provides an immediate response to the deviation, the integral unit accumulates historical deviations to eliminate steady-state errors, and an anti-integral saturation mechanism is configured to prevent excessive integral input. Simultaneously, the derivative unit senses the rate of change of the deviation to provide anti-advance damping.

[0081] Furthermore, based on the system dynamics simulation results and actual load tests, empirical values ​​for the proportional, integral, and derivative coefficients were pre-set. These parameters were determined through robot joint system identification experiments, demonstrating good stability and response speed. Specifically, the specific values ​​of the proportional, integral, and derivative coefficients were determined by fixing the robot joint servo motor to the test bench, inputting a step angle command, and recording the response overshoot and settling time under different coefficient combinations. The coefficient combination with the overshoot less than 5% and the shortest settling time was selected as the final value. Tests showed that the typical value for the proportional coefficient was 2.5, the typical value for the integral coefficient was 0.6, and the typical value for the derivative coefficient was 0.1. By substituting the deviation value into the PID control flow, the correction increment used to compensate for the control quantity was calculated. For example, when a slight deviation in facial expression intensity or joint angle is detected, the system adjusts the drive pulse width or output current to quickly converge the deviation to a reasonable range.

[0082] It's worth noting that the system further introduces a timeline fine-tuning mechanism to handle the cumulative latency difference between facial expression rendering and action execution. Specifically, by comparing the arrival time of the facial expression peak with the completion time of the joint movement, if an asymmetry in the time span between the two is detected, a uniform extension algorithm is used to dynamically adjust the instruction execution cycle. By calculating the adjustment amount and rewriting the timestamps of subsequent instructions, the overall synchronization error between facial expressions and body movements is controlled within 20 milliseconds. This synchronization accuracy has been verified through multimodal collaborative interaction tests, and the timing deviation is imperceptible to the human eye. This multimodal closed-loop correction mechanism ensures that the robot toy can present a coordinated and extremely natural final interaction effect in complex dynamic interactions.

[0083] In summary, this invention discloses an intelligent robot toy interaction control method based on multimodal perception. By using multimodal feature collaborative perception, dynamic prediction of intent sequence, adaptive fine-tuning of environment, and secondary planning optimization of motion trajectory, it effectively solves problems such as single interaction perception dimension, lagging emotion recognition, lack of coordination between facial expressions and actions, and abrupt switching of mechanical movements. It achieves high-precision coordination and smooth and natural expression of robot toy facial expression rendering and body movements, significantly improving the real-time performance, naturalness, and robustness of the interaction process.

[0084] Reference Figure 2 The second embodiment of the present invention provides an intelligent robot toy interaction control system based on multimodal perception, comprising: The data preprocessing module is used to collect multi-source raw data streams and preprocess the multi-source raw data streams to obtain a multi-source data set; The feature synchronization fusion module is used to extract multimodal features from the multi-source dataset and perform synchronous fusion processing on the multimodal features to obtain a multimodal feature vector. The comprehensive state modeling module is used to input the multimodal feature vector into a pre-trained long short-term memory network to obtain the sentiment hidden representation vector, and to use an attention mechanism to dynamically assign weights to the sentiment hidden representation vector to obtain a comprehensive state vector containing sentiment state labels and behavioral probability distributions. The facial expression parameter adaptive module is used to retrieve facial display driving parameters from a preset emotion mapping library based on the emotion state label, calculate fine-tuning coefficients using a pre-trained convolutional neural network model combined with real-time acquired ambient lighting and user distance, and adaptively fine-tune the facial display driving parameters to obtain a basic set of facial expression parameters. The motion planning and smoothing module is used to perform motion planning processing on the comprehensive state vector in combination with the robot's current motion state to obtain joint motion trajectories, and to perform motion smoothing processing on the joint motion trajectories to obtain joint motion trajectory parameters. The instruction optimization and integration module is used to synchronize the basic facial expression parameter set with the joint motion trajectory parameters in time to obtain a synchronous control instruction, perform a redundancy elimination operation on the synchronous control instruction to obtain an optimized control instruction, and perform bit-width mapping processing on the optimized control instruction to obtain a coherent control sequence. The drive feedback correction module is used to execute the coherent control sequence to drive the face display screen and the servo motors of each joint, collect sensor feedback data and perform deviation correction processing to obtain the final interactive effect.

[0085] It should be noted that the intelligent robot toy interactive control system based on multimodal perception provided in this embodiment of the invention is used to perform all the process steps of the intelligent robot toy interactive control method based on multimodal perception in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0086] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for interactive control of intelligent robot toys based on multimodal perception, characterized in that, include: Collect raw data streams from multiple sources and preprocess the raw data streams from multiple sources to obtain a multi-source data set; Multimodal features are extracted from the multi-source dataset, and the multimodal features are simultaneously fused to obtain a multimodal feature vector; The multimodal feature vector is input into a pre-trained long short-term memory network to obtain an emotion hidden representation vector. An attention mechanism is used to dynamically assign weights to the emotion hidden representation vector to obtain a comprehensive state vector containing emotion state labels and behavior probability distributions. Based on the emotional state label, facial display driving parameters are retrieved from a preset emotional mapping library. A pre-trained convolutional neural network model is used to calculate fine-tuning coefficients by combining real-time ambient lighting and user distance. The facial display driving parameters are then adaptively fine-tuned to obtain a set of basic expression parameters. The robot's current motion state is combined with the comprehensive state vector to perform motion planning processing to obtain the joint motion trajectory, and the joint motion trajectory is then subjected to motion smoothing processing to obtain the joint motion trajectory parameters; The basic facial expression parameter set and the joint motion trajectory parameters are synchronized in time to obtain a synchronization control command. Redundancy elimination is performed on the synchronization control command to obtain an optimized control command. Bit-width mapping is performed on the optimized control command to obtain a coherent control sequence. The sequential control sequence is executed to drive the facial display screen and the servo motors of each joint, and sensor feedback data is collected and deviation correction is performed to obtain the final interactive effect.

2. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The process of acquiring multi-source raw data streams and preprocessing them to obtain a multi-source data set includes: Simultaneously acquire multi-source raw data streams containing image data, audio data, pressure data, and acceleration data; The image data is used to extract features using a pre-trained facial keypoint regression model to obtain facial keypoint coordinates, and the image data is then subjected to adaptive histogram equalization to obtain structured image data. The audio data is subjected to a short-time Fourier transform to obtain the power spectral density, and the audio data is then subjected to noise reduction processing based on the power spectral density to obtain an effective sound segment; The pressure data and acceleration data are smoothed and filtered respectively to obtain filtered pressure data and filtered acceleration data. The facial key point coordinates, the structured image data, the effective sound segments, the filtered pressure data and the filtered acceleration data are then integrated with timestamps to obtain a multi-source data set.

3. The intelligent robot toy interactive control method based on multimodal perception according to claim 2, characterized in that, The step of extracting multimodal features from the multi-source dataset and simultaneously fusing the multimodal features to obtain a multimodal feature vector includes: The facial key point coordinates, structured image data, effective sound segments, filter pressure data, and filter acceleration data are extracted from the multi-source dataset. The distance feature is obtained by calculating the Euclidean distance of the facial key point coordinates, the pose estimation is performed on the structured image data to obtain the pose feature vector, and the distance feature and the pose feature vector are aggregated to obtain the image feature vector. Audio feature vectors are obtained by extracting Mel-frequency cepstral coefficients from valid sound segments; The pressure data and acceleration data are subjected to time-domain statistical feature extraction processing to obtain pressure feature vector and acceleration feature vector respectively, and the pressure feature vector and acceleration feature vector are subjected to vector concatenation processing to obtain motion feature vector; A synchronization feature set is obtained by performing timestamp alignment on the image feature vector, the audio feature vector, and the motion feature vector using a linear interpolation algorithm, and then performing vector concatenation on the synchronization feature set to obtain a multimodal feature vector.

4. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The process involves inputting the multimodal feature vector into a pre-trained long short-term memory network to obtain an emotion hidden representation vector. An attention mechanism is then used to dynamically assign weights to the emotion hidden representation vector, resulting in a comprehensive state vector containing emotion state labels and behavioral probability distributions, including: The multimodal feature vector is input into a pre-trained long short-term memory network for temporal dependency analysis to obtain the sentiment hidden representation vector, and the sentiment hidden representation vector is normalized exponentially mapped to obtain the sentiment state label. The attention mechanism is used to calculate the weights of the emotion hidden representation vector to obtain dynamic weight coefficients, and the emotion hidden representation vector is weighted and integrated according to the dynamic weight coefficients to obtain a weighted feature vector. A fully connected layer is used to perform a linear transformation on the weighted feature vector to obtain a behavior probability distribution, and a comprehensive state vector is constructed by combining the emotion state label and the behavior probability distribution as feature vectors.

5. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The process involves retrieving facial display driving parameters from a preset emotion mapping library based on the emotion state labels, calculating fine-tuning coefficients using a pre-trained convolutional neural network model combined with real-time acquired ambient lighting and user distance, and adaptively fine-tuning the facial display driving parameters to obtain a set of basic expression parameters, including: Based on the emotional state tags, the initial facial display driving parameters are obtained by key-value indexing of the preset emotional mapping library; Real-time ambient lighting and user data are acquired, and a pre-trained convolutional neural network model is used to perform nonlinear feature mapping on the ambient lighting and the distance to the user to obtain fine-tuning coefficients. The initial facial display driving parameters are numerically corrected based on the fine-tuning coefficients to obtain the target driving parameters. The target driving parameters are verified for inter-frame smoothness using a preset rendering engine to obtain a set of basic facial expression parameters.

6. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The process of combining the robot's current motion state with the comprehensive state vector to obtain joint motion trajectories, and then performing motion smoothing processing on the joint motion trajectories to obtain joint motion trajectory parameters, includes: Obtain the robot's current position and joint angle information as its current motion state; The target pose is mapped based on the comprehensive state vector to obtain the desired motion target, and the initial value of the joint motion trajectory is obtained by solving the spatial path in combination with the current motion state. The initial value of the joint motion trajectory is optimized by continuous constraint using a quadratic programming algorithm to obtain a smooth joint motion trajectory. The smooth joint motion trajectory is then sampled and quantized to obtain the joint motion trajectory parameters.

7. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The process of synchronizing the basic facial expression parameter set with the joint motion trajectory parameters in time to obtain a synchronization control command, performing redundancy elimination on the synchronization control command to obtain an optimized control command, and performing bit-width mapping processing on the optimized control command to obtain a coherent control sequence includes: The basic facial expression parameter set is timestamped using a linear interpolation algorithm to obtain a synchronous control command aligned with the joint motion trajectory parameters; The synchronization control commands are identified using a clustering algorithm, and the optimized control commands are obtained by eliminating redundant commands based on the identification results. The optimized control instructions are mapped to numerical ranges according to the communication protocol of the target hardware to obtain bit-width adaptation instructions, and the bit-width adaptation instructions are timestamped to obtain a coherent control sequence.

8. The intelligent robot toy interactive control method based on multimodal perception according to claim 1, characterized in that, The execution of the coherent control sequence drives the facial display screen and each joint servo motor to operate, collects sensor feedback data, and performs deviation correction processing to obtain the final interactive effect, including: The facial display drive command and the joint servo command in the coherent control sequence are distributed synchronously to drive the facial display screen and each joint servo to execute synchronously. The encoder data and current data of each joint servo motor are collected in real time as the sensor feedback data; The deviation between the joint servo command and the sensor feedback data is calculated using a PID control algorithm to obtain compensation parameters. The joint servo command is then corrected in real time based on the compensation parameters to obtain the final interactive effect.

9. A multimodal perception-based intelligent robot toy interactive control system, characterized in that, include: The data preprocessing module is used to collect multi-source raw data streams and preprocess the multi-source raw data streams to obtain a multi-source data set; The feature synchronization fusion module is used to extract multimodal features from the multi-source dataset and perform synchronous fusion processing on the multimodal features to obtain a multimodal feature vector. The comprehensive state modeling module is used to input the multimodal feature vector into a pre-trained long short-term memory network to obtain the sentiment hidden representation vector, and to use an attention mechanism to dynamically assign weights to the sentiment hidden representation vector to obtain a comprehensive state vector containing sentiment state labels and behavioral probability distributions. The facial expression parameter adaptive module is used to retrieve facial display driving parameters from a preset emotion mapping library based on the emotion state label, calculate fine-tuning coefficients using a pre-trained convolutional neural network model combined with real-time acquired ambient lighting and user distance, and adaptively fine-tune the facial display driving parameters to obtain a basic set of facial expression parameters. The motion planning and smoothing module is used to perform motion planning processing on the comprehensive state vector in combination with the robot's current motion state to obtain joint motion trajectories, and to perform motion smoothing processing on the joint motion trajectories to obtain joint motion trajectory parameters. The instruction optimization and integration module is used to synchronize the basic facial expression parameter set with the joint motion trajectory parameters in time to obtain a synchronous control instruction, perform a redundancy elimination operation on the synchronous control instruction to obtain an optimized control instruction, and perform bit-width mapping processing on the optimized control instruction to obtain a coherent control sequence. The drive feedback correction module is used to execute the coherent control sequence to drive the face display screen and the servo motors of each joint, collect sensor feedback data and perform deviation correction processing to obtain the final interactive effect.

Images