A method for generating control commands for bionic actuators based on cross-modal dynamic fusion

Abstract

This invention discloses a method for generating bionic actuator control commands based on cross-modal dynamic fusion, belonging to the field of human-computer interaction and bionic robotics. It collects user visual, speech, and physiological data through multimodal sensors, and obtains modal features through encoding and semantic alignment. Spatiotemporal registration of visual and physiological features is performed to form visual-physiological alignment features. A cross-modal attention mechanism is introduced to fuse speech features, outputting cross-modal attention fusion features. These features are input into a DRL policy network and combined with a cognitive graph knowledge base to infer and match the user's emotional actions in real time, generating an emotion vector. Finally, the emotion vector is solved into facial action unit parameters, and combined with the broadcast text generated by a large language model to construct an enhanced action parameter sequence, which is then parsed to obtain bionic actuator control commands. This invention improves decision-making speed and accuracy, achieves smooth and natural emotional transitions and highly bionic precision facial expression control, and further optimizes the humanoid interaction experience.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction technology, and in particular to a method for generating control instructions for a bionic actuator based on cross-modal dynamic fusion. Background Technology

[0002] For large biomimetic robots, the emotional interaction technology aims to achieve natural and fluent emotional communication between humans and the system. Existing solutions often employ modular designs, such as using independent vision and speech modules for emotion recognition, making decisions through rule bases or simple machine learning models, and relying on servo mechanisms to execute preset facial expressions. However, existing technologies have many shortcomings: (1) Modal conflict and poor fusion effect: The independent working time of each sensing mode is not synchronized and the feature space is inconsistent. Furthermore, fixed weight fusion is adopted. When the signal-to-noise ratio of a certain mode decreases (such as strong light or noise), the overall performance of the system drops sharply. (2) Low level of decision intelligence and slow response: Rule-based decision systems lack adaptive learning capabilities, cannot handle complex and dynamic interactive scenarios, have high decision process delays, and are difficult to achieve natural and smooth real-time interaction. Summary of the Invention

[0003] This invention provides a method for generating bionic actuator control instructions based on cross-modal dynamic fusion to overcome the above-mentioned technical problems.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A method for generating control commands for a bionic actuator based on cross-modal dynamic fusion, comprising the following steps: S1. Collect the user's visual data, voice data, and physiological data through several modal sensors, and encode the visual data, voice data, and physiological data and perform inter-modal semantic alignment processing to obtain semantically aligned visual features, voice features, and physiological features. S2. Perform spatiotemporal alignment and registration on the visual and physiological features of semantic alignment to obtain visual-physiological alignment features; S3. Perform cross-modal attention fusion on the visual physiological alignment features and the semantically aligned speech features to obtain cross-modal attention fusion features; S4. Input the cross-modal attention fusion features into a preset DRL policy network, and combine them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions that match the user's state, and then output the emotional vector. S5. Convert the emotion vector into facial motion unit parameters, obtain the text data to be played based on the speech data and the pre-trained large language model; obtain the enhanced motion parameter sequence based on the facial motion unit parameters and the text data to be played, and parse it to obtain the bionic actuator control command.

[0005] Furthermore, in S2, the specific steps for spatiotemporal alignment and registration of the semantically aligned visual and physiological features to obtain visual-physiological aligned features include: S21. Set up a sliding window and cache visual and physiological features through the sliding window, and sort all cached data by timestamp; S22. Perform linear interpolation or cubic spline interpolation on the physiological features within a sliding window to generate corresponding physiological feature estimates for the timestamp of each frame of visual features, thereby obtaining the interpolated physiological features. S23. Calculate the minimum cumulative distance based on visual features and interpolated physiological features. To obtain the cumulative distance matrix, the minimum cumulative distance The calculation formula is: ; In the formula, The first of the physiological characteristics i The eigenvalue and the visual feature of the th eigenvalue i Distance metric between eigenvalues; Based on the cumulative distance matrix, the endpoint backtracking is performed to determine the optimal nonlinear mapping path. The optimal nonlinear mapping path is the time point mapping relationship that minimizes the cumulative distance between physiological features and visual features. The visual features and interpolated physiological features are time-aligned according to the optimal nonlinear mapping path to obtain time-synchronized sequence pairs. S24. Perform keypoint topology mapping on the time-synchronized sequence pairs and the current frame visual data to obtain visual physiological alignment features.

[0006] Furthermore, in S24, the specific steps for performing keypoint topological mapping processing on the time-synchronized sequence pairs and the current frame visual data to obtain visual physiological alignment features include: Spatial mapping: The human body key points are identified from the current frame visual data using a preset key point detection model, and then a skeleton topology map is constructed. The human body key points include skeleton joints and sensor attachment reference points, wherein the sensor attachment reference points are the attachment reference points of physiological sensors in modal sensors. By combining time-synchronized sequence pairs, the location information of physiological sensors is mapped to sensor attachment reference points in the skeleton topology map; The positional information of the physiological sensors is transformed into the global visual space coordinate system where the skeleton topology map is located using an affine transformation matrix. The affine transformation matrix is ​​expressed as follows: , Where A is the rotation and scaling matrix, and b is the translation vector; x, y ( ) are the coordinates of the physiological sensor; x', y' () are the transformed coordinates; Dynamic binding: Key point trajectory tracking technology is used to predict the motion trend of the attachment reference points of physiological sensors in the skeleton topology map. This includes: when the displacement of the physiological sensor is less than a set threshold, the historical binding relationship established by the skeleton topology map and the position information of the physiological sensor is used as the binding result; when the displacement of the physiological sensor is greater than or equal to the set threshold, the three-dimensional point cloud of the surface of the physiological sensor shell and the corresponding contact area of ​​the human skin surface is obtained, and then the three-dimensional point cloud of the two is registered using the ICP point cloud matching method. The optimal affine transformation matrix from the physiological sensor coordinate system to the skeleton coordinate system is recalculated, thereby realizing the dynamic update of the binding result between the position of the physiological sensor and the skeleton topology map. Visual physiological alignment features are obtained based on the binding results between the skeleton topology map and the positions of physiological sensors.

[0007] Furthermore, in S3, the specific steps for cross-modal attention fusion of the visual physiological alignment features and the semantically aligned speech features include: Use any one of the following modalities as the query vector: spatiotemporally aligned visual features, physiological features, and semantically aligned speech features. Q The other two modes are key vectors. K Sum value vector V ; The attention mechanism is calculated based on the initial weights to obtain the cross-modal attention fusion features, which are represented as follows: , In the formula, d k The dimension of the key vector; As weight; This includes real-time calculation of the signal-to-noise ratio of visual, speech, and physiological data to adjust the initial weights, including: The effective signal power and noise power of visual data, speech data, and physiological data are calculated separately, and then the signal-to-noise ratio is calculated using the following formula: , In the formula, P signal The effective signal power for visual data, speech data, or physiological data;P noise Noise power for visual data, speech data, or physiological data; A dynamic weight adjustment strategy is implemented based on the signal-to-noise ratio calculation results, and the attention mechanism is recalculated based on the adjusted weights to update the cross-modal attention fusion features. The dynamic weight adjustment strategy includes: In a predefined strong light scene, the weight of visual features is reduced and the weight of voice features is increased according to a set ratio; In a predefined high-noise scenario, the weight of speech features is reduced and the weight of physiological features is increased according to a set ratio.

[0008] Furthermore, in S4, the specific steps of inputting the cross-modal attention fusion features into a preset DRL network and combining them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions matching the user's state, and then outputting the emotional vector, include: S41. Construct an environment state representation for the DRL policy network and query the cognitive graph knowledge base to obtain prior knowledge constraints, including: The cross-modal attention fusion features are used as the basic environmental state s0; Temporal context information, scene type identifier, and user personalized parameters are concatenated to the end of the basic environmental state s0 to construct an enhanced environmental state vector s with a total dimension of s0 + 24. Using the enhanced environmental state vector s as a query condition, semantic retrieval is performed in the cognitive graph knowledge base to obtain the emotional expression rules that best match the current scene and the user's national cultural background. Cultural weight coefficients, emotional intensity boundary constraints, and temporal smoothing parameters are extracted from the emotional expression rules. After concatenating, normalizing, and encoding the cultural weight coefficients, emotional intensity boundary constraints, and temporal smoothing parameters in sequence, a fixed-dimensional prior knowledge vector k is obtained. S42. Deep reinforcement learning decision-making based on the enhanced environment state vector s and prior knowledge vector k, including: The enhanced environment state vector s is concatenated with the prior knowledge vector k to obtain the concatenated state-knowledge vector [s; k], and the state-knowledge vector [s; k] is used as the input to the DRL policy network. The DRL policy network receives the concatenated state-knowledge vector [s;k] and processes it through a multi-layer neural network, including: using the prior knowledge vector k to initially constrain the search space, then combining it with the enhanced environment state vector s to calculate the optimal policy, and finally outputting an original emotional action a. raw ; DRL value networks receive raw emotional actions. raw And calculate the expected cumulative reward Q(s, a) rawThe expected cumulative reward Q(s, a) raw This is used to guide the optimization of the DRL policy network; S43. Employ a dynamic rule engine to process original emotional actions a raw Conduct multi-dimensional feasibility calibration, including: Conduct cultural adaptation calibration: Applying cultural weighting coefficients to the original emotional action a raw The attenuation process is performed using the following formula: I cultural = a raw × culture ; In the formula, culture It is the cultural weighting coefficient, which is the influence of the user's national cultural background on emotional actions; to I cultural Conduct physiological feasibility verification: Muscle movement conflicts are detected by using a pre-set mutual exclusion matrix of action units to obtain an optimized I that meets physiological feasibility. optimized ; to I optimized Perform time-series smoothing: A dynamic model based on pre-set mass-damping-stiffness parameters is used for I. optimized Temporal smoothing is performed to obtain a 128-dimensional temporally smoothed vector after multi-constraint calibration and temporal smoothing. S44. Perform structured encoding on the 128-dimensional temporal smoothing vector, where: A 0-31 dimensional vector represents the emotional valence, with a value range of [-1, 1], and is used to characterize the positive or negative orientation of emotions; A 32-63 dimensional vector represents arousal level, with values ​​ranging from [0,1], and is used to characterize physiological activation levels; A 64-95 dimensional vector represents the degree of dominance, with values ​​ranging from [0,1], and is used to characterize subjective control ability; A 96-127 dimensional vector represents cultural context features, with values ​​ranging from [0,1], and includes cultural weights, scene types, and user-personalized parameters; S45. Verify the 128-dimensional temporal smoothing vector of the structured coding: Perform numerical range verification to ensure that all dimension values ​​are within the theoretical boundaries; Perform internal consistency checks to verify the logical coherence between different emotional dimensions; A time series continuity test is performed, and a smoothness index is calculated with historical data using a sliding window. Add timestamps and quality score metadata to sentiment vectors that meet the quality standards to obtain standardized data packages, i.e., 128-dimensional sentiment vectors.

[0009] Further, in S5, the specific steps of converting the emotion vector into facial motion unit parameters, obtaining the text data to be played based on the speech data and the pre-trained large language model, and obtaining the enhanced motion parameter sequence based on the facial motion unit parameters and the text data to be played, and parsing it to obtain the bionic actuator control instructions include: S51. Decoding the emotion vector into emotion semantics and mapping it to a low-dimensional action space includes: An emotion vector is initially mapped to the original parameters of the facial action unit using a multilayer perceptron model. S52. The original parameters of the facial motion unit are smoothed and optimized using a preset damped spring vibration model, and the smoothed and optimized motion unit parameters are obtained. S53. Apply anatomical and physiological constraints to the smoothed and optimized motion unit parameters to generate the final motion unit parameters. Specific steps include: A sparse optimization objective function is established based on the existing FACS action unit mutual exclusion rules. By solving the sparse optimization objective function, a sparse parameter set is generated. The formula for the sparse optimization objective function is as follows: , In the formula, It is a 42-dimensional vector representing the final action unit parameters after sparse optimization that we need to solve; is the regularization coefficient, and it is greater than 0; For smoothing and optimizing the motion unit parameters; This is an L1 norm regularization term; Based on the sparse optimization objective function, a composite objective function is constructed that simultaneously considers accuracy, sparsity, mutual exclusion, and synergy. By solving the composite objective function, the optimized action unit parameters are obtained. The formula for the composite objective function is as follows: ; In the formula, It is a 42×42 action unit coordination matrix; It is a 42×42 action unit mutual exclusion matrix; The goal is to achieve a cooperative objective, which is a 42-dimensional vector. , and This is the collaborative weighting coefficient, used to control the strength of collaborative constraints; The optimized motion unit parameters; For the optimized action unit parameters After normalization, the final output is optimized parameters that conform to emotional intent and are physiologically feasible for biomimetic execution. S54. Generate the text to be played based on the speech data and the pre-trained large language model, and convert the text to be played into a phoneme sequence and the corresponding time-phoneme aligned file through the TTS engine. S55. Traverse the time-phoneme alignment file and assign a target lip position parameter to each phoneme during its duration based on the phoneme-visual position mapping table, and finally calculate the representative lip position parameter. S56. The facial motion unit parameters and lip shape view position parameters are fused using a fusion strategy to obtain an enhanced motion parameter sequence with timestamps that incorporates lip shape and emotion. S57. Convert the enhanced motion parameter sequence into multi-degree-of-freedom control instructions for the bionic actuator.

[0010] Furthermore, in S57, the specific steps for converting the enhanced motion parameter sequence into multi-degree-of-freedom control instructions for the bionic actuator include: S571. Obtain facial muscle biomechanical simulation data, establish a nonlinear mapping model for converting the abstract action unit space to the displacement space of a specific bionic actuator, and calibrate the nonlinear mapping model based on the facial muscle biomechanical simulation data to obtain the calibrated nonlinear mapping model. S572. Extract the fused facial motion unit parameter AU_final for each time step from the enhanced motion parameter sequence, input it into the calibrated nonlinear mapping model, and obtain the expected displacement field of the facial surface at the corresponding time. S573. Construct a kinematic model for the bionic actuator array with parallel or series structure, including a forward kinematic model and a Jacobian matrix; combine the desired displacement field of the facial surface, the forward kinematic model and the Jacobian matrix, and calculate the desired joint position that minimizes the position error of the bionic actuator end effector through inverse kinematics solution, as the position control command. S574. Differentiate the desired joint position to obtain the desired joint velocity. Combine the error between the current joint position and the desired position to generate a velocity feedback command through a proportional-integral-derivative control algorithm. At the same time, calculate the desired torque command based on a preset bionic actuator dynamics model. Combine the position control command, velocity feedback command, and desired torque command into a multi-degree-of-freedom control command for driving the bionic actuator.

[0011] Furthermore, the fusion strategy involves weighting the facial motion unit parameters and lip-sync parameters based on their respective effects on different facial regions. Specifically, this includes: For the lips and surrounding preset areas, setting the mouth shape viewpoint parameters through a dynamic weighting method has the highest priority; For predefined non-conflict regions, facial motion unit parameters are fully preserved; For a predefined common influence area, facial motion unit parameters and lip shape visual position parameters are balanced by dynamic weighting. The formula for calculating the dynamic weight AU_final is as follows: AU_final = ×AU_viseme + (1- )×AU_emotion; in, AU_viseme represents the 42-dimensional lip-sync viewpoint parameter; AU_emotion represents the 42-dimensional facial motion unit parameter.

[0012] Furthermore, the specific steps for encoding the visual data, speech data, and physiological data, as well as for inter-modal semantic alignment processing, include: The ViT encoder is used to convert visual data into visual features; the Wav2Vec2 encoder is used to convert speech data into speech features; and a temporal convolutional network encoder is used to convert physiological data into physiological features. The transformed visual features, speech features, and physiological features are mapped to the constructed shared feature space using the corresponding projection matrices. By maximizing the similarity between mapped visual, speech, and physiological features using an alignment loss function, the visual, speech, and physiological features are aligned in a shared feature space, resulting in feature-aligned visual, speech, and physiological features. The formula for the alignment loss function is as follows: , In the formula, Represents the feature vector of positive samples from any two different modalities. The similarity between them; τ Indicates temperature parameter; Represents the feature vector of a negative sample; j This represents the negative sample index.

[0013] Beneficial Effects: This invention employs a cross-modal attention fusion mechanism and a spatiotemporal alignment algorithm to obtain cross-modal attention fusion features. These features are input into a pre-defined DRL policy network, which outputs an emotion vector. By replacing the traditional rule engine with a real-time decision-making framework driven by the DRL policy network, the decision-making speed and accuracy are significantly improved. This invention rapidly and accurately calculates high-dimensional emotion vectors into executable action unit parameters, achieving smooth and natural emotional state transitions and providing a more intelligent and human-like emotional interaction experience. Based on facial action unit parameters and the acquired text to be broadcast, an enhanced action parameter sequence is obtained, which is then parsed to obtain bionic actuator control commands, ultimately achieving highly bionic and highly reliable precision facial expression control. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a bionic actuator control command generation method based on cross-modal dynamic fusion in this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This embodiment provides a method for generating bionic actuator control commands based on cross-modal dynamic fusion, such as... Figure 1 As shown, the specific steps include: S1. Collect the user's visual data, voice data, and physiological data through several modal sensors, and encode the visual data, voice data, and physiological data and perform inter-modal semantic alignment processing to obtain semantically aligned visual features, voice features, and physiological features. Specifically, in this embodiment, physiological data is collected through a wearable device, which integrates physiological sensors and an inertial measurement unit. The collected physiological data includes electroencephalogram (EEG) signals, electromyogram (EMG) signals, and motion data collected by the inertial measurement unit.

[0018] In a specific embodiment, the specific steps for encoding the visual data, speech data, and physiological data, as well as the inter-modal semantic alignment processing, include: The visual data, speech data, and physiological data are converted by different encoders to obtain initial visual features, speech features, and physiological features. Specifically, in this embodiment, a ViT encoder is used to convert visual data into visual features; a Wav2Vec2 encoder is used to convert speech data into speech features; and a temporal convolutional network encoder is used to convert physiological data into physiological features.

[0019] The transformed visual features, speech features, and physiological features are mapped to the constructed shared feature space using the corresponding projection matrices. By maximizing the semantic similarity between the mapped visual, speech, and physiological features using an alignment loss function, the initial visual, speech, and physiological features are aligned in a shared feature space, resulting in feature-aligned visual, speech, and physiological features. The formula for the alignment loss function is as follows: , In the formula, Represents the feature vector of positive samples from any two different modalities. The similarity between them is usually measured using cosine similarity; τ This represents a temperature parameter, with a value between 0.01 and 1.0, used to adjust the smoothness of the similarity distribution and prevent gradient explosion or vanishing. Represents the feature vector of a negative sample; j This represents the negative sample index, used for negative sample sampling in contrastive learning.

[0020] S2. Perform spatiotemporal alignment and registration on the visual and physiological features of semantic alignment to obtain visual-physiological alignment features; In a specific embodiment, the specific steps for performing spatiotemporal alignment and registration processing on semantically aligned visual features and physiological features to obtain visual-physiological aligned features include: S21. Set a 100ms sliding window and cache visual and physiological features through the sliding window, such as 10 frames of images + 1 GSR data point, and sort all cached data by timestamp. S22. Perform linear interpolation or cubic spline interpolation on the physiological features within a sliding window to generate corresponding physiological feature estimates for the timestamp of each frame of visual features, thereby obtaining the interpolated physiological features. S23. Calculate the minimum cumulative distance based on visual features and interpolated physiological features. To obtain the cumulative distance matrix, the minimum cumulative distance The calculation formula is: , In the formula, The first of the physiological characteristics i The eigenvalue and the visual feature of the th eigenvalue i Distance metric between eigenvalues; Based on the cumulative distance matrix, the endpoint backtracking is performed to determine the optimal nonlinear mapping path. The optimal nonlinear mapping path is the time point mapping relationship that minimizes the cumulative distance between physiological features and visual features. Specifically, the process of backtracking to the endpoint based on the cumulative distance matrix to determine the optimal nonlinear mapping path is a standard step in the Dynamic Time Warping (DTW) algorithm, and will not be elaborated here.

[0021] The visual features and interpolated physiological features are time-aligned according to the optimal nonlinear mapping path to eliminate timing misalignment caused by clock drift and motion, and obtain time-synchronized sequence pairs. S24. Perform keypoint topological mapping processing on the time-synchronized sequence pairs and the current frame visual data to obtain visual physiological alignment features, including: Spatial mapping: Using a pre-set keypoint detection model (HRNet or OpenPose), human keypoints are identified from the current frame visual data, and then a skeleton topology map is constructed. The human keypoints include skeleton joints and sensor attachment reference points, wherein the sensor attachment reference points are the attachment reference points of the physiological sensors in the modal sensors. Combining time-synchronized sequence pairs, the position information of the physiological sensors is mapped to the sensor attachment reference points in the skeleton topology map, such as the left wrist node. Specifically, in this embodiment, the acquisition of physiological sensor displacement is autonomously completed by the inertial navigation system integrated in the wearable device, relying on the vision-inertial navigation fusion method to ensure positioning accuracy.

[0022] The positional information of the physiological sensors is transformed into the global visual space coordinate system where the skeleton topology map is located using an affine transformation matrix. The affine transformation matrix is ​​expressed as follows: , Where A is the rotation and scaling matrix, and b is the translation vector; x, y ( ) are the coordinates of the physiological sensor; x', y' () are the transformed coordinates; Dynamic binding: Keypoint trajectory tracking techniques (such as Kalman filtering) are used to predict the motion trends of attachment reference points for physiological sensors in the skeleton topology map, including: When the displacement of the physiological sensor is less than a set threshold, the historical binding relationship established by the skeleton topology map and the position information of the physiological sensor is used as the binding result. When the displacement of the physiological sensor is greater than or equal to the set threshold, it indicates that its relative position with the human skin surface may have changed. At this time, the system will automatically perform position recalibration, that is: acquire the three-dimensional point cloud of the physiological sensor shell surface and the corresponding contact area human skin surface, and then use the ICP point cloud matching method to register the three-dimensional point clouds of the two (register the point clouds by iterative nearest point algorithm), and recalculate the optimal affine transformation matrix from the physiological sensor coordinate system to the skeleton coordinate system, thereby realizing the dynamic update of the binding result between the position of the physiological sensor and the skeleton topology map; the visual physiological alignment feature is obtained based on the binding result between the skeleton topology map and the position of the physiological sensor.

[0023] Specifically, this embodiment addresses the temporal drift problem caused by different sampling rates through a sliding window dynamic calibration mechanism, a dynamic resampling mechanism, and the DTW algorithm. Furthermore, it binds visual features with physiological features using a keypoint topology mapping method to achieve spatial registration, ensuring that the physiological signal at each moment is not only temporally aligned with the corresponding visual frame, but also that its spatial source has precise coordinates within the visual frame. The biomimetic multilayer transmission network based on biomechanical topology mapping can simulate the layered structure of human facial muscles.

[0024] S3. Perform cross-modal attention fusion on the visual physiological alignment features and the semantically aligned speech features to obtain cross-modal attention fusion features; In a specific embodiment, the specific steps for cross-modal attention fusion of the visual physiological alignment features and the semantically aligned speech features include: Use any modality (such as speech) from spatiotemporally aligned visual features, physiological features, and speech features as the query vector. Q The other two modalities (such as visual and physiological) are key vectors. K Sum value vector V ; The attention mechanism is calculated based on the initial weights to obtain the cross-modal attention fusion features, which are represented as follows: , In the formula, d k The dimension of the key vector; As weight; This includes real-time calculation of the signal-to-noise ratio (SNR) of visual, speech, and physiological data to adjust the initial weights, including: The effective signal power and noise power of visual data, speech data, and physiological data are calculated separately, and then the signal-to-noise ratio (SNR) is calculated using the following formula: , In the formula, P signal The effective signal power for visual, speech, or physiological data; P noise Noise power for visual data, speech data, or physiological data; Specifically, for visual data, filtering methods (such as low-pass filtering) are typically used to separate and estimate noise power and signal power. For speech data, noise power is usually estimated using silent segment detection; signal power can be approximated by calculating the energy of the speech segment after applying bandpass filtering, and then subtracting the estimated noise power. For physiological data, noise power is usually estimated using data from a baseline period without events; signal power is obtained by calculating the signal during the event-locked period. A dynamic weight adjustment strategy is implemented based on the SNR calculation results, and the attention mechanism is recalculated based on the adjusted weights to update the cross-modal attention fusion features. The dynamic weight adjustment strategy includes: In a predefined strong light scene, the weight of visual features is reduced and the weight of voice features is increased according to a set ratio; In a predefined high-noise scenario, the weight of speech features is reduced and the weight of physiological features is increased according to a set ratio.

[0025] Specifically, this embodiment dynamically calculates the signal-to-noise ratio (SNR) by real-time monitoring of the power spectral density of each modality signal (such as visual illumination intensity, speech background noise, and bioelectrical baseline fluctuations). When the visual feature SNR drops sharply under strong light (e.g., SNR < 10 dB when > 80 lux), the weight is adjusted downwards. When the speech feature SNR decreases in a noisy environment (e.g., signal-to-noise ratio < 15 dB), physiological features have higher weights due to their stronger anti-interference ability (EMG signal SNR > 25 dB). For example, in a strong light scene, a 30% decrease in visual weight leads to a 65% increase in acoustic weight (relying on the beamforming of the microphone array to resist light noise). In a high-noise scene, a 40% decrease in speech weight leads to a 70% increase in bioelectrical weight (utilizing the physiological specificity of EMG / EEG signals). This embodiment sets the weight constraint condition as a total weight of 1 to avoid modal conflicts.

[0026] Specifically, this embodiment introduces adaptive weight allocation based on real-time signal-to-noise ratio, which automatically increases the weight of high-confidence modalities under specific conditions, significantly improving the robustness and recognition accuracy of the system, and realizing high-precision, highly adaptive multimodal perception fusion.

[0027] S4. Input the cross-modal attention fusion features into a preset DRL network, and combine them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions that match the user's state, and then output the emotional vector. In a specific embodiment, the steps of inputting the cross-modal attention fusion features into a preset DRL network and combining them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions that match the user's state, and then outputting the emotional vector, include: S41. Construct an environment state representation for the DRL policy network and query the cognitive graph knowledge base to obtain prior knowledge constraints, including: The cross-modal attention fusion features are used as the basic environmental state s0; Add time context information, scene type identifier and user personalization parameters to the basic environment state s0 to construct an enhanced environment state vector s with a total dimension of s0+24. Specifically, the time context information includes: current time (hour normalized), weekday type (weekday / weekend / holiday), duration of current interaction, time interval since last interaction, time period of the day in which the current interaction occurs (early morning / morning / afternoon, season / month information, special time period identifier, time pressure identifier (such as near deadline). Scene type identifiers include: indoor / outdoor, public / private, social scene type, task scene, lighting conditions (bright / dark / overexposed), noise level (quiet / noisy / extremely high noise), space size, and number of people estimated. User personalization parameters include: age group (child / youth / middle-aged / elderly coding), gender, cultural background (East Asia / Europe / Middle East, etc.), personality traits, emotional expression habits (open / reserved), familiarity with the robot (first time / occasionally / frequently), current emotional baseline (calm / excited / fatigued), and special state indicators.

[0028] Using the enhanced environment state vector s as the query condition, semantic retrieval is performed in the cognitive graph knowledge base to obtain emotional expression rules that are deeply related to the current scene and the user's cultural background; Specifically, the cognitive graph knowledge base uses cultural foundational data, such as Hofstede cultural dimension data, to annotate the intensity of facial expressions across different cultural backgrounds. Because users come from different cultural backgrounds, the same emotion can be expressed differently in different cultural contexts, for example: East Asian culture: negative emotions such as anger and sadness tend to be expressed in a restrained and introspective manner. Western culture: Emotions are expressed more openly and directly; Middle Eastern culture: Emotional expression may be accompanied by specific ritualistic actions.

[0029] The cultural weight coefficient, emotional intensity boundary constraint, and temporal smoothing parameter are extracted from the emotional expression rules. After the cultural weight coefficient, emotional intensity boundary constraint, and temporal smoothing parameter are sequentially concatenated, normalized, and encoded, a 512-dimensional prior knowledge vector k is obtained. S42. Deep reinforcement learning decision-making based on the enhanced environment state vector s and prior knowledge vector k, including: The enhanced environment state vector s is concatenated with the 512-dimensional prior knowledge vector k to obtain the concatenated state-knowledge vector [s; k], and the state-knowledge vector [s; k] is used as the input of the DRL policy network. The DRL policy network receives the concatenated state-knowledge vector [s;k] and processes it through a multi-layer neural network, including: using the prior knowledge vector k to initially constrain the search space, then combining it with the enhanced environment state vector s to calculate the optimal policy, and finally outputting an original emotional action a. raw ; DRL value networks receive raw emotional actions. raw And calculate the expected cumulative reward Q(s, a) raw The expected cumulative reward Q(s, a) raw This is used to guide the optimization of the DRL policy network; S43. Employ a dynamic rule engine to process original emotional actions a raw Conduct multi-dimensional feasibility calibration, including: Conduct cultural adaptation calibration: Applying cultural weighting coefficients to the original emotional action a raw Attenuation processing is performed, with different attenuation coefficients used for different cultural backgrounds. The formula is as follows: I cultural = a raw × culture ; In the formula, culture It is the cultural weighting coefficient, which is the influence of the user's national cultural background on emotional actions; to I cultural Conduct physiological feasibility verification: Muscle movement conflicts are detected by using a pre-defined mutual exclusion matrix of action units to avoid physiologically impossible facial expressions, resulting in an optimized I-mode that meets physiological feasibility. optimized ; to I optimized Perform time-series smoothing: A dynamic model based on pre-set mass-damping-stiffness parameters is used for I. optimizedTemporal smoothing is performed to achieve a natural transition of emotional states and ensure the smoothness of emotional changes, resulting in a 128-dimensional temporally smoothed vector after multi-constraint calibration and temporal smoothing. S44. Perform structured encoding on the 128-dimensional temporal smoothing vector, where: A 0-31 dimensional vector represents the emotional valence, with a value range of [-1, 1], and is used to characterize the positive or negative orientation of emotions; A 32-63 dimensional vector represents arousal level, with values ​​ranging from [0,1], and is used to characterize physiological activation levels; A 64-95 dimensional vector represents the degree of dominance, with values ​​ranging from [0,1], and is used to characterize subjective control ability; A 96-127 dimensional vector represents cultural context features, with values ​​ranging from [0,1], and includes cultural weights, scene types, and user-personalized parameters; Specifically, each dimension uses a standardized numerical range to ensure consistency.

[0030] S45. Verify the 128-dimensional temporal smoothing vector of the structured coding: Perform numerical range verification to ensure that all dimension values ​​are within the theoretical boundaries; Perform internal consistency checks to verify the logical coherence between different emotional dimensions; A time series continuity test is performed, and a smoothness index is calculated with historical data using a sliding window. Add timestamps and quality score metadata to sentiment vectors that meet the quality standards to obtain standardized data packages, i.e., 128-dimensional sentiment vectors.

[0031] Specifically, in this embodiment, the construction process of the cognitive graph knowledge base includes: Obtain raw data from four main channels, including: Social text data: such as public texts on platforms like Weibo and Twitter, used to capture natural emotional expressions with cultural tags; Audiovisual script data: such as movie and TV scripts, providing contextualized emotional interaction samples with rich context; Scientific datasets: such as AffectNet and other public datasets, which provide multimodal data annotated by experts and containing sentiment labels or PAD (pleasure-activation-dominance) vectors; Expert Rule Base: This base contains fundamental rules manually constructed by cultural psychology experts. It includes norms, taboos, and intensity regulation patterns of emotional expression across different cultural backgrounds. Each rule includes the applicable cultural scope, specific context, corresponding emotional pattern, and the confidence level given by the expert. These rules are stored in a structured format for easy retrieval and recall. For example: "In East Asian cultures, directly expressing anger is generally considered impolite." Knowledge extraction processing includes: Using LDA topic modeling, sentiment analysis, and named entity recognition techniques, knowledge triples in the format of <subject, sentiment, intensity> or <action, is, expression of sentiment> are extracted from the unstructured raw data mentioned above, ultimately obtaining structured knowledge units. For example, from the sentence "After winning the game, he celebrated by pumping his fist and smiling broadly," <he, joy, 0.9> and <pumping his fist, is, expression of joy> can be extracted.

[0032] S5. The 128-dimensional emotion vector is converted into 42-dimensional facial motion unit parameters through an inverse mapping model. Based on the 42-dimensional facial motion unit parameters and the obtained text to be broadcast, an enhanced motion parameter sequence with precise timestamps that integrates lip movements and emotions is obtained. The sequence is then parsed to obtain bionic actuator control instructions.

[0033] In a specific embodiment, in S5, the 128-dimensional emotion vector is converted into 42-dimensional facial motion unit parameters. Based on the 42-dimensional facial motion unit parameters, the text data to be broadcast is obtained based on the speech data and the pre-trained large language model. The specific steps of obtaining an enhanced motion parameter sequence based on the facial motion unit parameters and the text data to be broadcast, and parsing it to obtain the bionic actuator control instructions include: S51. Decode the 128-dimensional emotion vector into emotion semantics and map it to a low-dimensional action space, including: A multilayer perceptron model is used to initially map a 128-dimensional emotion vector into 42-dimensional facial motion unit raw parameters. Specifically, in this embodiment, a three-layer perceptron model is used to initially map the 128-dimensional emotion vector into the original parameters of the 42-dimensional facial action unit.

[0034] Specifically, the mapping relationship of the multilayer perceptron model is obtained by pre-training based on the facial anatomy database, ensuring that the parameters of each action unit correspond to the activation intensity of a specific facial muscle group, with values ​​ranging from 0 to 1.

[0035] S52. The original parameters of the 42-dimensional facial motion unit are smoothed and optimized using a pre-set damped spring vibration model to simulate the physiological dynamics of facial muscles and ensure the natural continuity of facial expression changes, and to obtain the smoothed and optimized motion unit parameters. Specifically, the damped spring vibration model treats the original parameters of each facial motion unit as an independent vibration subsystem. Its mass parameter reflects the emotional inertia of the muscle group, the damping coefficient controls the smoothness of the control state changes, and the stiffness coefficient affects the stability of the expression. Using the motion unit state from the previous moment as initial conditions and the original parameters of the 42-dimensional facial motion units as the target equilibrium position, the damped spring vibration model calculates the smooth trajectory of each motion unit at the current moment under the combined effects of inertia, damping, and stiffness by solving second-order differential equations. This process effectively eliminates parameter jumps, simulates the velocity and acceleration limitations of real facial muscle movements, and makes the transition of expressions smooth and natural.

[0036] S53. Apply anatomical and physiological constraints to the 42-dimensional smoothed and optimized action unit parameters to generate final action unit parameters that conform to both emotional intent and physiological feasibility. Specific steps include: Based on the existing FACS action unit mutual exclusion rules, a sparse optimization objective function is established. By solving the sparse optimization objective function, antagonistic muscle combinations that are physiologically impossible to be strongly activated simultaneously are suppressed, thereby generating a sparse parameter set. The formula for the sparse optimization objective function is as follows: , In the formula, It is a 42-dimensional vector representing the final action unit parameters after sparse optimization that we need to solve; It is a regularization coefficient, and is greater than 0, used to balance data fidelity and sparsity; For smoothing and optimizing the motion unit parameters; This is an L1 norm regularization term; Based on a sparse optimization objective function, a composite objective function is constructed that simultaneously considers accuracy, sparsity, mutual exclusion, and synergy. This enhances the muscle group activities that typically occur synergistically when expressing specific emotions, ensuring the coordination and recognizability of facial expressions. By solving the composite objective function, the optimized 42-dimensional motion unit parameters are obtained. The formula for the composite objective function is as follows: ; In the formula, It is a 42×42 action unit coordination matrix; It is a 42×42 action unit mutual exclusion matrix; The goal is to achieve a cooperative objective, which is a 42-dimensional vector. , and This is the collaborative weighting coefficient, used to control the strength of collaborative constraints; The optimized motion unit parameters; For parameters Normalization is performed to ensure that all values ​​are within the safe operating range of the underlying bionic actuator. The final output is a 42-dimensional optimized parameter that is sparsely optimized and co-enhanced and can be directly used for bionic execution, which is both emotionally intentional and physiologically feasible. Specifically, this embodiment uses a sparse optimization objective function to suppress antagonistic muscle combinations that are physiologically impossible to be strongly activated simultaneously, and a composite objective function to enhance the activity of muscle groups that typically synergistically occur when expressing specific emotions, ultimately obtaining a 42-dimensional facial motion unit parameter vector optimized by multiple objectives. .

[0037] S54. Generate the text to be played based on the speech data and the pre-trained large language model, and convert the text to be played into a phoneme sequence and the corresponding time-phoneme aligned file through the TTS (text-to-speech) engine. S55. Traverse the time-phoneme alignment file and assign a target lip position parameter to each phoneme during its duration based on the phoneme-lip position mapping table, and finally calculate the 42-dimensional lip position parameters representing the pure lip shape. Specifically, each phoneme (such as the vowel / a / and the consonant / p / ) corresponds to a mouth shape posture, or "sight position". The system has a pre-stored phoneme-sight position mapping table. By traversing the time-phoneme alignment file, a target sight position parameter is assigned to each phoneme within its duration, and finally, the 42-dimensional mouth shape sight position parameter representing the pure mouth shape is calculated.

[0038] S56. The 42-dimensional facial motion unit parameters and the 42-dimensional lip shape view position parameters are fused using a fusion strategy to obtain an enhanced motion parameter sequence with precise timestamps that integrates lip shape and emotion. In a specific embodiment, the fusion strategy involves weighting the 42-dimensional facial motion unit parameters and the 42-dimensional lip-shape viewing parameters based on their respective effects on different facial regions. Specifically, this includes: For the lips and surrounding key areas, to ensure speech clarity, the mouth shape viewpoint parameters are set to the highest priority through dynamic weighting, and conflicting facial motion unit parameters are covered or weakened. For non-conflict areas (such as eyebrows and forehead), the 42-dimensional facial motion unit parameters are fully preserved. For areas of common influence (such as the cheek), the 42-dimensional facial motion unit parameters and the 42-dimensional mouth shape visual position parameters are balanced by dynamic weighting. The formula for calculating the dynamic weight AU_final is as follows: AU_final = ×AU_viseme + (1- )×AU_emotion; in, AU_viseme represents the 42-dimensional lip-sync viewpoint parameter; AU_emotion represents the 42-dimensional facial motion unit parameter.

[0039] Specifically, in this embodiment, to completely eliminate the influence of emotion parameters and ensure absolute lip-sync accuracy, the lip-sync viewpoint parameter is weighted at 1, while the facial motion unit parameter is weighted at 0, thus covering the facial motion unit parameter. In practice, to reduce the influence of emotion parameters without completely eliminating them, the lip-sync viewpoint parameter is given a high weight, and the facial motion unit parameter a low weight, such as: AU_final = 0.8 × AU_viseme + 0.2 × AU_emotion, thereby weakening the facial motion unit parameter.

[0040] S57. Convert the enhanced motion parameter sequence into multi-degree-of-freedom control instructions for the bionic actuator, specifically including: S571. Obtain facial muscle biomechanical simulation data, establish a nonlinear mapping model for converting the abstract action unit space to the displacement space of a specific bionic actuator, and calibrate the nonlinear mapping model based on the facial muscle biomechanical simulation data to obtain the calibrated nonlinear mapping model. Specifically, facial muscle biomechanical simulation data is obtained using mature biomechanical simulation technology. The main process is as follows: Model building: Based on medical imaging or anatomical data, a multi-layered three-dimensional digital facial model containing skin, fat, muscle, and bone is built in professional software (such as ANSYS, Abaqus, Artisynth, etc.). Assign physical properties: Set material mechanical parameters such as elasticity and viscosity close to those of real biological tissues for each layer of tissue (such as skin and muscle); Drive and Calculation: The software simulates the contraction of specific facial muscles (drive signals), and then uses physics engines such as finite element analysis to calculate the resulting soft tissue deformation, stress distribution, etc., and finally outputs simulation data such as displacement and velocity of key points on the facial surface for model calibration.

[0041] S572. Extract the fused facial motion unit parameter AU_final for each time step from the enhanced motion parameter sequence, input it into the calibrated nonlinear mapping model, and obtain the expected displacement field of the facial surface at the corresponding time. S573. Construct a kinematic model for the bionic actuator array with parallel or series structure, including a forward kinematic model and a Jacobian matrix; combine the desired displacement field of the facial surface, the forward kinematic model and the Jacobian matrix, and calculate the desired joint position that minimizes the position error of the bionic actuator end effector through inverse kinematics solution, as the position control command. S574. Differentiate the desired joint position to obtain the desired joint velocity. Combine the error between the current joint position and the desired position to generate a velocity feedback command through a proportional-integral-derivative control algorithm. At the same time, calculate the desired torque command (including gravity compensation, inertial force compensation, etc.) based on a preset bionic actuator dynamics model. Combine the position control command, velocity feedback command, and desired torque command into a multi-degree-of-freedom control command for driving the bionic actuator.

[0042] Specifically, this embodiment also performs security verification and real-time distribution of the multi-degree-of-freedom control commands, and initiates a sensory closed loop.

[0043] Specifically, the safety verification includes: before sending the command, collision detection, singularity avoidance, workspace boundary checks, and joint limit protection are performed to ensure the physical safety of the command. After successful verification, the discrete control command stream is distributed to the corresponding actuator controller via a real-time communication bus. Simultaneously, a feedback closed loop based on facial marker visual sensors and electromyography sensors is initiated to monitor the deviation between the actual movement and the expected trajectory in real time, and the control command is fine-tuned online using a proportional-integral-derivative control algorithm to compensate for model errors and external disturbances, ultimately achieving accurate, stable, and natural facial emotion expression.

[0044] S6. Drive the simulator to move according to the multi-degree-of-freedom control command to complete the target expression and broadcast the voice, so as to realize emotional interaction.

[0045] In a specific embodiment, the specific steps for driving the simulation actuator to move according to the multi-degree-of-freedom control command include: S61. Receive the multi-degree-of-freedom control command through a preset three-order bionic transmission network and execute the corresponding action, and drive the bionic facial actuator to generate the target expression; The bionic facial actuator includes a linear motor, a servo motor, and a servo electric cylinder; Specifically, in this embodiment, the physical state of the bionic facial actuator is monitored in real time using a fiber optic grating sensor, including: Dual-grid differential compensation technology is used to improve the accuracy of tension measurement and eliminate the interference of temperature drift on the measurement results. Real-time monitoring of the grating wavelength offset difference, i.e.: , in, It is the wavelength offset of the fiber grating. and The difference between them and This is the sensitivity coefficient. and This represents the change in tension.

[0046] Specifically, since the two gratings are in the same temperature environment, the effect of temperature changes on their wavelengths is similar. By calculating the difference in wavelength offset between the two gratings, the common effects of temperature factors can be effectively offset, thereby improving the tension measurement accuracy to ±0.01N. Through precise wavelength measurement and effective temperature compensation, the tension change borne by the bionic facial actuator can be accurately calculated based on the wavelength change, thereby achieving precise monitoring of the actuator's physical state and feeding the monitoring data back to the control system in real time, forming an inner closed loop.

[0047] S62, Adaptive PID control and real-time error compensation, including: An adaptive PID algorithm is used to dynamically adjust the gain parameter according to the load changes of the bionic facial actuator. The formula is as follows: , in, This is the initial proportional gain. For adaptive coefficients, This refers to the instantaneous tracking error; A Lyapunov optimization controller is used to minimize the position error function. , It is a positive definite gain matrix, and the gradient of the control quantity is adjusted in real time to make it... V Decrease over time to ensure system stability and rapid convergence.

[0048] S63. System thermal management and lifespan protection, including: Dynamic derating based on the motor temperature rise-torque decay model: establishing the motor winding temperature rise... With current Differential equations: , in, For winding resistance, For heat dissipation coefficient, For surface area, Heat capacity; Specifically, by monitoring the motor temperature in real time, when the temperature rise exceeds a threshold, such as 60°C, the torque output is automatically reduced according to the pre-calibrated temperature-torque curve, such as by 15%, in order to balance performance and heat load and ensure the long-term reliability of the system.

[0049] Specifically, as shown in the table below, from multimodal recognition accuracy and temporal alignment efficiency to various indicators such as emotion decision-making, emotion transfer, special effects fusion, and micro-expression control, this embodiment significantly outperforms existing technologies in core dimensions such as accuracy, real-time performance, naturalness, and precision. This indicates that this embodiment comprehensively improves the performance of the emotion interaction system, providing technical support for more natural and accurate emotion computing applications.

[0050]

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating biomimetic actuator control commands based on cross-modal dynamic fusion, characterized in that, The specific steps include: S1. Collect the user's visual data, voice data, and physiological data through several modal sensors, and encode the visual data, voice data, and physiological data and perform inter-modal semantic alignment processing to obtain semantically aligned visual features, voice features, and physiological features. S2. Perform spatiotemporal alignment and registration on the visual and physiological features of semantic alignment to obtain visual-physiological alignment features; S3. Perform cross-modal attention fusion on the visual physiological alignment features and the semantically aligned speech features to obtain cross-modal attention fusion features; S4. Input the cross-modal attention fusion features into a preset DRL policy network, and combine them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions that match the user's state, and then output the emotional vector. S5. Convert the emotion vector into facial motion unit parameters, obtain the text data to be played based on the speech data and the pre-trained large language model; obtain the enhanced motion parameter sequence based on the facial motion unit parameters and the text data to be played, and parse it to obtain the bionic actuator control command. In S5, the emotion vector is converted into facial motion unit parameters, and the text data to be played is obtained based on the speech data and the pre-trained large language model. The specific steps for obtaining an enhanced motion parameter sequence based on the facial motion unit parameters and the text data to be played, and parsing it to obtain the bionic actuator control instructions, include: S51. Decoding the emotion vector into emotion semantics and mapping it to a low-dimensional action space includes: An emotion vector is initially mapped to the original parameters of the facial action unit using a multilayer perceptron model. S52. The original parameters of the facial motion unit are smoothed and optimized using a preset damped spring vibration model, and the smoothed and optimized motion unit parameters are obtained. S53. Apply anatomical and physiological constraints to the smoothed and optimized motion unit parameters to generate the final motion unit parameters. Specific steps include: A sparse optimization objective function is established based on the existing FACS action unit mutual exclusion rules. By solving the sparse optimization objective function, a sparse parameter set is generated. The formula for the sparse optimization objective function is as follows: ， In the formula, It is a 42-dimensional vector representing the final action unit parameters after sparse optimization that we need to solve; is the regularization coefficient, and it is greater than 0; For smoothing and optimizing the motion unit parameters; This is an L1 norm regularization term; Based on the sparse optimization objective function, a composite objective function is constructed that simultaneously considers accuracy, sparsity, mutual exclusion, and synergy. By solving the composite objective function, the optimized action unit parameters are obtained. The formula for the composite objective function is as follows: ； In the formula, It is a 42×42 action unit coordination matrix; It is a 42×42 action unit mutual exclusion matrix; The goal is to achieve a cooperative objective, which is a 42-dimensional vector. , and For collaborative weighting coefficients; The optimized motion unit parameters; For the optimized action unit parameters After normalization, the final output is optimized parameters that conform to emotional intent and are physiologically feasible for biomimetic execution. S54. Generate the text to be played based on the speech data and the pre-trained large language model, and convert the text to be played into a phoneme sequence and the corresponding time-phoneme aligned file through the TTS engine. S55. Traverse the time-phoneme alignment file and assign a target lip position parameter to each phoneme during its duration based on the phoneme-visual position mapping table, and finally calculate the representative lip position parameter. S56. The facial motion unit parameters and lip shape view position parameters are fused using a fusion strategy to obtain an enhanced motion parameter sequence with timestamps that incorporates lip shape and emotion. S57. Convert the enhanced motion parameter sequence into multi-degree-of-freedom control instructions for the bionic actuator.

2. The bionic actuator control command generation method based on cross-modal dynamic fusion according to claim 1, characterized in that, In S2, the specific steps for spatiotemporal alignment and registration of semantically aligned visual and physiological features to obtain visual-physiological aligned features include: S21. Set up a sliding window and cache visual and physiological features through the sliding window, and sort all cached data by timestamp; S22. Perform linear interpolation or cubic spline interpolation on the physiological features within a sliding window to generate corresponding physiological feature estimates for the timestamp of each frame of visual features, thereby obtaining the interpolated physiological features. S23. Calculate the minimum cumulative distance based on visual features and interpolated physiological features. To obtain the cumulative distance matrix, the minimum cumulative distance The calculation formula is: ； In the formula, The first of the physiological characteristics i The eigenvalue and the visual feature of the th eigenvalue i Distance metric between eigenvalues; Based on the cumulative distance matrix, the endpoint backtracking is performed to determine the optimal nonlinear mapping path. The optimal nonlinear mapping path is the time point mapping relationship that minimizes the cumulative distance between physiological features and visual features. The visual features and interpolated physiological features are time-aligned according to the optimal nonlinear mapping path to obtain time-synchronized sequence pairs. S24. Perform keypoint topology mapping on the time-synchronized sequence pairs and the current frame visual data to obtain visual physiological alignment features.

3. The bionic actuator control command generation method based on cross-modal dynamic fusion according to claim 2, characterized in that, In S24, the specific steps for performing keypoint topological mapping processing on the time-synchronized sequence pairs and the current frame visual data to obtain visual physiological alignment features include: Spatial mapping: The human body key points are identified from the current frame visual data using a preset key point detection model, and then a skeleton topology map is constructed. The human body key points include skeleton joints and sensor attachment reference points, wherein the sensor attachment reference points are the attachment reference points of physiological sensors in modal sensors. By combining time-synchronized sequence pairs, the location information of physiological sensors is mapped to sensor attachment reference points in the skeleton topology map; The positional information of the physiological sensors is transformed into the global visual space coordinate system where the skeleton topology map is located using an affine transformation matrix. The affine transformation matrix is ​​expressed as follows: ， Where A is the rotation and scaling matrix, and b is the translation vector; x,y ( ) are the coordinates of the physiological sensor; x',y' () are the transformed coordinates; Dynamic binding: Key point trajectory tracking technology is used to predict the motion trend of the attachment reference points of physiological sensors in the skeleton topology map. This includes: when the displacement of the physiological sensor is less than a set threshold, the historical binding relationship established by the skeleton topology map and the position information of the physiological sensor is used as the binding result; when the displacement of the physiological sensor is greater than or equal to the set threshold, the three-dimensional point cloud of the surface of the physiological sensor shell and the corresponding contact area of ​​the human skin surface is obtained, and then the three-dimensional point cloud of the two is registered using the ICP point cloud matching method. The optimal affine transformation matrix from the physiological sensor coordinate system to the skeleton coordinate system is recalculated, thereby realizing the dynamic update of the binding result between the position of the physiological sensor and the skeleton topology map. Visual physiological alignment features are obtained based on the binding results between the skeleton topology map and the positions of physiological sensors.

4. The bionic actuator control command generation method based on cross-modal dynamic fusion according to claim 1, characterized in that, In S3, the specific steps for cross-modal attention fusion of the visual physiological alignment features and the semantically aligned speech features include: Use any one of the following modalities as the query vector: spatiotemporally aligned visual features, physiological features, and semantically aligned speech features. Q The other two modes are key vectors. K Sum value vector V ; The attention mechanism is calculated based on the initial weights to obtain the cross-modal attention fusion features, which are represented as follows: ， In the formula, d k The dimension of the key vector; As weight; This includes real-time calculation of the signal-to-noise ratio of visual, speech, and physiological data to adjust the initial weights, including: The effective signal power and noise power of visual data, speech data, and physiological data are calculated separately, and then the signal-to-noise ratio is calculated using the following formula: ， In the formula, P signal The effective signal power for visual data, speech data, or physiological data; P noise Noise power for visual data, speech data, or physiological data; A dynamic weight adjustment strategy is implemented based on the signal-to-noise ratio calculation results, and the attention mechanism is recalculated based on the adjusted weights to update the cross-modal attention fusion features. The dynamic weight adjustment strategy includes: In a predefined strong light scene, the weight of visual features is reduced and the weight of voice features is increased according to a set ratio; In a predefined high-noise scenario, the weight of speech features is reduced and the weight of physiological features is increased according to a set ratio.

5. The biomimetic actuator control command generation method based on cross-modal dynamic fusion according to claim 4, characterized in that, In S4, the specific steps of inputting the cross-modal attention fusion features into a preset DRL network and combining them with a preset cognitive graph knowledge base to calculate the bionic robot's emotional actions that match the user's state, and then outputting the emotional vector, include: S41. Construct an environment state representation for the DRL policy network and query the cognitive graph knowledge base to obtain prior knowledge constraints, including: The cross-modal attention fusion features are used as the basic environmental state s0; Temporal context information, scene type identifier, and user personalized parameters are concatenated to the end of the basic environmental state s0 to construct an enhanced environmental state vector s with a total dimension of s0 + 24. Using the enhanced environmental state vector s as a query condition, semantic retrieval is performed in the cognitive graph knowledge base to obtain the emotional expression rules that best match the current scene and the user's national cultural background. Cultural weight coefficients, emotional intensity boundary constraints, and temporal smoothing parameters are extracted from the emotional expression rules. After concatenating, normalizing, and encoding the cultural weight coefficients, emotional intensity boundary constraints, and temporal smoothing parameters in sequence, a fixed-dimensional prior knowledge vector k is obtained. S42. Deep reinforcement learning decision-making based on the enhanced environment state vector s and prior knowledge vector k, including: The enhanced environment state vector s is concatenated with the prior knowledge vector k to obtain the concatenated state-knowledge vector [s;k], and the state-knowledge vector [s;k] is used as the input to the DRL policy network. The DRL policy network receives the concatenated state-knowledge vector [s;k] and processes it through a multi-layer neural network, including: using the prior knowledge vector k to initially constrain the search space, then combining it with the enhanced environment state vector s to calculate the optimal policy, and finally outputting an original emotional action a. raw ; DRL value networks receive raw emotional actions. raw And calculate the expected cumulative reward Q(s, a) raw The expected cumulative reward Q(s, a) raw This is used to guide the optimization of the DRL policy network; S43. Employ a dynamic rule engine to process original emotional actions a raw Conduct multi-dimensional feasibility calibration, including: Conduct cultural adaptation calibration: Applying cultural weighting coefficients to the original emotional action a raw The attenuation process is performed using the following formula: I cultural = a raw × culture ; In the formula, culture It is the cultural weighting coefficient, which is the influence of the user's national cultural background on emotional actions; to I cultural Conduct physiological feasibility verification: Muscle movement conflicts are detected by using a pre-set mutual exclusion matrix of action units to obtain an optimized I that meets physiological feasibility. optimized ; to I optimized Perform time-series smoothing: A dynamic model based on pre-set mass-damping-stiffness parameters is used for I. optimized Temporal smoothing is performed to obtain a 128-dimensional temporally smoothed vector after multi-constraint calibration and temporal smoothing. S44. Perform structured encoding on the 128-dimensional temporal smoothing vector, where: A 0-31 dimensional vector represents the emotional valence, with a value range of [-1, 1], and is used to characterize the positive or negative orientation of emotions; A 32-63 dimensional vector represents arousal level, with values ​​ranging from [0,1], and is used to characterize physiological activation levels; A 64-95 dimensional vector represents the degree of dominance, with values ​​ranging from [0,1], and is used to characterize subjective control ability; A 96-127 dimensional vector represents cultural context features, with values ​​ranging from [0,1], and includes cultural weights, scene types, and user-personalized parameters; S45. Verify the 128-dimensional temporal smoothing vector of the structured coding: Perform numerical range verification to ensure that all dimension values ​​are within the theoretical boundaries; Perform internal consistency checks to verify the logical coherence between different emotional dimensions; A time series continuity test is performed, and a smoothness index is calculated with historical data using a sliding window. Add timestamps and quality score metadata to sentiment vectors that meet the quality standards to obtain standardized data packages, i.e., 128-dimensional sentiment vectors.

6. The bionic actuator control command generation method based on cross-modal dynamic fusion according to claim 5, characterized in that, In S57, the specific steps for converting the enhanced motion parameter sequence into multi-degree-of-freedom control instructions for the bionic actuator include: S571. Obtain facial muscle biomechanical simulation data, establish a nonlinear mapping model for converting the abstract action unit space to the displacement space of a specific bionic actuator, and calibrate the nonlinear mapping model based on the facial muscle biomechanical simulation data to obtain the calibrated nonlinear mapping model. S572. Extract the fused facial motion unit parameter AU_final for each time step from the enhanced motion parameter sequence, input it into the calibrated nonlinear mapping model, and obtain the expected displacement field of the facial surface at the corresponding time. S573. Construct a kinematic model for the bionic actuator array with parallel or series structure, including a forward kinematic model and a Jacobian matrix; combine the desired displacement field of the facial surface, the forward kinematic model and the Jacobian matrix, and calculate the desired joint position that minimizes the position error of the bionic actuator end effector through inverse kinematics solution, as the position control command. S574. Differentiate the desired joint position to obtain the desired joint velocity. Combine the error between the current joint position and the desired position to generate a velocity feedback command through a proportional-integral-derivative control algorithm. At the same time, calculate the desired torque command based on a preset bionic actuator dynamics model. Combine the position control command, velocity feedback command, and desired torque command into a multi-degree-of-freedom control command for driving the bionic actuator.

7. The bionic actuator control command generation method based on cross-modal dynamic fusion according to claim 6, characterized in that, The fusion strategy involves weighting the facial motion unit parameters and lip-sync parameters based on their respective effects on different facial regions. Specifically, this includes: For the lips and surrounding preset areas, setting the mouth shape viewpoint parameters through a dynamic weighting method has the highest priority; For predefined non-conflict regions, facial motion unit parameters are fully preserved; For a predefined common influence area, facial motion unit parameters and lip shape visual position parameters are balanced by dynamic weighting. The formula for calculating the dynamic weight AU_final is as follows: AU_final = ×AU_viseme + (1- )×AU_emotion; in, AU_viseme represents the 42-dimensional lip-sync viewpoint parameter; AU_emotion represents the 42-dimensional facial motion unit parameter.

8. The method for generating bionic actuator control commands based on cross-modal dynamic fusion according to claim 1, characterized in that, The specific steps for encoding the visual data, speech data, and physiological data, as well as performing inter-modal semantic alignment processing, include: The ViT encoder is used to convert visual data into visual features; the Wav2Vec2 encoder is used to convert speech data into speech features; and a temporal convolutional network encoder is used to convert physiological data into physiological features. The transformed visual features, speech features, and physiological features are mapped to the constructed shared feature space using the corresponding projection matrices. By maximizing the similarity between mapped visual, speech, and physiological features using an alignment loss function, the visual, speech, and physiological features are aligned in a shared feature space, resulting in feature-aligned visual, speech, and physiological features. The formula for the alignment loss function is as follows: ， In the formula, Represents the feature vector of positive samples from any two different modalities. The similarity between them; τ Indicates temperature parameter; Represents the feature vector of a negative sample; j This represents the negative sample index.

Images