Model training method, humanoid robot dynamic control method based on multi-modal data fusion and related equipment

Abstract

The invention discloses a model training method, a humanoid robot dynamic control method based on multi-modal data fusion and related equipment, and the method comprises the steps: carrying out the preliminary training of a neural network model through a first-stage training sample set, obtaining an initial motion instruction, and carrying out the dynamic control of a humanoid robot based on the execution feedback information after the humanoid robot executes the initial motion instruction. Generating a sample optimization result, constructing a corresponding second-stage training set, carrying out the secondary training of the neural network model through the second-stage training set, obtaining a robot control model, and achieving the model training of multi-modal data fusion; the action control instruction is output through the robot control model according to the dynamically collected multi-mode sensing data, the humanoid robot is controlled to execute the target action based on the action control instruction so as to synchronously simulate the action of an operator, and the action simulation efficiency and accuracy of the humanoid robot in a complex scene are remarkably improved. The method can be widely applied to the technical field of robot control.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a model training method, a dynamic control method for humanoid robots based on multimodal data fusion, and related equipment. Background Technology

[0002] Among related technologies, existing humanoid robot control technologies have the following limitations:

[0003] 1) Limitations of single-modal data: Existing technologies rely on observation data from slave sensors (images, joint encoders, etc.) and master motion data, without fusing the operator's biological and sensory signals, resulting in motion mapping lag (delay ≥50ms);

[0004] 2) Poor environmental adaptability: Lacking tactile feedback and dynamic adjustment of environmental perception, it is prone to motion errors (error rate ≥15%) in scenarios such as grasping fragile items and walking on complex terrain.

[0005] 3) Insufficient sample quality control: The lack of a clear preprocessing mechanism for training samples leads to slow model training convergence (iteration count ≥ 2000 rounds).

[0006] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0007] The main objective of this application is to propose a model training method, a dynamic control method for humanoid robots based on multimodal data fusion, and related equipment.

[0008] To achieve the above objectives, one aspect of this application proposes a model training method, which includes the following steps:

[0009] Obtain the first-stage training sample set; the first-stage training sample set includes multiple multimodal sensing data, including operator biological and perceptual data, robot perception data, and target action execution data;

[0010] Synchronous preprocessing and feature fusion are performed on each of the multimodal sensing data to generate a fused feature vector corresponding to each of the multimodal sensing data.

[0011] For each of the multimodal sensing data, based on the fused feature vector, a corresponding initial action command is output through a neural network model, and the humanoid robot is controlled to perform the target action based on the initial action command;

[0012] Collect execution feedback information after the humanoid robot performs the target action, and optimize the initial action command and training sample weights based on the execution feedback information and the target action execution data to generate corresponding sample optimization results;

[0013] The multimodal sensing data and the corresponding sample optimization results are associated and bound to construct the second-stage training sample set.

[0014] The neural network model is trained using the second-stage training sample set to obtain the corresponding robot control model.

[0015] In some embodiments, the operator's biological and sensory data includes electroencephalogram (EEG) signals, electromyogram (EMG) signals, auditory command data, visual motion data, and joint control data; the robot's sensory data includes tactile data and environmental perception data.

[0016] In some embodiments, the step of synchronously preprocessing and fusing features on each of the multimodal sensing data to generate a fused feature vector corresponding to each of the multimodal sensing data includes:

[0017] The multimodal sensing data is spatiotemporally aligned and noise filtered to obtain preprocessed multimodal sensing data.

[0018] Feature extraction and fusion are performed on the multimodal sensing data to obtain the fused feature vector.

[0019] In some embodiments, the step of outputting corresponding initial action commands through a neural network model based on the fused feature vector for each of the multimodal sensing data, and controlling the humanoid robot to perform a target action based on the initial action commands, includes:

[0020] The fused feature vector is input into the first encoding layer of the neural network model, and the multimodal basic features are initially associated through an adaptive attention mechanism, outputting a primary association feature vector representing the underlying association between modalities;

[0021] The primary correlation feature vector is input into the second encoding layer of the neural network model. Local attention window processing is applied to the primary correlation feature vector to enhance its temporal series continuity, associate action features at adjacent time points, and output the corresponding temporal feature vector.

[0022] The temporal feature vector is input into the third encoding layer of the neural network model, and a cross-modal attention mechanism with visual features as query vectors is applied to the temporal feature vector to output the corresponding cross-modal feature vector.

[0023] The cross-modal feature vector is input into the fourth encoding layer of the neural network model, and a nonlinear transformation is performed on the cross-modal feature vector to output a higher-order feature vector after nonlinear transformation.

[0024] The higher-order feature vector is input into the fifth encoding layer of the neural network model and calculated through a global self-attention mechanism to model the dependency between features at different time steps in the action sequence and output the corresponding global context feature vector.

[0025] The global context feature vector is input into the sixth encoding layer of the neural network model. The global context feature vector is mapped to the multi-dimensional joint space through a dynamically configured weight matrix, and the initial action command corresponding to all joints of the humanoid robot is output.

[0026] In some embodiments, the step of collecting execution feedback information after the humanoid robot performs a target action, and optimizing the initial action command and training sample weights based on the execution feedback information and the target action execution data to generate corresponding sample optimization results includes:

[0027] Based on the execution feedback information and the target action execution data, the action execution error is determined, including posture error and tactile pressure error;

[0028] Based on the action execution error, the weights of the training samples corresponding to the initial action command and the multimodal sensing data are dynamically adjusted to generate the sample optimization result, including the optimized initial action command and training sample weights.

[0029] In some embodiments, training the neural network model using the second-stage training sample set to obtain the corresponding robot control model includes:

[0030] The multimodal sensing data in the second-stage training sample set are input into the neural network model. Based on the training sample weights corresponding to each multimodal sensing data, the model training process is optimized, and the action prediction control command corresponding to each multimodal sensing data is output.

[0031] Based on the action prediction control command corresponding to each of the multimodal sensing data, the training sample weights, and the optimized control command, a composite loss is calculated. Based on the composite loss, the model parameters of the neural network model are optimized through backpropagation to obtain the robot control model.

[0032] To achieve the above objectives, another aspect of this application proposes a dynamic control method for a humanoid robot based on multimodal data fusion, the method comprising the following steps:

[0033] Dynamically acquire multimodal sensing data, which includes the current operator's biological and sensory data and the current robot's sensory data;

[0034] The multimodal sensing data is input into the robot control model, and corresponding motion control commands are output; the robot control model is trained using the model training method described above.

[0035] Based on the motion control instructions, the humanoid robot is controlled to perform the target motion to synchronously mimic the operator's actions.

[0036] To achieve the above objectives, another aspect of this application proposes a humanoid robot dynamic control device based on multimodal data fusion, the device comprising:

[0037] The first module is used to dynamically collect multimodal sensing data, which includes the current operator's biological and sensory data and the current robot's sensory data.

[0038] The second module is used to input the multimodal sensing data into the robot control model and output corresponding motion control commands; the robot control model is trained using the model training method described above.

[0039] The third module is used to control the humanoid robot to perform target actions in a synchronous manner to mimic the operator's actions based on the motion control instructions.

[0040] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the model training method or the humanoid robot dynamic control method described above.

[0041] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned model training method or humanoid robot dynamic control method.

[0042] The embodiments of this application include at least the following beneficial effects: This application provides a model training method, a humanoid robot dynamic control method, device, electronic device, and program product based on multimodal data fusion. This solution performs synchronous preprocessing and feature fusion on the multimodal sensor data in the first-stage training sample set. Through the fused feature vectors of the multimodal sensor data, the neural network model is initially trained to obtain initial action commands. Based on the execution feedback information after the humanoid robot executes the initial action commands, the initial action commands and training sample weights are optimized to generate sample optimization results and construct the corresponding second-stage training set. The neural network model is then trained a second time through the second-stage training set to obtain the robot control model. This achieves multimodal data fusion model training, solving the problems of low motion mapping accuracy and poor environmental adaptability in traditional single-modal training, and significantly improving the prediction accuracy of the model. The robot control model outputs motion control commands based on dynamically collected multimodal sensor data. Based on the motion control commands, the humanoid robot is controlled to execute target actions to synchronously imitate the operator's actions, achieving synchronous motion imitation of the robot. This significantly improves the efficiency and accuracy of motion imitation of the humanoid robot in complex scenarios, as well as the robustness of task execution. Attached Figure Description

[0043] Figure 1 This is a flowchart of the model training method provided in the embodiments of this application;

[0044] Figure 2 This is a schematic diagram of the neural network model in the embodiments of this application;

[0045] Figure 3 This is a flowchart of a humanoid robot dynamic control method based on multimodal data fusion provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of the humanoid robot dynamic control device based on multimodal data fusion provided in the embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0049] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0050] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] The model training method or the humanoid robot dynamic control method based on multimodal data fusion provided in this application relates to the field of information technology. The model training method or the humanoid robot dynamic control method based on multimodal data fusion provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the model training method or the humanoid robot dynamic control method based on multimodal data fusion, but is not limited to the above forms.

[0053] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0054] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0055] Figure 1 This is an optional flowchart of the model training method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0056] Step S101: Obtain the first-stage training sample set; the first-stage training sample set includes multiple multimodal sensing data, including operator biological and perceptual data, robot perception data, and target action execution data;

[0057] Step S102: Perform synchronous preprocessing and feature fusion on each multimodal sensing data to generate a fused feature vector corresponding to each multimodal sensing data.

[0058] Step S103: For each multimodal sensor data, based on the fused feature vector, the corresponding initial action command is output through the neural network model, and the humanoid robot is controlled to perform the target action based on the initial action command.

[0059] Step S104: Collect execution feedback information after the humanoid robot performs the target action; based on the execution feedback information and the target action execution data, optimize the initial action command and training sample weights, and generate the corresponding sample optimization results.

[0060] Step S105: Associate and bind the multimodal sensing data and the corresponding sample optimization results of each multimodal sensing data to construct the second-stage training sample set.

[0061] Step S106: Train the neural network model using the second-stage training sample set to obtain the corresponding robot control model.

[0062] Steps S101 to S106 as shown in the embodiments of this application improve the accuracy of the model output results through the collaborative mechanism of multimodal data fusion, model training and dynamic feedback adjustment. This solves the problems of low motion mapping accuracy and poor environmental adaptability in traditional single-modal training (such as existing technologies that only rely on visual and motion data), and enables humanoid robots to accurately imitate complex movements and adapt to the environment.

[0063] In some embodiments, operator biosensory data includes electroencephalogram (EEG) signals, electromyogram (EMG) signals, auditory command data, visual motion data, and joint control data; robot sensory data includes tactile data and environmental sensory data.

[0064] Specifically, the first-stage training set is constructed through a data acquisition module, which includes an electroencephalogram (EEG) sensor (10-20 system EEG leads), an electromyogram (EMG) sensor (sampling rate 2000Hz), a binocular vision sensor (including an infrared depth camera), a directional auditory sensor, a motion capture device (multi-node IMU), a 6×6 array of tactile sensors at the robot end effector, and a lidar environmental sensor.

[0065] Multimodal sensor data is collected through the data acquisition module, as detailed below:

[0066] The EEG sensor is attached to the operator's motor cortex (leads C3 and C4) with conductive adhesive to make the impedance ≤5kΩ. It is used to collect the operator's EEG signals, such as motor cortex μ waves (8-12Hz), and to predict the grasping intention (triggering the action command 300ms in advance).

[0067] Electromyography (EMG) sensors are attached to the operator's biceps, triceps, and other muscle groups, parallel to the direction of muscle fibers, to collect the operator's EMG signals, such as monitoring the operator's arm muscle activation.

[0068] The binocular vision sensor and the robot base establish a mapping relationship between pixel coordinates and robot coordinate system through eye-hand calibration (reprojection error ≤ 1 pixel), which is used to collect the operator's visual motion data and reconstruct the operator's three-dimensional motion posture (error ≤ 2 mm), as the basis for motion commands;

[0069] Directional auditory sensors are used to collect auditory command data from operators, such as voice commands;

[0070] Motion capture equipment: collects joint control data of the operator, such as the angles of the operator's various joints;

[0071] The tactile sensor is fixed to the robot's end effector via an elastic beam and is calibrated with the robot flange using force-position calibration (error ≤ 0.5N). It is used to collect tactile data of the humanoid robot, such as detecting the distribution of the humanoid robot's gripping pressure to avoid damage to fragile items (threshold ≤ 5N).

[0072] LiDAR environmental sensor: Collects environmental perception data, such as the location and distance of obstacles in the environment where the humanoid robot is located (e.g., within a 5m radius).

[0073] In step S102 of some embodiments, the multimodal sensing data is spatiotemporally aligned and noise filtered to obtain preprocessed multimodal sensing data. Then, the multimodal sensing data is feature extracted and fused to obtain a fused feature vector.

[0074] Alternatively, its specific implementation method is as follows:

[0075] Spatiotemporal alignment: Based on the GPS timing module, the controller clocks of all sensors are synchronized, so that the data timestamp error is ≤ a preset time threshold (e.g., 10ms);

[0076] Noise filtering: EEG signals (Electroencephalogram) are filtered using a 50Hz notch filter, and EMG signals (Electromyogram) are denoised using db4 wavelet.

[0077] Feature extraction: μ-wave energy values ​​are extracted from EEG signals, root mean square (RMS) values ​​are extracted from EMG signals, and pose features are extracted from visual signals using ResNet50.

[0078] Feature fusion: The data fusion processor uses a weighted algorithm to generate a fused feature vector F, specifically expressed by the following formula:

[0079]

[0080] Where F is the fused feature vector, and N is the number of modal categories in the multimodal sensing data, including EEG, EMG, visual, auditory, motor, and tactile-environment modalities; w i f is the weighting factor for the i-th mode; i Let i be the feature vector of the i-th mode (e.g., μ-wave energy value for EEG features, root mean square value for EMG features), where i takes values ​​from 1 to N.

[0081] The weighting factor for each mode is calculated based on the preset signal-to-noise ratio (SNR) for each mode, as expressed by the following formula:

[0082]

[0083] Among them, P Signal,i Let P be the signal power of the i-th mode. noise,i Let be the noise power of the i-th mode.

[0084] In some embodiments, step S103 may include, but is not limited to, steps S201 to S206.

[0085] Step S201: Input the fused feature vector into the first encoding layer of the neural network model, perform preliminary association of multimodal basic features through an adaptive attention mechanism, and output a primary association feature vector representing the underlying association between modalities.

[0086] Step S202: Input the primary correlation feature vector into the second encoding layer of the neural network model, perform local attention window processing on the primary correlation feature vector to enhance its temporal series continuity, associate action features at adjacent time points, and output the corresponding temporal feature vector.

[0087] Step S203: Input the temporal feature vector into the third encoding layer of the neural network model, apply a cross-modal attention mechanism with visual features as the query vector to the temporal feature vector, and output the corresponding cross-modal feature vector;

[0088] Step S204: Input the cross-modal feature vector into the fourth encoding layer in the neural network model, perform a nonlinear transformation on the cross-modal feature vector, and output the high-order feature vector after the nonlinear transformation.

[0089] Step S205: Input the high-order feature vector into the fifth encoding layer in the neural network model, and perform calculation through the global self-attention mechanism to model the dependency relationship between features at different time steps in the action sequence, and output the corresponding global context feature vector.

[0090] Step S206: Input the global context feature vector into the sixth encoding layer in the neural network model, and map the global context feature vector to the multi-dimensional joint space through a dynamically configured weight matrix, and output the initial action commands corresponding to all joints of the humanoid robot.

[0091] In some embodiments, refer to Figure 2 The neural network model described above is an improved Transform model, which contains six encoding layers. The specific functions of each encoding layer are as follows:

[0092] 1) First coding layer:

[0093] Input data: fused feature vector F (128 dimensions) and location encoding information;

[0094] Intermediate processing: The self-attention mechanism is used to initially correlate the basic features of multimodalities (EEG μ-wave energy, EMG root mean square value) to capture the underlying correlations between modalities;

[0095] Output: Integrated primary association features (128 dimensions);

[0096] 2) Second coding layer:

[0097] Input data: Primary association features output from the first layer;

[0098] Intermediate processing: A local attention window (window size = 4) is used to enhance the continuity of the time series, focusing on associating action features at adjacent moments (such as the electromyographic change trend between the previous 0.1s and the current moment);

[0099] Output: Time-continuous feature vectors (dimension 128), i.e., time-series feature vectors;

[0100] 3) Third coding layer:

[0101] Input data: Temporal feature vectors output from the second layer;

[0102] Intermediate processing: Emphasize the dominant role of visual action posture through a cross-modal attention mechanism (visual features are query vectors, and other modalities are key-value pairs) and weaken low signal-to-noise ratio modalities (such as noisy auditory commands);

[0103] Output: Visual-dominated cross-modal features (128 dimensions), i.e., cross-modal feature vectors;

[0104] 4) Fourth coding layer:

[0105] Input data: Cross-modal feature vectors output from the third layer;

[0106] Intermediate processing: A feed-forward network (FFN) (with 256 hidden layers and ReLU activation) is introduced to perform non-linear transformations on the features, enhancing the ability to express complex action patterns (such as curve grasping trajectories).

[0107] Output: High-order eigenvectors after nonlinear transformation (dimension 128);

[0108] 5) Fifth coding layer:

[0109] Input data: High-order feature vectors output from the fourth layer;

[0110] Intermediate processing: By using a global attention mechanism to associate full-sequence features (EEG prediction and end-effector tactile feedback throughout the entire action cycle), a global match between action intention and execution effect is achieved;

[0111] Output: Global context feature vector (128 dimensions);

[0112] 6) Sixth coding layer:

[0113] Input data: Global context feature vector (128 dimensions) output from the fifth layer;

[0114] Intermediate processing: An adaptive projection layer is used to map features to the multi-dimensional joint space of the target robot through a configurable weight matrix (dimension 128×M, where M is the total number of joints of the humanoid robot, supporting dynamic configuration); configuration parameters are imported through an external configuration file, supporting different models such as M=12 (small humanoid robot), M=18 (standard humanoid robot), M=32 (high-precision collaborative humanoid robot); combined with layer normalization (normalization range [-1,1]) for stable output, it adapts to the physical limitations of different robot joint angles (such as rotation range ±180° or ±90°);

[0115] Output: M-dimensional initial motion commands (including target angles for the M joints of the humanoid robot).

[0116] In step S105 of some embodiments, the action execution error, including posture error and tactile pressure error, is determined based on the execution feedback information and target action execution data. Based on the action execution error, the training sample weights corresponding to the initial action command and multimodal sensing data are dynamically adjusted to generate sample optimization results, including optimized initial action command and training sample weights. The training sample weights are used to adjust the contribution of different samples in the loss function to handle data imbalance or emphasize important samples.

[0117] Optionally, based on the action execution error, the initial action instruction is dynamically optimized using a correction formula, as follows:

[0118] Δθ=k1·e pose +k2·e force ,

[0119] Where Δθ is the joint angle correction amount (rad), e pose The pose error is the difference (in rad) between the robot's current pose and the target pose after performing the target action. force For tactile pressure error, k1 is the preset posture feedback weight (e.g., 0.8) and k2 is the preset force feedback weight (e.g., 0.3).

[0120] In some embodiments, step S106 may include, but is not limited to, steps S301 to S302.

[0121] Step S301: Input the multimodal sensing data from the second-stage training sample set into the neural network model, optimize the model training process based on the training sample weights corresponding to each multimodal sensing data, and output the action prediction control command corresponding to each multimodal sensing data.

[0122] Step S302: Calculate the composite loss based on the motion prediction control command, training sample weights, and optimized control command corresponding to each multimodal sensing data. Based on the composite loss, optimize the model parameters of the neural network model through backpropagation to obtain the robot control model.

[0123] In some embodiments, each multimodal sensing data is associated and bound with its corresponding fused feature vector and optimized initial action command to construct a second-stage training sample set. In this second-stage training sample set, each sample is a data combination of "multimodal sensing data - fused feature vector - optimized initial action command".

[0124] The neural network model is iteratively trained using the second-stage training sample set, combined with the composite loss function and backpropagation algorithm, until the model converges or the model error rate is less than a preset threshold (e.g., 5%). Training is then stopped, and the trained neural network model, i.e., the robot control model mentioned above, is obtained.

[0125] The model training method of this invention will be described in detail and explained below, using a specific application example of a robot mimicking an operator grasping a glass:

[0126] 1) Collect multimodal sensing data: The operator's grasping intention (μ wave energy ≥ 0.8mV2) is collected through the EEG sensor, the biceps brachii activation is shown by the electromyography signal (RMS = 0.5mV), the operator's hand posture when approaching the glass is collected through the visual sensor (10cm above the glass), and the pre-contact pressure (3N) of the humanoid robot grasping the glass is collected through the tactile sensor.

[0127] 2) Generate a fusion feature vector F based on multimodal sensing data, and trigger the initial action command (finger opening angle 30°) through the neural network model to control the robot to perform the initial glass grasping;

[0128] 3) Feedback Adjustment: The detected motion execution error is: tactile pressure error e force =1N and attitude error e pose =0.02rad, the posture feedback weight is 0.8, the force feedback weight is 0.3, then the corresponding joint correction angle Δθ = 0.8 × 0.02 + 0.3 × 1 = 0.316rad. The initial motion command is optimized by this joint correction angle, and the optimized initial motion command, i.e. the motion optimization command, is generated. The motion optimization command is used to control the robot to grasp the glass again, so that the robot's grasping force on the glass is stabilized at 3N.

[0129] 4) After iteratively training the neural network model using the second-stage sample set constructed by combining "multimodal sensing data - fused feature vectors - action optimization instructions", the model's capture success rate increased from the initial 60% to 95%.

[0130] Figure 3 This is an optional flowchart of a humanoid robot dynamic control method based on multimodal data fusion provided in the embodiments of this application. Figure 3 The method may include, but is not limited to, steps A101 to A103.

[0131] Step A101: Dynamically collect multimodal sensing data, which includes the current operator's biological and perceptual data and the current robot's perceptual data;

[0132] Step A102: Input the multimodal sensing data into the robot control model and output the corresponding motion control commands; the robot control model is trained using the model training method described above.

[0133] Step A103: Based on motion control instructions, control the humanoid robot to perform the target action to synchronously imitate the operator's actions.

[0134] In some embodiments, multimodal sensor data is received through a robot control model. By fusing operator physiological signals (EEG / EMG), multisensory data (visual / auditory), and robot proprioceptive feedback (tactile / environmental), corresponding motion control commands are generated to control the robot to perform synchronous motion imitation. This enables advanced prediction of the operator's motion intentions and dynamic optimization of robot motion planning, overcoming the limitations of single-modal training in existing technologies and significantly improving the motion imitation accuracy and environmental adaptability of humanoid robots.

[0135] Please see Figure 4 This application also provides a humanoid robot dynamic control device based on multimodal data fusion, which can implement the above-mentioned method. The device includes:

[0136] The first module is used to dynamically collect multimodal sensing data, which includes the current operator's biological and sensory data and the current robot's sensory data.

[0137] The second module is used to input multimodal sensing data into the robot control model and output corresponding motion control commands; the robot control model is trained using the model training method described above.

[0138] The third module is used to control the humanoid robot to perform target actions based on motion control commands in order to synchronously imitate the operator's actions.

[0139] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0141] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0143] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0144] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0145] The input / output interface 903 is used to implement information input and output;

[0146] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0147] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0148] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0150] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0152] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0154] The humanoid robot dynamic control method, device, electronic device, storage medium, and program product based on multimodal data fusion provided in this application embodiment synchronously preprocesses and fuses features of various multimodal sensor data in the first-stage training sample set. Using the fused feature vectors of the various multimodal sensor data, a neural network model is initially trained to obtain initial action commands. Based on the execution feedback information after the humanoid robot executes the initial action commands, the initial action commands and training sample weights are optimized, generating sample optimization results and constructing a corresponding second-stage training set. The neural network model is then trained a second time using the second-stage training set to obtain a robot control model. This achieves multimodal data fusion model training, solving the problems of low action mapping accuracy and poor environmental adaptability in traditional single-modal training, and significantly improving the model's prediction accuracy. The robot control model outputs action control commands based on dynamically acquired multimodal sensor data, controlling the humanoid robot to execute target actions to synchronously imitate operator actions, achieving synchronous action imitation of the robot. This significantly improves the efficiency and accuracy of humanoid robot action imitation in complex scenarios, as well as the robustness of task execution.

[0155] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0156] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0159] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0160] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0162] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that, The method includes the following steps: Obtain the first-stage training sample set; the first-stage training sample set includes multiple multimodal sensing data, including operator biological and perceptual data, robot perception data, and target action execution data; Synchronous preprocessing and feature fusion are performed on each of the multimodal sensing data to generate a fused feature vector corresponding to each of the multimodal sensing data. For each of the multimodal sensing data, based on the fused feature vector, a corresponding initial action command is output through a neural network model, and the humanoid robot is controlled to perform the target action based on the initial action command; Collect execution feedback information after the humanoid robot performs the target action, and optimize the initial action command and training sample weights based on the execution feedback information and the target action execution data to generate corresponding sample optimization results; The multimodal sensing data and the corresponding sample optimization results are associated and bound to construct the second-stage training sample set. The neural network model is trained using the second-stage training sample set to obtain the corresponding robot control model.

2. The method according to claim 1, characterized in that, The operator's biological and sensory data includes electroencephalogram (EEG) signals, electromyogram (EMG) signals, auditory command data, visual motion data, and joint control data; the robot's sensory data includes tactile data and environmental perception data.

3. The method according to claim 1, characterized in that, The step of synchronously preprocessing and fusing features of each of the multimodal sensing data to generate a fused feature vector corresponding to each of the multimodal sensing data includes: The multimodal sensing data is spatiotemporally aligned and noise filtered to obtain preprocessed multimodal sensing data. Feature extraction and fusion are performed on the multimodal sensing data to obtain the fused feature vector.

4. The method according to claim 1, characterized in that, The step of, for each of the multimodal sensing data, outputting corresponding initial action commands through a neural network model based on the fused feature vector, and controlling the humanoid robot to perform target actions based on the initial action commands, includes: The fused feature vector is input into the first encoding layer of the neural network model, and the multimodal basic features are initially associated through an adaptive attention mechanism, outputting a primary association feature vector representing the underlying association between modalities; The primary correlation feature vector is input into the second encoding layer of the neural network model. Local attention window processing is applied to the primary correlation feature vector to enhance its temporal series continuity, associate action features at adjacent time points, and output the corresponding temporal feature vector. The temporal feature vector is input into the third encoding layer of the neural network model, and a cross-modal attention mechanism with visual features as query vectors is applied to the temporal feature vector to output the corresponding cross-modal feature vector. The cross-modal feature vector is input into the fourth encoding layer of the neural network model, and a nonlinear transformation is performed on the cross-modal feature vector to output a higher-order feature vector after nonlinear transformation. The higher-order feature vector is input into the fifth encoding layer of the neural network model and calculated through a global self-attention mechanism to model the dependency between features at different time steps in the action sequence and output the corresponding global context feature vector. The global context feature vector is input into the sixth encoding layer of the neural network model. The global context feature vector is mapped to the multi-dimensional joint space through a dynamically configured weight matrix, and the initial action command corresponding to all joints of the humanoid robot is output.

5. The method according to claim 1, characterized in that, The process involves collecting execution feedback information after the humanoid robot performs a target action, optimizing the initial action command and training sample weights based on the execution feedback information and the target action execution data, and generating corresponding sample optimization results, including: Based on the execution feedback information and the target action execution data, the action execution error is determined, including posture error and tactile pressure error; Based on the action execution error, the weights of the training samples corresponding to the initial action command and the multimodal sensing data are dynamically adjusted to generate the sample optimization result, including the optimized initial action command and training sample weights.

6. The method according to claim 1, characterized in that, The step of training the neural network model using the second-stage training sample set to obtain the corresponding robot control model includes: The multimodal sensing data in the second-stage training sample set are input into the neural network model. Based on the training sample weights corresponding to each multimodal sensing data, the model training process is optimized, and the action prediction control command corresponding to each multimodal sensing data is output. Based on the action prediction control command corresponding to each of the multimodal sensing data, the training sample weights, and the optimized control command, a composite loss is calculated. Based on the composite loss, the model parameters of the neural network model are optimized through backpropagation to obtain the robot control model.

7. A dynamic control method for a humanoid robot based on multimodal data fusion, characterized in that, The method includes the following steps: Dynamically acquire multimodal sensing data, which includes the current operator's biological and sensory data and the current robot's sensory data; The multimodal sensing data is input into the robot control model, and corresponding motion control commands are output; the robot control model is trained using the model training method described in any one of claims 1 to 6. Based on the motion control instructions, the humanoid robot is controlled to perform the target motion to synchronously mimic the operator's actions.

8. A dynamic control device for a humanoid robot based on multimodal data fusion, characterized in that, The device includes: The first module is used to dynamically collect multimodal sensing data, which includes the current operator's biological and sensory data and the current robot's sensory data. The second module is used to input the multimodal sensing data into the robot control model and output corresponding motion control commands; the robot control model is trained by the model training method described in any one of claims 1 to 6. The third module is used to control the humanoid robot to perform target actions in a synchronous manner to mimic the operator's actions based on the motion control instructions.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images