Humanoid robot target tracking interactive control method and related equipment

Abstract

This application discloses a target tracking and interactive control method and related equipment for a humanoid robot, belonging to the field of humanoid robots. The method includes: acquiring current and historical multimodal data of a moving target, acquiring current and historical posture data of the robot, generating multimodal fusion data and dynamically voxelizing it, generating an encoding matrix and inputting it into a feature extraction model, acquiring a target dynamic feature map and inputting it into a prediction model, obtaining trajectory prediction results, generating and executing motion control commands. This application achieves fusion tracking of the moving target and the humanoid robot's multimodal data, extracts the target's dynamic feature map using a spatiotemporal feature extraction network model, and then uses a target tracking prediction model constructed with a Kalman filter algorithm and attention mechanism to predict the moving target's trajectory. This effectively improves the tracking accuracy and prediction accuracy of the moving target, thereby enhancing the accuracy of the humanoid robot's interactive control.

Description

Technical Field

[0001] This application relates to the field of humanoid robot technology, and in particular to a humanoid robot target tracking interactive control method and related equipment. Background Technology

[0002] Currently, humanoid robots have become core equipment in industrial automation and home services.

[0003] In related technologies, humanoid robots need to frequently interact with dynamic targets (such as workpieces on conveyor belts in industrial settings, users' hands in home settings, and handling boxes in warehouse settings). However, in practical applications, it has been found that traditional interactive control methods for humanoid robots suffer from problems such as high tracking errors and large prediction errors.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] This application provides a humanoid robot target tracking and interactive control method and related equipment, which can effectively improve the tracking accuracy and prediction accuracy of moving targets, thereby improving the interactive control accuracy of humanoid robots.

[0006] On one hand, embodiments of this application provide a humanoid robot target tracking and interactive control method, the method comprising the following steps:

[0007] The system acquires in real time the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period, and acquires the current posture data of the humanoid robot and the historical posture data within a preset time period; wherein, the multimodal data includes point cloud data, image data and depth data;

[0008] By combining the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data, multimodal fusion data is generated.

[0009] The multimodal fusion data is dynamically voxelized and encoded to generate a multimodal encoding matrix;

[0010] The multimodal coding matrix is ​​input into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model;

[0011] The target dynamic feature map is input into the target tracking prediction model to obtain the trajectory prediction result output by the target tracking prediction model; wherein, the trajectory prediction result includes position coordinates, motion speed and motion trajectory;

[0012] Based on the interaction requirements of the moving target to be tracked, and combined with the trajectory prediction results output by the target tracking prediction model, motion control commands for the humanoid robot are generated and executed.

[0013] The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model, and the target tracking prediction model is built on the basis of the Kalman filter algorithm and the attention mechanism.

[0014] Optionally, the step of combining the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fusion data includes:

[0015] A three-dimensional coordinate system is constructed based on the current posture data of the humanoid robot;

[0016] By combining the historical posture data of the humanoid robot, the difference between the heading angle of the humanoid robot within a preset time period and the current heading angle, as well as the torso position deviation of the humanoid robot, are determined, and the rotation transformation matrix and translation transformation matrix are calculated.

[0017] Based on the rotation transformation matrix and the translation transformation matrix, the coordinate transformation of the historical multimodal data in the three-dimensional coordinate system is completed, and the data is merged with the current multimodal data to obtain multimodal fusion data.

[0018] Optionally, the step of dynamically voxelizing and encoding the multimodal fused data to generate a multimodal encoding matrix includes:

[0019] Based on the speed of the moving target, the pre-set initial voxel size is corrected to obtain the dynamic voxel size;

[0020] Based on the three-dimensional coordinate system, a voxel space is constructed according to the dynamic voxel size, and the multimodal fusion data is mapped to the voxel space;

[0021] The number of point clouds, the average number of image pixels, and the average depth within each voxel are counted to construct a multimodal coding matrix.

[0022] Optionally, the spatiotemporal feature extraction network model includes a temporal coding submodule and a spatial coding submodule that are connected alternately in sequence;

[0023] The number of the temporal coding submodules is the same as the number of the spatial coding submodules;

[0024] The temporal coding submodule is a three-dimensional convolutional neural network model; the spatial coding submodule is a two-dimensional convolutional neural network model, combined with an attention mechanism.

[0025] Optionally, the step of generating and executing motion control commands for the humanoid robot based on the interaction requirements of the moving target to be tracked, combined with the trajectory prediction results output by the target tracking prediction model, includes:

[0026] Based on the trajectory prediction results output by the target tracking prediction model, and according to the interaction requirements of the moving target to be tracked, the motion vector of the humanoid robot is generated.

[0027] Based on the DH parameter method for humanoid robot arms, the motion vector is decomposed into joint angle changes by matrix inversion;

[0028] Based on the joint amplitude limit range, the joint angular velocity is calculated according to the change in joint angle and the preset movement time;

[0029] The torque parameters are calculated based on the joint load parameters.

[0030] By combining the changes in joint angles, the joint angular velocities, and the torque parameters, motion control commands for the humanoid robot are generated and executed.

[0031] Optionally, after generating and executing motion control commands for the humanoid robot based on the interaction requirements of the moving target to be tracked, combined with the trajectory prediction results output by the target tracking prediction model, the method further includes:

[0032] Obtain the execution result information of the humanoid robot in response to the motion control command;

[0033] When the error of the execution result information is greater than a preset threshold, the motion control command of the humanoid robot is dynamically corrected based on the PID algorithm, and the dynamically corrected motion control command is executed.

[0034] On the other hand, embodiments of this application provide a humanoid robot target tracking and interactive control device, the device comprising:

[0035] The data acquisition module is used to acquire the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period in real time, and to acquire the current posture data of the humanoid robot and the historical posture data within a preset time period; wherein, the multimodal data includes point cloud data, image data and depth data;

[0036] The data fusion module is used to combine the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fused data;

[0037] The matrix generation module is used to dynamically voxelize and encode the multimodal fusion data to generate a multimodal encoding matrix;

[0038] The feature extraction module is used to input the multimodal encoding matrix into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model;

[0039] The trajectory prediction module is used to input the target dynamic feature map into the target tracking prediction model and obtain the trajectory prediction result output by the target tracking prediction model; wherein, the trajectory prediction result includes position coordinates, motion speed and motion trajectory;

[0040] The instruction execution module is used to generate and execute motion control instructions for the humanoid robot based on the interaction requirements of the moving target to be tracked and the trajectory prediction results output by the target tracking prediction model.

[0041] The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model, and the target tracking prediction model is built on the basis of the Kalman filter algorithm and the attention mechanism.

[0042] On the other hand, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0043] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0044] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0045] This application embodiment achieves fusion tracking by acquiring multimodal data of moving targets and humanoid robots, extracts dynamic feature maps of targets through a spatiotemporal feature extraction network model, and then uses a target tracking prediction model constructed with Kalman filtering algorithm and attention mechanism to predict the trajectory of moving targets. This can effectively improve the tracking accuracy and prediction accuracy of moving targets, thereby enhancing the interactive control accuracy of humanoid robots. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the implementation environment of a humanoid robot target tracking interactive control method provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating a humanoid robot target tracking and interactive control method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a target tracking interactive control and execution feedback process provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a humanoid robot target tracking interactive control device provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0052] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0053] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] Currently, humanoid robots have become core equipment in industrial automation and home services.

[0056] In related technologies, humanoid robots need to frequently interact with dynamic targets (such as workpieces on conveyor belts in industrial settings, users' hands in home settings, and handling boxes in warehouse settings). However, in practical applications, it has been found that traditional interactive control methods for humanoid robots suffer from problems such as high tracking errors and large prediction errors.

[0057] In summary, the technical problems existing in the relevant technologies need to be improved.

[0058] In view of this, this application provides a humanoid robot target tracking interactive control method and related equipment. By acquiring multimodal data of the moving target and the humanoid robot for fusion tracking, and extracting the target dynamic feature map through a spatiotemporal feature extraction network model, the target tracking prediction model constructed by the Kalman filter algorithm and attention mechanism is used to predict the trajectory of the moving target. This can effectively improve the tracking accuracy and prediction accuracy of the moving target, thereby improving the interactive control accuracy of the humanoid robot.

[0059] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0060] The specific implementation methods of the embodiments of this application will be described in detail below with reference to the accompanying drawings. First, a humanoid robot target tracking interactive control method provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0061] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the implementation environment for a humanoid robot target tracking interactive control method provided in this application embodiment. In this implementation environment, the main hardware and software components involved include a terminal processor 110 and a server 120.

[0062] Specifically, the terminal processor 110 may be equipped with a control program for a humanoid robot target tracking interactive control method, and the server 120 serves as the backend server for this control program. The terminal processor 110 and the backend server 120 are connected for communication. The humanoid robot target tracking interactive control method provided in this embodiment can be executed on the terminal processor 110 side.

[0063] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0064] In addition, server 120 can also be a node server in a blockchain network.

[0065] The terminal processor 110 and the server 120 can establish a communication connection via a wireless network. This wireless network uses standard communication technologies and / or protocols. The network can be the Internet or any other network, including but not limited to a Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, or any combination of wireless networks, private networks, or virtual private networks. Furthermore, these hardware and software components can use the same or different communication connection methods; this application does not impose specific limitations in this regard.

[0066] Of course, this is understandable. Figure 1 The implementation environment described in this application is only one of the optional application scenarios for the humanoid robot target tracking and interactive control method provided in this embodiment. The actual application is not fixed. Figure 1 The software and hardware environment shown is not specifically limited in this application.

[0067] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a humanoid robot target tracking and interactive control method provided in an embodiment of this application, specifically including but not limited to steps 100 to 600.

[0068] Step 100: Acquire the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period in real time, and acquire the current posture data of the humanoid robot and the historical posture data within a preset time period.

[0069] The multimodal data includes point cloud data, image data, and depth data.

[0070] In this embodiment of the application, a moving target refers to a moving target that the humanoid robot needs to track and interact with. For example, when the humanoid robot is performing the task of grasping a workpiece on a conveyor belt in an industrial scenario, the workpiece on the conveyor belt is the moving target to be tracked.

[0071] Furthermore, the multimodal data mainly includes point cloud data, image data, and depth data. The current multimodal data is acquired in real time through sensor devices mounted on the humanoid robot, while historical multimodal data within a preset time period can be obtained by reading previously acquired data. For example, the preset time period can be set to a preset number of historical frames (e.g., 5 frames) with the current frame as the marker point.

[0072] In practical applications, a LiDAR installed on a humanoid robot can collect current frame point cloud data of a moving target and acquire multiple frames of point cloud data within a preset historical frame range. Similarly, an RGB camera installed on the humanoid robot can collect current frame image data of a moving target and acquire multiple frames of image data within a preset historical frame range. A depth camera installed on the humanoid robot can also collect current frame depth data of a moving target and acquire multiple frames of depth data within a preset historical frame range. The depth data collected by the depth camera is used to measure the spatial distance between various points on the surface of the moving target and the location of the depth camera on the humanoid robot.

[0073] Furthermore, the joint angles, torso position, and heading angle of the humanoid robot can be obtained as posture data by using joint attitude sensors built into the servo motors of each joint of the humanoid robot.

[0074] Step 200: Combine the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fusion data.

[0075] In this embodiment, the three-dimensional shape, visual texture, and distance information of the target are collected by LiDAR, RGB camera, and depth camera. This information can be combined with the robot's own posture sensor data and a reference coordinate system can be constructed. The historical multi-frame data can be transformed into the same coordinate system to achieve spatiotemporal alignment of multimodal data, make up for the information shortcomings of a single sensor, and improve the integrity of the target features.

[0076] Specifically, as an optional implementation, the step of combining the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fusion data includes:

[0077] A three-dimensional coordinate system is constructed based on the current posture data of the humanoid robot;

[0078] By combining the historical posture data of the humanoid robot, the difference between the heading angle of the humanoid robot within a preset time period and the current heading angle, as well as the torso position deviation of the humanoid robot, are determined, and the rotation transformation matrix and translation transformation matrix are calculated.

[0079] Based on the rotation transformation matrix and the translation transformation matrix, the coordinate transformation of the historical multimodal data in the three-dimensional coordinate system is completed, and the data is merged with the current multimodal data to obtain multimodal fusion data.

[0080] In this embodiment of the application, the joint angles, body position and heading angle of the current posture data of the humanoid robot can be extracted to construct a three-dimensional reference coordinate system, wherein the center of the humanoid robot's torso is the origin, the forward direction of the humanoid robot is the X-axis, the horizontal rightward direction perpendicular to the forward direction of the humanoid robot is the Y-axis, and the Z-axis is perpendicular to the ground and pointing upward.

[0081] Furthermore, for the i-th frame of historical data, based on the motion posture data of the humanoid robot in that frame, the rotation transformation matrix and translation transformation matrix of the historical frame coordinate system relative to the reference coordinate system are calculated.

[0082] The rotation transformation matrix can be expressed as: ,in, The translation transformation matrix is ​​determined by the difference between the heading angle of the humanoid robot within a preset time period and the current heading angle, based on the deviation of the humanoid robot's torso position.

[0083] Furthermore, it can be achieved through formulas The coordinates of each point in the historical frame point cloud data Transform the coordinates to the reference coordinate system P(x,y,z), where, Let be the rotation transformation matrix. This is the translation transformation matrix.

[0084] Furthermore, the above steps are repeated to complete the coordinate transformation of image data and depth data, and finally merged with the current multimodal data to obtain multimodal fusion data.

[0085] For example, a reference coordinate system O-XYZ is established with the center of the robot's torso in frame t as the origin O. For the historical frame t-1, the difference in heading angle between this frame and frame t is calculated. Poor trunk position Therefore, the rotation transformation matrix can be calculated. for .

[0086] Furthermore, the above formula can be used to determine the value of a point in the point cloud data of frame t-1. Convert to coordinates in the reference coordinate system Then, the coordinates of point cloud data, image data, and depth data from all historical frames are transformed and finally merged to obtain multimodal fusion data.

[0087] Step 300: Perform dynamic voxelization encoding on the multimodal fusion data to generate a multimodal encoding matrix.

[0088] In this embodiment of the application, the merged multimodal fusion data can be further dynamically voxelized and encoded to generate a multimodal encoding matrix.

[0089] For example, the step of dynamically voxelizing and encoding the multimodal fusion data to generate a multimodal encoding matrix includes:

[0090] Based on the speed of the moving target, the pre-set initial voxel size is corrected to obtain the dynamic voxel size;

[0091] Based on the three-dimensional coordinate system, a voxel space is constructed according to the dynamic voxel size, and the multimodal fusion data is mapped to the voxel space;

[0092] The number of point clouds, the average number of image pixels, and the average depth within each voxel are counted to construct a multimodal coding matrix.

[0093] In the embodiments of this application, the voxel size can be dynamically adjusted according to the motion characteristics of the moving target, such as the motion speed, to obtain a dynamic voxel size.

[0094] Specifically, the dynamic voxel size adjustment formula can be expressed as follows: ),in, For the initial voxel size, This is the voxel adjustment factor (the value can be set from 0.1 to 0.3). Let be the velocity of the moving target in frame t. Let be the dynamic voxel size of the t-th frame.

[0095] Furthermore, based on the three-dimensional reference coordinate system constructed in the above steps, a voxel space is constructed, and the point cloud data in the multimodal fusion data is mapped to the voxel space. Then, the number of point clouds, the average number of image pixels, and the average depth in each voxel are counted, and finally, a multimodal coding matrix is ​​generated.

[0096] Therefore, this application can dynamically adjust the voxel size according to the movement speed of the moving target (the faster the speed, the larger the voxel size, reducing the amount of computation; the slower the speed, the smaller the voxel size, preserving details), which can achieve adaptive adjustment and solve the contradiction between efficiency and accuracy at different movement speeds with a fixed voxel size.

[0097] Step 400: Input the multimodal coding matrix into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model.

[0098] The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model.

[0099] In this embodiment, a spatiotemporal feature extraction network model is constructed, and a multimodal coding matrix is ​​input into the spatiotemporal feature extraction network model. The target motion temporal features and target morphological spatial features are extracted sequentially through the temporal coding submodule and spatial coding submodule in the spatiotemporal feature extraction network model to obtain the target dynamic feature map.

[0100] In practical applications, the spatiotemporal feature extraction network model is built on the basis of the convolutional neural network model (CNN). Specifically, it can include multiple sequentially connected temporal coding sub-modules and spatial coding sub-modules, and the number of temporal coding sub-modules and spatial coding sub-modules is the same.

[0101] For example, the spatiotemporal feature extraction network model includes four temporal coding sub-modules and four spatial coding sub-modules, and each temporal coding sub-module and spatial coding sub-module are alternately connected. For example, after the multimodal coding matrix is ​​input into the spatiotemporal feature extraction network model, it passes through the first temporal coding sub-module, the first spatial coding sub-module, the second temporal coding sub-module, the second spatial coding sub-module, and finally the fourth temporal coding sub-module before being output.

[0102] Furthermore, the temporal coding submodule is constructed using a 3D CNN, which captures the motion trend features of multiple frames of data in the time-space dimension through 3D convolutional kernels. The spatial coding submodule is constructed using a 2D CNN and combined with an attention mechanism, which can focus on key spatial regions.

[0103] Furthermore, by constructing a dataset covering the motion data of industrial workpieces / household targets and setting a hybrid loss function combining mean squared error (MSE) and cross-entropy loss, the spatiotemporal feature extraction network model can be pre-trained through multiple rounds of iterative training. Normalization layers and ReLU activation functions can be set after each temporal encoding submodule and spatial encoding submodule to accelerate network convergence and enhance feature representation capabilities.

[0104] Therefore, this application can effectively improve the feature recognition accuracy of moving targets by jointly extracting spatiotemporal features through a spatiotemporal feature extraction network model.

[0105] Step 500: Input the target dynamic feature map into the target tracking prediction model and obtain the trajectory prediction result output by the target tracking prediction model.

[0106] The trajectory prediction results include position coordinates, movement speed, and movement trajectory; the target tracking prediction model is built on the basis of Kalman filtering algorithm and attention mechanism.

[0107] In the embodiments of this application, the target tracking prediction model is mainly built on the Kalman filter algorithm and attention mechanism. By inputting the target dynamic feature map into the target tracking prediction model, the position coordinates, speed and trajectory of the moving target in the next frame can be effectively predicted.

[0108] Specifically, by determining the state transition matrix and the control input matrix, and combining the current frame position vector and control vector (which can be determined based on the current motion velocity vector of the humanoid robot), the position coordinates, motion velocity, and motion trajectory of the moving target in the next frame can be calculated.

[0109] Therefore, compared to traditional target tracking methods that only use a single Kalman filter and make linear predictions based on the target's historical position, which cannot adapt to complex movements such as variable acceleration and turning, this application uses Kalman filtering to make "prior predictions" of the target's position, velocity, and other states, providing a basic dynamic trend estimate for tracking. Then, it uses an attention mechanism to adaptively focus on the target's key areas (such as target edges and parts with drastic changes in motion), providing "posterior correction" weights for the "prior predictions" of Kalman filtering, improving tracking accuracy in complex scenarios. At the same time, it reflects the target's morphological characteristics through the target's dynamic feature map, significantly reducing target tracking prediction errors in scenarios with target occlusion and rapid movement.

[0110] Step 600: Based on the interaction requirements of the moving target to be tracked, and combined with the trajectory prediction results output by the target tracking prediction model, generate and execute motion control commands for the humanoid robot.

[0111] In this embodiment, the requirements of interactive tasks (grabbing, following, and avoiding) can be preset, and the trajectory prediction results output by the target tracking prediction model can be combined to generate joint motion control commands for the humanoid robot, and control the humanoid robot to execute the relevant commands to complete the interactive control tasks of the moving target.

[0112] Therefore, this application achieves fusion tracking of moving targets and humanoid robots by acquiring multimodal data, extracting dynamic feature maps of targets through a spatiotemporal feature extraction network model, and then using a target tracking prediction model constructed with Kalman filtering algorithm and attention mechanism to predict the trajectory of moving targets. This can effectively improve the tracking accuracy and prediction accuracy of moving targets, thereby enhancing the interactive control accuracy of humanoid robots.

[0113] For example, the step of generating and executing motion control commands for the humanoid robot based on the interaction requirements of the moving target to be tracked, combined with the trajectory prediction results output by the target tracking prediction model, includes:

[0114] Based on the trajectory prediction results output by the target tracking prediction model, and according to the interaction requirements of the moving target to be tracked, the motion vector of the humanoid robot is generated.

[0115] Based on the DH parameter method for humanoid robot arms, the motion vector is decomposed into joint angle changes by matrix inversion;

[0116] Based on the joint amplitude limit range, the joint angular velocity is calculated according to the change in joint angle and the preset movement time;

[0117] The torque parameters are calculated based on the joint load parameters.

[0118] By combining the changes in joint angles, the joint angular velocities, and the torque parameters, motion control commands for the humanoid robot are generated and executed.

[0119] In the embodiments of this application, based on the interaction requirements of the moving target to be tracked, such as when the interaction task is to grasp a workpiece, the requirement of the grasping task can be further determined to be to determine the current position of the humanoid robot's hand, and based on the trajectory prediction results output by the target tracking prediction model, the target prediction position can be determined, and then the motion vector of the humanoid robot's hand can be calculated.

[0120] Furthermore, based on the DH parameter method for humanoid robot arms, the link lengths, torsion angles, offsets, and joint angles between different joints of the humanoid robot are determined, kinematic equations are established, and the hand motion vector is decomposed into the angle changes of joints such as shoulder, elbow, and wrist through matrix inversion.

[0121] Furthermore, based on the joint maximum angular velocity limit table, the joint angular velocity is calculated according to the joint angle change and the preset motion time. When the joint angular velocity is about to exceed the joint maximum angular velocity limit, the preset motion time can be extended. The joint torque can also be determined by combining joint load parameters (workpiece weight, joint friction coefficient).

[0122] Finally, joint motion control commands containing angle, angular velocity, and torque are generated and executed by the humanoid robot.

[0123] Specifically, as an optional implementation, after generating and executing motion control commands for the humanoid robot based on the interaction requirements of the moving target to be tracked, combined with the trajectory prediction results output by the target tracking prediction model, the method further includes:

[0124] Obtain the execution result information of the humanoid robot in response to the motion control command;

[0125] When the error of the execution result information is greater than a preset threshold, the motion control command of the humanoid robot is dynamically corrected based on the PID algorithm, and the dynamically corrected motion control command is executed.

[0126] In the embodiments of this application, please refer to Figure 3 , Figure 3 This is a flowchart illustrating a target tracking interactive control and execution feedback process provided in an embodiment of this application. After acquiring multimodal data, performing coordinate system alignment and transformation, and then fusing the data to obtain multimodal fusion data, dynamic voxelization encoding is performed sequentially. The data is then input into a spatiotemporal feature extraction network model for joint spatiotemporal feature extraction, input into a target tracking prediction model for target trajectory prediction, and motion control commands for the humanoid robot are generated based on the prediction results.

[0127] It can control humanoid robots to execute relevant motion control commands, and obtain the execution result information of humanoid robots to execute motion control commands through sensor devices such as force sensors and joint position sensors. Then, it calculates the error between the execution result information and the actual interactive task target. When the error is greater than a preset threshold, the motion control commands can be dynamically corrected and fed back based on the PID algorithm.

[0128] Therefore, compared with traditional control methods that mostly adopt a linear process of "tracking-decision-execution" without planning joint movements in advance based on the target prediction results, resulting in long interaction response time, this application effectively shortens the interaction response time and improves the interaction control accuracy of humanoid robots by forming a closed loop of "prediction-execution-feedback-correction".

[0129] Please see Figure 4 , Figure 4 This is a schematic diagram of a humanoid robot target tracking interactive control device provided in an embodiment of this application. This application also provides a humanoid robot target tracking interactive control device that can implement the above-mentioned humanoid robot target tracking interactive control method. The device includes:

[0130] The data acquisition module 410 is used to acquire in real time the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period, and to acquire the current posture data of the humanoid robot and the historical posture data within a preset time period; wherein, the multimodal data includes point cloud data, image data and depth data;

[0131] The data fusion module 420 is used to combine the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fused data;

[0132] The matrix generation module 430 is used to perform dynamic voxelization encoding on the multimodal fusion data to generate a multimodal encoding matrix;

[0133] Feature extraction module 440 is used to input the multimodal encoding matrix into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model;

[0134] The trajectory prediction module 450 is used to input the target dynamic feature map into the target tracking prediction model and obtain the trajectory prediction result output by the target tracking prediction model; wherein, the trajectory prediction result includes position coordinates, motion speed and motion trajectory;

[0135] The instruction execution module 460 is used to generate and execute motion control instructions for the humanoid robot based on the interaction requirements of the moving target to be tracked and the trajectory prediction results output by the target tracking prediction model.

[0136] The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model, and the target tracking prediction model is built on the basis of the Kalman filter algorithm and the attention mechanism.

[0137] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] Please see Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0139] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0140] The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application.

[0141] The input / output interface 503 is used to implement information input and output;

[0142] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0143] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0144] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0146] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0148] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0150] This application provides a humanoid robot target tracking interactive control method and related equipment, which fuses and tracks multimodal data of moving targets and humanoid robots, extracts dynamic feature maps of targets through a spatiotemporal feature extraction network model, and then uses a target tracking prediction model constructed with Kalman filtering algorithm and attention mechanism to predict the trajectory of moving targets. This can effectively improve the tracking accuracy and prediction accuracy of moving targets, thereby enhancing the interactive control accuracy of humanoid robots.

[0151] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0152] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0155] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0156] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0158] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A humanoid robot target tracking interactive control method, characterized in that, The method includes the following steps: The system acquires in real time the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period, and acquires the current posture data of the humanoid robot and the historical posture data within a preset time period; wherein, the multimodal data includes point cloud data, image data and depth data; Multimodal fusion data is generated by combining the current multimodal data, the historical multimodal data, the current posture data, and the historical posture data; wherein, a three-dimensional coordinate system is constructed based on the current posture data of the humanoid robot. The multimodal fusion data is dynamically voxelized and encoded to generate a multimodal encoding matrix, including: Based on the speed of the moving target, the pre-set initial voxel size is corrected to obtain the dynamic voxel size; Based on the three-dimensional coordinate system, a voxel space is constructed according to the dynamic voxel size, and the multimodal fusion data is mapped to the voxel space; The number of point clouds, the average number of image pixels, and the average depth within each voxel are counted to construct a multimodal coding matrix. The multimodal coding matrix is ​​input into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model; The target dynamic feature map is input into the target tracking prediction model to obtain the trajectory prediction result output by the target tracking prediction model; wherein, the trajectory prediction result includes position coordinates, motion speed and motion trajectory; Based on the interaction requirements of the moving target to be tracked, and combined with the trajectory prediction results output by the target tracking prediction model, motion control commands for the humanoid robot are generated and executed. The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model, and the target tracking prediction model is built on the basis of the Kalman filter algorithm and the attention mechanism.

2. The method according to claim 1, characterized in that, The process of combining the current multimodal data, the historical multimodal data, the current attitude data, and the historical attitude data to generate multimodal fusion data includes: By combining the historical posture data of the humanoid robot, the difference between the heading angle of the humanoid robot within a preset time period and the current heading angle, as well as the torso position deviation of the humanoid robot, are determined, and the rotation transformation matrix and translation transformation matrix are calculated. Based on the rotation transformation matrix and the translation transformation matrix, the coordinate transformation of the historical multimodal data in the three-dimensional coordinate system is completed, and the data is merged with the current multimodal data to obtain multimodal fusion data.

3. The method according to claim 1, characterized in that, The spatiotemporal feature extraction network model includes a temporal coding submodule and a spatial coding submodule that are connected alternately in sequence; The number of the temporal coding submodules is the same as the number of the spatial coding submodules; The temporal coding submodule is a three-dimensional convolutional neural network model; the spatial coding submodule is a two-dimensional convolutional neural network model, combined with an attention mechanism.

4. The method according to claim 1, characterized in that, Based on the interaction requirements of the moving target to be tracked, and combined with the trajectory prediction results output by the target tracking prediction model, motion control commands for the humanoid robot are generated and executed, including: Based on the trajectory prediction results output by the target tracking prediction model, and according to the interaction requirements of the moving target to be tracked, the motion vector of the humanoid robot is generated. Based on the DH parameter method for humanoid robot arms, the motion vector is decomposed into joint angle changes by matrix inversion; Based on the joint amplitude limit range, the joint angular velocity is calculated according to the change in joint angle and the preset movement time; The torque parameters are calculated based on the joint load parameters. By combining the changes in joint angles, the joint angular velocities, and the torque parameters, motion control commands for the humanoid robot are generated and executed.

5. The method according to claim 1, characterized in that, After generating and executing motion control commands for the humanoid robot based on the interaction requirements of the moving target to be tracked, combined with the trajectory prediction results output by the target tracking prediction model, the process further includes: Obtain the execution result information of the humanoid robot in response to the motion control command; When the error of the execution result information is greater than a preset threshold, the motion control command of the humanoid robot is dynamically corrected based on the PID algorithm, and the dynamically corrected motion control command is executed.

6. A humanoid robot target tracking and interactive control device, characterized in that, The device includes: The data acquisition module is used to acquire the current multimodal data of the moving target to be tracked and the historical multimodal data within a preset time period in real time, and to acquire the current posture data of the humanoid robot and the historical posture data within a preset time period; wherein, the multimodal data includes point cloud data, image data and depth data; The data fusion module is used to combine the current multimodal data, the historical multimodal data, the current posture data, and the historical posture data to generate multimodal fused data; wherein, a three-dimensional coordinate system is constructed based on the current posture data of the humanoid robot; The matrix generation module is used to perform dynamic voxelization encoding on the multimodal fusion data to generate a multimodal encoding matrix, including: Based on the speed of the moving target, the pre-set initial voxel size is corrected to obtain the dynamic voxel size; Based on the three-dimensional coordinate system, a voxel space is constructed according to the dynamic voxel size, and the multimodal fusion data is mapped to the voxel space; The number of point clouds, the average number of image pixels, and the average depth within each voxel are counted to construct a multimodal coding matrix. The feature extraction module is used to input the multimodal encoding matrix into the spatiotemporal feature extraction network model to obtain the target dynamic feature map output by the spatiotemporal feature extraction network model; The trajectory prediction module is used to input the target dynamic feature map into the target tracking prediction model and obtain the trajectory prediction result output by the target tracking prediction model; wherein, the trajectory prediction result includes position coordinates, motion speed and motion trajectory; The instruction execution module is used to generate and execute motion control instructions for the humanoid robot based on the interaction requirements of the moving target to be tracked and the trajectory prediction results output by the target tracking prediction model. The spatiotemporal feature extraction network model is built on the basis of a convolutional neural network model, and the target tracking prediction model is built on the basis of the Kalman filter algorithm and the attention mechanism.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images